Sensory Evaluation: Applications and Opportunities in the Product Development Process
Sensory evaluation is a science that measures, analyses and interprets the responses of people to products as perceived by the senses. For decades sensory scientists have researched and developed methods to capture the reactions of people to various kinds of stimuli and better understand the perceptual process, while others have used sensory information to identify successful consumer products. The stimulus can be as simple as a purified chemical used to study perception or it can be a more complex mixture used in the manufacture of a food, a beverage or a cosmetic intended for sale to a consumer. While some sensory professionals continue to study basic processes, many others apply their knowledge to the evaluation of consumer products, contributing product sensory information to a brand’s market strategy. The articles in this issue are focused primarily on this latter topic, to demonstrate the application of sensory resources to the product development process along with discussion about the potential uses and misuses of the obtained sensory information.
Before describing some examples, it is useful to provide a brief review of the science behind sensory evaluation. This is essential because there still is a lack of familiarity with the basic principles of the science and specifically how physiology, psychology, experimental design and statistics impact the process. While this may be surprising to some, it is evident when new ideas are presented and adopted without appreciation for whether they have a scientific basis. A recent example is the idea that one should field a discrimination test using a large number of consumers (as many as 100 has been mentioned) based solely on product use. Empirically it has been observed that about 30% of consumers who are high frequency consumers of a particular food type, cannot discriminate differences at better than chance. For most consumers the discrimination test is a behavioral challenge, i.e. consumers have to learn how to take such a test. Following the aforementioned plan, keeping in mind that consumers are not all skilled and have no prior test experience, there is a high likelihood of not finding a difference when, in fact, there is (sometimes referred to as ß risk) an error that has serious business consequences. Proposals such as these to avoid the sensory qualifying process, reflects a lack of knowledge of the science and familiarity with the sensory literature. Sensory professionals have a responsibility to assess new ideas and methods for their applicability to their company’s products before considering changes to their practices, hence the need for a brief review of the science.
As stated at the outset, knowledge of the science of sensory evaluation must be an integral part of the test planning process. One must be clear as to objectives, the availability of qualified subjects, the method selected, and the appropriate design and analyses. In sensory analytical tests, repeated trials are often ignored without appreciation for the loss of information/conclusions derived from the results. Confidence in a decision based on the obtained data is essential. A test with no repeated trials is no different than completing a single chemical analysis of a product and presenting it as a valid representation of the chemical composition of that product.
Sensory evaluation relies on a limited set of resources to function effectively, and these include subjects, methods, facilities and capabilities in experimental design and analysis. To be effective in providing actionable product knowledge, one must have the resources available and know how to use them. For a more extensive discussion see Sidel and Stone (2006) or Stone and Sidel (2004).
People are an essential part of the evaluation process. This is “good” because there are many people available but “bad” because people are different from each other in many ways, and especially in their sensory skills. In addition, because all of us have senses, there is a tendency to assume we are all experts, especially those involved in developing a product or part of a development team. This leads to situations in which decision makers dissatisfied with a sensory test result reinterpret the results to meet their expectations, assuming they are better qualified than the consumer.
Regardless of the type of test, subjects must be qualified to participate. The qualifications are two-fold, the first being an average or above average user of the product type and the second is empirical, i.e. the subject must demonstrate that he/she can discriminate differences at better than chance with the product category being tested. This screening/qualifying process takes 3 or 4 sessions (~60-90 minutes each) and about 30 to 40 trials before one can be confident that a specific individual is qualified. As mentioned, about 30% of those who volunteer fail to meet the better-than-chance qualifying criterion. Once qualified, performance is monitored based on test results and serves as a basis for participation in future tests. This is not unlike the demographic and product usage profile that qualifies a consumer to participate in larger scale marketing tests. These latter recruiting criteria have been shown to increase sensitivity and provide results that enable a decision to be reached with greater confidence. Regardless of the test type, failure to follow qualifying procedures increases the risk of concluding there is no difference when, in fact, there is (commonly referred to as ß risk). Testing errors lead to decisions that are not correct and this leads to a loss of confidence in sensory and sensory professionals. While space limitations preclude a more comprehensive discussion about this topic, the aforementioned references will be helpful to the interested reader.
There are two basic categories of sensory methods: analytical and affective. They are separate types of methods because they answer different questions. On occasion, one encounters requests to combine a difference with a preference test without appreciating the difficulties in analysis and interpretation of the results. Such requests should be rejected vigorously.
Analytical methods include discrimination (or difference) and descriptive methods. The former methods answer the question: were the differences between products perceived and at what confidence level? The latter method identifies the specific types of differences and their magnitudes.
A. Discrimination methods include: paired, duo-trio, triangle, and dual standard, to name the most frequently used methods. Each offers advantages and disadvantages, but no one of them is more sensitive than the other. The choice of method is determined by the objective and the type of product, i.e. a product with a lingering effect will reduce sensitivity and one should select a method that limits the number of samplings. The paired test requires the fewest number of samplings (1 pair), followed by the duo-trio (2 pairs), the triangle (3 pairs) and the dual standard (4 pairs). All too often the selection is based on other criteria that reduce test sensitivity and increase ß risk. A common practice is to select the triangle test based on the mistaken belief that it is more sensitive because p=0.333 vs 0.50 for the duo-trio and paired tests. The problem with this approach is that the test is a behavioral/perceptual test, not a statistical test. If sensitivity were based solely on the probability value, then a four sample test would be more sensitive, p=0.25, etc. No evidence has been published that supports the sensitivity of the triangle test. Further to this point, sensory fatigue and test complexity directly impacts sensitivity especially for foods and beverages.
A second requirement is that the design of a test must incorporate repeated trials to further increase test sensitivity and enhance the likelihood that the difference will be detected. For discrimination testing, the recommended practice is for each subject to provide two judgments, i.e. to evaluate a set of products and make a decision, and after a specified rest interval (usually 2-3 minutes) the process is repeated. This yields two judgments for a subject and enables testing of sensitivity and reliability within and across the subjects, by serving order, etc. which further enhances confidence in the decision.
B. Descriptive Analysis methods include: Flavor Profile®, Texture Profile®, Spectrum Analysis®, and QDA®. These methods provide product descriptions with numerical measures of strength. All these methods are described in the literature as are many other methods. All the others are combinations of existing methods, claiming to make use of the best of each. Since there are fundamental differences between them, such claims should be viewed with caution. Whatever method is chosen, the sensory staff must understand the process sufficiently to reach an informed decision. There are two basic issues to consider: the first is whether the subjects develop a language to describe the products or the sensory staff trains the subjects to respond to specific attributes, and second, how the responses are analysed. A detailed discussion of this topic can be found in Stone and Sidel (1994).
Descriptive analysis is probably the most powerful and useful of sensory methods. It identifies all of a product’s perceived characteristics along with measures of the strengths of those characteristics. It provides a “signature” or a “fingerprint” of a product. When that information is combined with physical, chemical measures, preferences and imagery, it becomes an integral part of a company’s product market strategy. It is especially useful to know what product attributes best fit the imagery and which formulation optimises those attributes.
Figure 1. This is a sensory map of four (of more than 15) competitive white wines evaluated by a trained QDA® panel of 12 subjects. Products were served following a balanced block design. Each subject scored each product four times. Means computed and the products plotted from the center = 0 or none of that attribute to the maximum where the product line crosses the attribute line. There were more than 30 attributes measured but those shown were obtained from a PCA and identified as most important in increasing preferences. Attributes with a + meant increasing the strength. The best liked wine was Brand C003 and least liked was Brand X14.
It is a small panel procedure using 15 or fewer subjects but it requires screening, as described for discrimination but also training to develop a language and provide measures of strength. These kinds of panels are especially useful, able to evaluate products in more typical use situations and not restricted to a laboratory. A typical sensory map is shown in Figure 1. The map shows the specific product attribute (sensory characteristics) differences in quantitative terms that allow for comparisons among products and with knowledge of the preferences from targeted consumers allows for correlation, single and multivariate analysis to better understand the product basis for preference differences.
C. There are two types of affective methods, the paired preference and the 9-point hedonic scale. Examples of each method can be found in the cited literature. From a sensory perspective, the 9-point hedonic scale is more useful because it provides a measure of liking for each product, the magnitude of the difference in liking among the products and enables use of parametric statistics such as the analysis of variance to identify significant product differences. Overall it is a more efficient methodology enabling one to test multiple products vs multiple paired comparisons. However, if one wants to know which of two products are preferred, then the paired comparison is more appropriate.
Method selection should be determined primarily by the objectives for a test. Other factors that impact choice are the time line, availability of qualified subjects, and, of course, availability of the product.
With these resources, sensory professionals can make a more significant contribution to a company’s business and especially the product development process. To be most effective, however, sensory professionals need to be organised and be a part of the development team from the beginning. Sensory professionals have to be able to provide actionable product information and manage it effectively vs simply reporting a statistically significant difference. In this next section I will discuss the ways in which sensory can contribute in a meaningful way.
Developing products that are successful is a difficult and time consuming process. The published evidence shows that the rate of success is very low so there always is a search for ways to increase the rate of success. This low rate of success has been attributed to many factors some acting alone but most often reflecting a combination of problems. Competition is intense, time lines are short, strategies change, targets move, raw materials are unavailable, etc. It is an imperfect process because so many of the core issues change as a result of new information. Yet, the process of product development through to market introduction starts out in a logical way and most follow a logical sequence of activities. Many of the elements in the process change in unpredictable ways. For example, the initial concept changes or the target market changes, regulatory issues occur with a particular ingredient. While project teams meet regularly to share this kind of information, its impact on a product’s sensory characteristics is unknown unless the sensory staff is a part of the team and can quickly obtain needed information. If this information is not available, project team memory is used to make decisions.
Whether one is on the marketing side or the technical side, certain kinds of information are essential to success. At the outset of any development effort, sensory descriptive information such as shown In Figure 1 is essential. While no preference information is available, the descriptive information serves as a baseline about the product and all related formulations. Since it is quantitative, comparisons with future product results are easily achieved. It is acknowledged that product sensory information is just one part of a much larger information pool; however, all other things being equal, if the sensory experience of a product does not satisfy expectation, the product will not be purchased in sufficient volume to build and maintain a business and it will fail. For more discussion about this topic see Stone and Sidel (2007).
Using sensory resources
In the early stages of a product development project, most information is obtained from focus groups and related qualitative sessions with targeted consumers. Consumers, in
Figure 2. A preference and sensory map listing product preferences (numerical values from a 9-point hedonic scale) and the important sensory attributes that best satisfied different consumer segments. The important sensory attributes were obtained from use of Factor Analysis and multivariate analyses. For more detail on use of these techniques, see Stone and Sidel (2004)
small groups of 8-12, talk about a concept in many different ways, e.g. what it might look like, what kinds of characteristics, benefits and uses, etc. A person identified as the moderator functions as the facilitator to encourage conversation, to be sure that all aspects of the idea have been explored. In most instances, there are many of these sessions until the project manager is satisfied. The information is summarised and is used to initiate actual product development or an early version of the product may have already been prepared, in which case some further formulation will be done. As mentioned earlier, a sensory descriptive analysis has to be completed using as many of the formulations as is possible. This type of multi-product test is especially effective in identifying formulations that change product attributes in a direction consistent with the concept and also adds to the product baseline data. As the development effort is refined, the market strategy updated, and the project moves closer to larger scale testing, the sensory results continue to provide data that measures the effects of various changes, ingredient and/or process. When consumer preference and related information is available, regression analyses can identify which product characteristics best fit with the concept and target population. Figure 2 displays the integration of these kinds of information in a more complex map.
Sensory professionals need to be able to develop and manage this kind of information to expect to be effective in product development efforts, and for the development itself to be successful. All too often sensory professionals are unprepared or lack the resources to field tests, obtain quantitative information and communicate it effectively, leaving project managers concerned about the value of sensory information. It is hoped that more discussion about these challenges will lead to more effective use of sensory product knowledge in development efforts.
In the subsequent articles, other authors will describe procedures and practices they are using in product development. Their perspectives and experiences will be helpful as you become involved in product development activities at your companies.
Stone, H and Sidel, JL (2004) Sensory Evaluation Practices, 3rd edn. London/New York: Academic Press/Elsevier.
Sidel, JL and Stone, H (2006) Sensory science: methodology. In Hui, YH (ed). Handbook of Food Science, Technology and Engineering. Boca Raton, FL: CRC Press; 57-1 ̶ 57-24.
Stone, H & Sidel, JL (2007) Sensory research and consumer-led food product development. In MacFie, H (ed), Consumer-led Food Product Development. Boca Raton, FL: CRC Press; 307-320.
Dr Herbert Stone is Senior Advisor and former President of Tragon Corporation, 350 Bridge Parkway, Redwood Shores, CA 94065, USA. He is a former President of the [US] Institute of Food Technologists (2004-06) and is a Fellow of the International Academy of Food Science and Technology; E-mail: firstname.lastname@example.org