Product-performance testing presents a lot of challenges, but if a test method is developed correctly, it will consistently reveal valuable information and insights. Performance results are only as good as the method used to gather them.

Many companies support claims of “better,” “faster” and “more-efficient” products through performance-based testing. In the best-case scenarios, performance tests are extremely relevant to consumers, as they reveal how products perform in real-world settings. Unfortunately, biased and unrealistic product tests are a major problem– and bogus claims can come to light when a competitor, regulatory agency, or even consumer files a lawsuit against the offending company.

At bb7, product testing is one of our specialties. It’s one of the core offerings that makes us truly unique among product-development firms. Here is our expert advice on how to design and deploy a product-performance test, one that will fairly and accurately represent real-world performance.

WHAT MAKES A GOOD PRODUCT TEST?

The main goal of any product performance test should be to make the test as consumer-relevant as possible while still maintaining enough control over the system to gather meaningful, accurate and repeatable data.

Many challenges can arise while attempting to take the uncontrolled world of the consumer and map it within a scientific test method. With too much control, a test can become completely unrealistic, so that the data gathered will not accurately mirror a real-life scenario. On the other hand, if there is not enough control, the data is also useless. Little can be determined from a test with numerous uncontrolled variables influencing the results.

FOUR STEPS TO A BETTER TEST

1. Determine the question(s) the test is attempting to answer

Many factors contribute to a product’s overall performance. Those qualities include – but aren’t limited to — speed, effectiveness, power consumption, lifespan, performance deterioration and ease-of-use. A single test can’t measure all those things simultaneously, so it’s necessary to decide on a focal point for your testing.

When possible, it’s best to design tests that measure data quantitatively versus qualitatively. Quantitative data are objectively measured with numerical values, while qualitative data consist of subjective descriptions. If a test is designed to determine “how fast” or “how effective” something is, qualitative data would be based on the tester’s perception. That could introduce their associated bias as well as variability between testers. The same tester’s rating behavior can even change after several trials.

2. Find out what is relevant to consumers through user research

Good product testing should focus on mimicking real-world environments and use cases. Information about these factors can be gleaned from several different sources, including test standards and user research. When setting up a testing framework, test standards are useful, as they can provide guidance on how similar tests have been executed. However, it is important to always approach those test procedures – even the standardized ones – with a critical eye.

User research is one of the most relevant ways to gather data on typical use cases, and it can take several forms: Surveys, in-home studies and user testing in a controlled lab.

• Surveys represent a fast and easy way to gather research data from many users. However, there are instances in which survey respondents’ information is not captured accurately or quantitatively enough to create a sound test method.
• In-home studies are useful in observing how products are used in realistic uncontrolled settings. They can provide insights on how consumers interact with product, and the approach can be more useful than a survey. That’s because many consumers will oversimplify or omit steps when recounting product usage in a survey. Observational research may also reveal unintended use cases and unmet needs.
• User testing in a controlled lab can be used to gather specific data to mimic use behavior. For example, if the test is investigating how a user wipes up a spill with a towel, your test can determine the downward forces applied and the speed and motion of the moving towel. Variation in speed and force between users is to be expected, so multiple individuals and demographics need to be measured to determine the forces and speeds used in the test procedure.

3. Standardize the test

One of the most difficult aspects of any test method is standardization of the test itself. The control of variables that can influence test results is essential, and depending on the type of product you’re testing, it can be incredibly tricky to achieve.

As the test method is developed one needs to continuously consider any sensitivities inherent in the method.  For example, if one is running a test to determine the efficiency of an engine, the temperature of the environment in which the test is run in will have a substantial impact on the results.  Thus, that variable will need to be controlled by specifying the appropriate temperature range within the method.

Another example is performing a mechanical evaluation on a plastic part.  There are a variety of plastic materials that absorb water from the environment, which then influences the material’s mechanical performance.  To prevent this behavior from influencing test results the appropriate humidity control will need to be established.

4. Clarify the test method

The test method itself should be written clearly and precisely. The ultimate goal should be to document the test such that it will produce repeatable results regardless of the individual performing the test.

There should be no ambiguity that could cause a test executor to make assumptions or deviate from the intended procedure. One way that a test method can be vetted for ambiguity is to recruit individuals who have no prior knowledge of the test and ask them to perform the test using only the test procedure. Then they should be observed carefully, and without interruption while they perform it. Any deviation from the procedure or confusion of the tester should be noted, and then any issues that were raised respecting the procedure instructions should be used to modify and clarify the test document.

EVERY PIECE OF A PERFORMANCE TEST MATTERS

The key to success is keeping your testing method consumer-relevant, accurate and repeatable. Each of those elements is crucial, and each of them can be addressed in specific ways.

The consumer-relevant portion can be addressed with user research. The accurate and repeatable requirements can be provided by standardizing the highly variable aspects of your testing procedure. Depending on the test method, product-performance testing can range from meaningful to worthless, but if proper care is taken, the results can be extremely useful for developing and marketing new products.