Print This Page

9. Quality assessment methods

9.3 Conducting listening tests

To achieve a relevant quality assessment about the audio quality and sound quality of an audio system, we propose the following conditions to be met for valid listening tests:

1) Tests must be controlled: all factors other than the audio system must be either removed or kept constant:
    

* sound source (live musicians can not be used)
* acoustic environment
* listening position & angle
* visible environment
* temperature and humidity
* smell and taste

2) at least two listening sessions must be performed per listening test to allow differential analysis.
     * A single session referencing to memory is not valid.

3) tests must be blind
    
* The test subjects must not know to what reference they are listening to

4) audio materials must be shorter than 20 seconds

5) If different signal chains are used, their total gain must be calibrated within 0.5 dB

Significance
The abilities and characteristics of the human auditory system differ strongly from individual to individual, but also over time. Single listening tests (with multiple sessions) only provide a quality assessment of a system that is valid only for the test subject at the time of the test. To achieve statistical significance in order to generate statements that are valid for an average audience at all times, listening tests and sessions can be performed multiple times, applying general scientific statistical principles (eg. analysis of variance, χ² tests)

Analysis - statements on audio quality and sound quality
The results from valid listening tests can identify audio quality issues in the Performance processes of a system, and sound quality issues in the Response processes of a system. However, there is no translation table available to translate hearing sensations to physical phenomenon in a system’s circuits or software. Statements on physical phenomenon can not be made based only on listening test results. At best, electronic measurements can be proposed - based on listening test results - to find a possible physical cause of the perceived quality issue. Only if a physical cause can be confirmed, a valid statement can be made correlating the hearing experience to the physical phenomenon. All assuming that the listening tests were ‘controlled’ - conducted under the conditions proposed in table 901.

In this white paper, we strongly advise not to draw direct conclusions about physical phenomenon in networked audio systems based on listening tests. A valid conclusion can only be drawn after confirming a found cause for the listening tests results - normally by conducting further listening tests varying the found cause parameters. We even more strongly not advise to draw any conclusion in general based on uncontrolled listening tests.

Return to Top