Introduction

In order to evaluate the quality of human-system interaction, testers commonly need to measure Usability, Workload, Training, and Trust. As is the case for all measurement, testers should measure these concepts as precisely as possible, using validated scales to minimize measurement error. Here, we provide an overview of scales approved by DOT&E to measure each of the above concepts during operational test and evaluation.

For each scale, we detail:

  • What the scale measures
  • Some relevant academic reference(s)
  • Administration guidance
  • Scoring guidance
  • Score interpretation guidance

Note that some concepts of interest, such as Situational Awareness, do not have approved scales and must be measured behaviorally. If you have any questions, please contact the Test Science team, testscience2@ida.org for advice.

A Note on Survey Administration

Surveys, like other scientific instruments, only produce interpretable data when used correctly. Here are some best practices that can improve data integrity:

  1. Administer the survey to the people you want to learn about. If you want to learn about typical users of your product, then administer your survey to typical users; don’t administer it to “golden” users or, worse, the product designers. It can also be useful to record demographic data for consideration in subsequent analysis.
  2. Avoid biasing your respondents toward particular answers. We won’t learn anything new if we tell the respondents what to say – whether we do so intentionally or unintentionally. Don’t tell your respondents what you’re hoping to hear. Do tell them that you’re interested in their opinions and unique experiences.
  3. Minimize the total number of surveys given. Answering surveys is hard work—often done shortly after performing other, harder work. Respondents can easily become tired, frustrated, and/or irritated, and this can lead to inaccurate responses and low-quality data. It is better to get a few, accurate responses than a pile of meaningless ones.
  4. Acknowledge the limitations of surveys. It’s important to keep in mind that surveys are subjective measures and should be used in conjunction with other measures such as physiological data or task performance data to gain a more complete understanding.