Overview
Training effectiveness is a question of how well a training protocol prepares the user to interact with the system. We describe two scales for helping to improve training procedures. The Operational Assessment of Training Scale (OATS) is a tool for measuring the relevance and efficacy of training, while the Diagnostic Survey of Training (DSoT) helps to identify necessary changes in the amount of various types of training.
Summary of Endorsed Scales
| Scale Name | Acronym | Advantages | Disadvantages | Subscales | Number of Items | |
| Operational Assessment of Training Scale | OATS | Construct subscales | Currently undergoing validation | Relevance, Efficacy | 6 | |
| Diagnostic Survey of Training | DSoT | Helpful for improving training | Not validated. Only used as a supplement | Overall | 4 |
Operational Assessment of Training Scale (OATS)
The OATS helps to measure the relevance and efficacy of provided training.
Administration
We suggest administering the OATS both immediately after training and at the end of test. This allows for characterization of any changes in user opinions about training as they gain more experience with the task.
Directions: Instruct the respondent to read each statement carefully and indicate the extent to which they agree or disagree using the scale provided.
Survey
Scoring
Each subscale may be scored separately, resulting in two scores: one for Training Relevance and one for Training Efficacy. To calculate a subscale score, take the average of all subscale items. It is also appropriate to take the average of all items as a measure of general training quality.
Interpretation
Higher scores indicate that training is more relevant and/or effective, respectively.
Reference
Vickers, B.D., Porter, D.J., Wojton, H., & Haga, R (2022, April 26-28). Measuring Training Efficacy: Structural Validation of the Operational Assessment of Training Scale (OATS) [Conference presentation]. Defense and Aerospace Test and Analysis Workshop 2022, Alexandria, VA, United States.
Diagnostic Survey of Training (DSoT)
The DSoT helps to identify ways to improve a training protocol. It should be used as a supplement to the OATS or other training diagnostics. The DSoT should not be used instead of validated training measuring. The DSoT is sometimes referred to as the Training Structure Assessment (TSA).
Administration
The DSoT should be administered soon after training while the course remains fresh in the mind of the participant.
Directions: Choose the option that best describes what you think should happen to each of the aspects of training listed below.
Survey
The DSoT consists of two surveys with 8 items and 1 item, respectively. The first survey addresses the content of training, while the second survey addresses the training instructor.
Survey about the training content:
Survey about the training instructor:
Scoring
The DSoT does not need to be scored.
Interpretation
The item responses indicate the user’s opinion about that feature of the training protocol. The items should not be averaged. Instead, each response can be used diagnostically on its own. For example, if your users respond with low values for Pace of the course training (indicating that the pace of training should be decreased), then you may conclude that the course moves too quickly. You might address this by discussing ways to slow down the pace of training, such as by incorporating additional examples.
Reference
Not applicable
