Set Test Goals Select Response Variables Select Factors
Response variables (otherwise known as dependent variables) measure the outcome of a test. Responses often include measures of system performance, effectiveness, or suitability. Recorded response variables provide the test data you will later analyze and provide answers to the questions that drive your design. Similar to deciding on test goals, selecting response variables is a collaborative process among many stakeholders, including testers, subject matter experts, requirements representatives, program managers, and statisticians. There are many options for response variables, some better than others. Most tests will have multiple responses to measure, as outcomes such as effectiveness and suitability are often complex, multidimensional constructs. The key is to select the most important and informative responses, as these will be used to size (i.e., determine the scale and cost) of your test and enable the testers to draw definitive conclusions from the test data. There are a number of best practices described below that can help you select response variables.
Best practices for selecting response variables
- Draw from all available resources including requirements documents, concept of employment documents, Action Officer operational and testing experience, operator and maintainer experience, and other stakeholders.
- Choose variables that aid in the determination of mission capability or provide a meaningful measure of system performance, effectiveness, or suitability.
- Choose relevant variables that capture the reason we are acquiring the system (i.e., make sure the response provides important information that relates to the test goals).
- Choose responses that lend themselves well to experimental design. This means they are:
Measurable: The response of interest can be observed and recorded at a reasonable cost and without affecting the test outcome. You may ultimately be interested in effectiveness, but you must provide a detailed and precise definition of how effectiveness is measured. Additionally, you should strive to make observations as unobtrusive as possible.
Valid: Measures should directly address the test objectives and be carried out with some degree of precision and consistency. Just because data is available and convenient to collect doesn't mean it is relevant to your goals.
Informative: Resulting data is detailed and provides insight into the question at hand. Continuous as opposed to binary data (see below) is best suited to provide informative data.
Data Types
Different response variables and measurement techniques result in different kinds of data. Some data are more informative than others, and choosing more informative variables can reduce resource costs by 50% or more! (See the JCAD Case Study for more on cost reduction). Common data types and their characteristics are summarized below.
Categorical
Categorical data, also commonly referred to as qualitative, provides information on the nature or quality of objects. Categorical data often conveys only names of different categories (aka: nominal data), but sometimes also identifies whether the objects or categories can be placed in an order (ordinal data). Two subtypes of categorical data are common in operational testing.Binary: Binary data are a special type of categorical data consisting of 2 possible responses (e.g., Hit/Miss; Detect/Non-detect). These types of responses offer the least amount of information. For example, you may ultimately be interested in whether a projectile hit or missed it's target, but having such information does not help you determine why some missed, or what other factors are most closely related to accuracy.
Ordinal: Ordinal data provides information on how units are ranked or ordered, but not how big of a difference there is between each element (e.g., Rank order of top performers, 5 star rating scales, Letter grades). This is more informative than binary data, but the ordered categories do not convey as much information as a continuously measured response.
Numerical
Numerical, or quantitative, data indicate how much of something there is (e.g., strength, magnitude, size), rather than what type it is. Numerical data often results from counting or taking physical measures, which create further subtypes.Discrete: Discrete data consists of counts or frequencies. These values usually take on whole numbers.
Continuous: Continuous data is quantitative and can take on an infinite number of values (e.g., Detection range, Time until event). This is the most informative type of data and requires fewer test resources to arrive at quality answers. Continuous data can be broken into interval and ratio scales, with both having equal distance between scale points, but only the latter having a meaningful 0 points (Absolute 0 indicates a complete absence of the measured entity on the ratio scale, but may mean only relatively little on an interval scale).
Data are also treated differently throughout analysis depending on their type. Visit the Data Exploration and Visualization section to learn more about the implications of data type.Example Response Variables for Common System Types
Replacing Binary with Continuous Response Variables & the JCAD Case Study
Whereas binary response variables are often readily available and of interest, some clever planning can generate substitute continuous measure to maximize test efficiency. Read the JCAD Chemical Agent Detector test and analysis case study to learn more about substituting continuous for binary response variables.
