Best Practices Plan & Design

A well planned and designed test could contribute to the success of it and to limiting possible incidentals. Hence, a set of best practices for planning and designing a test are presented in this section. Some of these advices pertain only to the panning phase or to the designing phase, but others can be associated to both. Therefore, we first present best practices for planning a test, then the recommendations that can be used while planning and designing, and finally, the best practices for designing a test.

Best Practices: Plan

Identify all stakeholders upfront

Do not limit test goals to verifying a single, narrowly-defined requirement in static conditions

Use multiple response variables when needed

Where possible, use continuous response metrics rather than binanry or categorical metrics

Keep in mind the system’s concept of operations (CONOPS) and users’ tactics, techniques, and procedures (TTPs)

Best Practices: Plan & Design

Use continuous factors when possible to cover the operational envelope

Acknowledge that complex systems rarely operate themselves

Develop protocol and training

Best Practices: Design

Avoid using single hypothesis tests to determine test adequacy when sizing tests

Design tests around the superset of response variables

Include all relevant factors in the test design

Use sequential experimentation approaches

Be careful with loss of information

Ensure the test is providing the right answer to the right question

Follow the test-run order

Above all, remember that you are working in a team. In order to maximize the team effort, communication among experts is important. For example, the test might not go as planned. In this case, communication between operational testers, users, and analysis experts is key in order to properly interpret the results and understand what information is lost or different based on the change in text execution.