A Chief Resilience Officer’s (CrO’s) Guide to Best-Practice Crisis Exercising and Testing

Introduction

A Chief Resilience Officer (CrO) has their work cut out for them. Threat levels are increasing across all metrics. Natural disasters have become more common and devastating. Political systems, both national and international, have become more volatile, introducing even more stress to already unpredictable financial markets. Adding to it all, the industrialization of cybercrime, with the introduction and now-platforming of ransomware attacks, means that a cyberattack is a matter of when not if.

Given the perilous risk picture, the best defense a CrO has is offense, proactive measures to ensure the resilience of the organization. A discrete set of processes that resilience teams develop before a crisis strikes to manage its effects, crisis management plans, or resilience plans, have always been measures that organizations wield to assess and address their vulnerabilities to avoid or minimize the impact of crises.

And so, naturally, as crises increase in kind and cost, planning becomes more instrumental – but not just the plans themselves, the exercise and testing of the plans, as well. Here, though, there’s been an historic disconnect between an organization’s needs – a solid resilience posture – and its inputs – often untested crisis management plans.

In their roles, CrO’s are now traversing that divide, quickly realizing that all might not be well with their crisis exercising and testing programs. What can CrO’s do? We devote this guide to helping CrO’s develop best-practice crisis exercising and testing programs at their organizations.  

What are crisis management exercises?

Crisis management exercises themselves are the proactive events that strengthen the crisis management capability, by helping responders hone their skills, learn from mistakes, and ultimately get better at managing crisis situations before they happen.

No doubt CrO’s understand the value in engaging in regular crisis management exercises. But they have likely also realized that there’s little in the way of consensus about what constitutes a successful crisis management exercise.

Why does that matter? Well, organizations, without such a consensus, often find themselves investing time and resources in crisis management exercises that don’t move the needle – sound familiar?

Worse still, a pattern of unsatisfactory crisis management exercises often has the effect of delegitimizing the crisis management exercise capability itself. Fewer lessons are learned from individual exercises, rendering crisis responders less prepared when an actual crisis hits.

Fortunately, there’s some academic literature that can help instruct CrO’s how to get the best out of their crisis management exercises.

Success criteria for crisis management exercises

One of the key factors for successful crisis management exercising is simply picking the right exercise (we delve in to the many types later in the guide). The choice of the broader exercise format depends on an organization’s needs. Operations-based exercises, for instance, make sense when there’s a concrete need for strong realism and real-time simulation of the crisis. In contrast, organizations and entities might opt for discussion-based exercises when prioritizing a seminar-based approach with discussion.

Beyond the choice of the right exercise, the research suggests that several other factors predominate in whether a crisis management exercise is successful. Those factors include:

• Having specific, achievable goals, decided upon as early as possible in the planning process, is essential for the successful conduct of an exercise and extracting learnings afterwards.
• Selection and development. Closely related to exercise goals are scenario selection and development. Indeed, certain exercise goals are easier to achieve in certain exercise formats.

Another aspect of development is the decision about levels of detail and realism in the exercise. It might make theoretical sense to introduce as much detail as possible in a scenario. It has been shown, though, that too many details overload participants with too much information, causing frustration during the actual exercise.    

• Similarly, the goals and type of exercise will determine who participates beyond key decision makers. There’s a robust literature (scholarly and anecdotal) about the importance of a strong moderator or facilitator.

Beyond that role, organizations must also factor in how many people will participate in the exercise, which will influence the format of a given exercise. Add to that, the precise scenario an organization chooses and the goals it seeks to achieve must be recalibrated based on the participants involved in the scenario.

Introducing ISO/DIS 22398

Given the preamble, what should CrO’s be doing specifically to make maximum use of the controlled, risk managed environment of exercises and testing? Here, we turn to international, business resilience standard, ISO 22398, which describes the exact procedures necessary for planning, implementing, managing, evaluating, reporting, and improving exercises, as well as the testing designs needed to assess the crisis-readiness of an organization.

In introduction, the standard argues that organizations should codify specific policies stipulating that exercises, testing, and implementation procedures should lead to corrective action. To this end, organizations should:

1. Develop exercise performance objectives to define the direction and scope of exercises and testing.
2. Implement the procedures that trigger a review based on the critique of an exercise, test, and actual events. Scenarios should reflect the objectives of the exercise.

Establishing the foundation and other key standard sections

From there, the standard instructs complying organizations to conduct a needs and gap analysis to establish the need for exercises and testing in the first place. That might sound like overkill. However, such an analysis effectively signals the role of exercises and testing in managing business risks, helping stakeholders (including senior leaders) understand that conducting exercises and testing is needed to manage risks.

What questions should CrO’s ask to get started? Common questions include:

• Does the plan address requirements for exercises and testing?
• Can the plan promote consensus with interested parties?
• Does the plan offer an opportunity to reach and interact with its target group(s) and potentially address their interests?
• Does the plan provide an opportunity to address multiple issues in depth?
• Does the plan focus on key issues?
• Does the plan provide information tailored to the target group(s)?
• Is the plan practical and relatively easy to implement?
• Does the plan provide for information transfer at relatively low cost?
• Is the plan easy to update?
• Is the effectiveness of the plan measurable?
• Is the plan a good vehicle for education?
• Is the plan creating a constructive and supportive atmosphere?
• Is the plan an effective way to get publicity or increase public awareness?
• Does the plan conform to the organization's constraints?

To help move organizations away from generic tests, the gap analysis will point the CrO toward the kind of exercise (out of the many available options) that the program should be deploying. Exercises organizations might undertake include:  

Besides type, exercises themselves can be broken down into discussion or operations based, as mentioned earlier. Of course, even those two categories include multiple sub-categories, examples of which include:

What are the testing stages?

For CrO’s, the standard doesn’t provide a play-by-play for each specific type of scenario. It does, however, give organizations a set of six generic stages through which exercises go through that might be important for CrO’s to know and appreciate as they attempt to codify crisis exercises across the business. Those stages include:

1. Run-through

A joint exercise prior to the start of the “real” exercise that helps ensure that all members of the exercise team receive the same initial information. This review should be brief and contain only information that is vital to ensure that the participants can perform as planned during the conduct of the exercise. The lead evaluator should be a participant in this process. It’s also critical that a similar review occurs with the control team to remain synchronized with scenario changes and to facilitate the implementation of the exercise director’s guidance as the exercise proceeds.

2. Start-up briefing

An integral part of exercise hazard control, where the organization clearly communicates the reasons for an exercise intervention (both crisis and non-crisis) to all participants. The start-up briefing should be used to avoid confusion between simulated and actual events.

3. Launch

At this stage, the organization checks the communications that will be used to launch, stop (temporarily), and terminate exercises and testing prior to the scheduled launch. The methods for communicating launch, stoppage, and termination of exercises should be explained during the start-up briefing.

4. Wrap up

Here, the organization will use the same communications for launching and temporarily stopping the exercise to terminate the exercise altogether.

5. Post-exercise briefing

The stage devoted to gathering information from actual exercises and testing to provide valuable information concerning the validity of the plan, the resources that were available, how the resources were used, and the transfer of behavior learned in training. The same format for the critique of an exercise or test will be used for an actual incident. During the post-exercise debriefing, special attention should be given to the functioning of the exercise organization and the exercise planning process.

6. Observation

The evaluators of the exercise should have knowledge of the expected performance. They should have prepared observation forms, which should contain the exercise performance objective and allow for notes to be taken during the exercise.

A closer look at what to do once the exercise is completed

As CrO’s well know, the primary purpose of exercises and testing is to inform stakeholders which practices are working as planned and which are not, making the often-neglected after-action report the most important deliverable of the entire process.  

Of course, CrO’s will have heard of the after-action report, a staple of post-crisis analysis. The post-testing after-action report does something similar, in that it (a) gives organizations an overview of the exercises and testing performed; (b) reports on any successes against performance objectives; (c) elucidates what went well; (d) lays out the issues identified; and (e) lists subsequent remediation actions to be taken and by whom.

Of course, post-testing after-action reports differ in substance from post-crisis after-action reports; the former, by definition, details what happens in the more controlled exercise environment. What, then, are discussion points one might see in the former but not the latter? Discussions might include:

• The set-up and staging of the exercise (project management versus crisis management)
  • Experiences of the participants with respect to the set-up (first impressions and the evaluation forms)
• Exercise aims or objective of testing
• Constraints on the exercises and testing process
• Exercise performance objectives
• Type of exercises and testing
• Choice of a location
• List of preparation participants
• Expert opinion concerning the quality of the exercise
• Conclusions regarding the validities of the exercise and the durability of the exercise aims
• Evaluation of the exercises and testing performances
• Recommendations for the next exercise
• Self-reflection of the participants, taking into account the adaptation of the exercise aims
• Operational performances, competencies, and learning experience of participants      

Finally, CrO’s know how valuable their crisis management plans are. But planning isn’t done once the plan is developed. More than ever, rigorous testing is needed to ensure that plans and responders can perform under pressure.

In this piece, we’ve sought to lay out what a rigorous, best-practice testing program should look like. One final component is crisis management software with exercise management functionality to test your organization's readiness and ensure your teams are prepared to handle any situation that comes their way.

Where to find such a solution? Consider Noggin. Thanks to integrated threat intelligence, response plan activation, team collaboration, and post crisis reviews, our crisis management software empowers organizations to plan, coordinate, and streamline their response efforts to minimize the negative consequences of an incident, crisis, or emergency and return operations to normal as quickly as possible.

But don’t just take our word for it - request a software demonstration to see Noggin in action for yourself.