How to Design an Effective Cyber Exercise

In today’s blog, we’ll discuss how to design an effective cyber exercise as a follow-up to our post on the benefits of cyber exercises and how to get approvals.

A good starting point is to consider why you want to conduct a cyber exercise. Cybersecurity teams and upper management often consider cyber exercises a “nice to have,” not a “must-have.” However, a simulated cybersecurity simulation, usually structured as a half-day or full-day tabletop or roundtable event, can be a beneficial way to see how prepared your organization is. 

Consider your organization’s biggest threats or concerns: Is it a ransomware attack? Insider threats? Nation-state actors? Answering that question is a key step to making sure that you determine what you want to build an exercise for and why to get all stakeholders aligned and on board. 

Three Phases of Cyber Exercise Development

One way to look at cyber exercises is to consider professional sports teams. Why do they practice when they’re already pros? Could you imagine how badly they would perform if they never practiced? They’d never make it to the playoffs, much less to superstar status.

Developing a simulated cyber exercise allows you to figure out organizational strengths, weaknesses and differences of opinion before an actual security incident happens. The more specific you can be with your goals and objectives, the more valuable the exercise will be. Let’s take an in-depth look at how you can do this.

We recommend a three-phase approach to cyber exercise development: Pre-planning, Event Planning and Post-planning. Each phase serves a vital role in creating an effective exercise.

1. Pre-planning consists of items that must be decided upon and coordinated before starting the exercise. Pre-planning is the most critical step since it will determine the exercise’s value to the participants and your organization.
2. Event Planning consists of items needed during the event to make it flow as envisioned. This phase defines how well you can achieve the pre-planning goals versus how much time you spend overcoming any limitations of an exercise.
3. Post-planning consists of items that will close out the exercise while reinforcing its value to the organization. Post-planning is the time to gather all the results and feedback and give the teams time to discuss their roles with each other.

Pre-Planning

Consider the following during the Pre-planning phase:

Scope of Participants 

Who is the exercise supposed to be training? This list should include active participants, support participants and exercise managers. Determine which support participant roles will require people to be present and which will be simulated.

For example, if the exercise includes an Incident Response (IR) activity, the cyber personnel who would find the incident and do the preliminary investigation and/or mitigation would be active participants. The roles of people from your legal team, upper management, a representative from your insurance company and law enforcement personnel would be support participants and/or could be simulated virtually or by having other employees play those parts.

Who will manage the exercise? In most instances, at least some of the people involved know what is supposed to take place in the exercise so they can monitor what’s happening. They should be ready to coordinate with people outside of the activity in case of technical issues or if the participants miss anything.

Another thing to consider is how many personnel will participate. In most companies or agencies, teams are involved in a single role, each with various skills. Will the cyber exercise actively utilize a member from each team, an entire team, multiple teams, or some blend? This discussion often includes cost arguments and staffing constraints. Regulatory requirements can also play a role in this area.

What types of actors are needed, and how will they be included in the exercise? Actors are defined as people who are required to support the realism of a task within the training without being part of the metrics. As this article recommends, ”ask yourself what they would know in a given situation, what’s important to them and what motivates them.” For example, will you simulate users or have a group of actual users role-play daily tasks? If you use a group of real people, then they are actors.

If you have remote sites, partners and/or external customers who need to be factored into the exercise, how will they be included/simulated? Adding different “political” actors and levels of “demanding” actors adds to the exercise’s realism and fun. For example, you could simulate a vice president on a business trip who wants to get something done as quickly as possible.

Scope of Facilities

There’s usually an established communication pattern if the exercise occurs in the players’ standard offices. However, daily work and interruptions are hard to control. Flow is hard to observe. Is there a classroom or training space available? If so, what must be considered to make it usable for the exercise? And with many people now working remotely, how will you include them or simulate their roles?

• Tip: Consider power supply, HVAC, communication methods, evacuation, restrooms and access needs.

Scale

The better an exercise can duplicate reality, the more value it will have for the participants. Scale encompasses all the details required for the exercise: access to policies, procedures, checklists, “normal traffic” and standard tools for each task. It will also include access to monitoring, logging, and ticketing systems. This is where you’ll discuss what you need regarding time, money, resources, and personnel. At this point, you’ll often need to negotiate a balance between what you want and what you can do.

For example, the company- or agency-owned backup tools may only be licensed for your production network. Since using the production network for an exercise is not normal, you legally cannot use that software license. The question then becomes, do you buy another license, use a similar tool or remove that tool from the exercise? There may be an evaluation license that you can use. Many vendors can provide temporary licenses or sell lower-cost “lab” licenses. Determine the number of servers and workstations, accounts with what privilege level, what software will be available and what security will be applied to the range. Here are a few other things to consider:

• Can you use the same IP space, naming convention and workstations? If you are in a virtual or cloud environment, can you clone production?
• Based on the deviation from production, determine if you want to allow the participants to do a dry run to get familiar with the exercise environment and its differences.

Rules of Engagement (RoE)

Most exercises include a communications channel just for the exercise that should not be modified during the event. There may be “known” passwords for the observers that cannot be changed. Add any procedures and/or checklists that won’t work in the exercise environment to the RoE so the players know.

Determine what will be in-exercise and out-of-exercise communications. To help an exercise seem real, keeping the participants within the exercise is helpful. You can create phone, email, or chat rosters of the participants by role so they don’t have to keep discussing that they are part of the exercise. Add realism into the communication from the support participants to maintain the illusion as much as possible. If the active participants are focused on the exercise instead of the tasks, help them role-play their way back.

Exercise Goals

Your goals are the common starting point. The other items within the pre-planning section will often determine the scope of your goals. Working on this section requires constant review as the other items are decided, and the limitations are identified. Consider the following:

• List the training/regulatory requirements.
• List at least one measure/metric for each goal.
  • Determine the collection method to score the measure/metric.
• Creating a “story” for any multi-step event helps the realism of the event.
  • For example, changing permissions or creating a group of accounts, folders and/or shares can be the result of a new remote site, departmental reorganization, merger with another company, new partnership, etc. People like to understand the “why” of tasks, so adding the background helps players stay engaged.

Event Planning

This phase is about how you can help the exercise run smoothly and anticipate the participants’ expectations. Even though it is pre-event, it is a separate phase because the decisions have mostly been made during pre-planning. It’s time to ensure each task can be accomplished as envisioned.

• Tip: The more similar it is to the production network, the more realistic the training.

Build and test user and admin workstations: Is there a baseline build for the participants’ workstations in production? Can that baseline be used in the exercise?

Build and test network devices: Is there a standard/common network device(s) in production, and can that device(s) be included in the exercise? After that is determined, you can build and test the standard communication method: email, chat, phone, or something else.

Develop a test plan for each task: Compare it to the RoE to ensure it can be accomplished. The goal is to ensure that the environment works and that the injects/tasks can be completed successfully to test/document any deviation from the procedure and/or identify the need for simulation. Be sure to confirm that the metrics/measures can be collected. 

Tip: Don’t give an unfair advantage to some participants by letting the personnel who tested it beforehand be active participants. You can use the staff who will manage the exercise as part of the test. This will give them experience in what should occur.

How will familiarization with the exercise environment be handled?: Differences will affect the participants unless the exercise is performed on the production network (not common). In some cases, the participants are just expected to figure it out during the exercise. In other cases, a task based on each role of the participant can be used to gain familiarity with the environment structure.

Most of these tasks should be simple, like logging into a given system or reading a certain log. The purpose is to get the participants used to moving around the new network.

How long will the exercise run?: Keeping various groups busy simultaneously is ideal but hard to achieve. Accomplishing the exercise goal requires tasks running in parallel and tasks requiring multiple teams to coordinate.

One method is to decide the time for active exercise and build the exercise to match the time. In this case, add the ability to start, pause and end. Plan an extra hour or two for this if the exercise will run four hours or more, and at least half an hour for shorter exercises. The other method is to look at the goals created in Pre-planning and determine the time needed to accomplish them.

Consider the following:

• Which teams can run simultaneously, and which teams are needed to complete each task? Use filler tasks based on daily operations to keep other teams occupied.
• Consider adding a capability to monitor the exercise without the participant’s knowledge.
• Consider options like an observation backchannel. If virtual, determine how to monitor participants without them noticing.

The goal here is not to tip the participants that something is about to happen while confirming that the task actually happens and recording how the participant(s) responds. If you can’t do this, create a plan to record tasks so you can gather metrics.

Develop and test scripts for the actors to interact with the participants: If actors are added to the exercise, they can be too polite to the participants. In other words, the sense of “I need to do my job” is missing. Consider added players who are going to be difficult and/or demanding. Add players who demand quick responses at the expense of following procedure, like the earlier example of a traveling vice president who needs information ASAP for a client meeting.

If you have standard forms that need to be completed for certain tasks, have the filled forms ready so the return can be handled quickly. Consider common errors in the forms that can be submitted initially and a corrected form that can be submitted if the error is caught.

How will traffic be generated?: An “empty” range makes all actions too visible. What does your organization consider as “normal” traffic? Determine how to run it during the exercise.

A common mistake is to replay and loop captured traffic. This is easy to identify and filter out. You’ll want to define what the following items look like and plan accordingly:

• Normal tasks – Daily operations
  • Use as filler tasks to keep everyone occupied.
  • Use to mask non-normal tasks.
  • Use to validate and familiarize the range prior to the exercise.
• Non-normal tasks – Injects, malware, virus, attacks, etc.
  • Pre-plan the tools needed to identify them.
    • Confirm the tools’ configuration to ensure the identification happens as expected.
    • For example, most Intrusion Detection Systems (IDS) will NOT identify Social Security Numbers by default.
  • Pre-plan the files needed to trigger against the tools.
  • Be able to reset the range after testing to leave no remnant of the non-normal tasks after initial testing.

Post Planning

This phase is all about wrapping up the exercise. Consider the political aspects, the exercise aspects and the people aspects. In short, look at the decisions made in the Pre-Planning and Event Planning phases and get some feedback.

The political aspect concerns upper management, active participants, and support participants. Here are a few considerations:

• Upper management needs to be presented with successes and findings to justify the expense of the exercise.
• Active participants want to feel that they have an opportunity for success, not just a failure. In many exercises, the participants feel that they were set up for failure due to the exercise format. If that is the majority feedback, the cause should be analyzed before any future exercises.
• Support participants should also gain some understanding of their interactions with the active participants and should be asked for improvements to their notification and/or involvement in real-world events.

The exercise aspect is to learn how the environment, simulations, support and active tasks worked out. Ask the following questions, record what was monitored and report on it:

• What went well?
• What needs to be improved or replaced in future exercises?
• What task was realistic to your daily job?
• What task exposed the most memorable insights? Why?
• Was the communication between teams effective?

The people aspect includes how participants felt about the exercise. It’s always interesting to get feedback from the actors and support participants as well. The following questions should be addressed:

• How well did participants notice the “non-normal” tasks?
• Are there any “heroes” from the exercise?
• Discuss the realism of the exercise from the participants’ perspective.
• Were there any serious shortcomings of the exercise?  Include comments about things that were intentionally left out of the exercise during pre-planning and ask participants if something on that list impacted the exercise negatively.
• Did the information flow in a usable manner? 

At a minimum, it’s strongly advised to prepare a survey for participants to complete after the exercise. Consider different surveys for the monitoring team, the support participants and the actors. This way, you can tailor the information you gather from different perspectives.

Summary

Cyber exercises are crucial to determining how prepared your organization is for potential attacks. Creating an effective plan before you begin the exercise can make the exercise even more effective for your organization. Separating the exercise into three distinct phases: Pre-planning, event planning and post-planning can help scope and track the development of the exercise. 

You’ll want to determine who should be involved and what their roles are, where it will happen, the scale of the exercise, the rules and the goals. Once you determine these, you can plan out how you want the exercise to happen. Once it is complete, it is very important to determine what the outcomes are, report on them and use the results to plan future exercises.

Don’t let a lack of planning be the reason you are unprepared when cyber attacks hit. Use this guide to plan your next cyber exercise, and read more about cyber exercises in other Motorola Solutions cybersecurity blogs.

Motorola Solutions offers a wide range of Advisory Services, including Cyber Exercises, to help organizations prepare for and prevent cyber attacks. Contact us for more information.