First published in Security Magazine.

By Claire Meyer 18 December 2023 Focus on Cyber Incident Response

When it comes to cybersecurity, prevention is a losing battle, says Patrick Hayes. But preparedness can enable organizations to stay resilient and recover when times get tough—including during an incident.

“The reality is business is risk,” says Hayes, who has spent 30 years in the cybersecurity industry, including as an enterprise security architect, CISO, and now chief product officer at ThirdWave Innovations. “If you want to outperform your competitors, if you want to grow your market, you need to take risks.”

While deterrence and prevention are still worthy goals, organizations need to spend much more time on incident response and recovery preparedness, he adds. New regulatory measures, including those through the U.S. Securities and Exchange Commission (SEC), are forcing the issue, but security needs to go beyond compliance, especially where organizations’ unique business models are concerned.

“The security program is largely influenced by the strategy of the business, and the strategy of the business sets the stage for risk tolerance,” Hayes says. “While regulatory compliance is good as a minimum standard for the masses, it doesn’t take into account the strategies associated with each of the businesses that are governed by it.”

Some businesses may be especially risk averse, while others are more risk tolerant. Some are willing to test out new technologies to grow faster or to streamline a key supply chain to achieve a shorter time to market. In some cases, the risk pays off. In others, it puts the organization in a precarious position, particularly around securing digital assets and processes.

This is where cyber incident response comes into play.

Expect to Escalate

Not every cyber incident reaches the SEC reporting threshold of being “material” to the value of an organization, and not every incident requires a full-blown crisis management response. An IT issue, such as a system getting knocked offline by a load balancer issue, is not necessarily a security breach. While it might require future changes to avoid repeat issues, it doesn’t reach a crisis designation, Hayes says. But if a key system is knocked offline due to outside interference or an unknown system error, that is worth speedy investigation and a potential escalation up the chain of command.

It’s important to have these thresholds nailed down well in advance, so staff are not forced to make snap judgment calls in the heat of the moment, he says.

Organizations should set criteria for when a technical incident should activate a broader crisis response, says Mike Barcomb, director of executive cybersecurity exercises at the SANS Institute. Those criteria will be tailored to the needs and unique functions of the organization, but they could include a financial threshold (a predetermined dollar amount is met), reputational element (the news hits a major media outlet or gets shared on social media), or other operational impacts (an incident hampers a key production line).

“If any one of these criteria is met, then we have a crisis,” Barcomb says. Intermediate actions are possible, especially while discovery is underway. For example, early signs that an incident could become material could warrant an early warning to key leaders. Regardless, the goal is to enable a faster and more effective response without each level in the organization trying to solve the problem alone before alerting leaders.

“In companies that have those plans in place, it’s a quicker and more effective response versus those organizations that don’t have it defined, they haven’t practiced it, and they don’t know people’s roles and responsibilities,” he says. “There, it’s chaotic. It’s a lot of craziness that’s happening. And it’s costing the organization. Because of that time to respond, the attacker’s probably siphoning off more data. The wrong messaging’s getting out—the media’s got it, social media’s got it, and they’re controlling the message.”

Barcomb boils incident response preparedness down to a straightforward action: “Have your team, your people, and your plan, and practice as much as you can.”

Build Trust through Crisis Communications

Cybersecurity teams are also likely to face broad pressures during incident response, exacerbated by the volume and severity of incidents, says Tony Scott, CEO of Intrusion Inc. and former U.S. federal chief information officer.

First, accurate information is a challenge, especially early on in an incident. “Usually, the first information you get is wrong because the investigation is not complete, and you don’t know what you don’t know,” Scott says.

“You also have the fear factor. There’s been some rather public blaming kinds of things with major incidents, where information that was relayed early on turned out to be wrong, incomplete, or minimizing a situation when it turned out to be something much bigger,” he continues. “I think there’s a human reluctance to be too forthcoming when you don’t have sufficient facts. That’s a big issue. All of us engaged in this game have got to get used to a world where sooner is better, even if it needs to be revised later on.”

That proactive outreach is an important component to trust—an invaluable commodity before, during, and after an incident. In past incidents, security teams sat on information for months as they researched the event and tried to get the exact details nailed down, but this delay regularly backfires, Barcomb says. When you lose control of your messaging about response and recovery, it’s hard to get that trust back.

Here, it makes sense to set clear expectations with stakeholders inside and outside the organization.

Clear communication during a crisis is essential—it can’t be a one-and-done message, Scott says. Especially in the heat of the moment, people are likely to forget key details from your prior message, or they might not be familiar with a key concept you’re referencing. Plan to provide more information and repeat the key details to keep your stakeholders informed. “Situational awareness is probably the main skill executives need to have,” he adds. “In any given situation, you’ve got to be right on top of it in terms of who are the affected stakeholders and what’s our strategy going to be to manage those expectations.”

Start by preparing a stakeholder map that outlines who your stakeholders are (including employees, shareholders, and customers), what to communicate with them about, how to reach them, when you will contact them, and who is responsible for keeping them updated during a crisis, recommends Barcomb.

Then establish a cadence for communication, share that plan with stakeholders, and follow through. Establish the current facts as you know them to your constituents, Barcomb says, even if you need to add caveats, such as “these facts are likely to change as we learn more about the breach.”

Keeping the door open for information to change also gives investigators and auditors more leeway to dig deeper into the incident, Hayes says. Often, cybersecurity and IT teams are pressured to resolve the issue as quickly as possible so the organization can issue an all-clear signal. But that short-term strategy can backfire in the end.

“The reality is if you’re faced with a security incident and you try and recovery systems too quickly, you may actually miss containment of a situation,” he says. “Worse, you might actually be covering it up. You might damage the evidence that you have that would allow you to actually find something.”

Practice Differently

As in many security scenarios, improvement regularly comes from practice, whether during real-life incidents or training. And for crisis-level cyber incidents, organizations should practice in multiple ways.

Tabletop exercises are an effective place to start, but they aren’t necessarily basic. The setup is as important as the exercise itself to ensure that the scenario fits the organization’s model and feels realistic but not commonplace to participants, Scott says. You don’t want it to be easy for participants to write off the exercise, saying “well, that couldn’t happen here.”

Throw wrenches into exercises—whether they are tabletops or practical simulations—to test out how people will respond and to expose different challenges and areas for improvement, Scott recommends. Try changing locations, reshuffling participants, or giving confusing information. Also bring in departments that might not be overtly connected to cybersecurity.

In one narrowly tailored exercise, someone interrupted the plan and recommended bringing in colleagues who manage supply chain issues, Scott recalls.

“It was an eye-opener,” he says. The request for additional participation uncovered how connected many seemingly disparate functions of the organization are, how issues can quickly domino, and how valuable it can be to bring in partners security doesn’t regularly liaise with.

You will also learn a lot about your colleagues and stakeholders personally, Hayes says. “The last thing you want to do is find out how your peers are going to respond to a pressure situation without ever rehearsing it in the past. You learn a lot about an individual when they’re under pressure,” he adds.

Be sure to loop in frontline cybersecurity professionals, too, Scott says. These individuals are highly motivated to learn about cyber incident response and get it right, but they often want to get it right personally, rather than knowing when the time is right to loop in additional resources.

“The biggest learning I’ve ever had from exercises is the desire for people at every level to try and solve the problem themselves and the failure to ask for help,” Scott says. “If you think of an organization that has, say, four levels in its management structure, every manager at every level is going to try and solve the problem him or herself, because they don’t want to bother their boss. That times time, and it takes energy. If every level tries to do that in the face of a big problem, you’re not going to know about it for a long time.

“One of the things that we’ve practiced in the organizations I’ve been involved in is to communicate quickly up the food chain,” he continues. “Create awareness on anything that even remotely smells like it could be a big issue. Then have the confidence in the management layers to know when to say, ‘I can’t resolve it at my level. I need help.’ But create awareness quickly. Good news travels fast, bad news has to travel even faster, and you have to practice that because it doesn’t come naturally.”

Exercise participants will also learn how stakeholders value different priorities and processes, and when they are willing to accept risk. In addition, the context behind decisions matters, Hayes says. If a retailer’s e-commerce site might have a vulnerability, should the retailer shut it down or revert to backups until the investigation is complete? What if it’s a busy holiday shopping weekend? Having some early knowledge of stakeholders’ business drivers will prepare cybersecurity teams to switch over to backup plans and alternatives, rather than getting blindsided when leaders side with business imperatives over system lockdowns.

Hayes also recommends appointing an incident commander who can calm tense situations, follow preset procedures, and serve as the objective referee for tough calls, such as deciding how to balance breach containment with business continuity and regulatory requirements.

“Panic isn’t going to solve the situation,” Hayes says.

In addition to scenarios that stretch your capabilities, organizations can add atypical training elements as part of broader exercises, such as preparing to deal with members of the media.

“Tabletop exercises don’t often give you the experience of having to deal with the press or being interviewed or ambushed on your way out of the office,” Scott says. “I think it’s useful to have a real, honest-to-goodness reporter come stick a microphone in your face and say, ‘Hi Mr. Scott, can you talk to us?’”

He recommends recording individuals in the exercise as they speak to the reporter or give a mock press conference about the incident and then reviewing the footage. Consider how the speaker comes across, whether the information seems valuable or not, and how the message reflects back on the organization.

“I think that’ll send most people back to the drawing board right away once they’ve experienced that,” Scott adds.

And don’t stop the exercise at the point of the attack being resolved; organizations should practice recovery, too, Barcomb says.

Exercise participants should understand their roles and responsibilities during recovery, and these practice sessions enable them to set up contingencies for when recovery goes awry. For instance, if your ransomware recovery plan is to have system backups, what do you do if those backups are contaminated? Do you have a plan to keep backups secure, validate they are not contaminated, and test them? Do you know how long it would take?

“Practice, practice, practice,” Barcomb emphasizes. “Don’t wait for the incident to be that driving factor to determine how long it takes.”

After the Storm

Whether they are after a training exercise or a live incident, after-action steps are essential.

An after-action review (or lessons learned, “hot wash,” or debrief) “is the most important thing for organizations who have gone through an attack or have practiced through some assimilation exercise,” Barcomb says. “It is where you bring those key players who actually participated or were involved in that incident or that training exercise through that process where you talk about what went well, what didn’t go well, and where we need to improve.

“I think it’s great to have that in a report, so it’s documented,” he continues. “Then I think more important is to assign tasks or actions to people for those areas where we need to make improvements or fill gaps. Have a project manager who is going to drive that through completion. If you don’t, you’re still going to be at the same state from a response perspective as you were from the original attack or that training exercise.”

While official actions will likely be assigned to security, IT, and other stakeholders after an event, security leaders should also solicit and collect feedback from individuals and groups not directly involved in the response, Scott says.

You will likely get plenty of direct feedback, especially if you look for it. “People are going to tell you how they feel, what they see, and so on,” Scott says. “The press is going to tell you usually, and you’ll see what you got right and what you got wrong or somewhere in between. Another great source now is social media. There are very active places where you can go see how the public at large is reacting. Then I would always recommend making some direct calls. This could be to key trusted advisors, people outside your organization. You could pick a handful of people to just call up and have a conversation: ‘How’s this looking to you? Do you think we’re doing an effective job here? What suggestions do you have?’”

Pick people from a wide cross-section of stakeholders so you have a variety of viewpoints to learn from.

“There’s no substitute for that direct conversation, and it’s important to pay attention to both the facts and the perceptions that people have,” Scott continues. “But you also have to ask questions about their emotional reaction to something.”

This doesn’t necessarily mean whether you got the facts right or if you responded quickly enough, but people’s emotional reaction can tell you a lot about how your efforts and your organization are perceived. Did people feel you did a credible job in responding? Did your messaging help clarify the situation, or did it feel more like you were trying to avoid responsibility? Were your statements more about optics and PR than providing actionable information? Did your messaging actually get through to people?

“It’s about good communications,” Scott says. “It’s choice of words, it’s optics, it’s presence, it’s frequency. All of those things help in a tough situation. The worst thing is the one-and-done message. You show up and then never show up again—that doesn’t work very well.”

Check your own feelings and state of mind for feedback as well. Self-awareness is another essential skill in incident response, but it’s less commonly addressed, says Scott.

“You need to know your own capabilities and your own emotional intelligence on these things,” he says. “Be aware of how the crisis is affecting you. Are you eating right? Are you sleeping right? Are you getting angry easily? Then do things to adjust appropriately. Once or twice a day, go take a good look in the mirror and ask ‘Hey, how am I doing?’ It’s pretty important, because it’s easy to get caught up in these things.”

Claire Meyer is managing editor for Security Management.