by Jen Neville
Utilities and government agencies, including Western, have been battle-tested during the past six years to ensure their staff and systems are able to respond to and fend off cyber and physical attacks. Furthering its preparedness, Western joined more than 200 utilities for the third, bi-annual GridEx Exercise, or GridEx III, simulation of a catastrophic, calculated attack on the bulk electric system.
“Successfully managing a crisis of this magnitude requires practice, and this gave Western an opportunity to do that,” Chief Information Security Officer Jim Ball, who served as incident commander for the exercise.
Hosted by North American Electric Reliability Corporation, the Nov. 18-19 exercise was designed to test participating entities’ ability to respond to cyber or physical security incidents and provide improvements to local, regional and national grid security programs.
During the two-day scenario, 100 Western employees designated as “registered players” for the exercise and numerous “incidental” players were inundated with exercise injects, mostly through email and realistic videos and photos that described simulated incidents and events occurring in Western’s grid from large-scale physical attacks to cybersecurity hacks and attempted malware implants.
“When we’re talking about the bulk electric system, it is imperative to fully know your environment, appropriate response personnel, communication plans and incident response procedures to respond properly to emergency situations,” said Cyber Security Information Assurance Supervisor Kevin Schulz, who led the planning and implementation of GridEx III at Western. “An exercise such as GridEx tests an organization’s preparedness in these and many other areas, identifying lessons learned to implement corrective actions that strengthen our response capability.”
Simulated disasters unfold
The first day, each utility underwent an onslaught of physical attacks including a substation fire, ammonia spill, drone activity and an active shooter scenario.
“Everything happened so quickly; it was a little hectic getting the initial emergency operation centers set up,” said Emergency Management Program Manager Tiffani DeFore, who observed the exercise. “But once they were established, it was amazing to see how smoothly information flowed between each regional EOC.”
Western staff quickly coordinated information and responses across its 15-state region and interacted with other utilities. “Crisis management and emergency operations are things you have to practice to be good at,” added Ball. “I was impressed by the cohesiveness of the team. The players had not worked together before in this type of situation, but they came together during the exercise and it worked well.”
During the past six months, a team of 28 Western employees, who served as exercise “planners,” orchestrated the scenarios and prepared for the exercise. The planners created realistic situations and challenges to test the system, as well as the staff’s procedures. “GridEx promotes tactical daydreaming at both the planner and player level,” said Colorado River Storage Project Energy Management and Marketing Office Manager Steve Johnson, who was a planner for the exercise. “Through the process of preparing and executing the drill, we wanted to test how our staff would handle things like breaks in normal communication channels and responding to critical cyber and physical events.”
The exercise ended for Western midway through the second day with copycat attacks mostly against IT systems. By the end, as the chaos subsided and the players had strong control of managing the last few situations, the team began strategizing how to bring Western back to normal operations.
A few employees from Pacific Northwest National Laboratory watched the exercise. “We at PNNL appreciated the opportunity to observe. You’re managing across 15 states and that’s not a trivial thing,” said PNNL Senior Power Engineer Mark Rice. “We learned a lot about how a utility responds under crisis and we can take that back and incorporate that in our research. This has been great, and we will look for other opportunities to partner.”
Thinking about the lessons learned—what worked well and what can be improved—the planning team is already developing ideas for continuing to test Western’s staff and system during more regular ongoing activities, as well as when the next GridEx opportunity rolls around.