“Exercise. Exercise. Exercise. Increased activity on social media by several extremist groups calls for protest, vandalism and attacks on equipment and personnel. Their goal is disruption of the power grid in the WAPA area."
This alarming message is an example of an “inject," or scenario that participants experienced as part of a recent nationwide training drill.
Over two days in November, WAPA, the Department of Energy and other utilities and organizations that support the power grid took part in the North American Electric Reliability Corporation's GridEx VI emergency exercise.
The immersive simulation gives the electric industry, government agencies and other organizations a chance to practice implementing their emergency response and recovery plans in a safe environment. Due to the COVID-19 pandemic, this was the first year that WAPA held the event in a virtual setting.
More than 120 WAPA “players" from across all regions, the Colorado River Storage Project Management Center, Headquarters and a wide range of functional groups tested their knowledge and skills while planners and evaluators observed their actions. At the same time, thousands of others across the utility industry and government participated in parallel events.
During the WAPA exercise, employees were challenged when a series of injects simulating cyberattacks and physical attacks took down, within the simulation, various IT, communication and electric power systems, eventually threatening employees' physical security.
As impacts from protestors and cyberattackers escalated throughout the exercise, players had to figure out how to work around disruptions in the communications systems upon which they normally rely, such as email, cellphones and the internet.
“When communications went down during the exercise, we talked about how we would need to use sat phones," Supervisory Electrical Engineer Zea Flores said, referring to portable satellite phones. “We need better training, familiarity and set-up configuration for some of these tools to stay in touch."
“At times we can become over-reliant on our technologies, and this served as a reminder to all of IT's staff of what happens when these tools are unavailable," Senior Vice President and Chief Information Officer Mike Montoya said.
One of the overarching purposes of GridEx is to identify areas for improvement and propose solutions to implement before a real emergency hits.
“We need to revisit our continuity of operations plans to make sure they work in a virtual environment, as well as have more holistic plans for communications policies and procedures for when different comms go out," Operations Manager Jon Aust said.
“Some of the approaches we could deploy include leveraging WAYS tickets and having common dashboards across cyber," Information Technology Specialist Lee Ebreo said, referencing WAPA's IT customer-request tracking platform. “A common monitoring system to oversee various systems, including WAYS, would be a great place to start."
During the “hot wash" after the exercise, evaluators shared observations of how the players fared.
“What jumped out at me was folks not having cross-functional knowledge of what other parts of WAPA do and, most importantly, what corrective actions we could take," IT Cybersecurity Specialist Jim Ball said.
“We've become so dependent on technology that I think we need to take a step back and look to see how we can manually fly the plane when all the automated controls fail," said Vice President of Technical Services Steve Yexley.
Players also discussed ways in which the exercise could be enhanced.
“Have a consistent approach to releasing injects to everyone," Physical Security Specialist Justin Ramsay suggested. “There was often confusion when an inject was announced to participants who aren't directly involved in that function."
WAPA used the Cisco WebEx platform to run the exercise. Breakout rooms created the virtual spaces for various organizational units to meet, such as regional emergency operations centers, which would be stood up during a real emergency.
“A drawback of using WebEx was that the chat thread would be lost when moving to another breakout room, so Teams might work better," said Power Operations Advisor Cory Danson. WAPA recently adopted Microsoft Teams as part of its Modern Workplace initiative.
WAPA had good reasons to practice using WebEx and Teams to conduct emergency operations.
“In a real-life loss of other comms, the way our emergency groups were structured with various breakout rooms could work in the real world as an emergency comms tool, as long as we have internet," said Supervisory Information Technology Specialist Kim Kelly.
Ultimately, GridEx VI served as a stimulating environment to analyze how WAPA and the broader electric utility industry can improve their resilience and react to adverse conditions.
“We had some great discussions about how the Southwest Power Pool's grid would have responded to the renewable energy generation impacts," Supervisory Power System Dispatcher John Roemen said.
WAPA's participants generally found GridEx to be a valuable use of time and resources.
“I tend to scrutinize our investments of time along the same cost-versus-benefit metric I apply to WAPA's investments of capital," Senior Vice President and Chief Financial Officer Mike Peterson said. “In my assessment, the two days invested in GridEx was time well spent."
As in past exercises, communications emerged as an area of critical importance and one that WAPA continues to improve.
“This was the first GridEx where people acknowledged and tried to address the internal communications piece," said Public Affairs Specialist Lisa Meiman. During the exercise, players made a concerted effort to practice notifying employees internally of the unfolding scenarios. “Usually that's something that's left behind in the speed of trying to respond to the external events."
“Highlighting communication practices has been a general theme in every exercise we've had," Executive Vice President and Chief Operating Officer Kevin Howard said. “These exercises really help us practice critical communication skills—knowing who to reach out to and how to make information-sharing occur."
The event continues to challenge and inspire both the players, whose role involves just two days every two years, and the 24 planners, who spend countless hours preparing for it.
“I'm very impressed with the level of engagement from all the players on this, and I'm very thankful for our team of planners who have worked so hard over the past nine months to bring this together," said Emergency Management Specialist Tarra Keathley, who helped organize the event.
WAPA's GridEx planning group is working on the after-action report to share with senior leaders in early 2022.
Note: The author is a public affairs specialist.
During a real-life electric grid emergency, Operations staff, such as Power System Dispatcher Ernie Salcido, must respond quickly to ensure the balance
of load and generation is maintained. They must train at least once a year on techniques and strategies to restore the electrical system from blackout
conditions and meet mandatory certification standards. (Photo courtesy of Scott Gregory.)