The move to incorporate CIPv5 standards within business practices
by Jennifer Neville, Oct. 31, 2016
|Overview of CIPv5/6 Implementation
Goal: Ensure strong protection of WAPA's critical infrastructure with measurable compliance to meet NERC's CIP versions 5 and 6 reliability standards
Number of CIP requirements: 38 (accounting for almost 9% of the NERC Standards which apply to WAPA)
Start date: December 2014
Initial deadline: April 1, 2016
Revised deadline: July 1, 2016
Number of sub-projects: 27
Number of project participants: 57
Estimated project work hours: 41,000
Actual project work hours: about 39,000
Many utilities across the country were focused this year on meeting the North American Electric Reliability Corporation compliance goal for incorporating Critical Infrastructure Protection Version 5 Standards, or CIPv5. As the July 1 deadline ticked closer, WAPA was well positioned to complete the overall project on schedule and ensure the standards were integrated into its business practices.
"Nothing like this has been attempted before," said WAPA's CIPv5 Project Manager John Work. "This is certainly a defining moment for us. Ultimately, we have a way to measure the success of our business practices and strong standards for delivering reliable power and ensuring strong security, both physical and cyber."
He continued, "Our entire IT infrastructure is more secure than it was a year ago for both our operations and our business systems, and it is still improving. We are continuing to build on our strong program and processes by adopting changes to best practices as they come up."
Reflecting back on the project from its early beginnings in 2014 to the challenging changes toward the end, it is evident that dedication to business, technology and organizational excellence brought the effort across the goal line in June 2016.
Early effort charts multiple unknowns
NERC's CIPv5 standards, which were approved by the Federal Energy Regulatory Commission in November 2013, dramatically increased the requirements for protecting the bulk electric system's cyber assets by expanding the scope of eligible assets and adding more prescriptive requirements to the now 10 Critical Infrastructure Protection Reliability Standards.
In 2014, WAPA assessed the requirements and determined how to ensure compliance with the new standards before the initial April 1, 2016, industry-wide deadline. "There were a lot of unknowns getting started, so it was hard to plan all the details and specific steps in the beginning. Instead we were focused on helping folks understand the end goal and that, as we go forward, we are looking for WAPA-wide solutions," said WAPA Reliability Compliance Program Manager Chris Johnson. "Any variations from a WAPA-wide process would need to be conscious decisions that are justified, and verified with senior managers."
Enter the CIPv5 Implementation Team, whose goal was to identify issues and barriers early on, as well as coordinate the steps across regions and systems to integrate compliance activities into normal work activities. "Focusing on our good business practices as a principle, we knew that integrating compliance into everyday business activities would be key," said Johnson. "The decision to tighten our business controls gave us the ability to build on our program, while meeting the standards. We told folks, 'If we have good business practices, it will take care of 90 percent of the compliance needs.'"
The CIPv5 Implementation Team—made up of 26 core members and an additional 31 participants from Operations, Maintenance, IT, Security and other programs—broke down the effort into 27 subordinate projects to standardize the process for determining and tracking the specifics of WAPA's cyber and physical assets. "Our objective was to provide transparency into CIPv5 programs, processes, teams and workgroups," explained Work, who led the CIPv5 Implementation Team. "We wanted everyone to have a clear understanding of their corporate efficiency, functional interrelationships, and our end goal – to verify that we are protecting the bulk electric system following NERC critical infrastructure protection standards."
Sub-projects ranged in scope from configuration management to cyber asset management and change control. "We recommended people for those projects. The effort was added to their already-heavy schedules," said Upper Great Plains Reliability Compliance Manager Doug Brown. "I applaud them and their managers for the sacrifices they made for the project. A lot of people put in a lot of hard work."
IT Evolution enters midstream
In August 2015, WAPA's Information Technology underwent a large change to realign programs through a consolidation called IT Evolution. With so many people from IT supporting the implementation of common tools for the CIPv5 project, IT Evolution bought with it some challenges and an accelerant. "In the moment, when the realignment happened, there was some confusion about what that would mean for the project. Would the same people stay on their projects, even if they had new responsibilities? How would the efforts be transitioned to new players? These were questions we needed to answer, while still moving the project forward," said Work.
"In the end, the timing of the IT Evolution was a great catalyst for taking the project to the next level," Work continued. "IT's shift to a WAPA-wide focus helped us concentrate even more on WAPA-wide solutions. Folks like IT Vice Presidents James Philips, Jim Ball and Greg Hansen stepped up to lead their teams from across WAPA toward decisions and standardized solutions. These decisions were streamlined because of the Evolution and they were key to keeping the various projects on track to meet the April 1, 2016, deadline."
Brown shared his perspective on the changes and WAPA's security position. "We are closer as an agency because of the CIPv5 project and we are stronger for it. Before CIPv5, each region researched and evaluated the changing standards and the impact they would have on their group, then they developed processes and procedures to meet their own needs. Now with WAPA-wide processes, the support and the relationships that were built help us to leverage our knowledge and activities across all regions. This isn't just helping us right now, but will also help us in the future when we have new hardware or software, a new project or when another compliance standard is introduced. Now we have the relationships and we talk about the compliance issues with one another."
Teams demo first results in December
Looking back at specific moments during the 18-month project, Johnson identified the December 2015 face-to-face meeting in Phoenix as an exciting event. All 26 core team members met to review specific projects, identify what progress had been made and get feedback from teammates on the results.
"We were seeing deliverables for the first time as a team. We were testing, reviewing and discussing products and putting some meat on the documents we had been talking about for the last year," said Johnson.
Work added, "We were starting to see how the sub-groups were self-governing and communicating on different products. That accountability and responsibility at the sub-project level was so important for making sure the gears meshed between the different projects. If they had not done that, we would not have finished the project."
The meeting created further momentum for the overall effort and inspired the teams to continue collaborating to solve problems.
"John Work was instrumental in keeping us on the path," said Brown. "He reached out to people, asked for due dates and provided accountability for those due dates. It looked like a bunch of isolated projects, but as it developed we saw how they had to dovetail into each other. And then we could see the big picture and how far we were into meeting the goal."
NERC announces version 6
As WAPA and other utilities were implementing CIPv5 measures, the ink from the policy side had not yet dried. The new world of cyber assets raised a lot of questions, which NERC was responding to as quickly as possible. However, the requests for clarifications and the interpretations from NERC created some hiccups in implementation and some delays for completion. In some cases, guidance previously received from the reliability organizations was completely reversed by NERC. This affected many entities, including WAPA. "In a few cases our direction was in direct contrast to the new interpretations of the standards," said Work. "We had to evaluate what was cohesive with sound business practices and ensure we were meeting the intent, rather than simply the explicit letter of the standard."
For instance, would a virtual server be a sufficient redundancy for a server that is associated with critical assets? Regardless of the policy, it was a commonsense business practice put into place to help WAPA support its critical infrastructure.
On January 21, 2016, FERC approved version 6 of the CIP standards that brought more requirements into scope. The effective date for the new requirements was March 31. Because the changes required more time and effort, FERC responded to industry questions by extending the CIP implementation date to July 1.
"That was a pivotal point in the implementation effort," Work reflected. "We went back to review our efforts, and made some adjustments to ensure we incorporated the new [version 6] requirements into our final results."
The extension also gave the Implementation Team time to look at the gaps and seams between the 27 sub-projects, and make sure some plans were in place for aligning the efforts into the larger project.
27 projects completed
In May, Work was confident that the team would complete the project after a final face-to-face meeting. "At that point, we had tested some completed projects from January, and we were optimistic that we would have all the measures and a complete system in place to be compliant," said Work.
The meeting gave the group time to focus on what would still be needed after the July 1 go-live date. "We knew that we were still going to need to train employees on the new procedures. That is something we will need to incorporate for both our general WAPA-wide training and for key individuals with specific roles in maintaining the system."
As the team turned the program over to be maintained through normal business practices, the underlying systems streamlined the process for demonstrating how WAPA is protecting its systems and assets. "Most people who were involved before CIPv5 appreciate the improvements and products rolled out after the CIPv5 transition," said Brown.
Administrator and CEO Mark A. Gabriel congratulated the CIPv5/6 Implementation Team, saying, "This has been a journey from the time we first estimated 40,000 person hours would be required to meet the standards and the completion of these tasks, ahead of time. Your efforts in WAPA-wide coordination have set a strong example for breaking traditional business-line barriers and implementing new tools, processes and procedures to improve WAPA's compliance activities. Thank you all for your support in meeting this monumental task."
Since the implementation, WAPA is working to be more proactive in the standards development process.