By Lisa Meiman
In June, WAPA selected a new vendor to provide supervisory control and data acquisition and energy management system, or SCADA/EMS, software to all of WAPA’s regions.
SCADA and EMS are two of the most critical information technology systems used at WAPA. It is the grid operator’s primary tool to monitor and operate the grid 24/7/365.
“This project touches every region and Headquarters,” said IT Vice President of SCADA James Phillips. “Its impact will be felt for the next 15-20 years. This is not an easy project. The size, the scope and the competing needs are daunting. Dozens of personnel from multiple disciplines and multiple locations were involved with evaluating and choosing the best product for WAPA as a whole.”
Monarch, from Open Systems International, is the software that was selected over the competition after a year and half of evaluation and procurement negotiations for a commercial off-the-shelf SCADA/EMS platform.
With the new common platform, WAPA’s control centers will be able to partner with each other and learn from each other in ways that are not currently possible.
“For us, this project was critical for a technology refresh, to modernize the SCADA/EMS solution WAPA-wide and bring it to industry standards,” said IT Project Manager Tom Jamrowski, who works under the Innovative Management Concepts contract. “It was also an opportunity to bring the people together who use the systems from across WAPA to make this decision. And, finally, we wanted to standardize the tool that was being used in Dispatch.”
Today, WAPA has three separate SCADA systems, each with a primary and backup control center for a total of six control centers. The three distinct systems evolved independently, with different hardware, software and maintenance agreements.
Installing a common SCADA platform in WAPA’s control centers will improve operational flexibility; increase reliability and cybersecurity; and avoid future support and administrative costs related to supporting multiple SCADA vendors and contracts.
“This doesn’t change how Power System Operations does business,” said Power System Dispatcher Patrick Reamy, who was one of 14 members of the evaluation team and is also serving on the new standardization team. “It’s a software to operate the system in a safe and reliable state. This is a much better tool and will hopefully give us better visibility, alarming and hierarchical views to be able to perform our jobs.”
The evaluation team that selected the vendor was made up of eight operations personnel representing all four regions who will use the software in their daily jobs, four SCADA support employees and two other people from IT evaluating the network, cybersecurity and infrastructure compatibility of each of the vendors.
“Probably the biggest challenge was that we haven’t done this kind of major IT purchase before,” said Jamrowski. “There were no previous projects to look back to and learn from. We had to make up a lot of stuff to make this happen. We were grateful to have a lot of executive support to get over the roadblocks in this project.”
The team used a full and open solicitation, meaning all vendors that believed their product could meet WAPA’s needs could submit a quotation explaining their product’s capabilities.
“We used a multi-faceted approach to selecting a vendor,” said Contract Specialist David Halla. “We provided our basic requirements as part of the solicitation. The vendors submitted technical information for our review and then we watched a demonstration of their product based upon scenarios developed by the technical evaluation team. It was good to get healthy competition and see what was truly available in the market side by side to make this best value decision for all of WAPA.”
The novelty of the project added additional complexity to the acquisition process.
“The request for quotation was new and different,” continued Halla. “Instead of steel poles or circuit breakers where initial ordering requirements are clearly defined up front, we had a lot of different implementation strategy decisions that had to be made only after we received the quotes. The request for quotation was developed in a manner to provide WAPA that flexibility and it was a unique approach for that.”
Although only four vendors expressed interest, the volume of technical information submitted was immense.
“We received hundreds of pages to review and score,” said Jamrowski. “We had eight categories in the request to understand the maturity of the vendor in areas, such as cybersecurity, past experience and training, and vendors had to provide documentation in every category, totaling between 30-50 pages each. Every evaluation team member had to evaluate those answers and score them.”
Once that was done, the evaluation team plus the Procurement specialists met at the Electric Power Training Center in Golden, Colorado, in November 2019 to view demonstrations from each vendor.
“You can imagine people reading all this information; it all kind of melds together in your head,” said Jamrowski. “When you can see it demonstrated on a screen in front of you, that is where the rubber meets the road.”
After the demonstrations and scoring all responses, the team met again at the EPTC in January. The objective was to select the vendor using a “best value” approach based first on technical excellence, then cost and finally past performance. There was a week of analysis and debate about which vendor was the best solution for WAPA. Eventually, Monarch emerged as the top choice.
“It was a major accomplishment to be able to walk out with 100% alignment,” said Jamrowski. “That doesn’t mean that everyone got their first choice, but everyone was satisfied with the process and the final decision. This is very important because this team will be the advocates of change in the future when we go to implementation.”
The final negotiations, justifications and internal approvals for the selection took more than two months before the contract was awarded. During much of evaluation, the team was under a non-disclosure agreement, meaning they could not discuss the process outside the evaluation team.
“I was very happy with the team and how they interacted,” said Jamrowski. “This was a WAPA-wide pick, and there is a lot of value in that. Everyone has to go through the same change, and everyone gets the tool that they said they wanted.”
“This was a huge effort and pretty far reaching in terms of involving people both supporting and using the system for the evaluation,” added Halla. “By having the entire 14-member team from different regions and disciplines sit in a room together, we were able to confidently make the best decision for WAPA and its future.”
Philosophy, style guide development underway
One of the next steps is producing common display configuration standards and support processes to implement the Monarch systems in all of the control centers, a task assigned to the Operator Information Standardization team.
The purpose of the standardization phase is to develop configurations and support processes that provide operational efficiencies, reduce human errors, increase system resilience and enhance WAPA’s ability to recover quickly from outages.
The standardization team is made up of 12 Operations and SCADA employees representing all of the regions. Most of them are new to the project as the evaluation team members rotated out. Their primary goals are to determine the WAPA-wide human-machine interface style and philosophy guidelines for the displays, including how they look and function.
“The collaboration is important because no region wants to feel left out and like the other region or regions are imposing their will,” said Reamy. “All four regions have valuable input and one region may be doing something the others aren’t, and it might be a better way.”
The team will define the same color scheme and figures to be used across WAPA, and although the new system will permit for regional differences, operators from one region will be able to work in a different control center with minimal training.
“When I look at my Mead Substation in the new software, it shouldn’t be very different than when I look at Oahe Substation in Upper Great Plains or a similar substation in Sierra Nevada,” said Reamy. “There should be an overlying standardization on how the substation will be laid out. That’s what we are trying to get to.”
The standardization team is also working with industry partners, such as the Electric Power Research Institute, in their quest to apply leading practices based on human performance studies and the OSI vendor. One thing that supports the development of the new standards is that everyone goes through a change with the new software.
“We may not always agree, but we need to have alignment,” said Reamy. “It helps that everyone will be stepping into this fresh with a brand-new shiny product. Everyone will have to change. The group of people we have are forward thinkers and will do the best for WAPA now and in the future.”
This Operator Information standardization team anticipates completing its tasks by the end of 2020, paving the way for implementation preplanning to start later this year.
There are other WAPA-wide standardization efforts that will be launched as the team learns the new product and tries to gain additional operational efficiencies.
Implementation, the beginning of the end
The transition to the new software is projected to take about 16-24 months per region and will occur on a staggered schedule. Sierra Nevada will be the first to begin implementation preplanning in late 2020, followed by Upper Great Plains starting in late 2021 and finally Rocky Mountain and Desert Southwest beginning in late 2023.
“We anticipate the implementation phase to take about five years total in addition to the time we have already spent on the project,” said Jamrowski. “The lessons learned with each implementation will help improve later transitions. Keeping people going in a long-running project is hard. All these people have their day jobs they are trying to get done, but this is very important to WAPA’s future.”
Many on the team see the opportunity for more beneficial changes once the common SCADA system is established, including standardizing the switching process used by the craft to manually take equipment in and out of service.
“I hope it will start to snowball so that we’re performing maintenance and switching the same way,” said Reamy. “It will be better WAPA-wide for Maintenance and Operations personnel involved. Our Maintenance crews are traveling to different regions all the time, so if all the switching is unified, it is going to be easier for them. There won’t be the discrepancies like there are right now. Even Loveland and Phoenix are different in switching.”
Implementing common tools, a core tenet in the Strategic Roadmap 2024 Critical Pathway of Business, Technology and Organizational Excellence, will result in fewer maintenance, service, installation, employee training and other redundant costs paid separately per region for the same service. This cost avoidance will allow WAPA to redirect limited resources to other pressing priorities.
“Watching these people coalesce into an effective workgroup made me proud to be associated with them,” said Phillips.
“In addition to supporting WAPA’s highest goals, this project demonstrates innovation, stewardship, partnership and transparency—principles that are the foundation of IT at WAPA,” said Senior Vice President and Chief Information Officer Mike Montoya. “There is no doubt in my mind that WAPA’s greatest asset is its employees. They are capable of designing, deploying and maintaining whatever system WAPA needs as the industry evolves.”
Note: Meiman is a public affairs specialist.