by Leah Shapiro
A team of dedicated Information Technology employees solved a problem in October that is estimated to save WAPA $50,000.
Staff in the Montrose, Colorado, Energy Management and Marketing Office work around the clock to make sure power is available when WAPA customers need it. To perform this mission-critical work, Energy Management and Marketing Specialists, or merchants, rely on information from the Open Access Technology International network, or OATINet (pronounced OH-tee-net), the connection that allows the EMMO access to OATI’s web-tag system. These services, and the network connection, are vital for Power Marketing and the balancing area’s functionality.
Rocky Mountain Business Systems Manager Tonya Spencer explained, “The system processes energy requests from different entities in the electrical market in real time. As new requests come in via OATI, marketers use their in-house scheduling system, called TIGER [Transmission, Interchange, Generation, Energy and Reserves], to process the information and to verify that the transactions have been successfully completed.”
Whenever Montrose real-time merchants tried to access OATI display screens, they experienced long wait times—usually more than 25 seconds per query. Montrose EMMO Real-Time Supervisor Tim Vigil explained the significance of the slow connection: “Real-time merchants constantly balance load and generation each hour for their respective water project. They engage in transactions each hour to either sell excess generation or purchase to supplement a deficit of generation, and everything has to net to zero. They need to decide quickly if we need to buy or sell energy. To accomplish this, they have to open and review these tags frequently. With a 30-second load time, there were instances when the delay caused merchants to risk meeting the time deadline for these transactions. If you can’t buy transmission quickly—and you can’t meet the tagging deadline of 20 minutes before the start of the transaction hour—somebody ends up short and somebody ends up long. This can result in financial penalties for both parties and affect system operators who are responsible for balancing the entire balancing authority. It’s bad for everybody.”
Senior Vice President and Chief Information Officer Dawn Roth Lindell observed the lag time during a visit to the Montrose office, Sept. 24. Merchants shared their frustration with the slowness between their workstations and OATI and confessed that they had nicknamed the network “SLOW-ti.”
Start at beginning, make no assumptions
Roth Lindell shared, “What I saw appeared to be a problem with our local network, not a problem with OATI. But honestly, I didn’t care where the problem was or whose fault it was; I just wanted it fixed.” She met with Network Specialist Nick Gonzales, who showed her a local network diagram. Roth Lindell said, “It looked good; it was clearly built correctly.”
In the days that followed, Gonzales gathered a team of people to get to the bottom of the “OATI latency” issue, which ultimately required coordination with RM’s network team and system administrators, network peers at OATI and vendors including Sprint, Verizon and Century Link.
The Rocky Mountain network team began troubleshooting the physical components, such as wires and interfaces, and performed a ground-up analysis of all the systems used for the OATI function in Montrose. Gonzales explained, “We also requested that OATI perform analysis of their equipment located in Montrose. This analysis revealed the root cause of the problem.”
It turned out to be a simple solution: Century Link, who leases a line from OATI, had to re-seat an Optical Carrier-48 card inside their network.
The testing to discover the problem required WAPA IT staff to put a dynamic failover solution in place to route real-time traffic through Loveland, which allowed marketers to perform their jobs without interruption. The failover was in place from Oct. 6-14. RM Supervisory Control and Data Acquisition Manager Randy Curtis shared, “The dynamic failover solution has proven on multiple occasions to be invaluable for real-time functions in both Montrose and Loveland because it performed as designed without end-user knowledge or intervention by RM IT personnel.”
Every second counts
From start to finish, the problem was resolved in less than a month. Real-time merchants noticed an improvement immediately after the card was re-seated. Their screens now refresh in four seconds. Due to this fix, operators now save about 21 seconds every time they refresh their screen. Considering the number of operators and the number of times they refresh their screens, the cost and time saving over a year is significant.
Having all this time back allows real-time merchants to operate in a more proactive manner and perform their jobs better. Vigil explained, “Merchants now have more time to analyze their required operating reserves to determine how much excess reserve capacity can be sold on the open market. They can analyze reservoir elevations to determine the best economic generation profiles for the remainder of the day. They can go out and engage in additional transactions and buy and deliver energy to preference customers that make requests through our contracts. It basically gives them more time to optimize their economic positions as well as plan and analyze for future transactions.”
Supervisory Energy Management and Marketing Specialist Ken Otto, who works for the Colorado River Storage Project EMMO in Montrose, shared, “We appreciate the efforts of WAPA’s entire IT community, especially our Montrose and Loveland staff, who are dedicated to assuring we have the necessary software and hardware tools to provide Western’s customers with their firm electric service at all times.”
Roth Lindell shared, “I’m grateful we have network people of Nick’s caliber to do the research that is required to get to the bottom of a problem of this scope. Nick did a great job of coordinating, problem solving, identifying errors and resolving the issue.”
Gonzales shared, “I enjoyed coordinating internal and external IT personnel to discuss the latency issue, determine the best approach to use to identify the root problem, and finally resolve the problem after it was definitively identified.”
Continuous improvement more than buzzword
The primary takeaway from this issue is that processes, procedures and communication can always be improved. Each WAPA support function likely has many opportunities to improve service to its business partners. The team involved in solving this particular problem identified technical and procedural recommendations that will address some of the communication gaps they identified.
Roth Lindell shared, “Loveland and Montrose have a great team of IT support staff. They have really impressed me with their creative problem solving and dogged determination to continuously improve.”