The Colonial pipeline ransomware shutdown: A wake-up call for energy cybersecurity

How digital hostage-taking can bring about the breakdown of fossil fuel networks — and threaten the power grid.
By Jeff St. John

  • Link copied to clipboard

The U.S. energy infrastructure’s exposure to cyberattack was thrown into stark relief over the weekend when Colonial Pipeline closed down its 5,500-mile, Texas-to-New Jersey fossil fuel network in response to a ransomware attack.

Now cybersecurity experts are looking at what the country’s most severe energy-sector hack to date could mean for the broader electricity sector — and what industry and government agencies can do to prepare for an increasingly common cyberthreat.

The effectiveness of ransomware in taking down a physical infrastructure system flies in the face of the narrative we’ve heard for the past 10 years” from pipeline operators, said Jonathon Monken, a principal at Converge Strategies and an Army reservist supporting the U.S. Homeland Security Department’s National Cybersecurity and Communications Integration Center.

That oft-repeated narrative holds that pipelines could be manually operated even if ransomware attacks locked up the pipeline operator’s IT systems, making recovery possible in a matter of days. The contention was one Monken heard often in his past role as senior director of system resiliency and strategic coordination at PJM, the country’s biggest interstate grid operator.

I spent the better part of two years on a task force trying to unpack the level of vulnerability and disruption to natural-gas pipelines that were the major source of delivery to a lot of the power plants in PJM territory,” he said in a Monday interview.

Most operators were worried about what they categorize as advanced persistent threats” from countries such as Russia and China, which for years have been reported to have already infiltrated U.S. energy industry enterprise IT systems. If those state-sponsored hackers can bridge from IT systems to the utility operational technology systems that run power plants or pipelines, they could actively sabotage those operations, much like hackers tied to Russia did to Ukraine’s grid in 2015, Monken said.

Ransomware attacks, which exploit vulnerabilities in IT networks to steal data, encrypt files and render computers inoperable, and then demand payment to release them, have always been regarded as a threat. But industry discussions about them were usually concluded with the assertion that if something like that does happen, there wouldn’t be a need to shut down the entire system,” he said.

Gas operators always tout their ability to operate the system manually,” he said. “‘There are physical valves we turn. It’s just a matter of keeping pressure in the system.’” But, according to Monken, this event with Colonial shows that is not true.”

From taking computers hostage to disabling energy infrastructure

The costs of ransomware attacks have been growing, with victims ranging from private businesses to government agencies and universities. A March report from cyberthreat researchers at Palo Alto Networks’ Unit 42 found that average ransomware demands have tripled over the past year to $312,000 per affected company, while the highest ransom paid doubling from $5 million to $10 million. An Eastern European group called DarkSide has been identified as playing a role in the Colonial Pipeline attack; it has reportedly made a lucrative business of this digital extortion racket around the globe.

But the shutdown of a pipeline supplying about 45 percent of the East Coast’s gasoline stands out for its potentially devastating impacts. While reports of fuel shortages hadn’t emerged as of Monday afternoon, the U.S. Department of Transportation issued an emergency declaration in 17 states and Washington, D.C. to lift normal operating restrictions on trucks to carry fuel to areas that may face shortages.

Colonial reported that it proactively took certain systems offline to contain the threat, which temporarily halted all pipeline operations and affected some of our IT systems.” On Monday, it reported that it expects to be able to restore full service by the end of this week, which could help the country avoid severe fuel shortages.

How can the seizure of enterprise IT assets cause a weeklong shutdown? Monken offered several hypotheses, though he cautioned that they are merely speculative.

First, it’s likely that a failure of very basic cyber-hygiene” granted hackers access to Colonial’s network, he said. An employee clicked on a bad link, used a bad password [or] a basic security patch wasn’t done.”

These types of boring” failures are linked to most of the worst hacks out there, he said. But because they’re so hard to police, there has to be more attention paid to the fact that you have to have a much stronger contingency plan for a ransomware attack — and that it can’t be just a we don’t negotiate with terrorists’ stance.”

That’s because ransomware can spread very rapidly across an enterprise IT network, he said. In 2015, a cyberattack on Saudi Aramco crippled more than 35,000 computers in a matter of hours, forcing the national oil company to resort to phone calls and faxes to conduct its business.

While that attack was not linked to an extortion effort, its crippling effect on business operations highlights the potential for a similar impact to Colonial’s pipeline business, even if pipeline operations themselves weren’t affected.

Imagine, he said, trying to conduct the business of selling 45 percent of the East Coast’s gasoline like a racetrack bookie with a chalkboard behind him, trying to calculate the odds in real time. […] There’s no way you can maintain the fidelity of those transactions. […] They’re not going to hand over hundreds of millions of dollars of product with no guarantee they’ll get paid for it.”

Minding the IT-OT gap

Some kind of government intervention to backstop this financial threat could allow operations to continue, he said.

But that wouldn’t do much good if the malware used in the attack was able to spread to operational technology [OT] devices that haven’t been property isolated from the broader enterprise IT network, he said.

Even when they’re in manual mode, there’s somebody there [operating] it, [and] they still have an IT component to access the local system,” he said. These human-machine interfaces” are vital links in a 5,500-mile pipeline network.

They don’t have enough people in the entire company to fan everybody out to every single asset in their system and stand there with a walkie-talkie, turning cranks when they’re directed to,” he said.

Critical infrastructure operators take pains to create air gaps” between these mission-critical controllers and their internet-connected enterprise networks, he said. But internal OT networks can be inadvertently or purposely connected to broader IT networks in ways that compromise those gaps.

If Colonial had any…evidence to suggest the ransomware was making it into their human-machine interfaces, I could see them pulling the plug immediately,” he said. Replacing thousands of standard desktop or notebook computers wrecked by a ransomware attack is easy compared to replacing specialty devices purpose-built for industrial control systems, according to Monken.

Steps to prevent ransomware attacks — and to recover from them

Critical infrastructure operators, such as PJM and the country’s other transmission grid operators, have taken these kinds of risks into account, Monken said.

PJM’s strategy [is based on] something called a golden image’ — an air-gapped backup of the entire energy management system,” which it can use to rebuild its system in case of catastrophic attack. The grid operator has multiple backup control sites and contingency plans for replacing workstations crippled by cyberattack. It has also developed fallback strategies to operate its wholesale energy markets in event of losing large portions of its IT infrastructure to sustain continuity of operations in that environment.”

PJM is fortunate in this regard [because]…the physical infrastructure is limited to [its] operations centers,” he said. If you’re a company like Colonial and have tens of thousands of workstations…you have to figure out those subsets of core infrastructure, those hardware and software assets, that absolutely have to be functional.”

While the gasoline and diesel fuel that fills most of Colonial’s pipeline network can be stored and trucked, a similar attack on the U.S. natural-gas pipeline network could be more disruptive, halting just-in-time supplies to power plants across the country. The result could be a cyberattack-induced version of the February winter freeze that throttled natural-gas supplies and led to multiday blackouts for millions of Texas residents, Monken said.

The Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency highlighted the potential risk in a report last year, which described how a ransomware attack on an unnamed pipeline operator penetrated the OT system of a natural-gas compressor station, forcing a two-day shutdown of a pipeline. Cybersecurity firm Dragos has cast doubt on this version of events, however, noting that the incident described in the agency’s report appears to describe an occurrence at a U.S. Coast Guard facility, not a natural-gas facility.

The American Gas Association industry group and its Canadian counterpart have been actively researching, analyzing and compiling information about this incident” via the jointly operated Downstream Natural Gas Information Sharing and Analysis Center, an association spokesperson said in a Monday email. The AGA has also been collaborating with federal government partners and industry since early Saturday morning.”

Aligning cybersecurity efforts for electric grids and pipelines

U.S. electric utilities and grid operations must abide by mandatory cybersecurity efforts under the North American Electric Reliability Corporation Critical Infrastructure Protection standards. But pipeline networks are subject only to voluntary cybersecurity standards under the purview of the U.S. Transportation Safety Administration, the same agency responsible for airport security.

As of 2019, TSA had only six full-time employees tasked with pipeline cybersecurity, rendering it unable to ensure the security of dangerous and susceptible natural gas pipeline infrastructure,” U.S. Senator Ed Markey (D-Mass.) said in a Monday tweet.

The Federal Energy Regulatory Commission regulates interstate transmission networks and has authority over large portions of interstate gas pipelines, but it lacks oversight over pipeline cybersecurity. In 2019, then-FERC Chair Neil Chatterjee testified to a Senate committee that a successful cyberattack on the natural gas pipeline system could have a significant impact on the electric grid.”

On Monday, current FERC Chair Richard Glick issued a statement reiterating that encouraging pipelines to voluntarily adopt best practices is an inadequate response to the ever-increasing number and sophistication of malevolent cyber actors.”

The Biden administration last month launched a 100-day plan between the Cybersecurity and Infrastructure Security Agency, the Department of Energy and the electricity industry to review electricity system cybersecurity. This effort was aimed in part at addressing the threat of bulk power system equipment that may contain internal components embedded with cybersecurity threats, most likely from foreign governments.

Karen Wayland, president of the utility and technology vendor group GridWise Alliance, said that utilities and grid operators have to expect that hackers are going to get into your system.” The group is seeking $2 billion for cybersecurity job training, threat monitoring and technology deployment as part of a $50 billion proposal for federal grid spending under the Biden administration’s infrastructure plan.

A 2020 report from DOE’s Lawrence Livermore National Laboratory found that cybersecurity efforts in the electric industry are currently outpacing” those of the U.S. oil and natural gas industry. It identified the need for a coherent, comprehensive, multilayered strategy” to close that gap.

Mandatory standards may help, Monken said, but developing them is a decades-long process.” In the meantime, there’s got to be an open acknowledgement of what the threat is,” he said. This type of malware used in an attack like this is readily available on the dark web.”

Investing in cybersecurity to prevent such attacks, and in the resources to recover from those that succeed, can be an enormously expensive proposition,” he noted. If you’re a commodity business, you may find yourself…saying, I’m going to roll the dice and hope that doesn’t happen’” — much like a generator in Texas saying, It’s not going to get that cold.’”

(Lead photo: Quinten de Graff) 

Jeff St. John is director of news and special projects at Canary Media. He covers innovative grid technologies, rooftop solar and batteries, clean hydrogen, EV charging and more.