On February 5, 2021, a hacker gained access to the water treatment system of Oldsmar, Florida, and attempted to increase the levels of sodium hydroxide, putting thousands of city residents at risk of being poisoned. According to city officials, operators onsite noticed the intrusion and took immediate action to reduce the levels back to normal.
It’s not really fair for the engineers and managers at Oldsmar, but they have now become the answer to a question they never thought would be asked. How vulnerable, really, are the infrastructure sites that provide the stuff of life — clean water, in this case — to cybersecurity attacks?
As it turns out, these sites are tremendously vulnerable, and not because the most skillful nation-state attackers have suddenly taken an interest. Unlike the high-profile “critical infrastructure” sites, which benefit from support from Homeland Security and dedicated budgets for security personnel and tools, most public utilities don’t enjoy resources for anything related to security. No staff; no tools; no external reviews; they simply hope for the best because, so far, nobody has recognized their importance.
It could have been any one of thousands of sites across the country in similar situations. Much of the public discourse following the attack has focused on “mistakes” Oldsmar made, but that’s not a fair treatment of the subject matter. We don’t blame the driver when their car is destroyed after a deer runs in front of them on a dark road. These things come out of nowhere, they are essentially unpredictable, and while the technology exists to reduce the risk, it’s generally beyond the reach of most vehicle owners.
The reason Oldsmar used TeamViewer, a remote access desktop software, was that it didn’t cost them anything extra. With no latitude in typical budgets, and review and approval processes that are unwilling or unable to invest in a whole-risk perspective, operators are left with no alternative. Sometimes they need to get into the plant from outside. The people who need to do it are senior-level management, with significant responsibility and experience, and they can’t be constrained from doing what needs to be done.
Cybersecurity practitioners are quick to highlight some low-cost methods to reduce the risk. That’s better than pitching big-dollar solutions, of course, but every plant will make a simple calculation about how much time they can spare. It comes down to one or two headcounts — can I take my senior automation engineer off his normal routine for a week to make some changes? Maybe the answer is ‘yes’ for some plants, and in the wake of Oldsmar perhaps they will, but for a lot of plants, the answer is no. They don’t have business continuity or disaster recovery plans because they can’t afford the resources required to implement them properly.
All that said, every plant should consider prioritizing at least an assessment of how much like Oldsmar they may or may not be. Implementing a strong password on a remote access tool is better than not having one. Driving at least some minor change into the heart of old, embedded habits, in an effort to reveal specific vulnerabilities is a good habit for any manager, whether or not they specialize in cybersecurity.
In recent years the OT cybersecurity industry as a whole has seen an enormous influx of venture capital — over $500M at last count. Most of the companies in the space, having raised significant amounts of capital, are pressured to generate returns quickly. This means they tend to price their solutions as premium solutions, and ultimately the selection bias of such a strategy leads them all to pursue the same small set of very high-end customers. Most industrial and automation sites are completely overlooked, dismissed because “they can’t afford it” or “they don’t get it” or similar tropes.
The reality, however, is far different. The United States alone has something in excess of 30,000 community water systems, and 99%+ of them are broadly similar to Oldsmar in terms of their preparedness for cybersecurity incident response. Again, this isn’t a criticism, it’s a reflection of the reality of ratepayer-driven services where the mystery of how it all works is distilled down to a penny or two per month per customer.
The federal government’s Solarium Commission, created in 2018 to study the nation’s critical infrastructure and its ability to deal with cybersecurity threats, focused on water systems as a particular target. When its report was published in 2020, it recommended extending support to all water systems, not just the one that services the US Capitol and the White House, because attacking water is a great way to pursue terrorist attacks. Literally, everyone depends on clean water, and if you can’t trust what comes out of your tap, your options are limited if you’re not already on a well or rainwater cisterns.
The good news in all this is that solutions don’t have to be expensive, and they don’t have to come from overcapitalized Silicon Valley darlings. There are plenty of inexpensive solutions and free lessons to be learned, all of which can be implemented with existing staff and expertise.
The first thing to do is ask some basic questions:
Do I know what’s on my network? Do I know who has access to it?
Do I know how people get into those devices, both internally and externally?
Do I have good passwords in place? Do I change them periodically?
Can I verify all of the above, either myself or via someone else?
Do I know what to do with the answers to all of these questions?
The last question isn’t meant to be a gotcha. Any plant manager can learn, to at least a basic level, what to do with the answers to questions 1-4 without hiring a consultant or otherwise writing a check. Infrastructure operators, as a matter of due diligence and maintaining the public trust, should be able to take themselves out of the “wide open to exploit” category by addressing those questions, first.
From there, learn to pinpoint your specific needs as your program evolves. While it’s easy to find OT cybersecurity solutions that cost more than your entire annual operating budget, that’s not the only way. Companies that offer modular industrial cybersecurity solutions provide a holistic approach that is explicitly designed not to require you to hire new staff with new skills in order to be successful. A modest level of budget spend should be in scope for even the smaller utilities, and any responsible supplier will maximize your utility’s overall chances of success by encouraging you to focus on what you can do yourself, first, before looking to outside assistance.
The landscape for networked infrastructure has changed dramatically, and it’s easy to miss that evolution when you’re focused on internal activities. Think about email — when it first appeared, by definition everyone with an account on ARPAnet was not only a sophisticated user, but they also had a US government security clearance. This is why the underlying standards for email didn’t have a separate security layer; every user was trusted by default. Automation and industrial control systems are essentially no different because it was believed at the time they were built that they’d exist in a private system with no risk of external access. And yet, here we are.
If you’re a control systems manager, the one lesson to take from this piece is that it’s important to understand the world around you has changed, particularly if you’ve never given much thought to security until the Oldsmar news hit the wire last week. This isn’t the type of problem with an “Easy Button” solution, but nor is it so esoteric that you have no practical options. Look for solutions that fit alongside what you already have, without demanding huge and expensive infrastructure upgrades. Look for providers who understand ICS and OT, and aren’t just tacking on buzzwords to their IT-centric products. Look for stories and case studies that don’t depend on the most elaborate and complex planning and sophistication. None of that is a reflection of the reality most American utilities face every day, and those systems are the heart of the public safety infrastructure. You deserve a forum in which your needs and constraints are respected and understood.