Stories about computer security tend to go viral when they bridge the vast divide between geeks and luddites, and this week’s news about a hacker who tried to poison a Florida town’s water supply was understandably front-page material. But for security nerds who’ve been warning about this sort of thing for ages, the most surprising aspect of the incident seems to be that we learned about it at all.

Spend a few minutes searching Twitter, Reddit or any number of other social media sites and you’ll find countless examples of researchers posting proof of being able to access so-called “human-machine interfaces” — basically web pages designed to interact remotely with various complex systems, such as those that monitor and/or control things like power, water, sewage and manufacturing plants.

And yet, there have been precious few known incidents of malicious hackers abusing this access to disrupt these complex systems. That is, until this past Monday, when Florida county sheriff Bob Gualtieri held a remarkably clear-headed and fact-filled news conference about an attempt to poison the water supply of Oldsmar, a town of around 15,000 not far from Tampa.

Gualtieri told the media that someone (they don’t know who yet) remotely accessed a computer for the city’s water treatment system (using Teamviewer) and briefly increased the amount of sodium hydroxide (a.k.a. lye used to control acidity in the water) to 100 times the normal level.

“The city’s water supply was not affected,” The Tampa Bay Times reported. “A supervisor working remotely saw the concentration being changed on his computer screen and immediately reverted it, Gualtieri said. City officials on Monday emphasized that several other safeguards are in place to prevent contaminated water from entering the water supply and said they’ve disabled the remote-access system used in the attack.”

In short, a likely inexperienced intruder somehow learned the credentials needed to remotely access Oldsmar’s water system, did little to hide his activity, and then tried to change settings by such a wide margin that the alterations would be hard to overlook.

“The system wasn’t capable of doing what the attacker wanted,” said Joe Weiss, managing partner at Applied Control Solutions, a consultancy for the control systems industry. “The system isn’t capable of going up by a factor of 100 because there are certain physics problems involved there. Also, the changes he tried to make wouldn’t happen instantaneously. The operators would have had plenty of time to do something about it.”

Weiss was just one of a half-dozen experts steeped in the cybersecurity aspects of industrial control systems that KrebsOnSecurity spoke with this week. While all of those interviewed echoed Weiss’s conclusion, most also said they were concerned about the prospects of a more advanced adversary.

Here are some of the sobering takeaways from those interviews:

• There are approximately 54,000 distinct drinking water systems in the United States.
• The vast majority of those systems serve fewer than 50,000 residents, with many serving just a few hundred or thousand.
• Virtually all of them rely on some type of remote access to monitor and/or administer these facilities.
• Many of these facilities are unattended, underfunded, and do not have someone watching the IT operations 24/7.
• Many facilities have not separated operational technology (the bits that control the switches and levers) from safety systems that might detect and alert on intrusions or potentially dangerous changes.

So, given how easy it is to search the web for and find ways to remotely interact with these HMI systems, why aren’t there more incidents like the one in Oldsmar making the news? One reason may be that these facilities don’t have to disclose such events when they do happen.

NO NEWS IS GOOD NEWS?

The only federal law that applies to the cybersecurity of water treatment facilities in the United States is America’s Water Infrastructure Act of 2018, which requires water systems serving more than 3,300 people “to develop or update risk assessments and emergency response plans.”

There is nothing in the law that requires such facilities to report cybersecurity incidents, such as the one that happened in Oldsmar this past weekend.

“It’s a difficult thing to get organizations to report cybersecurity incidents,” said Michael Arceneaux, managing director of the Water ISAC, an industry group that tries to facilitate information sharing and the adoption of best practices among utilities in the water sector. The Water ISAC’s 450 members serve roughly 200 million Americans, but its membership comprises less than one percent of the overall water utility industry.

“Some utilities are afraid that if their vulnerabilities are shared the hackers will have some inside knowledge on how to hack them,” Arceneaux said. “Utilities are rather hesitant to put that information in a public domain or have it in a database that could become public.”

Weiss said the federal agencies are equally reluctant to discuss such incidents.

“The only reason we knew about this incident in Florida was that the sheriff decided to hold a news conference,” Weiss said. “The FBI, Department of Homeland Security, none of them want to talk about this stuff publicly. Information sharing is broken.”

By way of example, Weiss said that not long ago he was contacted by a federal public defender representing a client who’d been convicted of hacking into a drinking water system. The attorney declined to share his client’s name, or divulge many details about the case. But he wanted to know if Weiss would be willing to serve as an expert witness who could help make the actions of a client sound less scary to a judge at sentencing time.

“He was defending this person who’d hacked into a drinking water system and had gotten all the way to the pumps and control systems,” Weiss recalled. “He said his client had only been in the system for about an hour, and he wanted to know how much damage could his client really could have done in that short a time. He was trying to get a more lenient sentence for the guy.”

Weiss said he’s tried to get more information about the defendant, but suspects the details of the case have been sealed.

Andrew Hildick-Smith is a consultant who served nearly 20 years managing remote access systems for the Massachusetts Water Resources Authority. Hildick-Smith said his experience working with numerous smaller water utilities has driven home the reality that most are severely under-staffed and underfunded.

“A decent portion of small water utilities depend on their community or town’s IT person to help them out with stuff,” he said. “When you’re running a water utility, there are so many things to take care of to keep it all running that there isn’t really enough time to improve what you have. That can spill over into the remote access side, and they may not have a IT person who can look at whether there’s a better way to do things, such as securing remote access and setting up things like two-factor authentication.”

Hildick-Smith said most of the cybersecurity incidents that he’s aware of involving water facilities fall into two categories. The most common are compromises where the systems affected were collateral damage from more opportunistic intrusions.

“There’ve been a bunch of times where water systems have had their control system breached, but it’s most often just sort of by chance, meaning whoever was doing it used the computer for setting up financial transactions, or it was a computer of convenience,” Hildick-Smith siad. “But attacks that involved the step of actually manipulating things is pretty short list.”

The other, increasingly common reason, he said, is of course ransomware attacks on the business side of water utilities.

“Separate from the sort of folks who wander into a SCADA system by mistake on the water side are a bunch of ransomware attacks against the business side of the water systems,” he said. “But even then you generally don’t get to hear the details of the attack.”

Hildick-Smith recalled a recent incident at a fairly large water utility that got hit with the Egregor ransomware strain.

“Things worked out internally for them, and they didn’t need to talk to the outside world or the press about it,” he said. “They made contact with the Water ISAC and the FBI, but it certainly didn’t become a press event, and any lessons they learned haven’t been able to be shared with folks.”

AN INTERNATIONAL CHALLENGE

The situation is no different in Europe and elsewhere, says Marcin Dudek, a control systems security researcher at CERT Polska, the computer emergency response team which handles cyber incident reporting in Poland.

Marcin said if water facilities have not been a major target of profit-minded criminal hackers, it is probably because most of these organizations have very little worth stealing and usually no resources for paying extortionists.

“The access part is quite easy,” he said. “There’s no business case for hacking these types of systems. Quite rarely do they have a proper VPN [virtual private network] for secure remote connection. I think it’s because there is not enough awareness of the problems of cybersecurity, but also because they are not financed enough. This goes not only for the US. It’s very similar here in Poland and different countries as well.”

Many security professionals have sounded off on social media saying public utilities have no business relying on remote access tools like Teamviewer, which by default allows complete control over the host system and is guarded by a simple password.

But Marcin says Teamviewer would actually be an improvement over the types of remote access systems he commonly finds in his own research, which involves HMI systems designed to be used via a publicly-facing website.

“I’ve seen a lot of cases where the HMI was directly available from a web page, where you just log in and are then able to change some parameters,” Marcin said. “This is particularly bad because web pages can have vulnerabilities, and those vulnerabilities can give the attacker full access to the panel.”

According to Marcin, utilities typically have multiple safety systems, and in an ideal environment those are separated from control systems so that a compromise of one will not cascade into the other.

“In reality, it’s not that easy to introduce toxins into the water treatment so that people will get sick, it’s not as easy as some people say,” he said. Still, he worries about more advanced attackers, such as those responsible for multiple incidents last year in which attackers gained access to some of Israel’s water treatment systems and tried to alter water chlorine levels before being detected and stopped.

“Remote access is something we cannot avoid today,” Marcin said. “Most installations are unmanned. If it is a very small water or sewage treatment plant, there will be no people inside and they just login whenever they need to change something.”

SELF EVALUTION TIME

Many smaller water treatment systems may soon be reevaluating their approach to securing remote access. Or at least that’s the hope of the Water Infrastructure Act of 2018, which gives utilities serving fewer than 50,000 residents until the end of June 2021 to complete a cybersecurity risk and resiliency assessment.

“The vast majority of these utilities have yet to really even think about where they stand in terms of cybersecurity,” said Hildick-Smith.

The only problem with this process is there aren’t any consequences for utilities that fail to complete their assessments by that deadline.

Hildick-Smith said while water systems are required to periodically report data about water quality to the U.S. Environmental Protection Agency (EPA), the agency has no real authority to enforce the cybersecurity assessments.

“The EPA has made some kind of vague threats, but they have no enforcement ability here,” he said. “Most water systems are going to wait until close the deadline, and then hire someone to do it for them. Others will probably just self-certify, raise their hands and say, ‘Yeah, we’re good.’”

Microsoft today rolled out updates to plug at least 56 security holes in its Windows operating systems and other software. One of the bugs is already being actively exploited, and six of them were publicized prior to today, potentially giving attackers a head start in figuring out how to exploit the flaws.

Nine of the 56 vulnerabilities earned Microsoft’s most urgent “critical” rating, meaning malware or miscreants could use them to seize remote control over unpatched systems with little or no help from users.

The flaw being exploited in the wild already — CVE-2021-1732 — affects Windows 10, Server 2016 and later editions. It received a slightly less dire “important” rating and mainly because it is a vulnerability that lets an attacker increase their authority and control on a device, which means the attacker needs to already have access to the target system.

Two of the other bugs that were disclosed prior to this week are critical and reside in Microsoft’s .NET Framework, a component required by many third-party applications (most Windows users will have some version of .NET installed).

Windows 10 users should note that while the operating system installs all monthly patch roll-ups in one go, that rollup does not typically include .NET updates, which are installed on their own. So when you’ve backed up your system and installed this month’s patches, you may want to check Windows Update again to see if there are any .NET updates pending.

A key concern for enterprises is another critical bug in the DNS server on Windows Server 2008 through 2019 versions that could be used to remotely install software of the attacker’s choice. CVE-2021-24078 earned a CVSS Score of 9.8, which is about as dangerous as they come.

Recorded Future says this vulnerability can be exploited remotely by getting a vulnerable DNS server to query for a domain it has not seen before (e.g. by sending a phishing email with a link to a new domain or even with images embedded that call out to a new domain). Kevin Breen of Immersive Labs notes that CVE-2021-24078 could let an attacker steal loads of data by altering the destination for an organization’s web traffic — such as pointing internal appliances or Outlook email access at a malicious server.

Windows Server users also should be aware that Microsoft this month is enforcing the second round of security improvements as part of a two-phase update to address CVE-2020-1472, a severe vulnerability that first saw active exploitation back in September 2020.

The vulnerability, dubbed “Zerologon,” is a bug in the core “Netlogon” component of Windows Server devices. The flaw lets an unauthenticated attacker gain administrative access to a Windows domain controller and run any application at will. A domain controller is a server that responds to security authentication requests in a Windows environment, and a compromised domain controller can give attackers the keys to the kingdom inside a corporate network.

Microsoft’s initial patch for CVE-2020-1472 fixed the flaw on Windows Server systems, but did nothing to stop unsupported or third-party devices from talking to domain controllers using the insecure Netlogon communications method. Microsoft said it chose this two-step approach “to ensure vendors of non-compliant implementations can provide customers with updates.” With this month’s patches, Microsoft will begin rejecting insecure Netlogon attempts from non-Windows devices.

A couple of other, non-Windows security updates are worth mentioning. Adobe today released updates to fix at least 50 security holes in a range of products, including Photoshop and Reader. The Acrobat/Reader update tackles a critical zero-day flaw that Adobe says is actively being exploited in the wild against Windows users, so if you have Adobe Acrobat or Reader installed, please make sure these programs are kept up to date.

There is also a zero-day flaw in Google’s Chrome Web browser (CVE-2021-21148) that is seeing active attacks. Chrome downloads security updates automatically, but users still need to restart the browser for the updates to fully take effect. If you’re a Chrome user and notice a red “update” prompt to the right of the address bar, it’s time to save your work and restart the browser.

Standard reminder: While staying up-to-date on Windows patches is a must, it’s important to make sure you’re updating only after you’ve backed up your important data and files. A reliable backup means you’re less likely to pull your hair out when the odd buggy patch causes problems booting the system.

So do yourself a favor and backup your files before installing any patches. Windows 10 even has some built-in tools to help you do that, either on a per-file/folder basis or by making a complete and bootable copy of your hard drive all at once.

Keep in mind that Windows 10 by default will automatically download and install updates on its own schedule. If you wish to ensure Windows has been set to pause updating so you can back up your files and/or system before the operating system decides to reboot and install patches, see this guide.

And as always, if you experience glitches or problems installing any of these patches this month, please consider leaving a comment about it below; there’s a better-than-even chance other readers have experienced the same and may chime in here with some helpful tips.