Globally, hundreds of thousand of organizations running Exchange email servers from Microsoft just got mass-hacked, including at least 30,000 victims in the United States. Each hacked server has been retrofitted with a “web shell” backdoor that gives the bad guys total, remote control, the ability to read all email, and easy access to the victim’s other computers. Researchers are now racing to identify, alert and help victims, and hopefully prevent further mayhem.

On Mar. 5, KrebsOnSecurity broke the news that at least 30,000 organizations and hundreds of thousands globally had been hacked. The same sources who shared those figures say the victim list has grown considerably since then, with many victims compromised by multiple cybercrime groups.

Security experts are now trying to alert and assist these victims before malicious hackers launch what many refer to with a mix of dread and anticipation as “Stage 2,” when the bad guys revisit all these hacked servers and seed them with ransomware or else additional hacking tools for crawling even deeper into victim networks.

But that rescue effort has been stymied by the sheer volume of attacks on these Exchange vulnerabilities, and by the number of apparently distinct hacking groups that are vying for control over vulnerable systems.

A security expert who has briefed federal and military advisors on the threat says many victims appear to have more than one type of backdoor installed. Some victims had three of these web shells installed. One was pelted with eight distinct backdoors. This initially caused a major overcount of potential victims, and required a great deal of de-duping various victim lists.

The source, who spoke on condition of anonymity, said many in the cybersecurity community recently saw a large spike in attacks on thousands of Exchange servers that was later linked to a profit-motivated cybercriminal group.

“What we thought was Stage 2 actually was one criminal group hijacking like 10,000 exchange servers,” said one source who’s briefed U.S. national security advisors on the outbreak.

On Mar. 2, when Microsoft released updates to plug the four Exchange flaws being attacked, it attributed the hacking activity to a previously unidentified Chinese cyber espionage group it called “Hafnium.” Microsoft said Hafnium had been using the Exchange flaws to conduct a series of low-and-slow attacks against specific strategic targets, such as non-governmental organization (NGOs) and think tanks.

But by Feb. 26, that relatively stealthy activity was morphing into the indiscriminate mass-exploitation of all vulnerable Exchange servers. That means even Exchange users that patched the same day Microsoft released security updates may have had servers seeded with backdoors.

Many experts who spoke to KrebsOnSecurity said they believe different cybercriminal groups somehow learned of Microsoft’s plans to ship fixes for the Exchange flaws a week earlier than they’d hoped (Microsoft originally targeted today, Patch Tuesday, as the release date).

The vulnerability scanning activity also ramped up markedly after Microsoft released its updates on Mar. 2. Security researchers love to tear apart patches for clues about the underlying security holes, and one major concern is that various cybercriminal groups may have already worked out how to exploit the flaws independently.

AVERTING MASS-RANSOMWARE

Security experts now are desperately trying to reach tens of thousands of victim organizations with a single message: Whether you have patched yet or have been hacked, backup any data stored on those servers immediately.

Every source I’ve spoken with about this incident says they fully expect profit-motivated cybercriminals to pounce on victims by mass-deploying ransomware. Given that so many groups now have backdoor web shells installed, it would be trivial to unleash ransomware on the lot of them in one go. Also, compromised Exchange servers can be a virtual doorway into the rest of the victim’s network.

“With the number of different threat actors dropping [web] shells on servers increasing, ransomware is inevitable,” said Allison Nixon, chief research officer at Unit221B, a New York City-based cyber investigations firm.

So far there are no signs of victims of this mass-hack being ransomed. But that may well change if the exploit code used to break into these vulnerable Exchange servers goes public. And nobody I’ve interviewed seems to think working exploit code is going to stay unpublished for much longer.

When that happens, the exploits will get folded into publicly available exploit testing kits, effectively making it simple for any attacker to find and compromise a decent number of victims who haven’t already patched.

CHECK MY OWA

Nixon is part of a group of security industry leaders who are contributing data and time to a new victim notification platform online called Check My OWA (Outlook Web Access, the Internet-facing Web component of Exchange Server machines).

Checkmyowa.unit221b.com checks if your Exchange Server domain showed up in attack logs or lists of known-compromised domains.

Perhaps it’s better to call it a self-notification service that is operated from Unit221B’s own web site. Enter an email address at Check My OWA, and if that address matches a domain name for a victim organization, that email address will get a notice.

“Our goal is to motivate people who we might otherwise have never been able to contact,” Nixon said. “My hope is if this site can get out there, then there’s a chance some victim companies are notified and take action or can get att

If the email’s domain name (anything to the right of the @ sign) is detected in their database, the site will send that user an email stating that is has observed the email domain in a list of targeted domains.

“Malicious actors were able to successfully compromise, and some of this information suggested they may have been able to install a webshell on an Exchange server associated with this domain,” reads one of the messages to victims. “We strongly recommend saving an offline backup of your Exchange server’s emails immediately, and refer back to the site for additional information on patching and remediation.”

“We have observed your e-mail domain appears in our list of domains the malicious actors were able to successfully compromise, and some of this information suggested they may have been able to install a webshell on an Exchange server associated with this domain,” is another message the site may return.

Nixon said Exchange users can save themselves a potentially nightmarish scenario if they just back up any affected systems now. And given the number of adversaries currently attacking still-unpatched Exchange systems, there is almost no way this won’t end in disaster for at least some victims.

“There are researchers running honeypots to [attract] attacks from different groups, and those honeypots are getting shelled left and right,” she said. “The sooner they can run a backup, the better. This can help save a lot of heartache.”

Oh, and one more important thing: You’ll want to keep any backups disconnected from everything. Ransomware has a tendency to infect everything it can, so make sure at least one backup is stored completely offline.

“Just disconnect them from a computer, put them in a safe place and pray you don’t need them,” Nixon said.

Sometimes when a complex story takes us by surprise or knocks us back on our heels, it pays to revisit the events in a somewhat linear fashion. Here’s a brief timeline of what we know leading up to last week’s mass-hack, when hundreds of thousands of Microsoft Exchange Server systems got compromised and seeded with a powerful backdoor Trojan horse program.

When did Microsoft find out about attacks on previously unknown vulnerabilities in Exchange?

Pressed for a date when it first became aware of the problem, Microsoft told KrebsOnSecurity it was initially notified “in early January.” So far the earliest known report came on Jan. 5, from a principal security researcher for security testing firm DEVCORE who goes by the handle “Orange Tsai.” DEVCORE is credited with reporting two of the four Exchange flaws that Microsoft patched on Mar. 2.

Reston, Va.-based Volexity first identified attacks on the flaws on Jan. 6, and officially informed Microsoft about it on Feb. 2. Volexity now says it can see attack traffic going back to Jan. 3. Microsoft credits Volexity with reporting the same two Exchange flaws as DEVCORE.

Danish security firm Dubex says it first saw clients hit on Jan. 18, and reported their incident response findings to Microsoft on Jan. 27.

In a blog post on their discovery, Please Leave an Exploit After the Beep, Dubex said the victims it investigated in January had a “web shell” backdoor installed via the “unifying messaging” module, a component of Exchange that allows an organization to store voicemail and faxes along with emails, calendars, and contacts in users’ mailboxes.

“A unified messaging server also allows users access to voicemail features via smartphones, Microsoft Outlook and Outlook Web App,” Dubex wrote. “Most users and IT departments manage their voicemail separately from their email, and voicemail and email exist as separate inboxes hosted on separate servers. Unified Messaging offers an integrated store for all messages and access to content through the computer and the telephone.”

Dubex says Microsoft “escalated” their issue on Feb. 8, but never confirmed the zero-day with Dubex prior to the emergency patch plea on Mar. 2. “We never got a ‘real’ confirmation of the zero-day before the patch was released,” said Dubex’s Chief Technology Officer Jacob Herbst.

How long have the vulnerabilities exploited here been around?

On Mar. 2, Microsoft patched four flaws in Exchange Server 2013 through 2019. Exchange Server 2010 is no longer supported, but the software giant made a “defense in depth” exception and gave Server 2010 users a freebie patch, too. That means the vulnerabilities the attackers exploited have been in the Microsoft Exchange Server code base for more than ten years.

The timeline also means Microsoft had almost two months to push out the patch it ultimately shipped Mar. 2, or else help hundreds of thousands of Exchange customers mitigate the threat from this flaw before attackers started exploiting it indiscriminately.

Here’s a rough timeline as we know it so far:

• Jan. 5: DEVCORE alerts Microsoft of its findings.
• Jan. 6: Volexity spots attacks that use unknown vulnerabilities in Exchange.
• Jan. 8: DEVCORE reports Microsoft had reproduced the problems and verified their findings.
• Jan. 11: DEVCORE snags proxylogon.com, a domain now used to explain its vulnerability discovery process.
• Jan. 27: Dubex alerts Microsoft about attacks on a new Exchange flaw.
• Jan. 29: Trend Micro publishes a blog post about “Chopper” web shells being dropped via Exchange flaws (but attributes cause as Exchange bug Microsoft patched in 2020)
• Feb. 2: Volexity warns Microsoft about active attacks on previously unknown Exchange vulnerabilities.
• Feb. 8: Microsoft tells Dubex it has “escalated” its report internally.
• Feb. 18: Microsoft confirms with DEVCORE a target date of Mar. 9 (tomorrow) for publishing security updates for the Exchange flaws. That is the second Tuesday of the month — a.k.a. “Patch Tuesday,” when Microsoft releases monthly security updates (and yes that means check back here tomorrow for the always riveting Patch Tuesday roundup).
• Feb. 26-27: Targeted exploitation gradually turns into a global mass-scan; attackers start rapidly backdooring vulnerable servers.
• Mar. 2: A week earlier than previously planned, Microsoft releases updates to plug 4 zero-day flaws.
• Mar. 2: DEVCORE researcher Orange Tsai (noted for finding and reporting some fairly scary bugs in the past) jokes that nobody guessed Exchange as the source of his Jan. 5 tweet about “probably the most serious [remotely exploitable bug] I have ever reported.”
• Mar. 3: Tens of thousands of Exchange servers compromised worldwide, with thousands more servers getting freshly hacked each hour.
• Mar. 4: White House National Security Advisor Jake Sullivan tweets about importance of patching Exchange flaws, and how to detect if systems are already compromised.
• Mar. 5, 1:26 p.m. ET: In live briefing, White House press secretary Jen Paski expresses concern over the size of the attack.
• Mar. 5, 4:07 p.m. ET: KrebsOnSecurity breaks the news that at least 30,000 organizations in the U.S. — and hundreds of thousands worldwide — now have backdoors installed.
• Mar. 5, 6:56 p.m. ET: Wired.com confirms the reported number of victims.
• Mar. 5, 8:04 p.m. ET: Former CISA head Chris Krebs tweets the real victim numbers “dwarf” what’s been reported publicly.
• Mar. 6: CISA says it is aware of “widespread domestic and international exploitation of Microsoft Exchange Server flaws.”
• Mar. 7-Present: Security experts continue effort to notify victims, coordinate remediation, and remain vigilant for “Stage 2” of this attack (further exploitation of already-compromised servers).

Update, 12:11 p.m. ET: Correct link to Dubex site (it’s Dubex.dk). Also clarified timing of White House press statement expressing concern over the number of the Exchange Server compromises. Corrected date of Orange Tsai tweet.