A raid on U.S. government Outlook accounts over the summer was blamed on Chinese hackers and a stolen signing key, and Microsoft has now released details about exactly how the incident began.
Microsoft has traced the signing key theft back to a “crash dump” of a signing system in the production network that mistakenly had the key in it, and that was moved from the isolated production environment to the company’s internet-facing network without anyone being aware of the issue. A breach of a Microsoft engineer’s work account by the Chinese hackers then yielded access to the crash dump and the embedded signing key.
Chinese hackers gained access in mid-May, no classified information obtained
Initial reports about the incident were downplayed by both Microsoft and the government despite the scope of officials and departments that appeared to be penetrated, with insistences that the Outlook accounts that were accessed were not used for classified materials. Microsoft appears to be confirming that only unclassified information was available to the Chinese hackers by way of the stolen signing key.
The Chinese hackers, a group Microsoft has labeled as “Storm-0558,” appear to have been randomly fortunate in compromising an account of an engineer with access to the debugging process for the crashed signing system. The report seems to confirm that the signing key should never have been outside of the isolated production environment, but that a previously unknown race condition error caused it to both be included and to not be detected by automated security systems before moving to the more public portion of the network.
The errant movement of the signing key took place in April 2021, and Microsoft only indicates that “sometime after” that the engineer’s account was compromised and the Chinese hackers made off with it. The report indicates that the company does not have specific logs to track back to exactly where and when the theft took place.
The Chinese hackers compromised some 25 organizations in total, some of them state or local government agencies or private companies. There is still little information about other sources of compromise beyond the several federal agencies and officials that were named as victims. The attackers were limited to logging into Outlook accounts via Outlook.com and Outlook Web Access.
Signing key mishap involved extended chain of failures
The attack was only possible with this particular signing key due to an additional security oversight: an API used to validate signatures apparently had overly broad permission due to failure to implement libraries differentiating between consumer and enterprise key types. Microsoft says that the API has since been updated.
There is still some amount of mystery surrounding the signing key incident. Microsoft offered no concrete description of how the engineer’s account was compromised in the first place, and some security analysts have noted that it is unusual for an organization of this type and size to not have logs in place allowing for a more thorough investigation of the breach’s ultimate origin.
The breach was serious enough to trigger an investigation by the U.S. Cyber Safety Review Board (CSRB), which will also look at other similar large cloud environments to review identity management and authentication procedures. Sen. Ron Wyden has called for the CSRB to specifically investigate Microsoft’s handling of its security and signing keys, and for the Department of Justice to examine if the company reached a level of negligence that would violate federal law.
The incident continues a fairly rough stretch for the security of Microsoft’s cloud-based platforms and hosted software solutions, dating back to the string of Exchange Server vulnerabilities that emerged and caused mass chaos toward the end of 2020. Other low points include the scraping of 500 million LinkedIn users, the Cosmos DB flaw that exposed thousands of Azure customer accounts, the “BlueBleed” data leak that exposed hundreds of thousands of customer files due to a misconfigured endpoint, and a 2022 breach by the Lapsus$ criminal gang that raised the question of Bing and Cortana being compromised.
There is also no shortage of Chinese hackers roaming cyberspace on espionage missions, but Storm-0558 appears to be a particularly advanced group and one that focuses on compromising US and European government agencies (with a particular interest in Taiwan and anything relating to the country’s Uyghur minorities). Storm-0558 is thought to carefully reconnoiter its targets and is generally only a threat to major tech companies.
Zane Bond, Head of Product at Keeper Security, notes that the attackers almost certainly must have a been a state-backed crew from a wealthy nation given their approach: “This was an attack that used uncommon tactics with a significant amount of time invested into its success. The breach is catastrophic, without a doubt– highly sensitive government employee emails were compromised- this incident will likely amplify calls from the cybersecurity community for Microsoft to strengthen its cloud security. However, this is a relatively rare incident. The strength of this well-resourced attacker allowed them to capitalize on analysis of a memory dump to obtain these highly sensitive keys. The average hacker would not have been able to accomplish this, and the average organization is not likely to be affected by this type of highly targeted attack.”
But though state-backed hacking teams are generally not a threat to the average organization, Ted Miracco (CEO of Approov) notes that the incident is a prompt to ensure that credentials are regularly being rotated: “The two most disconcerting parts of the report are that: Storm-0558 could forge tokens to access email accounts of high-level officials; and that the breach persisted for years without being discovered. This would lead one to question how many other accounts are being compromised today with forged tokens, and how do you go about identifying additional compromised accounts? The findings reinforce that constant vigilance is required to stay ahead of sophisticated attackers, and keys and tokens need to be rotated frequently to prevent persistent access to compromised accounts.”