CircleCI is one of the most widely used CI/CD platforms in the world, to the point that a serious enough security breach could create industry-wide catastrophes. That nightmare scenario appears to be unfolding as the company is advising clients to rotate secrets in the wake of an apparent intrusion.
The company has rotated all GitHub and Bitbucket OAuth tokens, but the security breach leaves most of its clients with a pile of time-consuming cleanup of assorted keys, tokens and variables. CircleCI says that it remains safe for users to build, but advises that additional layers of security be added to CI/CD pipeline configurations.
CircleCI security breach creates major headache for clients
The incident opened with an alert published on January 4, which provided very minimal details about the security breach but urged CircleCI clients to rotate secrets stored in the system. The breach may have taken place as early as December 21 as CircleCI urged customers to review their logs for unauthorized access going back to that date. The company also advised that Project API tokens had been invalidated and would need to be replaced.
An update the following day assured customers that it was safe to continue to build, and that CircleCI had taken a number of steps in response to the security breach: a full audit of system access, rotation of all production machines and cycling of all access keys, and an investigation involving the company’s third party partners. The company also reiterated that clients should rotate secrets and provided instructions for changing project variables, user API tokens, project SSH keys and other elements (an “Inspector Tool” was also uploaded to GitHub to assist in this process).
A follow-up post on January 7 indicated that a forensic investigation remained ongoing. In the meantime, some CircleCI clients are emphasizing that it is vital to rotate secrets ASAP as they are already reporting the abuse of stolen credentials in the wild (such as unauthorized access to Amazon AWS accounts).
Rush to rotate secrets as attackers pounce on exposed credentials
There is not yet any demonstrated connection between the two security breaches, but in September GitHub reported that attackers were attempting to phish users by pretending to be CircleCI and asking users to sign in to agree to an updated terms of service covering the authentication partnership between the two companies. CircleCI also published a “reliability update” on December 21 documenting its planned improvements, but there is not yet any connection between that and the recent security breach.
This is not the first time in recent history that CircleCI users may have been prompted to rotate secrets as a result of a security breach, though this recent attack appears to have had a much bigger impact. An August 2019 security breach resulted in the compromise of some usernames, email addresses and IP addresses associated with BitBucket and GitHub accounts. That incident was traced to a compromised third party vendor and did not appear to leak any customer authentication tokens, but some customers did experience the leak of repo and branch names that may have contained sensitive business information.
There are thus good reasons to not be fully confident in CircleCI security at this point, but jumping ship to competitors may also not be particularly appealing to customers. One of the biggest alternatives, Travis CI, suffered a security breach last year that was arguably much worse and also required clients to rotate secrets on short notice to avoid a disaster. Travis CI also failed to inspire confidence in that incident by being tight-lipped and evasive about exactly what happened in the immediate wake of the breach.
Fast, thorough and accurate breach notification has unfortunately become the rare exception rather than the norm, not just for CI/CD services but all manner of online platforms. As demonstrated by a number of major breaches that closed out 2022, ranging from LastPass to Twitter, a carefully worded initial press release that makes the security breach seem relatively minor often turns into a revised account weeks or months later that reports much more extensive damage.
That same careful wording (and lack of specific detail) is present in the CircleCi notification. While that doesn’t automatically imply that greater damage is being concealed or going undiscovered, it is concerning given a string of recent cases that have unfolded in this way. That and an advisory to “rotate secrets immediately” while being cagey about exactly what happened no doubt throws up a huge red flag for some security professionals.
Greg Notch, Chief Information Security Officer at Expel, is one of those that believes this incident should be treated as if more bad news is forthcoming, and perhaps put a system in place to rotate secrets more regularly in anticipation of future security breaches: “Key takeaway from this incident? Anticipate risk in your security planning. Security teams can prepare for situations like this with periodic tabletop exercises. These simulations reveal the potential impact of an attack like this-helping your team flex their incident response (IR) muscles and prepare an action plan. At Expel, because of exercises like these we’ve done in the past, our team has daily automated rotations in place on our highest-risk credentials in CircleCI. This means that if keys were exfiltrated, attackers would only have 24 hours to use those privileged credentials before they became useless. We also have a “canary” system in place to tell us if a credential gets used unexpectedly. Finally, if you have questions or concerns about potential exposure for your organization, contact your managed detection and response (MDR) partner for the latest updates.”
