Analyst working late using dashboard to look at SIEM rules and threat coverage

SIEM Rules’ Threat Coverage Is Far Less Than What’s Expected; 84% of MITRE ATT&CK Threats Are Not Covered

Organizations often purchase Security Information and Event Management (SIEM) products and expect it to serve as a comprehensive and largely “turnkey” security solution. A new study from AI-based security engineering firm CardinalOps suggests that a combination of broken and poorly configured SIEM rules are providing organizations with a far lower level of threat coverage than they believe they presently have.

Tested with the attacks contained in the MITRE ATT&CK framework, a security industry standard catalog of known threats used to test system readiness, the researchers found that SIEMs were completely unprepared for 84%. A substantial amount of SIEM rules, 25% in total, are broken and will never trigger. However, organizations are generally not aware of this. Additionally, SIEMs tend to eat up staff time with false positives given that only 15% of rules are generating 95% of alerts.

The data was drawn from some of the industry’s biggest SIEM vendors (Splunk, IBM Qradar, SumoLogic) in deployments at 10 large multinational corporations that do billions of dollars in annual sales. These organizations had a median of 95 daily tickets and a mean of 7,278.

SIEM rules in shambles

The report notes that there is an existing perception in the security world that SIEM deployments tend to be problematic and underperform, but this study not only provides more in-depth data on the phenomenon (with one of the largest data sets of this nature to date) but also indicates that more modern deployments are continuing to leave gaping holes in threat coverage.

While there is an ever-increasing need for automation in cybersecurity response, the data indicates that SIEM systems are not keeping pace. The SIEM system is generally the central element of the Security Operations Center (SOC), serving the primary function of flagging potential threats to take substantial manual workload off of staff. While a properly functioning SIEM deployment is ideal, most seem to be far from the mark.

SIEM systems are often sold as a comprehensive, “set and forget” solution to company decision makers (a $3 billion global annual market). That perception is part of the problem. Many organizations assume that the SIEM rules are coming out of the box properly configured and providing wide-ranging threat coverage. They also assume that the system will reduce technician workload, when it may in fact increase it by way of greater volumes of false positives generated by malfunctioning SIEM rules.

MITRE ATT&CK, a security knowledgebase that professionals use to catalog observed threat techniques and tactics, provides numerous options for defeating SIEM systems. The research finds that 84% of the techniques listed by MITRE ATT&CK are not accounted for by SIEM rules in the average setup. CardinalOps believes that this number is misleading, however; because proper defense against many MITRE ATT&CK methods requires the proper configuration of multiple SIEM rules in coordination with each other, the average level of threat coverage is probably lower than reported.

It is true that MITRE ATT&CK is not the only standard by which to judge SIEM readiness, and that the two do not even have any particular design or protocol to facilitate interoperability (MITRE ATT&CK  lists a number of techniques that cannot even be addressed by SIEM rules). And it is true that no SIEM can realistically be expected to have 100% threat coverage even if perfectly managed. The sheer amount of known real world attack types cataloged by it that do not line up with SIEM rules is troubling, however.

There appear to be systemic problems with SIEM rules. The report finds that 25% of the threat detection rules can be expected to be broken and never actually trigger, due largely to fields not being extracted correctly or sources not sending the expected log data. And 15% are “noisy” and are collectively generating 95% of all alerts, indicating that misconfigured SIEM rules are a primary source of time-consuming and stressful false positives.

Keeping up with threat coverage

The “out of the box” misperception of SIEM threat coverage also contributes to a failure to update properly. Even the best SIEM software is meant to be continuously maintained and updated due to ongoing changes in internal IT infrastructure combined with the rapidly-evolving external threat landscape. However, the report finds that on average organizations add only one new SIEM rule per month. 78% of organizations disable the default rules set by the vendor, but most are not able or willing to devote the resources to keep up with unique organizational needs. And, on average, each full-time security engineer at an organization can be expected to be assigned to 63 log sources.

Researchers found that SIEMs were completely unprepared for 84% of #cyberattacks in the MITRE ATT&CK framework and 25% of rules will never trigger. #cybersecurity #respectdata Click to Tweet

The central takeaway from this report is that there are no “perfect security tools” out there for organizations to identify, buy and then forget about after setting up; many SIEMs are quite capable, but are completely dependent on ongoing management. There is no way around devoting sufficient manpower to keeping up with both inter-organizational changes and the current threat coverage landscape.


Senior Correspondent at CPO Magazine