IT Controls Benchmark
How can you tell if your security and IT controls are really effective? How do you measure security effectiveness? How do you prove that IT security controls help increase IT operating effectiveness and efficiency and that they don’t just slow the business down? How can you differentiate high performing security and IT operations organizations from those that are low performing?
These are all questions that historically did not have great answers. Most of the better answers resembled platitudes instead of actionable advice (for example, “Make sure the business cares about security”).
Coming up with better answers is what Kevin Behr and I set out to do in 2000 when we co-founded the IT Process Institute. They wanted to advance the quantitative science in IT operations and security to help organizations answer these very questions and to create meaningful guidance that was tested with the same empirical rigor that the Lean Manufacturing researchers conducted at MIT in the 1980s.
Our hypothesis was that if we could analyze high performing IT and security organizations and what they did, they could discover and recommend specific actions, with fair confidence that these actions would produce measurable results.
After benchmarking over 1500 IT organizations from 2004 until 2009, we concluded two surprising things about high performing IT organizations:
High Performers Are More Than 4x More Productive
- High performers maintain a posture of compliance
- They have 4x fewer repeat audit findings:
By the time an auditor is issuing a finding for the third year in a row, they are saying, "We have a problem... and the problem might be you..." - They spend less than 3x less time on audit preparation effort:
They don't have the wave of work that precedes the audit, where management is scurrying to either prove that nothing bad happened, or recreate the illusion of documented controls.
- They have 4x fewer repeat audit findings:
- High performers find and fix security breaches faster
- They are 5x more likely to detect security breaches by automated control:
They have a system of internal controls that detects bad things happening, as opposed to relying on an external auditor, a customer or a newspaper headline notifying management that there is a problem. - They are 5x times less likely to have breaches result in a loss event:
When security breaches happen, which happen to everyone as Murphy's Law doesn't discriminate , high performers are 5x more likely detect and correct the problem before it turns into a loss event (e.g., reputational, financial, contractual, etc.)
- They are 5x more likely to detect security breaches by automated control:
- When high performers implement changes…
- They authorize and implement 14x times more changes:
Virtually every business decision results in at least one IT change, so the number of changes that an IT organization can make has a high correlation with its productivity... - They have one-half the change failure rate:
They are much better at having the IT change actually achieve its desired outcome, without causing an episode of unplanned work - They have one-quarter the change failure rate:
And when their changes fail, as Murphy's Law dictates will happen inevitably, they are much better at fixing the problems the first time, let alone not making it worse... - 10x faster MTTR for Sev 1 outages:
Which results in much shorter outage times, as observed by the customer!
- They authorize and implement 14x times more changes:
- When high performers manage IT resources…
- They have one-third the amount of unplanned work
This is important, as unplanned work isn't free... It comes at the expense of planned work! - They complete 8x more projects
So they complete more projects, with far higher project due date performance... - They manage 6x more applications
And at the conclusion of most IT projects, there is a new application to manage, which leads to an ever increasing number of IT services that must be supported
- They have one-third the amount of unplanned work
Three Controls Predict 60% Of Performance
This is the astonishing finding: IT controls really do predict IT performance. And just by asking three questions, you can predict all of the metrics that we discussed above. Those three questions are:
- To what extent does the organization define, monitor and enforce some standardized configuration strategy?
I.e., do they define a known, good build that is in a risk-reduced state, and hold management accountable for truing up production variances from that known, good state? - To what extent does the organization define, monitor and enforce a process culture?
I.e., is there tone at the top that states that the only acceptable number of unauthorized configurations and changes is zero? - To what extent is restricted access to production?
I.e., especially for DBAs and developers, because they're often the most likely to be making unauthorized production changes by backdoors that circumvent controls
On the one hand, it's astonishing that if you ask those three questions and put them into a spreadsheet, you can predict compliance performance, security performance, IT operational performance and project due date performance.
On the other hand, most people would agree that it is common sense that these three controls are prerequisites to good performance. Unfortunately, common sense is not common practice.
My 2009 Metricon Presentation
Shown below is my full 2009 Metricon presentation, where I describe how we benchmarked over 1500 IT organizations in over six waves of studies.