Metrics matter. Metrics are important because they tell you, stakeholders and budget planners how well you’re meeting your set goals. Metrics ensure that your program has visibility and is the only way to effectively communicate the value of your application security program. If you simply go through the AppSec motions of scanning and fixing, you have no insight into how effective your application security program is or if you’re hitting either your security goals or business goals.
Nobody said measuring security was going to be easy; Security and risk are simply hard to quantify. Is your risk level at 0% simply because you’ve never been hacked? Of course not. But security often only comes into the limelight when it fails, meaning that nobody pays attention until things go wrong. If this is how your board sees security, deciding on AppSec metrics that speak to those stakeholders is a key move towards higher transparency, a bigger budget, and a stronger application security program.
Why You Need Effective Application Security Metrics
Metrics show that your program is working. In order for your board and security stakeholders to get behind your program, you need concrete data that proves your organizational risk is being reduced and that security activities are done in parallel to overall business goals. Metrics, and their close cousin Key Performance Indicators, or KPIs, show the value of security and enable you to map security activities to business goals over time.
You also need metrics to provide accurate and consistent status updates on your program to your superiors. In many organizations, there is a gap in perception between how secure the CEO and board believe the organization is versus the CISO and security team. A full 63% of respondents to a Ponemon Security Metrics survey agreed that they only communicate when an incident occurs, which can devastate your AppSec program with a lack of budget and low confidence.
When CISOs only hold discussions with the board when high risk vulnerabilities or other issues are discovered, it completely erases any leverage they have in discussions surrounding your budget, personnel, and other resources. With the three biggest barriers to managing security effectively, according to the same Ponemon survey, being ‘insufficient resources or budget’, ‘lack of effective technology solutions’, and ‘lack of experts’ there seems to be a vicious cycle of only communicating failures, which leads to lack of budget, which leads to a lack of experts and effective solutions. The right metrics can break the cycle.
Lastly, you need metrics to ensure you’re on the right path now and in the future. You may not know what the future holds for your organization, but it’s vital your program is sustainable enough to deal with changes and new risks such as cloud and IoT. Use metrics to find lagging areas that need to be changed before these risks come into the picture, because a security team that works slower than the rest of the organization is not an effective security team at all.
Read more about why AppSec Metrics are essential here.
6 Essential AppSec Metrics:
So, now that you know you need metrics, where do you start? Here are six essential Application Security metrics that organizations with an AppSec program should be measuring.
- Number of initial vulnerabilities
Before making any changes to your AppSec program, record the number of vulnerabilities currently in your applications using SAST and manual testing. This number will be your baseline to help you relay the progress of your program to all your stakeholders.
- Time to detect vulnerabilities
This metric measures how effectively your team is discovering vulnerabilities, measuring the time from when a vulnerability is created until it is detected. This metric can be used in all progress reports to show how your program has improved this number over time.
- Effective ratio
The effective ratio, in short, measures how effective your resolution of vulnerabilities is by measuring the rate of vulnerabilities resolved to vulnerabilities reported. This metric is meant first and foremost to the CISO and the security team, but can also be reported to the board to show progress.
- Rate of vulnerability creation
Measuring the rate of vulnerability creation can help detect issues with developer’s security knowledge, and you can take it to the next level by pinpointing specific vulnerabilities that come up over and over. This metric can be used both to show the security team and developers the most common issues, and can be used to influence material being taught to developers. If, for example, SQL injections or XSS appear time and time again, it may be useful to offer a (mandatory) workshop to teach developers how to better avoid and spot these vulnerabilities.
- Number of vulnerabilities prevented from proactive activities
Use this metric to relate how many vulnerabilities the security team has detected through continuous security practices to prove your value to the C-suite and board.
- Compliance with industry regulations
This may be less of a metric and more of a checklist in many organizations, but it’s not any less important to report to your stakeholders. Compliance with industry regulations is a major business enabler, so make sure your board knows that your activities help keep the whole organization in compliance.
Tips on Crafting AppSec Metrics that Matter:
- Meet with security stakeholders and gain a better understanding of what metrics they’re looking for – and use language that speaks to them
Before you make any changes to your current metrics, your first step should be to meet with stakeholders that you’ve pinpointed as being able to help you succeed in your goals and create metric personas for each of them to help you should understand your ‘market’. Your CEO and other executives, for example, will most likely want to hear about how risk is reduced, and how compliance regulations are being met, while your CIO wants to hear more about how much downtime security issues cause and what your team is doing to prevent them. This way, as you recalibrate your current metrics and add new or modified measurements, you’ll know exactly what data your stakeholders want to see – and be able to give them the exact data what they want.
Beyond knowing what your stakeholders want to hear about, you’ll also want to explain the metrics in their ‘language.’ The same Ponemon survey quoted above found that 58% of respondents believe that the information they provide is too technical to be understood by those they present it to. Information that’s too technical, or on the other hand, too high-level, won’t be absorbed or acted on and the whole process becomes a waste of time and resources. Explain your results in a meaningful way to those your present them to.
- Track metrics that align to business goals
During the initial step of meeting with stakeholders, you’ll get a deeper understanding of their interests when it comes to security metrics and be able to better serve them with the exact measurements they want to know. Going a step further, mapping your metrics back to the goals of the business will help ensure your goals match the organization’s. Are you succeeding in hitting the targets most important to your board and the business goals they have? Making sure you give context to your metrics is a huge win in getting heard more. There are different tools you can adopt to more easily align security activities and initiatives to business goals, including the Goal-Question-Metric and the Business Model Canvas, so find one that works for you and your stakeholders.
- Dont measure just to measure
Lastly, make your metrics count. There’s a quote that says that doing nothing is better than being busy doing nothing, which can be perfectly applied to metrics. Measure only what matters to you and your board and then use those measurements to measure, manage, and adapt your program, as well as to gain visibility and budget. Measuring beyond that is just measuring to measure.
What metrics are you using and reporting with success? Share your thoughts below!