Full Report
A longish post, but this wasn’t going to fit into 140 characters. This is an argument pertaining to security metrics, with a statement that using pure vulnerability count-based metrics to talk about an organisation’s application (in)security is insufficient, and suggests an alternative approach. Comments welcome. Current metrics Metrics and statistics are certainly interesting (none of those are infosec links). Within our industry, Verizon’s Data Breach Investigations Report (DBIR) makes a splash each year, and Veracode are also receiving growing recognition for their State of Software Security (SOSS). Both are interesting to read and contain much insight. The DBIR specifically examines and records metrics for breaches, a post-hoc activity that only occurs once a series of vulnerabilities have been found and exploited by ruffians, while the SOSS provides insight into the opposing end of a system’s life-cycle by automatically analysing applications before they are put into production (in a perfect world… no doubt they also examine apps that are already in production). Somewhat tangentially, Dr Geer wrote recently about a different metric for measuring the overall state of Cyber Security, we’re currently at a 1021.6. Oh noes!
Analysis Summary
# Best Practices: Evolving Application Security Metrics Beyond Simple Vulnerability Counts
## Overview
These practices address the insufficiency of using pure vulnerability count-based metrics (total, critical count, etc.) to accurately measure an organization's application security posture. The focus is on adopting metrics that incorporate context, risk, attacker effort, and business impact to enable better decision-making.
## Key Recommendations
### Immediate Actions
1. **Stop Relying Solely on Raw Vulnerability Counts:** Immediately cease using the total number of vulnerabilities (High, Critical, Urgent) as the primary measure of application security risk for executive reporting.
2. **Contextualize Vulnerability Data:** Begin slicing vulnerability data by the *number of applications affected* rather than just the raw count. Calculate the percentage of applications exhibiting a specific vulnerability class (e.g., % of apps vulnerable to XSS).
3. **Document Report Sources:** Ensure all application security reports (internal scans, assessments, external scores) are meticulously logged and aggregated to allow for future trend analysis.
### Short-term Improvements (1-3 months)
1. **Incorporate Asset Value/Risk:** Start categorizing applications based on their business function (e.g., brochure-ware, customer-facing, transactional/financial systems). Tally vulnerabilities against these risk categories to understand where the most valuable assets are exposed.
2. **Track Mean Remediation Time by Hunter/Source:** Begin tracking how long it takes to remediate reported issues, potentially segmenting this data by the source (internal team finding vs. vendor assessment vs. penetration test) to evaluate team efficiency.
3. **Digitize Reporting:** Eliminate paper-based reporting and implement a centralized, queryable system (database or dedicated AppSec platform) to capture detailed security findings, metadata, and remediation status.
### Long-term Strategy (3+ months)
1. **Develop Attack Effort Modeling (Conceptual):** Investigate and begin prototyping metrics that approximate **attacker effort**. This involves pairing vulnerability severity with the technical complexity required for an external party to successfully exploit it.
2. **Incorporate Development Metadata:** Systematically capture and correlate security findings with relevant development attributes such as:
* Outsourcing status (Internal vs. Third-Party code).
* Development Lifecycle Model used (Agile, Waterfall).
* Team size, platform, core programming language, and frameworks used.
3. **Establish Benchmarking Against Peers:** Use external industry reports (like SOSS or DBIR) to benchmark the organization’s security metrics (e.g., % of apps with critical issues) for contextual comparison against anonymized competitor/industry data.
## Implementation Guidance
### For Small Organizations
- **Focus on Ratios:** Prioritize calculating the ratio of High/Critical vulnerabilities per application to get a baseline measure of quality, rather than focusing on total volume.
- **Manual Metadata Tagging:** Since formal tooling might be limited, mandate a simple spreadsheet or ticketing system where every finding must be tagged with the application’s primary business function (Low, Medium, High transactional risk).
### For Medium Organizations
- **Implement a Centralized Data Store:** Mandate the consolidation of findings from all scanners and testers into one repository to facilitate cross-application metric generation.
- **Start Trend Analysis:** Utilize data collected over the last year to identify which vulnerability classes (e.g., Injection, Auth Bypass) are most frequently appearing across the portfolio.
### For Large Enterprises
- **Automate Metadata Ingestion:** Integrate security testing tools directly with Configuration Management Databases (CMDB) or DevOps pipelines to automatically attach application lifecycle, outsourcing status, and system criticality to every finding.
- **Workshop-Driven Metric Validation:** Present derived, contextualized metrics (e.g., risk exposure per development team) to cross-functional stakeholders (Development, Risk, Business Units) and solicit qualitative feedback to validate the usefulness and accuracy of the metrics in driving remediation decisions.
## Configuration Examples
*No specific technical configuration examples were provided in the source text; the recommendations focus on data structuring and metric definition.*
## Compliance Alignment
The adoption of robust, risk-based metrics aligns indirectly with the goals of major standards by requiring systematic data management and risk assessment:
- **NIST CSF:** Supports the **Identify** function by requiring asset management and risk assessment.
- **ISO 27001/27002:** Supports Annex A.18 (Compliance) by mandating measurement and review of information security controls.
## Common Pitfalls to Avoid
- **Ignoring Context:** Measuring the security of 100 IoT devices the same way as 2 core banking platforms.
- **Data Silos:** Leaving vulnerability data trapped in disparate PDF reports or paper trails, making any trend analysis or aggregation impossible or excessively time-consuming.
- **Focusing on Input vs. Outcome:** Reporting on the *number of scans run* (input) instead of the *resulting risk reduction* (outcome).
- **Ignoring Attacker Perspective:** Measuring issues solely by technical severity without considering the practical effort an adversary would need to expend to exploit them.
## Resources
- **Verizon Data Breach Investigations Report (DBIR):** Useful for understanding post-hoc breach metrics and trends.
- **Veracode State of Software Security (SOSS):** Useful for understanding pre-production analysis and vulnerability trends across an application portfolio.
- **WASC Security Statistics Projects (Historical Context):** Provides examples of historical approaches to vulnerability classification reporting.