Full Report
[I originally wrote this blog entry on the plane returning from BlackHat, Defcon & Metricon, but forgot to publish it. I think the content is still interesting, so, sorry for the late entry :)] I’ve just returned after a 31hr transit from our annual US trip. Vegas, training, Blackhat & Defcon were great, it was good to see friends we only get to see a few times a year, and make new ones. But on the same trip, the event I most enjoyed was Metricon. It’s a workshop held at the Usenix security conference in San Francisco, run by a group of volunteers from the security metrics mailing list and originally sparked by Andrew Jacquith’s seminal book Security Metrics.
Analysis Summary
# Industry News: Focus on Security Metrics and Risk Quantification at Metricon 2011
## Summary
The article summarizes key takeaways from the Metricon 2011 workshop, an event focused on security metrics held in conjunction with the Usenix Security Conference. Highlights included discussions on practical methodologies for quantifying risk, critiques of existing security performance standards (like PCI), and the emerging role of cyber liability insurance as an alternative or complement to technical controls spending.
## Key Details
- **Date:** Post-BlackHat/Defcon timeframe; article published October 28, 2011.
- **Companies Involved:** Merkel (via Jake Kouns), various security research individuals/groups.
- **Category:** Industry discussion/Conference summary focusing on Security Metrics and Risk Management trends.
## The Story
The author found the Metricon workshop, focused on security metrics, to be a refreshing counterpoint to the offensive security focus of BlackHat and Defcon. Key presentations covered several emerging areas in security management:
1. **Wendy Nather** discussed quantifying the difficult aspects of risk and provided tactical advice on phrasing and presenting metrics, noting that common scaling (like 0-100) can unintentionally lead to suboptimal performance goals among practitioners.
2. **Josh Corman** argued that compliance frameworks like PCI can foster stagnation rather than true security improvement, likening it to an ineffective baseline standard, and highlighted the "half-life" of security advice as controls evolve.
3. **Jake Kouns** explored the growing market for **Cyber Liability Insurance**, suggesting it offers reasonable coverage for SMEs currently relying on minimal security (e.g., AV) and prompting a strategic question: why overspend on the "next big security product" when insurance offers a potentially cheaper risk transfer mechanism? He also noted the risk of moral hazard, where insurance disincentivizes control implementation.
4. **Allison Miller and Itai** detailed a practical, six-step methodology for developing applied risk analytics, moving from defining targets (using yes/no questions where possible) to data preparation, model training, assessment, and eventual automated deployment.
## Business Impact
### For the Companies Involved
* **SensePost (Author's firm):** Gained actionable insights and professional networking opportunities, particularly regarding risk measurement and the burgeoning cyber insurance sector, which can inform consulting strategies.
* **Merkel (via Jake Kouns):** Validated and promoted their involvement in the cyber insurance product management space, signaling that established security consultants are moving into risk transfer products.
### For Competitors
* Firms focused purely on selling technical defensive products or offensive penetration testing might face increased pressure to demonstrate quantifiable business value, as the Metricon discussions emphasized measurable risk reduction over mere control adoption.
### For Customers
* Customers gain validation for demanding more measurable, actionable security metrics rather than vague compliance reports.
* The rising availability and affordability of cyber insurance offer SMEs a viable (though potentially risky) option for managing catastrophic financial loss outside of traditional control spending.
### For the Market
* The emphasis at Metricon signifies a market maturation away from simply buying "security tools" toward integrating security into formal risk management frameworks, utilizing data science (as shown in Allison Miller’s talk).
* The growing consideration of cyber insurance signals a shift in how organizations externalize cybersecurity risk.
## Technical Implications
The technical focus was heavily skewed toward **Applied Risk Analytics**. Allison Miller's methodology details a prescriptive, measurable approach to security measurement, moving beyond simple vulnerability counts:
* **Data Variable Definition:** Establishing clear metrics (e.g., harvested credentials) to feed into predictive models.
* **Model Assessment:** Using concepts like False Positives/Negatives to calibrate the usefulness of metrics for automated decision-making (e.g., detecting account takeovers).
## Strategic Analysis
- **Market Positioning:** The industry discourse highlighted at Metricon suggests a strategic shift toward **Security as an Insurance/Financial Product**. Security vendors and consultants must adapt their positioning to align with quantifiable risk reduction rather than feature parity.
- **Competitive Advantage:** Firms that can successfully implement the repeatable, data-driven risk analytics championed by Miller and Itai will possess a significant advantage over those relying on anecdotal or compliance-based risk assessments.
- **Challenges:** The primary challenge is the industry's inertia regarding metrics, as noted by Corman regarding PCI creating a floor, not a ceiling. Successfully shifting budget holders' focus from compliance checklists to probabilistic risk models remains difficult.
## Industry Reactions
* **Analyst opinions:** The author views the focus on metrics as "refreshing" and vital for escaping the "hamster wheels of pain" associated with compliance-driven security.
* **Expert commentary:** The inclusion of cyber insurance as a serious component of risk strategy suggests acceptance among technical leaders that controls alone are insufficient or too costly to cover all eventualities.
## Future Outlook
* **Predictions and expectations:** Expect increased vendor focus on integrating their data outputs with standardized risk modeling platforms. Furthermore, as cyber insurance matures, underwriters will likely start demanding specific, measurable metrics (like those discussed at Metricon) as prerequisites for favorable policy terms.
* **What to watch for:** The performance and claims data from early cyber liability insurance policies will profoundly influence how both security spend and risk transfer strategies are budgeted in the coming years.
## For Security Professionals
Security practitioners must move beyond technical execution to understand the mathematics of risk assessment. Investing time in learning how to define clear inputs and interpret outputs from risk models will become crucial for influencing budgetary and strategic decisions within their organizations. The shift towards measurable outcomes benefits practitioners who can speak the language of the business (risk and finance).