Full Report
As 44Con 2012 starts to gain momentum (we’ll be there again this time around) I was perusing some of the talks from last year’s event… It was a great event with some great presentations, including (if I may say) our own Ian deVilliers’ *Security Application Proxy Pwnage*. Another presentation that caught my attention was Haroon Meer’s *Penetration Testing considered harmful today*. In this presentation Haroon outlines concerns he has with Penetration Testing and suggests some changes that could be made to the way we test in order to improve the results we get. As you may know a core part of SensePost’s business, and my career for almost 13 years, has been security testing, and so I followed this talk quite closely. The raises some interesting ideas and I felt I’d like to comment on some of the points he was making.
Analysis Summary
# Best Practices: Evolving Security Testing Beyond Traditional Penetration Testing
## Overview
These practices address the suggested shift in security validation methodologies, moving from a narrow focus on traditional Penetration Testing (which often fails to account for zero-day threats and broader risk contexts) towards a more comprehensive approach anchored in Threat Modeling integrated with technical validation. The goal is to make security assessments smarter, broader, more efficient, and more relevant to overall risk management.
## Key Recommendations
### Immediate Actions
1. **Clarify Testing Definitions:** Immediately cease using "Penetration Testing" as a blanket term for all security activities. Differentiate between **Vulnerability Assessment** (discovery) and **Penetration Testing** (exploitation/impact demonstration).
2. **Adopt a Phased Approach:** Ensure that "penetration testing" or "attack and penetration" is explicitly defined and used only as *one specific phase* within a broader security assessment process, not the entire process itself.
3. **Introduce "0-Day Emulation" Protocols:** Integrate a mechanism (referred to conceptually as '0-day cards') where testers can simulate the impact of unknown, high-impact vulnerabilities on specific systems without needing to possess or deploy an actual zero-day exploit.
### Short-term Improvements (1-3 months)
1. **Integrate Threat Modeling Workflow:** Begin the phased integration of systematic Threat Modeling before any technical testing commences. Use this as the primary driver for scoping and prioritizing validation efforts.
2. **Collaborative Model Development:** For all future or ongoing assessments, mandate joint development of the target environment model. This phase must involve both the testing team and client subject matter experts to capture both insider and outsider perspectives.
3. **Comprehensive Risk Enumeration:** Fully enumerate potential risks based on the developed environment model, creating a long, comprehensive list of hypothetical risks that explicitly includes non-public or novel threats (like zero-days).
### Long-term Strategy (3+ months)
1. **Establish Risk Prioritization Framework:** Develop and enforce a standardized methodology for sorting and grouping the enumerated hypothetical risks into actionable priority levels.
2. **Data-Driven Testing Prioritization:** Ensure that technical testing activities are strictly ordered based on the priority established during the Threat Modeling phase (proving or disproving the highest-priority hypothetical risks first).
3. **Continuous Feedback Loop:** Implement a cyclic process where remediation efforts are followed by targeted testing to gauge the effectiveness of the changes, thus incrementally improving organizational security posture over time.
## Implementation Guidance
### For Small Organizations
- **Focus on Core Threat Modeling:** Start by modeling the top 3-5 critical business processes or assets. Use a lightweight adaptation of the proposed methodology.
- **Prioritize Documentation:** Ensure the derived risk register (from threat modeling) serves as the central document for management review, making the testing results quantifiable and understandable to non-technical leaders.
- **Leverage Vendor Data:** When procuring external testing, explicitly require that the scope be driven by a preliminary threat model discussion rather than just a list of assets.
### For Medium Organizations
- **Formalize Methodology Adoption:** Adopt the core 6-step Threat Modeling approach (Model $\rightarrow$ Enumerate $\rightarrow$ Sort $\rightarrow$ Test $\rightarrow$ Remediate $\rightarrow$ Repeat) for all major system updates or new deployments.
- **Cross-Functional Team Training:** Train security staff and relevant development/operations leads in the threat modeling process to facilitate the development of the target environment model.
- **Track Risk Register Maturity:** Implement a system to track the lifecycle of identified risks, moving them from "Hypothetical" to "Validated" (or mitigated/accepted).
### For Large Enterprises
- **Enterprise-Wide Modeling Standard:** Formalize and standardize the Threat Modeling framework across the entire enterprise, potentially adjusting the Microsoft methodology template for internal use.
- **Integrate Testing Registry:** Ensure the tested risk register is directly integrated into the organization’s overall Enterprise Risk Management (ERM) system, linking technical validation outcomes to business impact metrics.
- **Mandatory Pre-Test Modeling Phase:** Make a successful, client-approved Threat Model review a mandatory prerequisite sign-off before any penetration or deep vulnerability test engagement can officially begin.
## Configuration Examples
*The provided text discusses methodologies and philosophical shifts, not specific technical configurations (like firewall rules or software settings). The closest guidance relates to process configuration:*
**Process Configuration: Threat Modeling Integration Checklist**
| Step | Actionable Item | Required Output |
| :--- | :--- | :--- |
| 1. Modeling | Define system boundaries, data flows, and trust levels. | System Model Diagram / Data Flow Diagram (DFD) |
| 2. Enumeration | Brainstorm all potential threats against the components identified in Step 1. | Comprehensive Hypothetical Risk List |
| 3. Prioritization | Rank risks based on potential business impact and exploitability likelihood. | Prioritized Risk Register |
| 4. Validation | Plan technical tests specifically designed to confirm or deny the highest-ranked hypothetical risks. | Test Plan Mapped to Risks (including 0-day emulation scenarios) |
| 5. Action | Define disposition (Remediate, Mitigate, Insure, or Inform Business). | Documented Risk Disposition Status |
## Compliance Alignment
- **NIST SP 800-160 (Systems Security Engineering):** The emphasis on initial modeling and systematic risk identification aligns closely with requirements for engineering security throughout the development lifecycle.
- **ISO/IEC 27001 (Information Security Management):** The iterative process of testing, understanding threats, and managing identified risks supports the continuous improvement cycle (Plan-Do-Check-Act) inherent in ISO 27001.
- **Microsoft SDL (Security Development Lifecycle):** The proposed methodology is explicitly mentioned as being roughly based on the Microsoft Threat Modeling approach, making this a direct alignment for organizations using that framework.
## Common Pitfalls to Avoid
1. **Treating Testing as Risk Management:** Do not conclude assessment reports by stating remediation of vulnerabilities *is* risk management. Testing only provides a *data point* for risk management; risk management requires factors beyond the client's direct control (e.g., operational context, external threats).
2. **Narrow Scope Tunnel Vision:** Avoid conducting penetration tests in a vacuum without the context provided by robust threat modeling, as this leads to testing only known weaknesses while ignoring potentially catastrophic unknown ones (the 0-day factor).
3. **Over-reliance on Technical Exploitation:** Do not allow the narrow definition of "penetration testing" (pure exploitation) to overshadow the broader goal of "security assessment" (learning about threats and defenses).
## Resources
- **Microsoft Threat Modeling Documentation:** Referenced as the foundational methodology for environmental modeling and risk enumeration. (Search term suggestion: "Microsoft Threat Modeling Methodology").
- **SensePost Corporate Threat Modelling Slides:** Referenced as the core approach documentation for enterprise application. (Search term suggestion: "SensePost Corporate Threat Modelling slides").