Full Report
A short while back, a discussion broke out on a mailing list about the nature of being a pen-tester. The discussion quickly gravitated towards the number of “security” companies where numbers of projects far out-weigh the interestingness of projects, leading rapidly to a cookie-cutter mentality to pen-test engagements.. Of course if you have spent any time in the industry, you already know this to be true.. the obvious danger with this is that you have a lot of unhappy pen-testers giving shoddy output to (eventually) very unhappy customers. Sadly this soon follows the well published “market for lemons” problem where eventually due to information asymmetry, bad products will soon push out good ones.. i.e. because its hard for customers to tell the difference between good pen-tests and lame pen-tests, eventually the market price drops towards low grade pen-tests (since the customer is paying for what they expect) and at the low prices, good pen-test teams will close shop and move on to other lines of work..
Analysis Summary
This analysis focuses on extracting actionable security best practices related to the *delivery and quality control* of penetration testing services, derived from the author's philosophy on ensuring consultants provide valuable, high-quality work rather than "cookie-cutter" assessments.
# Best Practices: Ensuring Quality and Value in Security Assessments
## Overview
These practices address the need for security assessment providers (and the customers procuring them) to move beyond standardized, low-effort, "cookie-cutter" penetration testing. The focus is on maintaining high professional standards, continuously improving techniques, and ensuring the output delivers measurable, actionable value to the client, thereby counteracting the "market for lemons" problem in the security consulting industry.
## Key Recommendations
### Immediate Actions
1. **Prioritize Value Over Volume:** Immediately refuse or critically re-scope any engagement where the projected effort contradicts the value delivered (i.e., turning down business where the client's money could be better spent elsewhere).
2. **Enforce Internal Quality Checks:** Mandate that analysts "double and triple check to make sure we are adding value" on all active engagements, even if it requires immediate rework or scope adjustment.
3. **Promote Intellectual Drive:** Ensure analysts are actively seeking technical challenges; foster an internal culture where providing "half-assed" work feels professionally unacceptable.
### Short-term Improvements (1-3 months)
1. **Cultivate Envelope-Pushing:** Establish formal processes or dedicated time slots to encourage and reward the research and application of novel attack techniques that go beyond basic scanning or documented vulnerabilities.
2. **Align Stakeholder Expectations:** For recurring assessments, proactively communicate how evolving attacker sophistication requires a shift in testing focus (e.g., moving from network vulnerability scans to complex application logic testing).
3. **Implement Peer Review of Techniques:** Institute a mandatory peer review system for complex attack methodologies discovered during testing to ensure technical soundness and document the emerging techniques internally.
### Long-term Strategy (3+ months)
1. **Invest in Culture Over Compliance:** Build a long-term organizational culture that reinforces continuous learning and technical excellence, recognizing this as a competitive advantage against firms relying solely on checklist compliance.
2. **Track Impact and Actionability:** Develop metrics to track whether client remediation efforts successfully address the specific, advanced findings delivered in the report. Favor clients who actively utilize high-level findings over those who only prioritize low-risk scanner outputs.
3. **Strategic Adaptation to Threat Evolution:** Continuously monitor the gap between "arcane" advanced attacks and "weaponized" industry attacks, ensuring testing methodologies proactively cover threats that will become common in 1-3 years.
## Implementation Guidance
### For Small Organizations
- **Focus on Niche Excellence:** Instead of attempting to cover every security domain superficially, focus resources on becoming the absolute best in one specialized area (e.g., complex Web Application Logic or Cloud Configuration).
- **Direct Client Communication:** Use preliminary scoping calls to genuinely assess the client's current security maturity. If automated tools cover 90% of their likely threats, recommend remediation first, then return for advanced testing later.
### For Medium Organizations
- **Formalized "Push the Envelope" Time:** Allocate a mandatory percentage (e.g., 10-20%) of analyst time toward self-directed research and developing new methodologies, separate from billable client work.
- **Team Health Checks:** Regularly assess analyst engagement. If technical staff appears bored or is resorting purely to automated scanning, initiate internal discussions about scope or role rotation immediately.
### For Large Enterprises
- **Establish an Internal Red Team Standards Board:** Create a body responsible for formalizing "best-in-class" methodology that intentionally exceeds current industry norms for assessment delivery.
- **Long-Term Relationship Auditing:** Ensure assessment contracts incorporate iterative testing cycles. This allows the security team to see the tangible benefit of their previous high-effort findings materialize into stronger defenses over time, justifying the cost of in-depth work.
## Configuration Examples
*(The provided text does not contain specific technical configuration examples related to security technologies, but rather operational and cultural configurations for a consulting business.)*
## Compliance Alignment
While the text focuses on exceeding standards, adherence to robust industry practices helps validate the thoroughness required:
- **NIST SP 800-115 (Technical Testing and Assessment Guidelines):** High-quality, in-depth testing directly supports the principles of thorough technical validation.
- **ISO/IEC 17025 (Testing and Calibration Laboratories):** While focused on labs, the underlying principles of competence, consistent methodology, and rigorous validation support the necessary quality framework.
## Common Pitfalls to Avoid
1. **Accepting Low-Value Engagements:** Do not accept projects purely for revenue if you know the required output will be mediocre ("cookie-cutter"). This erodes tester morale and damages client trust long-term.
2. **Ignoring Analyst Burnout/Boredom:** Failing to challenge technically skilled analysts leads to high attrition as they seek environments where they can "push the envelope."
3. **Short-Sighted Financial Focus:** Prioritizing immediate revenue over building long-term goodwill (by turning down unnecessary work) ultimately damages the company's reputation and market standing ("market for lemons").
4. **Believing Luck is Sustainable:** Relying solely on previous successes or inherent talent without consistent hard work and technique refinement is unsustainable in cybersecurity.
## Resources
The concepts discussed align with principles found in literature emphasizing technical rigor and personal dedication:
- **Richard Hamming's "You and Your Research":** A foundational text referenced for understanding the relationship between hard work, focus, and achieving greatness consistently.
- **Internal Knowledge Sharing Platforms:** Essential for documenting and sharing novel techniques discovered during "envelope-pushing," ensuring institutional knowledge retention.