Full Report
Cybersecurity is an incredibly broad topic. Even the category of offensive cybersecurity is quite broad. In this article, they do a comparison between code auditing and vulnerability research. Vulnerability research is all about understanding the practical threat landscape of a system or area of code. In this work, vulnerabilities are not enough. Instead, we care about how exploitable these bugs are and the real impact they can have given the constraints of real attackers. The output of a real proof of concept can even be helpful. Code auditing has the goal of improving security within an area of code over a given time frame. This is usually about finding the greatest number of bugs without an emphasis on real exploitability. Code quality or configuration improvements, like missing binary protections, can be found here as well as actual bugs. Both of these are valuable but serve different purposes. If it's a new codebase that's about to ship then a code audit to find many issues is a good idea. If vulnerability research was done on the codebase then they would likely find only a few horrible things but leave many risks and bugs still in there that weren't worth tracking down. Sometimes, it's the opposite though - vulnerability research is needed instead of a code audit. On a large codebase with lots of risk in a merger and acquisition or bug bounty are good examples of when this is necessary. If fuzzing is done on a library with little to no exposure to the outside world and lots of shadow bugs are found, it's not a realistic view of the security of the application. Instead, decisions should be made on the most impactful locations and bugs should attempted to be found in this. According to the author, the latter case is more likely to happen. A common issue is when the client signals that higher quantity of bugs is better than a few high impact ones, which leads to a code audit instead of vulnerability research. A good way to assess this (to me) is the likelihood and impact metrics. Overall, a good article on the differences between a code audit and vulnerability research! They are different things that are similar, leading to issues within various organizations.
Analysis Summary
# Research: Code auditing is not the same as vulnerability research
## Metadata
- Authors: [Author determined as individual researcher/opinion piece]
- Institution: [Self-published/Blog Post]
- Publication: PACIBSP | security
- Date: Nov 2, 2024
## Abstract
This analysis contrasts the goals, methodologies, and appropriate use cases for two distinct security activities: **Code Auditing** and **Vulnerability Research** (VR). While both involve analyzing code for weaknesses, Code Auditing prioritizes maximizing bug discovery within a fixed timeframe, often overlooking exploitability constraints. Conversely, Vulnerability Research focuses on understanding the practical threat landscape by deeply investigating exploitability, reachability, and the constraints faced by real-world attackers, producing knowledge-based outputs rather than high bug counts. Misalignment between organizational security goals and the chosen activity leads to wasted resources and flawed risk assessments.
## Research Objective
The primary objective is to clearly delineate the fundamental differences between Code Auditing and Vulnerability Research to help organizations correctly specify their security requirements and avoid inefficient resource allocation by requesting the wrong activity for a given security context.
## Methodology
### Approach
This work employs a comparative analytical approach, contrasting the inherent goals, typical outputs, and practical application contexts of Code Auditing versus Vulnerability Research based on the author's domain experience.
### Dataset/Environment
The analysis is conceptual, drawn from observations across various technical environments, including new codebases preparing for shipment, large codebases undergoing M&A, and library security assessments (e.g., via fuzzing).
### Tools & Technologies
The concepts discussed relate to the methodologies themselves, such as the use of fuzzing in discovery, but the paper primarily relies on conceptual frameworks rather than specific tooling outputs.
## Key Findings
### Primary Results
1. **Vulnerability Research (VR) Focus:** VR aims to understand the **practical threat landscape** of code. Its value lies in knowledge-based outputs regarding reachable attack surfaces, attacker constraints, and complex exploit flows, sometimes resulting in few but highly impactful findings (including PoCs).
2. **Code Auditing Focus:** Code Auditing prioritizes **maximizing the quantity of bugs** and security improvements (including configuration issues or missing binary protections) within a fixed time constraint, often downplaying deep exploitability investigation.
3. **Appropriate Scenarios:**
* **Code Audit is preferred** for new, pre-shipment codebases where the goal is general risk reduction before deployment (finding the highest volume of issues quickly).
* **Vulnerability Research is necessary** for high-risk scenarios like M&A integration or bug bounty programs where understanding the *most severe* exploitable risks is paramount.
4. **Misalignment Risk:** Organizations often request Code Audits (due to easy measurability via bug counts) when they actually require VR (for strategic risk assessment), leading to misleading security postures where easily found, low-impact bugs distract from critical, hard-to-find vulnerabilities.
### Supporting Evidence
The argument relies on illustrative examples: auditing a new codebase versus deep research on an isolated, unexposed library that yields many "shadow bugs" irrelevant to the overall application risk.
### Novel Contributions
The contribution is not a novel technical finding but a crucial **conceptual clarification** of security practices, emphasizing that the "quantity over quality" metric common in auditing is detrimental when strategic risk knowledge (VR) is required. Introducing **likelihood and impact metrics** as a way to assess the necessity of VR over auditing (for strategic work) is highlighted.
## Technical Details
Vulnerability research may involve significant technical effort to produce a **Proof of Concept (PoC)**, demonstrating real-world exploitability. In contrast, code auditing often settles for immediate, high-volume findings, such as identifying common bugs or configuration weaknesses (e.g., disabled security mitigations), without requiring complex exploit chain development.
## Practical Implications
### For Security Practitioners
Security teams must clearly articulate whether the goal is broad immediate risk reduction (Audit) or deep, strategic threat modeling (VR). Relying on bug volume as the primary success metric for strategic research efforts is counterproductive.
### For Defenders
If a security strategy relies on vulnerability data, ensure that data comes from VR efforts that model real attacker constraints, not just high-volume audits that might flag non-reachable defects.
### For Researchers
Researchers must be prepared to justify their output based on the client's stated need. If strategic knowledge is required, researchers must resist pressure to report high bug counts if deep exploration reveals few practical risks.
## Limitations
The paper notes the difficulty in objectively measuring the "goodness" of knowledge-based outputs from VR compared to quantifiable bug counts, which incentivizes organizations to lean toward audits even when inappropriate. The dependency on the researcher's intuition during an audit to make quick, correct calls on valuable surfaces is also noted as a potential point of failure.
## Comparison to Prior Work
This work builds upon established security testing paradigms but provides a direct critique of organizational misunderstanding and incentives that cause these two distinct disciplines to bleed into one another, often to the detriment of effective security posture management.
## Real-world Applications
- **Procurement of External Security Services:** Organizations can use this framework to write better Statements of Work (SOWs) for security assessments, specifying whether they need a high-volume quality review (Audit) or a targeted depth analysis (VR).
- **Internal Team Structuring:** Justifying dedicated VR teams whose successes are measured by impact reports rather than patch counts.
## Future Work
The author implies a need for the software security field to mature such that leaders can establish objective measures for the *quality* and *relevance* of knowledge gained from vulnerability research, reducing the reliance on easily countable, but often less impactful, vulnerability metrics.
## References
- [None explicitly listed, as this is a conceptual analysis piece mirroring practical experience.]
- [Related research - defanged URLs] N/A