Full Report
The blog post introduces a concept known as differential fuzzing. The idea is to generate input, process it through multiple pipelines and see if the output is the same. In blockchain protocols that must have the same data, such as consensus client protocols, this is a great way to find bugs. Asymmetric Research uses this for many things, like fuzzing the Solana validator client differences Agave and Firedancer. The target for this demonstration is around JSON parsing. Their chosen fuzzer is AFL. The majority of fuzzing is the same as with a regular AFL fuzzer - input generation and such. The main difference is that we have two JSON parse calls to make. If the results are different, then this results in a "crash"! While running the fuzzer, they ran into several bugs but they were unimportant for security. For instance, 222E2322 was an out of range number that was rejected for one parser but fine by another. Since this doesn't have security consequences, they patched out to accept this case as valid. This happened several times. It should be noted that using a fuzzer is an iterative process - it's not just letting it run once and forgetting about it. Eventually, they get a legitimate parsing difference. serde_json gets a parsing error while json-rust parsers it. After looking at a hexdump, it becomes clear - the vertical control tab (^K) is causing the issue. json-rust has a goal of being less strict than the other JSON parser, since this can cause friction at times. This bug is likely enough to cause a consensus divergence, which is pretty interesting. Many other parsing issues between implementations, like described by the Bishop Fox post, can lead to major security consequences by returning difference values. There are many ways to make this fuzzer better. Using structured input on JSON data, parallelization and checks for equivalence on the data. Overall, a great post discussing automatic bug finding via differential fuzzing.
Analysis Summary
# Tool/Technique: Differential Fuzzing (using AFL/LibAFL)
## Overview
Differential fuzzing is a testing technique used to uncover bugs, particularly consensus or logic discrepancies, by comparing the outputs generated by two or more different implementations processing the exact same input. In this context, it was applied to JSON parsing implementations within blockchain consensus clients (like Solana validator clients Agave and Firedancer) to find divergences that could lead to consensus failure or security issues.
## Technical Details
- Type: Technique / Tool (when combined with a specific fuzzer framework)
- Platform: Primarily demonstrated on Linux (Ubuntu 24.04) using a Rust environment targeting x86\_64-unknown-linux-gnu.
- Capabilities: Identifies differences in behavior between system components (e.g., two parsers) given identical inputs by flagging discrepancies as "crashes" or divergence points.
- First Seen: The concept of differential testing has been around, but its application here is specific to modern fuzzing frameworks and blockchain protocols.
## MITRE ATT&CK Mapping
This technique is related to security testing and vulnerability discovery rather than an active adversarial technique, so direct mapping to standard TTPs is weak. However, it relates to the discovery of flaws that could be exploited:
- **TA0005 - Defensive Evasion**: While not direct evasion, successful exploitation relies on discovering weaknesses inherent in implementation logic.
- **T1560.001 - Archive Collected Data: Archive via Utility**: Less relevant, but fuzzing often relies on manipulating structured inputs.
- **T1997 - Software Discovery**: (Conceptual mapping for tool understanding) Techniques used for deeper software analysis.
## Functionality
### Core Capabilities
- **Input Generation**: Utilizes the AFL fuzzer methodology (specifically leveraging LibAFL) to generate diverse input data (in this case, for JSON parsing).
- **Dual Execution Path**: Executes the same input through two distinct implementations simultaneously (e.g., `serde_json` parser vs. `json-rust` parser).
- **Difference Detection**: Triggers an alert (counted as a "crash" for the fuzzer) if the resulting outputs or states of the two pipelines are asymmetric or unequal.
### Advanced Features
- **Consensus Bug Hunting**: Specifically targeted at finding discrepancies in decentralized systems, such as validator clients, where even minor parsing differences can cause a network fork (consensus divergence).
- **Non-Security Bug Triage**: Successfully identifies and filters out non-security-critical bugs (e.g., parser 222E2322 rejection difference) to focus on meaningful divergences.
- **Specific Bug Identification**: Successfully pinpointed a bug caused by a non-standard control character (`vertical control tab (^K)`) being accepted by one parser (`json-rust`) but causing an error in the other (`serde_json`).
## Indicators of Compromise
N/A - This is a security testing methodology, not an attack tool or malware.
## Associated Threat Actors
N/A - This methodology is used by security researchers and developers (e.g., Asymmetric Research) for defensive purposes.
## Detection Methods
N/A - As a testing technique, it is used in development/audit environments. Detection is focused on monitoring the fuzzer's output logs for divergence flags.
## Mitigation Strategies
- **Code Auditing and Review**: Implementing rigorous peer review for data handling logic across multiple implementations.
- **Standardization**: Enforcing strict adherence to data format specifications (like JSON) across all interacting components; increasing parser strictness where necessary.
- **Iterative Fuzzing**: Recognizing fuzzing as an iterative process that requires continuous running, analysis, and refinement of harnesses.
- **Structured Fuzzing**: Improving the fuzzer by incorporating structured input generation for JSON data to target specific schema elements more effectively.
## Related Tools/Techniques
- **AFL (American Fuzzy Lop)**: The base fuzzing engine leveraged by LibAFL in this setup.
- **LibAFL**: The specific fuzzing framework utilized for instrumenting Rust binaries.
- **Standard Fuzzing**: Traditional fuzzing methods focusing on crashing a single target implementation.
- **Bishop Fox analysis**: Referenced article detailing JSON interoperability vulnerabilities.