Full Report
This is a continuation of a previous blog post on Solana fuzzing. They found two bugs in the node software via fuzzing and this post is about triaging the bugs. The first bug they found was an intricate memory leak. In Solana, the Instruction Meter is used to keep track of the amount of gas a program has left remaining. The meter is only checked at the end of a block, which should be fine. However, the ordering of operations matters here. During a previous security fix, the ordering was changed. If an out of gas exception occurred at the same time that a unknown symbol error happened, then the latter error was simply overwritten. As a result, the string was left along on the heap without a reference to it. Although Rust provides memory safety, it doesn't guarantee that all memory is cleaned up. By default, this was only seven bytes, which was not a huge deal. The string that gets allocated comes from the loaded ELF file - it's the name of the function. Since this is controllable, providing a very large name. The most we can do is 1M bytes being allocated. Using this in a loop on Solana over and over again will lead to the system being running out of RAM and crashing. The second bug was easy to trigger but hard to diagnose. The JIT terminated with Ok(0) but the interpreter returned with an access violation error. Juicy! Looks like there's an out-of-bounds write in the JIT somewhere. The violation was at 0x100000000 specifically. This memory address stores some read-only data present in the ELF provided. In order to find the bug, they used GDB to see when and why this was being written to. The program has access and bounds checks in it. Why is this failing? The instruction was being executed was cmp DWORD PTR [rax+0x19], 0x0. This is surprising because it's not an 8-bit operand!? Why!? The x86 instruction uses the opcode 0x81 but only for 16, 32, and 64-bit register operands. If you want to compare the 8-bit version, you must use the 0x80 opcode instead. This leads to the CPU performing an incorrect comparison and using unintended values around it. Neat! By breaking the comparison for the write using an 8-bit value, we can write a single byte to the read-only memory section. At first, the author thought this could be used to compromise the execution of a program. However, this only allows for modifying your own program and not others. Instead of the 1M they were hoping for, they got 100K for this bug (and the last one). Overall, it is a fantastic series! I enjoyed the Oracle approach to finding bugs that weren't the typical "fuzzier crashed" situation. Both bugs were relatively complicated and deep, yet not too many lines of code. Good strategy!
Analysis Summary
# Vulnerability: Memory Leak via Overwritten Out-of-Gas Exception in Solana rBPF Instruction Meter
## CVE Details
- CVE ID: Not assigned at the time of publication. (Referencing the associated Solana disclosures)
- CVSS Score: Not explicitly provided. Likely **Medium to High** severity due to potential DoS.
- CWE: CWE-401 (Memory Leak)
## Affected Systems
- Products: Solana rBPF (Rust virtual machine and JIT compiler for eBPF programs).
- Versions: Specifically tested against version `v0.2.21` of `solana-rbpf`.
- Configurations: Affected when an out-of-gas exception occurs simultaneously with an unknown symbol error during program execution, particularly when loading a malicious ELF file with a controllable function name.
## Vulnerability Description
The memory leak arises from an incorrect ordering of operations following an error condition within the Instruction Meter (which tracks remaining gas). A previous security fix altered the error handling sequence. If an **out-of-gas exception** occurred concurrently with an **unknown symbol error**, the unknown symbol error handler overwrote the out-of-gas exception's cleanup routine. This resulted in a dynamically allocated string (the name of the function from the loaded ELF file, which is controllable) being left unreferenced on the heap. By crafting an ELF with a deliberately linked, yet undefined, function having a very large name (up to 1MB), an attacker can repeatedly trigger this execution path, leading to cumulative memory exhaustion and eventual Denial of Service (DoS) by running the system out of RAM.
## Exploitation
- Status: Proof of Concept (PoC) developed and used for disclosure. Not explicitly stated as exploited in the wild.
- Complexity: **High**. Requires precise control over instruction sequencing and crafting a malicious ELF with a specific function name length and execution flow to hit the exact error interleaving condition.
- Attack Vector: Local/Programmatic (Requires loading and executing a malicious BPF/ELF program).
## Impact
- Confidentiality: None
- Integrity: None (Though continuous memory exhaustion impairs system operation)
- Availability: **High**. Leads to Denial of Service (DoS) by exhausting system RAM, causing the node to crash.
## Remediation
### Patches
- The vulnerability was patched within 4 hours of reporting in the `solana-rbpf` codebase. The specific version that contains the fix should be referenced from subsequent Solana releases following the report date (May 11, 2022).
### Workarounds
- No specific workarounds were detailed, as the fix was implemented rapidly. In principle, only validating ELF structure or ensuring correct error handling order would mitigate.
## Detection
- **Indicators of Compromise (IOCs):** Repeated allocation of memory corresponding to function names within the ELF structure; system instability or crash attributed to running out of RAM during BPF execution cycles.
- **Detection Methods and Tools:** Monitoring memory usage patterns during BPF program execution. The specific crashing sequence involves an instruction meter being depleted while simultaneously encountering a symbol lookup failure.
## References
- Vendor Advisories: Refer to subsequent Solana security advisories following the report date of May 11, 2022, pertaining to rBPF fuzzing results.
- Relevant Links:
- Original Research Site: hxxps://secret.club/2022/05/11/fuzzing-solana-2.html
- Affected rBPF Version Reference: hxxps://github.com/solana-labs/rbpf/releases/tag/v0.2.21
***
# Vulnerability: Out-of-Bounds Write in JIT Compiler Due to Incorrect x86 Opcode Selection
## CVE Details
- CVE ID: Not assigned at the time of publication.
- CVSS Score: Not explicitly provided. Likely **Medium** severity due to restricted scope of the write.
- CWE: CWE-190 (Integer Overflow or Wraparound) or CWE-787 (Out-of-bounds Write).
## Affected Systems
- Products: Solana rBPF JIT Compiler component within `solana-rbpf`.
- Versions: Tested against `v0.2.21` of `solana-rbpf`, likely affected in prior versions.
- Configurations: Triggered during JIT compilation/execution where specific comparison instructions are generated.
## Vulnerability Description
This vulnerability involves an out-of-bounds write occurring during the JIT translation of BPF instructions. The compiled assembly instruction intended to check an 8-bit value used the x86 opcode `0x81` (`cmp DWORD PTR [...]`), which specifies a 32-bit comparison operand size, instead of the correct opcode `0x80` for an 8-bit comparison. This opcode misuse caused the CPU to perform an unintended comparison operation, leading to the instruction using incorrect values, effectively bypassing bounds checks and allowing a **single byte write (100KB payload limit)** to a memory location mapped as read-only data within the provided ELF structure (specifically at address `0x100000000`). While the write target was read-only memory, the impact was limited to modifying data associated with the executing program, preventing code execution compromise in this context.
## Exploitation
- Status: PoC available specific to modifying own program state/data.
- Complexity: **High**. Requires deep understanding of JIT compilation artifacts and x86 instruction encoding to force the specific instruction generation error.
- Attack Vector: Local/Programmatic (Requires loading and executing a malicious BPF/ELF program).
## Impact
- Confidentiality: Low
- Integrity: Low (Limited to modifying the executing program's loaded data, not cross-program integrity compromise).
- Availability: Low (Though high frequency execution could cause instability).
## Remediation
### Patches
- The vulnerability was patched rapidly alongside the first bug. Refer to subsequent Solana releases post-May 11, 2022, addressing the rBPF JIT integrity.
### Workarounds
- No specific workarounds were detailed. Patching the instruction selection logic in the JIT compiler is the required fix.
## Detection
- **Indicators of Compromise (IOCs):** Unexpected write attempts to memory addresses within the ELF's read-only sections during JIT execution, specifically involving comparisons that mistakenly use 32-bit opcodes (`0x81`) where 8-bit opcodes (`0x80`) were intended.
- **Detection Methods and Tools:** Debugging JIT output during execution via GDB to inspect instruction encoding for comparisons. Detecting writes near the `0x100000000` address space during BPF execution.
## References
- Original Research Site: hxxps://secret.club/2022/05/11/fuzzing-solana-2.html