Full Report
ssh-agent is a program for hold private keys for authentication through ENV variables. Agent forwarding is the process of forwarding from further remote hosts, removing the need for authentication data to be stored on other machines. For obvious reasons, this is a slight security issue! It gives users the ability to bypass file permissions on a remote host; but, this is still widely used today. In our case, this gives a remote attack surface! While browsing the source code of ssh-agent, the authors noticed an interesting bug: a user who has access to the remotely forwarded host can call dlopen() and dlclose() on /usr/lib* on the main workstation. A local privilege escalation by Jann Horn would load a library from /tmp was discovered in 2016. This resulted in an allowlist being made to filter out types. Initially, they looked for a filter bypass or a directory traversal to this fix but couldn't find one. In 2010, Tavis Ormandy used a dlopen/dlclose primitive similar to this one in a local setting by abusing the side effects of the library loading and unloading process. The original author used local primitives (ENV variables, file writes, etc.) to get code execution. In this case, we only have the remote dlopen/dlclose primitive. Is this even exploitable? To determine this, they reviewed various default libraries that could be loaded. What they found was startling from manual review. These are listed below: Some libraries require an executable stack, which will make the make stack executable. 58 of these were found. Many libraries are undeletable once loaded. 16577 were found. Some libraries create a SIGSEVG signal handler and do not deregister that handler if closed. This turns into a use-after-free-like situation. 9 were found. Some libraries instantly crash if loaded, since they are not loaded in the proper/expected context. 2 were found. As soon as I read those primitives, I saw the potential for exploitation. Make the stack executable, create a signal handler in a mmaped section of memory that gets unmapped, reload code into that area, trigger the signal handler jump to executable stack for code execution. All of this sounds fine and dandy, but finding a proper path for this is tricky. So, the authors fuzzed the process with the various primitives from above. They injected their own signal handler with shellcode on the stack (0xCC opcode) to say if their shellcode as properly hit or not. They added their shellcode by writing to the socket of the connection directly. They found two separate chains for the signal handle use after free to get code execution. While looking at the crashes, they found more primitives though. The next primitive was a callback function UAF. A shared library would register a userland callback function within a core library. Once this was unloaded, a different library could be replaced in that memory space. Finally, a shared library would make a call to the core library function that triggers the callback in our code. Another primitive they found resulted from a strange SIGSEGV. A library is loaded, then a thread starts a timer in kernel-land. This library could be unloaded, but the timer never stops. Once the thread returns from execution, it was hop into unmapped code. Naturally, we can replace this with a different library to jump to the stack for execution. While fuzzing, they kept getting the error message ... overflowed sigaltstack. Sigaltstack allows a thread to define a new alternate signal stack for execution of signal handlers. Similar to the previous bug, the library was implementing a separate signal stack but not unregistering it. Once the signal was hit, improper code was being hit, creating a sigaltstack use-after-free. Although they found some primitives, they couldn't take this to code execution after loads of testing. Crazily enough, various combinations of library loads led to direct access over the instruction pointer. This happens as a consequence of signal handler overwriting then calling RET N. Once this occurs, the signal handler restoration process occurs, overwriting the userland addresses back to normal. In order to exploit this, they needed an ASLR leak to jump back to the proper location. They did not pursue this option further. Another fuzzing crash was puzzling to them. With specific combinations of libraries, crashes occurred without any of the other primitives from above. The library LLVM libunwind.so is loaded is several locations. Some other libraries load libgcc_s.so for handling C++ exceptions. If both of these are loaded and an exception occurs, LLVM's _Unwind_GetCFA will be called within libgcc_s.so instead of the LLVM library! These have different types of structures, leading to a super bizarre type confusion vulnerability. At the end, the authors mention there may be other ways to exploit this; they only looked on Ubuntu Desktop. Overall, another amazing article from Qualys that boggles my mind.
Analysis Summary
# Vulnerability: Remote Code Execution in OpenSSH's ssh-agent (Forwarded)
## CVE Details
- **CVE ID:** CVE-2023-38408
- **CVSS Score:** 8.1 (High) - *Calculated based on typical RCE impact in this context.*
- **CWE:** CWE-491 (Publicly Writeable Directory), CWE-824 (Access of Uninitialized Pointer), and CWE-94 (Code Injection via dlopen/dlclose primitives).
## Affected Systems
- **Products:** OpenSSH (specifically `ssh-agent` and `ssh-pkcs11-helper`).
- **Versions:** All versions prior to OpenSSH 9.3p2.
- **Configurations:**
- `ssh-agent` must be compiled with `ENABLE_PKCS11` (default in most distributions).
- The user must have `ssh-agent` forwarding enabled (via `ssh -A` or `ForwardAgent yes`).
- The environment must have specific shared libraries available in `/usr/lib*` that exhibit side effects during loading/unloading.
## Vulnerability Description
The vulnerability exists in the way `ssh-agent` handles PKCS#11 providers via the `ssh-pkcs11-helper` process. An attacker who has gained access to a remote host where a victim has forwarded their `ssh-agent` can request the loading and unloading of arbitrary shared libraries located in `/usr/lib*`.
While the loader is restricted to trusted paths, many legitimate libraries have unintended "side effects" when processed by `dlopen()` and `dlclose()`. These include:
- Executing constructor/destructor functions.
- Leaving "dangling" signal handlers or callback functions in memory after a library is unloaded (Use-After-Free).
- Setting the stack as executable (GNU_STACK).
- Triggering type confusion between `libgcc` and `libunwind` during exception handling.
By chaining these side effects, an attacker can manipulate the process memory layout to achieve Remote Code Execution (RCE) on the victim's local workstation.
## Exploitation
- **Status:** PoC developed and verified by Qualys; highly technical but reliable.
- **Complexity:** High (Requires specific library combinations and side-effect chaining).
- **Attack Vector:** Network (Remote via forwarded agent socket).
## Impact
- **Confidentiality:** High (Access to private keys and workstation data).
- **Integrity:** High (Ability to execute arbitrary commands).
- **Availability:** High (Potential to crash authentication services).
## Remediation
### Patches
- **OpenSSH 9.3p2:** Released July 19, 2023. This version addresses the issue by strictly limiting the helper process and hardening the loading mechanism.
### Workarounds
- **Disable Agent Forwarding:** Avoid using the `-A` flag or setting `ForwardAgent yes` in `~/.ssh/config`. Use Jump Hosts (`-J`) instead.
- **Restrict PKCS#11:** If forwarding is required, use the `PKCS11Provider` allow-list in `ssh_config` to limit which libraries can be loaded.
## Detection
- **Indicators of Compromise:**
- Unexpected errors in logs from `ssh-pkcs11-helper`.
- Segmentation faults in `ssh-agent` or its helper processes.
- Large numbers of `dlopen`/`dlclose` requests recorded in system auditing tools (e.g., `auditd`).
- **Detection Methods:** Monitor for unusual shared library loading activity originating from `ssh-pkcs11-helper`.
## References
- **Vendor Advisory:** [https://www.openssh.com/txt/release-9.3p2]
- **Qualys Security Advisory:** [https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh-forwarded-ssh-agent.php]
- **CVE Link:** [https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-38408]