Skip to main content

CVE-2023-3519, what you should know and how to fix your Netscaler ADC, NetScaler Gateway

Disclamer: This blogpost is made to help you understanding CVE-2023-3519 and how you can check if you are vulnerable and community guidelines how to fix your environment. This blogpost is not covering the details of the exploits out here as I have no interest in sharing what the red team is doing. All responsibility is your own. I highly recommend you read the blogpost and take action immediately, don’t hesitate.

Timeline for CVE-2023-3519

The timeline for CVE-2023-3519 will be updated as things progress. (last updated 25th July 2023)

June 2023

In June 2023, threat actors exploited this vulnerability as a zero-day to drop a webshell on a critical infrastructure organization’s non-production environment NetScaler ADC appliance. The webshell enabled the actors to perform discovery on the victim’s active directory (AD) and collect and exfiltrate AD data. The actors attempted to move laterally to a domain controller but network-segmentation controls for the appliance blocked movement.

The victim organization identified the compromise and reported the activity to CISA and Citrix. Citrix released a patch for this vulnerability on July 18, 2023.

18th July 2023

Citrix came out with a critical security message for Citrix ADC aka Netscaler, Citrix Access Gateway the 18th July 2023 which is CVE-2023-3519. By exploiting this vulnerability, a remote, unauthenticated attacker may be able to execute arbitrary code on a vulnerable system. At this time there is no public PoC, but the vulnerability has been observed being exploited in the wild.

Citrix has noted that the appliance must be configured as a Gateway (VPN virtual server, ICA Proxy, CVPN, RDP Proxy) or AAA virtual server to be vulnerable.

CVE-2023-3519 have a CVSS score 9.8 rating out of 10.

Citrix update creates CTX561482 which contains info about CVE-2023-3519.

Firmware builds have been released 18th July 2023 for Citrix ADC/Access Gateway versions
Citrix strongly recommends that customers on these versions install these updates immediately.

Description of Problem

Multiple vulnerabilities have been discovered in NetScaler ADC (formerly Citrix ADC) and NetScaler Gateway (formerly Citrix Gateway).

The following supported versions of NetScaler ADC and NetScaler Gateway are affected by the vulnerabilities: 

  • NetScaler ADC and NetScaler Gateway 13.1 before 13.1-49.13 
  • NetScaler ADC and NetScaler Gateway 13.0 before 13.0-91.13 
  • NetScaler ADC 13.1-FIPS before 13.1-37.159
  • NetScaler ADC 12.1-FIPS before 12.1-55.297
  • NetScaler ADC 12.1-NDcPP before 12.1-55.297

Note: NetScaler ADC and NetScaler Gateway version 12.1 is now End Of Life (EOL) and is vulnerable.

This bulletin only applies to customer-managed NetScaler ADC and NetScaler Gateway. Customers using Citrix-managed cloud services or Citrix-managed Adaptive Authentication do not need to take any action. 

CVE IDAffected ProductsDescriptionPre-requisitesCWECVSS
CVE-2023-3466Citrix ADC, Citrix GatewayReflected Cross-Site Scripting (XSS)Requires victim to access an attacker-controlled link in the browser while being on a network with connectivity to the NSIPCWE-208,3
CVE-2023-3467Citrix ADC, Citrix GatewayPrivilege Escalation to root administrator (nsroot)Authenticated access to NSIP or SNIP with management interface accessCWE-2698
CVE-2023-3519Citrix ADC, Citrix GatewayUnauthenticated remote code executionAppliance must be configured as a Gateway (VPN virtual server, ICA Proxy, CVPN, RDP Proxy) OR AAA virtual serverCWE-949,8

Jorren Geurts and Wouter Rijkborst, researchers at cybersecurity company Resillion, published a detailed technical analysis of the vulnerability, explaining how stacking specific commands in the NetScaler CLI allows any user with read-only permissions to obtain root privileges on the system.

19th July 2023

Cloud Software Group (Citrix) have build a IoC (Indicator of Compromise) script v3 to help organisations determine if they are potentially compromised with NetScaler ADC and NetScaler Gateway.

NetScaler customers/partners can get access to IoC script v3, when they create a support ticket at Citrix. The IOC script v3 helps validate if NetScaler customers are compromised also after if they have applied the new firmware, so it’s important to get tested the IOC script v3 in your NetScaler environment.

@deyda84 release a blogpost Checklist for NetScaler (Citrix ADC) CVE-2023-3519

20th July 2023

CISA has released an advisory with tactics, techniques, and procedures (TTPs) along with detection methods to help organizations, particularly those in the critical infrastructure segment, determine if their systems were compromised.

CISA release an advisory with tactics, techniques, and procedures (TTP) along with dectectio methods to help organisations, particular those in the critical infrastructure segment, determine if their NetScaler systems were compromised

CISA release pdf Threat Actors Exploiting Citrix CVE-2023-3519 to implement Webshells

Cloud Software Group (Citrix) have build a IoC (Indicator of Compromise) script v4 to help organisations determine if they are potentially compromised with NetScaler ADC and NetScaler Gateway.

NetScaler customers/partners can get access to IoC script v4, when they create a support ticket at Citrix. The IOC script v4 helps validate if NetScaler customers are compromised also after if they have applied the new firmware, so it’s important to get tested the IOC script v4 in your NetScaler environment.

21th July 2023

Assetnote release blogpost about Analysis of CVE-2023-3519 in Citrix ADC and NetScaler Gateway their analysis so far indicates that SAML has to be enabled for exploitation.

Mendicant release a blogpost about Exploitation of Citrix Zero-Day by Possible Espionage Actors (CVE-2023-3519)

Patrick Coble, Citrix CTP wrote a LinkedIn about CVE-2023-3519 CVE-2022-27513 and CVE-2023-3466

22th July 2023

CVE gets published on CVE-2023-3519 https://www.cve.org/CVERecord?id=CVE-2023-3519

NIST publish CVE-2023-3519 https://nvd.nist.gov/vuln/detail/CVE-2023-3519

Security researchers from the Shadowserver Foundation, a non-profit organization dedicated to enhancing internet security, revealed that at least 15,000 appliances were identified as exposed to attacks leveraging the flaw (CVE-2023-3519) based on their version information.

23th July 2023

Poppelgaard.com release blogpost about CVE-2023-3519

24th July 2023

Assetnote release blogpost Analysis of CVE-2023-3519 in Citrix ADC and NetScaler Gateway (Part 2)

25th July 2023

Cloud Software Group (Citrix) have build a IoC (Indicator of Compromise) script v5 to help organisations determine if they are potentially compromised with NetScaler ADC and NetScaler Gateway.

NetScaler customers/partners can get access to IoC script v5, when they create a support ticket at Citrix. The IOC script v5 helps validate if NetScaler customers are compromised also after if they have applied the new firmware, so it’s important to get tested the IOC script v5 in your NetScaler environment.

Why you should take CVE-2023-3519 serious

NIST GAVE THE CVE-2023-3519 A SCORE OF 9.8 OUT OF 10

Source:

Citrix Products affected

  • NetScaler ADC and NetScaler Gateway 13.1 before 13.1-49.13 
  • NetScaler ADC and NetScaler Gateway 13.0 before 13.0-91.13 
  • NetScaler ADC 13.1-FIPS before 13.1-37.159
  • NetScaler ADC 12.1-FIPS before 12.1-55.297
  • NetScaler ADC 12.1-NDcPP before 12.1-55.297

Note: It is important to be aware that NetScaler ADC and NetScaler Gateway version 12.1 has reached the End Of Life (EOL) status. This implies that this version is not only vulnerable but also exposes customers to potential risks. It is highly recommended that customers swiftly upgrade to a supported version, such as 13.0 or 13.1, to ensure a safer and more secure environment.

What should Customer do

Exploits of CVE-2023-3519 on unmitigated appliances have been observed. Cloud Software Group strongly urges affected customers of NetScaler ADC and NetScaler Gateway to install the relevant updated versions as soon as possible. 

NetScaler customers/partners should upgrade even the NetScalers that are not currently running AAA/GW. After the critical ones (NetScaler ADC running AAA/GW) have been patched, so that you do not unknowingly make yourself vulnerable down the road.

This is a critical patch from NetScaler and you should take the threat seriously and not wait as many customers/partner did back in 2019 (CVE-2019-19781) people waited until the public exploit came out.

Your NetScaler ADC/Gateway might already been compromised and to get this validated, you can create a support ticket at Citrix support and get IOC script v5 created by Cloud Software Group for CVE-2023-3519 and run on all your individual NetScaler appliances to validate if your NetScaler appliance is compromised. Make sure to run IOC script v5 as guided by Citrix and be careful to run it accurate as described in the IOC document.

Fix Timelines for firmware GA

Citrix have firmware updates in the form of refresh builds which are available across all supported versions of Netscaler ADC, Netscaler Gateway. Please refer to the table below for the expected release dates.

VPX/MPX/SDX – Customers with NetScaler ADC, NetScaler Gateway 12.1 & 13 & 13.1 can upgrade to latest 13.0 build 91.13 & 13.1 build 49.13 as security patch is GA.

FIPS VPX/MPX/SDX – Customers with NetScaler 12.1 & 13.1 can upgrade to latest 12.1-55.297 & 13.1-37.159
NDcPP VPX/MPX/SDX – Customer with NetScaler 12.1 can upgrade to latest 12.1-55.297

Note: NetScaler ADC and NetScaler Gateway version 12.1 is now End Of Life (EOL). Customers are recommended to upgrade their appliances to one of the supported versions that address the vulnerabilities. 

Netscaler ADC and Netscaler Gateway
VersionRefresh BuildExpected Release Date
13.113.1-49.1318th July 2023 (released)
13.013.0-91.1318th July 2023 (released)

Netscaler ADC and Netscaler Gateway FIPS/NDcPP
VersionRefresh BuildExpected Release Date
13.1-FIPS13.1-37.15918th July 2023 (released)
12.1-FIPS12.1-55.29718th July 2023 (released)
12.1-NDcPP12.1-55.29718th July 2023 (released)

Deep-dive insights about CVE-2023-3519

nothing GA.

Mitigation Steps for CVE-2023-3519

upgrade NetScaler ADC/Gateway to latest firmware from 18th July 2023

If you applied the firmware update for your NetScaler ADC or NetScaler Gateway CTX561482 after 18 July 2023, take ACTION

The chances of your environment being vulnerable is extremely high I would say 99.99%, so take no risk and check your environment ASAP.

Test IOC script v5 from Citrix support on your NetScaler ADC/AG
Reach out to Citrix support and get IOC script v5 for CVE-2023-3519 and run on all your individual NetScaler appliances to validate if your NetScaler appliance is compromised. Make sure to run IOC script v5 as guided by Citrix and be careful to run it accurate to document.

Check if you are compromised*:

*Below is reported by the community – below is a guidance but on own risk.

Check for files newer than last installation (web shells)

Note: Modify the -newermt parameter with the date that corresponds to your last installation

find /netscaler/ns_gui/ -type f -name *.php -newermt 20230501 -exec ls -l {} \;

find /var/vpn/ -type f -newermt 20230501 -exec ls -l {} \;

find /var/netscaler/logon/ -type f -newermt 20230501 -exec ls -l {} \;

find /var/python/ -type f -newermt 20230501 -exec ls -l {} \;

Check http error logs for abnormalities 

grep ‘\.sh’ /var/log/httperror.log*
grep ‘\.php’ /var/log/httperror.log*

Check shell logs for unusual post-ex commands

grep ‘/flash/nsconfig/keys’ /var/log/sh.log*

Find setuid binaries dropped

ind /var -perm -4000 -user root -not -path “/var/nslog/*” -newermt 20230501 -exec ls -l {} \;

Apache Log files

In addition, attempts to exploit the system leave traces in the Apache httpaccess log files. Those you can validate via:

shell cat /var/log/httpaccess.log | grep vpns | grep xml
shell cat /var/log/httpaccess.log | grep "/../"
shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml
shell gzcat /var/log/httpaccess.log.*.gz | grep "/../"

The following output is found on a system that was exploited:

However, a guarantee can never been given as attackers also might clean up their traces of the initial exploitation. A few more things to validate are…

Cron jobs

Check your crontab file for anomalies:

shell cat /etc/crontab
shell crontab -l -u nobody

Backdoor scripts

Running backdoors or other malicious tasks are often executed as Perl or Python scripts. Check for the presence of active running Perl or Python tasks:

shell ps -aux | grep python
shell ps -aux | grep perl

If you see more then the “grep” commands itself check the running scripts.

But beware, several NetScaler ADC system-native tasks might appear as well. Some run scheduled so run the query again a few seconds later. Some are permanent (custom monitoring scripts for Storefront for example). Check those scripts to make sure they weren’t altered.

Crypto miners

These are some of the checks that was done in previous exploits (CVE-2019-19781) which still is valid

Attacks can have installed crypto miners. You can identifiy those by looking at the CPU intense processes by running:

shell top -n 10

Should you see any other processes but NSPPE-xx displaying high CPU usage you might have found a crypto miner:

Bash logs

In addition to the apache logs, some payloads run bash commands which can be traced in the bash log files. Check for those via:

shell cat /var/log/bash.log | grep nobody
shell gzcat /var/log/bash.*.gz | grep nobody

Attackers will land with the rights of apache (which is “nobody”). So if they execute commands via bash – which a few of the payloads I’ve seen did – or if they spawn a remote shell they are likely to end up being logged here:

But beware, these logs rotate rather quickly (1-2 days) because ADC spams them with quite a bunch of messages each minute with its own scheduled tasks!

Apache error logs

A bit more tricky is the analysis of the apache error logs. These log failed execution attempts (and I’ve seen a few misspelled commands or syntax errors). You might be able to use the following filter to catch some events:

shell "cat /var/log/httperror.log | grep -B2 -A5 Traceback"
shell "gzcat /var/log/httperror.log.*.gz | grep -B2 -A5 Traceback"

The following is a failed attempts at executing python because the path wasn’t referenced correctly:

But again, this might only be one type of error, ideally you may want to review them manually, especially if you’re in doubt:

shell cat /var/log/httperror.log
shell gzcat /var/log/httperror.log.*.gz

Firewall

In addition to NetScaler ADC local indicators observe your surrounding firewalls for any irregular traffic. Most likely attackers will use the NetScaler ADC as a jump host to penetrate the network further.

Global Responder policy

Set up a global responder policy with no action and a log action to capture client IP, user agent, URL, arguments etc, and ship that out .. so at least there is a log of what has hit the VPN VServer, which you can look back on.

LDAP

Review AD logs for logon activities originating from the ADC IP with the account configured for AD connection. 

Review network/firewall logs for unexpected spikes in AD/LDAP/LDAPS traffic originating from the ADC (this may indicate AD/LDAP enumeration).

If you suspect you are compromised, take action:

Below is 2 solutions. Some like to reimage a VPX from scratch while others like to use their backup solution, so I have provided both solutions here.

Solution 1: reimage

  1. If compromised from above STEP do following:
  2. Test IOC script v5 from Citrix support on your NetScaler ADC/AG
    Reach out to Citrix support and get IOC script v5 for CVE-2023-3519 and run on all your individual NetScaler appliances to validate if your NetScaler appliance is compromised. Make sure to run IOC script v5 as guided by Citrix and be careful to run it accurately to document.
  3. Remove your NetScaler ADC/Gateway from network
  4. Create support ticket at NetScaler support
  5. Take a snapshot of potential vulnerable NetScaler ADC VPX for forensic analysis and further investigation
  6. Re-Image the NetScaler ADC (both in HA*) and re-import config
    its a slow process for MPX as you have to go through Citrix techsupport and might even need to get a new HDD with the clean image shipped
  7. Investigate all servers the NetScaler ADC has a connection to for further compromise.
    This might include the typical Citrix candidates like Storefront, etc. as well as DCs for LDAP but might also include a huge bunch of web- and application servers in a true NetScaler ADC deployment
  8. Change all NetScaler ADC local accounts passwords
    While stored hashed in the ns.conf those might be crackable
  9. Change all NetScaler ADC AD service accounts passwords
    While stored encrypted in the ns.conf those might be crackable
  10. Change all AD user account passwords that were using the NetScaler ADC to login
    All users? Yes, remember you are able to take wireshark traces on NetScaler ADC including SSL session keys which can then be used to decrypt the whole communications – including user login credentials
  11. Revoke and renew all SSL certificates (private/public)
    Private keys might have been downloaded. While private key passwords are stored encrypted in the ns.conf those might be crackable
  12. Upgrade to latest NetScaler ADC/AG build.
    Special note:
    if NetScaler ADC/AG 13.1 upgrade to build 13.1-49.13, to install the security vulnerability fixes
    if NetScaler ADC/AG 13.0 upgrade to build 13.0-91.13,  to install the security vulnerability fixes
    if NetScaler ADC/AG FIPS 13.1 upgrade to build 13.1-37.159, to install the security vulnerability fixes
    if NetScaler ADC/AG FIPS 12.1 upgrade to build 12.1-55.297 to install the security vulnerability fixes
    if NetScaler ADC/AG NDcPP 12.1 upgrade to build 12.1-55.297, to install the security vulnerability fixes
  13. Test IOC script v5 from Citrix support on your NetScaler ADC/AG
    Reach out to Citrix support and get IOC script v5 for CVE-2023-3519 and run on all your individual NetScaler appliances to validate if your NetScaler appliance is compromised.
    Make sure to run IOC script v5 as guided by Citrix and be careful to run it accurate as described in the IOC document.
    Rerun IOC script v5 on newly build ADC to check scan is OK and ADC/AG is not compromised.
  14. Enhance security and implement Citrix ADM from Citrix Cloud or On-prem.
  15. Run diagnostic and upload to cis.citrix.com to check if have recommendations from citrix to enhance
  16. Harden your NetScaler ADC/AG
    Example use this Best practices for NetScaler MPX, VPX, and SDX security guide from NetScaler
  17. Enable NetScaler ADC on internet
  18. Check for vulnerability see above
  19. Inform your country cyber security you have been compromised and take steps according to either GPDR or your data law in your country.

Solution 2: use snapshot from May 2023

  1. If compromised from above STEP do following:
  2. Remove your NetScaler ADC from network
  3. Take a snapshot of potential vulnerable NetScaler VPX for forensic analysis and further investigation
  4. Restore backup of VPX from May 2023 (both in HA)
  5. Investigate all servers the Citrix ADC has a connection to for further compromise.
    This might include the typical Citrix candidates like Storefront, etc. as well as DCs for LDAP but might also include a huge bunch of web- and application servers in a true ADC deployment
  6. Change all NetScaler ADC local accounts passwords
    While stored hashed in the ns.conf those might be crackable
  7. Change all NetScaler ADC AD service accounts passwords
    While stored encrypted in the ns.conf those might be crackable
  8. Change all AD user account passwords that were using the NetScaler ADC to login
    All users? Yes, remember you are able to take wireshark traces on ADC including SSL session keys which can then be used to decrypt the whole communications – including user login credentials
  9. Revoke and renew all SSL certificates (private/public)
    Private keys might have been downloaded. While private key passwords are stored encrypted in the ns.conf those might be crackable
  10. Upgrade to latest ADC/AG build.
    Special note:
    if NetScaler ADC/AG 13.1 upgrade to build 13.1-49.13, to install the security vulnerability fixes
    if NetScaler ADC/AG 13.0 upgrade to build 13.0-91.13,  to install the security vulnerability fixes
    if NetScaler ADC/AG FIPS 13.1 upgrade to build 13.1-37.159, to install the security vulnerability fixes
    if NetScaler ADC/AG FIPS 12.1 upgrade to build 12.1-55.297 to install the security vulnerability fixes
    if NetScaler ADC/AG NDcPP 12.1 upgrade to build 12.1-55.297, to install the security vulnerability fixes
  11. Test IOC script v5 from Citrix support on your NetScaler ADC/AG
    Reach out to Citrix support and get IOC script v5 for CVE-2023-3519 and run on all your individual NetScaler appliances to validate if your NetScaler appliance is compromised.
    Make sure to run IOC script v5 as guided by Citrix and be careful to run it accurate as described in the IOC document.
    Rerun IOC script v5 on newly build ADC to check scan is OK and ADC/AG is not compromised.
  12. Enhance security and implement Citrix ADM from Citrix Cloud or On-prem.
  13. Run diagnostic and upload to cis.citrix.com to check if have recommendations from citrix to enhance
  14. Harden your NetScaler ADC/AG
    Example use this Best practices for NetScaler MPX, VPX, and SDX security guide from NetScaler
  15. Enable NetScaler ADC on internet
  16. Check for vulnerability see above
  17. Inform your country cyber security you have been compromised and take steps according to either GPDR or your data law in your country.

CISA/Citrix recommendations

CVE-2023-3519 IOC script v5 can be received Citrix support if you are NetScaler customer/partner.

Cloud Software Group – Citrix
https://support.citrix.com/article/CTX561482/citrix-adc-and-citrix-gateway-security-bulletin-for-cve20233519-cve20233466-cve20233467

CISA
https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-201a#:~:text=e.g.%2C%20CLI).-,DETECTION%20METHODS,-Run%20the%20following

Community recommendations

Source

Sign up security alerts:

https://login.citrix.com/?url=https://support.citrix.com/user/alerts

BIG Thank You to the Citrix community

@ARNAUD_PAIN
@JANTYTGAT
@MANUELKOF
@DEYDA84
@VDIHACKER
@MBP_NETSCALER
@JENSHEERIN
@JOESHONK