Disclamer: This blogpost is made to help you understanding CVE-2019-19781 and how you can check if you are vulnerable and community guidelines how to fix your environment. This blogpost is not covering the details of the exploits out here as I have no interest in sharing what the red team is doing. All responsibility is your own. I highly recommend you read the blogpost and take action immediately, don’t hesitate.

Timeline for CVE-2019-19781

17th December 2019 (Warning and remediation plan from Citrix)

Citrix came out with a security message for Citrix ADC aka Netscaler, Citrix Access Gateway the 17th December 2019 which is CVE-2019-19781. By exploiting this vulnerability, a remote, unauthenticated attacker may be able to execute arbitrary code on a vulnerable system.

I tweeted that customers/partners should take action and after a few moments, Anton Van Pelt and Daniel Weppeler shared a responder policy was required. At that time CTX267679 didn’t have the responder policy, but it could be found on pastebin.com later it was official on CTX267679.

The majority of Citrix customers and partners implemented the responder policy.

8th January 2020

TripWire Release blog providing high-level details
Honeypot/Scanning Activity increases

10th January 2020

Project Zero India drops public bash PoC
TrustedSec release public python script with detections. 

11 January 2020 (Exploit kits gets GA)

In 2020 just before Citrix Summit (13-15th Jan 2020) the exploit kit was released for CVE-2019-19781. 11th January 2020 the exploit kit became weaponized, so it seem planned so that it would hit hard when all Citrix Partners was at Citrix Summit conference all week. Massive scanning of the internet began 11th January 2020 and the exploit kit evolved day by day and customers who hadn’t applied the mitigation steps according to CTX267679 was vulnerable. Citrix announced their patch timeline and FireEye observes multiple compromises using exploits. 

12th January 2020

Badpackets shared how massive the attack was and over 25.000 customers was vulnerable out of 100+K customers.
Source:

13th January 2020

The Cybersecurity and Infrastructure Security Agency (CICS) released a tool to check if your ADC/AG was vulnerable:
Source:

@Manuelkolloff release a blogpost Citrix ADC CVE-2019-19781 exploited! What now, which helped me find hackers in effected Citrix ADC/AG environments. Thank you Manuel for making this great blogpost, I helped Manuel with feedback and information such as if you been compromised was added 14th January 2020.

15th January 2020 (Citrix release verification tool)

Citrix releases verification tool to CVE-2019-19781
Source:

16th January 2020 (Citrix update CVE-2019-19781)

Citrix update CVE-2019-19781 including Citrix SD-WAN WANOP appliance, and In Citrix ADC Release 12.1 builds before 51.16/51.19 and 50.31, a bug exists that affects responder and rewrite policies bound to VPN virtual servers causing them not to process the packets that matched policy rules. Citrix recommends customers update to an unaffected build for the mitigation steps to apply properly

17th January 2020 (Citrix update CVE-2019-19781)

Citrix update CVE-2019-19781. In Citrix ADC and Citrix Gateway Release “12.1 build 50.28”, an issue exists that affects responder and rewrite policies causing them not to process the packets that matched policy rules. Citrix recommends that customers choose one from the following two options for the mitigation steps to function as intended:

1. Update to the refreshed “12.1 build 50.28/50.31” or later, OR
2. Apply the mitigation steps towards protecting the management interface as published in CTX267679. This will mitigate attacks, not just on the management interface but on ALL interfaces including Gateway and AAA virtual IPs.

Fermin J. Serna, Chief Information Security Officer at Citrix gives an gives updates on Citrix ADC, Citrix Gateway, SDWAN WAN OP vulnerability Source

19th January 2020 (Citrix update CVE-2019-19781)+security firmware GA for 11.1 build 11.1.63.15 & 12.0 build 12.0.63.13

Citrix update CVE-2019-19781.
Fixed builds have been released for Citrix ADC /Access Gateway versions 12.0 build 63.13 and 11.1 build 63.15.

Citrix strongly recommends that customers on these versions install these updates immediately.

Customers who have upgraded to fixed builds 12.0 build 63.13 or 11.1 build 63.15 do not need to retain the mitigation described in CTX267679.

These fixes also apply to Citrix ADC/Gateway Virtual Appliances (VPX) hosted on any of ESX, Hyper-V, KVM, XenServer, Azure, AWS, GCP or on a Citrix ADC Service Delivery Appliance (SDX).

It is necessary to upgrade all Citrix ADC/Gateway 11.1 instances (MPX or VPX) to build 11.1.63.15 to install the security vulnerability fixes.  It is necessary to upgrade all Citrix ADC/Gateway 12.0 instances (MPX or VPX) to build 12.0.63.13 to install the security vulnerability fixes.

Fermin J. Serna, Chief Information Security Officer at Citrix gives an gives updates on first permanent fixes available, timeline accelerated for Citrix ADC, Citrix Gateway, SDWAN WAN OP.

22th January 2020 (Citrix update CVE-2019-19781)+releases forensic tool+fix build for SDWAN WAN OP 10.2.6b and 11.0.3b

Citrix update CVE-2019-19781.

Citrix have together with FireEye released a forensic tool for CVE-2019-19781, you can download it here.

The free tool, available under the Apache 2.0 open source license, provides customers with increased awareness of potential compromise related to the CVE-2019-19781 vulnerability on their systems. The tool is designed to allow customers to run it locally on their Citrix instances and receive a rapid assessment of potential Indicators of Compromise based on known attacks and exploits.

The tool combines Citrix’s technical knowledge of the Citrix ADC and Gateway products and CVE-2019-19781 with industry-leading FireEye Mandiant’s forensics expertise and current knowledge of recent CVE-2019-19781 related compromises. FireEye Mandiant is not only at the forefront of cyberthreat intelligence and forensics, but has first-hand knowledge of the threat landscape and current exploits specific to CVE-2019-19781, making it an ideal partner for this important initiative.

Citrix also released the fixed firmware for Citrix SD-WAN WANOP 10.2.6b and 11.03b

To apply the security vulnerability fix, you need to upgrade all Citrix SD-WAN WANOP versions to build 10.2.6b or 11.0.3b as appropriate. These fixes are ONLY applicable to the SD-WAN 4000-WO, 5000-WO, 4100-WO, and 5100-WO platforms. All other SD-WAN PE and SD-WAN SE platforms are not impacted by this vulnerability and do not need to be patched.

Fermin J. Serna, Chief Information Security Officer at Citrix gives an gives updates and share forensic tool for Citrix ADC, Citrix Gateway, SDWAN WAN OP and release of fix for SD-WAN WAN OP

23th January 2020 (Citrix update CVE-2019-19781)+releases fix build for ADC/AG 12.1 build 55.18 and 13.0 build 47.24

Citrix update CVE-2019-19781.

Fixed builds have been released for Citrix ADC/Access Gateway versions 12.1 build 55.18 and 13.0 build 47.24.

Citrix strongly recommends that customers on these versions install these updates immediately.

Customers who have upgraded to fixed builds 12.1 build 55.18 and 13.0 build 47.24 do not need to retain the mitigation described in CTX267679.

Fermin J. Serna, Chief Information Security Officer at Citrix gives an gives updates on permanent fixes available for ADC/AG 12.1 & 13.0.

24th January 2020 (Citrix update CVE-2019-19781)+releases fix build for ADC/AG 10.5 build 70.12

Citrix update CVE-2019-19781.

Fixed builds have been released for Citrix ADC/Access Gateway versions 10.5 build 70.12

Citrix strongly recommends that customers on these versions install these updates immediately.

Customers who have upgraded to fixed builds 10.5 build 70.12 do not need to retain the mitigation described in CTX267679.

Fermin J. Serna, Chief Information Security Officer at Citrix gives an gives updates on permanent fixes available for ADC/AG 10.5

Why you should take CVE-2019-19781 serious

NIST GAVE THE CVE-2019-19781 A SCORE OF 9.8 OUT OF 10

Source:

(…this tool only checks if you applied the mitigation steps from Citrix, so if a hacker has already been in your system, then you are compromised, which many forget to think about. I can validate that I have fixed several client’s who thought they were safe but the hacker was in their system already when the mass scanning happened 11th January 2020.)

Thomas Poppelgaard

Citrix Products affected

• Citrix ADC and Citrix Gateway version 13.0 all supported builds 13.0.47.24
• Citrix ADC and Citrix Gateway version 12.1 all supported builds before 12.1.55.18
• Citrix ADC and Citrix Gateway version 12.0 all supported builds before 12.0.63.13
• Citrix ADC and Citrix Gateway version 11.1 all supported builds before 11.1.63.15
• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds before 10.5.70.12
• Citrix SD-WAN WANOP appliance models 4000-WO, 4100-WO, 5000-WO, and 5100-WO all supported software release builds before 10.2.6b and 11.0.3b

Customer with Citrix ADC /SD-WAN WANOP licenses.

You are entitle to implement a responder policy, which Citrix recommends, this apply to customers running ADC / AG 10.5 & 12.1 FIPS

Customers with ADC or Citrix Access Gateway 10.5 & 11.0 & 12.0 & 12.1 & 13 can upgrade to latest 10.5 build 70.12 11.1 build 63.15 & 12.0 build 63.13 & 12.1 build 55.18 & 13.0 build 47.24 as security patch is now GA and its not required to created responder policies for these build.

Important note for SD-WAN WAN Op customers:

To apply the security vulnerability fix, you need to upgrade all Citrix SD-WAN WANOP versions to build 10.2.6b or 11.0.3b as appropriate. These fixes are ONLY applicable to the SD-WAN 4000-WO, 5000-WO, 4100-WO, and 5100-WO platforms. All other SD-WAN PE and SD-WAN SE platforms are not impacted by this vulnerability and do not need to be patched.

The WANOP feature of SD-WAN Premium Edition is NOT impacted.

Customer with Citrix Access Gateway licenses.

Fix Timelines for when firmware GA

Citrix expects to have firmware updates in the form of refresh builds to be available across all supported versions of Citrix ADC, Citrix Gateway, before the end of January 2020. Please refer to the table below for the expected release dates.

VPX/MPX/SD – Customers with Citrix Access Gateway 10.5 & 11.0 & 12.0 & 12.1 & 13 can upgrade to latest 10.5 build 70.12 & 11.1 build 63.15 & 12.0 build 63.13 & 12.1 build 55.18 & 13.0 build 47.24 as security patch is now GA and its not required to created responder policies for these build.

Citrix SD-WAN WANOP fixed release now GA
To apply the security vulnerability fix, you need to upgrade all Citrix SD-WAN WANOP versions to build 10.2.6b or 11.0.3b as appropriate. These fixes are ONLY applicable to the SD-WAN 4000-WO, 5000-WO, 4100-WO, and 5100-WO platforms. All other SD-WAN PE and SD-WAN SE platforms are not impacted by this vulnerability and do not need to be patched.

NOTE: Customers who have upgraded to fixed builds do not need to retain the mitigation described in CTX267679.

Deep-dive insights about CVE-2019-19781

Reddit has a great blogpost where they describe the insights of cVE-2019-19781, all the exploits, tools and such. https://www.reddit.com/r/blueteamsec/comments/en4m7j/multiple_exploits_for_cve201919781_citrix/

Mitigation Steps for CVE-2019-19781

Source

Solution

The following configuration changes serve as a mitigation to the aforementioned vulnerability.

Standalone System

Run the following commands from the command line interface of the appliance or use putty, which I prefer always to create a responder action and policy: 

enable ns feature responder
add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""
add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403
bind responder global ctx267027 1 END -type REQ_OVERRIDE
save config 

Ensure that the changes apply to the management interfaces as well. From the command line interface, please run the following commands.

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0
shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"
reboot

HA Pair (On primary):

enable ns feature responder
add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""
add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403
bind responder global ctx267027 1 END -type REQ_OVERRIDE
save config 
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0
shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"
reboot

HA Pair (On secondary after primary comes up:)

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0
shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"
reboot

Cluster (On CLIP):

enable ns feature responder
add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""
add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403
bind responder global ctx267027 1 END -type REQ_OVERRIDE
save config 
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0
shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"
reboot

Cluster (Each cluster node):

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0
shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"
reboot

Cluster (Admin partition):

switch ns partition default
enable ns feature responder
add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""
add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403
bind responder global ctx267027 1 END -type REQ_OVERRIDE
save config
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0
shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"
reboot

Procedure to revert the changes (Standalone,CLIP, HA Primary)

unbind responder global ctx267027
rm responder policy ctx267027
rm responder action respondwith403
save config

Remove nsapi command from rc.netscaler. (Below command will search rc.netscaler file for the below pattern and remove the line that was originally added)

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=1
shell "sed -i '' '/skip_systemaccess_policyeval=0/d' /nsconfig/rc.netscaler"
reboot

The reboot, in each of the scenarios above, is not necessary to apply the policy but it is a precautionary step to ensure that if there are any open sessions,obtained via the vulnerability prior to policy application, are cleared.

Additional Information

Priority conflict

The priority given to the responder policy is 1. If there are any other responder policies bound with the same priority, the policy binding might fail. Customers are advised to adjust the priorities of other policies appropriately while making sure that the policy given here gets priority 1 

The ‘skip_systemaccess_policyeval’ Flag

This flag ensures that the responder policies are evaluated on the admin portal traffic.
If the admin portal IP is in a secured environment, this knob is not needed. 
Enabling this might cause some obstruction to some admin pages. In such a case, the customer can toggle the flag during their maintenance window and set it back to the value ‘1’.
 

Nodes that are removed from a cluster are vulnerable

When a cluster node is removed, its config is cleared. The above responder policies and hence the protection that comes with them are also cleared. Therefore, the node would lose the protections provided by these mitigation steps. 

Plugin download link from Admin UI

The current admin UI has a link to download the plugins (/vpns/scripts/vista/*.exe). This link has “vpns” in it and thus will not be accessible after this fix. 

/vpns/ in the backend url

If there is any backend webserver resource which has /vpns/ in its path, that resource will be blocked.

How do I check if my Citrix ADC/Access Gateway/SDWAN WANOP is vulnerable.

How do you check if your Netscaler is vulnerable and you didn’t apply responder policy CTX267679.

Tools you can use to scan if you are vulnerable use there:

CISA utility to check if customer is vulnerable

https://github.com/cisagov/check-cve-2019-19781
(requires Python 3.6 above)

CVE-2019-19781 – Verification Tool from Citrix

If you would like to scan from your own network, you can download this python script from Citrix
https://support.citrix.com/article/CTX269180 (requires Python 2.x or 3.x)

Community tools:

@zentura_cp took TrustedSec PoC code and build a Azure service which he made available for all Citrix customers/partners to use for free.

https://cve-2019-19781.azurewebsites.net

If you applied the responder policy according to CTX267679 after 10 January 2020, take ACTION

The chances of your environment being vulnerable is extremely high I would say 99.99%, so take no risk and check your environment ASAP.

Check if you are compromised:

(credits to Manuel Kolloff, he put together informations so its easy to validate if a hacker have been in your system’s, please take caution)

Template files

The exploits all write files to two different directories. Scan those via:

shell ls /netscaler/portal/templates/*.xml
shell ls /var/tmp/netscaler/portal/templates
shell ls /var/vpn/bookmark/*.xml

If you find files similar to the following you are likely to be compromised

#hint – /var/vpn/bookmark/*.xml, This folder can contain XML files (named <username>.XML) to support the personal bookmark functionality of the VPN virtual server. Validated if these are used but there is a chance is also because you have been compromised

Apache Log files

In addition, attempts to exploit the system leave traces in the Apache httpaccess log files. Those you can validate via:

shell cat /var/log/httpaccess.log | grep vpns | grep xml
shell cat /var/log/httpaccess.log | grep "/\.\./"
shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml
shell gzcat /var/log/httpaccess.log.*.gz | grep "/\.\./"

The following output is found on a system that was exploited:

However, a guarantee can never been given as attackers also might clean up their traces of the initial exploitation. A few more things to validate are…

Cron jobs

Attackers have been observed to obtain persistent access via scheduled tasks (“cron jobs” in Linux/BSD) to maintain their access even if the vulnerability gets patched. Check your crontab file for anomalies:

shell cat /etc/crontab
shell crontab -l -u nobody

The following is the output of a non-compromised system for you to compare:

Backdoor scripts

Running backdoors or other malicious tasks are often executed as Perl or Python scripts. Check for the presence of active running Perl or Python tasks:

shell ps -aux | grep python
shell ps -aux | grep perl

If you see more then the “grep” commands itself check the running scripts.

But beware, several Citrix ADC system-native tasks might appear as well. Some run scheduled so run the query again a few seconds later. Some are permanent (custom monitoring scripts for Storefront for example). Check those scripts to make sure they weren’t altered.

Crypto miners

Several attacks have been observed to install crypto miners. You can identifiy those by looking at the CPU intense processes by running:

shell top -n 10

Should you see any other processes but NSPPE-xx displaying high CPU usage you might have found a crypto miner:

Bash logs

In addition to the apache logs, some payloads run bash commands which can be traced in the bash log files. Check for those via:

shell cat /var/log/bash.log | grep nobody
shell gzcat /var/log/bash.*.gz | grep nobody

Attackers will land with the rights of apache (which is “nobody”). So if they execute commands via bash – which a few of the payloads I’ve seen did – or if they spawn a remote shell they are likely to end up being logged here:

But beware, these logs rotate rather quickly (1-2 days) because ADC spams them with quite a bunch of messages each minute with its own scheduled tasks!

Apache error logs

A bit more tricky is the analysis of the apache error logs. These log failed execution attempts (and I’ve seen a few misspelled commands or syntax errors). You might be able to use the following filter to catch some events:

shell "cat /var/log/httperror.log | grep -B2 -A5 Traceback"
shell "gzcat /var/log/httperror.log.*.gz | grep -B2 -A5 Traceback"

The following is a failed attempts at executing python because the path wasn’t referenced correctly:

But again, this might only be one type of error, ideally you may want to review them manually, especially if you’re in doubt:

shell cat /var/log/httperror.log
shell gzcat /var/log/httperror.log.*.gz

Firewall

In addition to Citrix ADC local indicators observe your surrounding firewalls for any irregular traffic. Most likely attackers will use the Citrix ADC as a jump host to penetrate the network further.

Attacks in the wild

Unfortunately, beside issuing warnings to all my customers well in advance, I have observed a few real world compromises. Luckily so far they all failed at some point – as far as I can tell.

I’ve observed the following behaviours in the wild so far:

• Retrieval of ns.conf
• Dropping of encoded Perl and CGI Backdoors
• Downloading of binary payloads which override the default httpd with a (likely) malicious version of apache
• Altering of crontab
• Retrieval of /etc/passwd
• Retrieval of the already present *.xml files
• Downloading of perl payloads and replacing existing Citrix ADC scripts
• Payloads that remove previously planted *.xml files and other artifacts
• Payloads that closed the door behind them by fixing newbm.pl after creating their own backdoor

Actually the SANS Institute published a very comprehensive list of payloads observed in the wild which covers most of the payloads I have seen as well, a good read if you want to dig deeper into what happens once a compromise took place: https://isc.sans.edu/diary/Citrix+ADC+Exploits%3A+Overview+of+Observed+Payloads/25704

Citrix forensic tool for CVE-2019-19781

This repository contains a utility for detecting compromises of Citrix ADC Appliances related to CVE-2019-19781. The utility, and its resources, encode indicators of compromise collected during FireEye Mandiant investigations.

In summary the utility will:

• do a best effort job at identifying existing compromise.

It will not:

• identify a compromise 100% of the time, or
• tell you if a device is vulnerable to exploitation.

Features of Citrix forensic tool

This scanner can identify:

• web server log entries indicating successful exploitation
• file system paths of known malware
• post-exploitation activity in shell history
• known malicious terms in NetScaler directories
• unexpected modification of NetScaler directories
• unexpected crontab entries
• unexpected processes
• ports used by known malware

Details of Citrix forensic tool

The Indicator of Compromise (IoC) Scanner for CVE-2019-19781 was jointly developed by FireEye Mandiant and Citrix based on knowledge gleaned from incident response engagements related to exploitation of CVE-2019-19781. The goal of the scanner is to analyze available log sources and system forensic artifacts to identify evidence of successful exploitation of CVE-2019-19781. There are limitations in what the tool will be able to accomplish, and therefore, executing the tool should not be considered a guarantee that a system is free of compromise. For example, log files on the system with evidence of compromise may have truncated/rolled, the system may have been rebooted, an attacker may have tampered with the system to remove evidence of compromise, and/or installed a rootkit that masks evidence of compromise, etc.

The output of the this tool will fall into one of three categories:

1. Evidence of compromise. This is the default. Any evidence that falls into this category indicates that a device was successfully compromised. This could be anything from executing commands that disclosure information (e.g. view the ns.conf or smb.conf configuration files), to installing a backdoor (e.g. NOTROBIN, a coin miner, etc.), or dropping a Perl-based web shell.
2. Evidence of successful vulnerability scanning (this could be authorized system administrator or unauthorized attacker). Any evidence that falls into this category indicates the system was in a vulnerable state (e.g. the mitigation had not been applied) and that at least the first step to exploit CVE-2019-19781 was successful.
3. Evidence of failed vulnerability scanning. Any evidence that falls into this category indicates that attempts to scan or exploit the system failed.

This tool is not guaranteed to find all evidence of compromise, or all evidence of compromise related to CVE-2019-19781. If indications of compromise are identified on systems, organizations should perform a forensic examination of the compromised system to determine the scope and extent of the incident. This tool is offered AS IS and without warranty.

Using Citrix forensic tool

You should download the standalone Bash script from the Releases tab of this repository. Copying the source directory to a Citrix ADC Appliance is possible but not recommended.

The IoC Scanner can be run directly on a Citrix ADC Appliance. In this mode, the tool will scan files, processes, and ports for known indicators. The tool writes diagnostic messages to the STDERR stream and results to the STDOUT stream. In typical usage, you should redirect STDOUT to a file for review. The tool must be run as root in live mode on a Citrix ADC Appliance.

For example:

$ sudo bash ./ioc-scanner-CVE-2019-19781-v1.0.sh > "/tmp/results-$(date).txt"

The tool is designed to be used with the following products:

• Citrix ADC and Citrix Gateway version 13.0
• Citrix ADC and Citrix Gateway version 12.1
• Citrix ADC and Citrix Gateway version 12.0
• Citrix ADC and Citrix Gateway version 11.1
• Citrix ADC and Citrix Gateway version 10.5
• Citrix SD-WAN WANOP software and appliance models 4000, 4100, 5000, and 5100

The IoC Scanner can also inspect a mounted forensic image. In this scenario, pass a command line argument specifying the path to the image root directory. You don’t have to be root to run in offline mode.

For example:

$ bash ./ioc-scanner-CVE-2019-19781-v1.0.sh /mnt/path/to/evidence/root/

In both modes, the tool will extract supporting code into a temporary directory; this directory will be deleted upon termination of the script. The tool does not make further changes to the system, although it may cause log entries to be generated.

Like all forensic analysis, prefer offline analysis against a dd image to live response. This will eliminate the likelihood that the tool causes relevant evidence to be overwritten.

Contributing to Citrix forensic tool

As you invent further ways to identify compromise, please consider contributing to this IoC Scanner. We would like to provide the most thorough, correct scanner as possible.

The primary goal is to report high confidence indicators of compromise. Because users may rely on the output of this tool to initiate further investigation, it’s important that we don’t send them on a wild goose chase. Therefore, activity such as simple scanning should not be reported in the default mode. Any evidence of an actor gaining access to the system, fetching information, or creating content should always be reported.

Design

Citrix provide this tool as a Bash script because it’s a common denominator across Citrix ADC Appliances. Here’s the feature matrix for Citrix ADC releases:

Although Citrix seen malware use Go to target FreeBSD/NetScaler, Go does not support FreeBSD 6.x.

Testing

Citrix maintain sparse file system images containing evidence of compromise in the ./tests/ directory. As you add IoCs to this tool, such as known paths or blacklisted content, please provide examples of the evidence for testing.

You can run the unit tests on a Linux or macOS system like so:

$ bash ./tests/test.sh
runnning test:  access-logs
runnning test:    xml-template
runnning test:  crontab
runnning test:    var-cron-tabs-nobody
runnning test:  error-logs
runnning test:  file-system
runnning test:    netscalerd
runnning test:    notrobin-tmp-init
runnning test:    notrobin-var-nstmp-nscache
runnning test:  ns-content
runnning test:    chr-encoded-template
runnning test:    copied-ns-conf
runnning test:    curl-in-template
runnning test:    perms
runnning test:    var-tmp-netscaler-portal-templates
runnning test:    var-vpn-bookmark
runnning test:    webshell-in-scripts
runnning test:  shell-history
runnning test:    bash_log
runnning test:    notice_log

Building

Once you’ve checked out the source repository, you can build a standalone script using the ./build.sh tool. This packages the primary script and supporting resources into a single bundle. Upon execution, it will extract to a temporary directory, execute from there, and then clean up.

To build:

$ bash ./build.sh > ioc-scanner-CVE-2019-19781-rev$(git rev-parse HEAD | cut -c 1-8).sh

Download Citrix forensic tool for CVE 2019 19781 here

Community Script

@_DanielWep created this community script that checks the Citrix ADC/AG/SDWAN WANOP if compromised by CVE-2019-19781 attacks and collects all file system informations.

The following files and logs will be checked:

• Template folders for XML files
• Apache Access logfiles
• Apache Error logfiles
• Cron Jobs
• Backdoor Scripts
• Crypto Miner
• Bash logfiles

If your Citrix ADC/SDWAN WAN OP/Access Gateway is compromised, take ACTION:

Note: If you have Citrix Access Gateway license you need to contact support to get a trial license for ADC so you can get responder policy configured. Customers with Citrix ADC/SDWAN WAN OP are entitled to create responder policies.

If you applied the responder policy according to CTX267679 after 10 January 2020 then I would recommend following steps

If compromised:

Below is 2 solutions. Some like to reimage a VPX from scratch while others like to use their backup solution, so I have provided both solutions here.

Solution 1: reimage

1. If compromised from above STEP do following:
2. Remove your Citrix ADC from network
3. Take a snapshot of potential vulnerable ADC VPX for forensic analysis and further investigation
4. Re-Image the Citrix ADC (both in HA) and re-import config
   its a slow process for MPX as you have to go through Citrix techsupport and might even need to get a new HDD with the clean image shipped
5. Investigate all servers the Citrix ADC has a connection to for further compromise.
   This might include the typical Citrix candidates like Storefront, etc. as well as DCs for LDAP but might also include a huge bunch of web- and application servers in a true ADC deployment
6. Implement Mitigation steps
   (only valid for mitigations to ADC/AG versions 12.1 FIPS)
   Note ADC/AG 10.5 build 70.12 & 11.1 build 11.1.63.15 & 12.0 build 12.0.63.13 & 12.1 build 55.18 & 13.0 build 47.24 as security patch is now GA+ SD-WAN WAN OP 10.2.6b and 11.0.3b is now GA, Customers who have upgraded to fixed builds do not need to retain the mitigation described in CTX267679.
7. Change all ADC local accounts passwords
   While stored hashed in the ns.conf those might be crackable
8. Change all ADC AD service accounts passwords
   While stored encrypted in the ns.conf those might be crackable
9. Change all AD user account passwords that were using the ADC to login
   All users? Yes, remember you are able to take wireshark traces on ADC including SSL session keys which can then be used to decrypt the whole communications – including user login credentials
10. Revoke and renew all SSL certificates
    Private keys might have been downloaded. While private key passwords are stored encrypted in the ns.conf those might be crackable
11. Upgrade to latest ADC/AG/SDWAN WAN OP build.
    Special note:
    if ADC/AG 10.5 upgrade to build 10.5.70.12, to install the security vulnerability fixes
    if ADC/AG 11.1 upgrade to build 11.1.63.15,  to install the security vulnerability fixes
    if ADC/AG 12.0 upgrade to build 12.0.63.13, to install the security vulnerability fixes
    if ADC/AG 12.1 upgrade to build 12.1 build 55.18 to install the security vulnerability fixes
    if ADC/AG 13.0 upgrade to build 13.0 build 47.24, to install the security vulnerability fixes
    if SDWAN WANOP before 10.2.6b upgrade to latest build, to install the security vulnerability fixes
    if SDWAN WANOP before 11.0.3b upgrade to latest build, to install the security vulnerability fixes
12. Enhance security and implement Citrix ADM from Citrix Cloud or On-prem.
13. Run diagnostic and upload to cis.citrix.com to check if have recommendations from citrix to enhance
14. Harden your Citrix ADC/AG/SDWAN WANOP
15. Enable ADC on internet
16. Check for vulnerability see above
17. Inform your country cyber security you have been compromised and take steps according to either GPDR or your data law in your country.

Solution 2: use snapshot from Dec 2019

1. If compromised from above STEP do following:
2. Remove your Citrix ADC from network
3. Take a snapshot of potential vulnerable ADC VPX for forensic analysis and further investigation
4. Restore backup of VPX from December 2019 (both in HA)
5. Investigate all servers the Citrix ADC has a connection to for further compromise.
   This might include the typical Citrix candidates like Storefront, etc. as well as DCs for LDAP but might also include a huge bunch of web- and application servers in a true ADC deployment
6. Implement Mitigation steps
   (only valid for mitigations to ADC/AG versions 12.1 FIPS)
   Note ADC/AG 10.5 build 70.12 &11.1 build 63.15 & 12.0 build 63.13 & 12.1 build 55.18 & 13.0 build 47.24 as security patch is now GA+ SD-WAN WAN OP 10.2.6b and 11.0.3b is now GA, Customers who have upgraded to fixed builds do not need to retain the mitigation described in CTX267679.
7. Change all ADC local accounts passwords
   While stored hashed in the ns.conf those might be crackable
8. Change all ADC AD service accounts passwords
   While stored encrypted in the ns.conf those might be crackable
9. Change all AD user account passwords that were using the ADC to login
   All users? Yes, remember you are able to take wireshark traces on ADC including SSL session keys which can then be used to decrypt the whole communications – including user login credentials
10. Revoke and renew all SSL certificates
    Private keys might have been downloaded. While private key passwords are stored encrypted in the ns.conf those might be crackable
11. Upgrade to latest ADC/AG/SDWAN WAN OP build.
    Special note:
    if ADC/AG10.5 upgrade to build 10.5.70.12, to install the security vulnerability fixes
    if ADC/AG11.1 upgrade to build 11.1.63.15,  to install the security vulnerability fixes
    if ADC/AG12.0 upgrade to build 12.0.63.13, to install the security vulnerability fixes
    if ADC/AG 12.1 upgrade to build 12.1 build 55.18 to install the security vulnerability fixes
    if ADC/AG 13.0 upgrade to build 13.0 build 47.24, to install the security vulnerability fixes
    if SDWAN WANOP before 10.2.6b upgrade to latest build, to install the security vulnerability fixes
    if SDWAN WANOP before 11.0.3b upgrade to latest build, to install the security vulnerability fixes
12. Enhance security and implement Citrix ADM from Citrix Cloud or On-prem.
13. Run diagnostic and upload to cis.citrix.com to check if have recommendations from citrix to enhance
14. Harden your Citrix ADC/AG/SDWAN WANOP
15. Enable ADC on internet
16. Check for vulnerability see above
17. Inform your country cyber security you have been compromised and take steps according to either GPDR or your data law in your country.

Community recommendations

• Arnaud Pain, Citrix CTP created a guideline how to fix your ADC
  http://arnaudpain.com/2020/01/17/citrix-cve-2019-19781-what-to-do/
  
• Patrick Coble, Citrix CTP created a detect Detecting Exploitation guide
  https://github.com/VDISEC/CVE-2019-19871-AuditGuide/blob/master/README.md

Source

Sign up security alerts:

https://login.citrix.com/?url=https://support.citrix.com/user/alerts

Kudos to citrix community

BIG Thank You to the Citrix community

@manuelkoff
@_DanielWep
@zentura_cp
@npreetz
@jensheerin
@VDIhacker
@mbp_netscaler
@virtuEs_IT
@antonvanpelt
@R_Kossen
@arnaud_pain