Report DIVD-2020-00001 - Citrix
Written on 13 Mar 2020 by Chris van 't Hof
Case lead:
Frank Breedijk
Case file: DIVD-2020-00001
On 17 December 2019 Citrix announced the presence of a critical vulnerability in all versions of their Application Delivery Controller (ADC) product, also known as NetScaler ADC, Citrix Gateway and Netscaler Gateway. These products are commonly used by organisations to allow employees to work remotely. An attacker could remotely exploit the vulnerability to execute arbitrary commands on these servers. Depending on circumstances, this access could be abused to gain access to data and applications of users and attack the internal network. Cases are known where this vulnerability has been exploited to infect organisations with ransomware.
The vulnerability was discovered by Mikhail Klyuchnikov, Web Application Penetration Tester at Positive Technologies and published in the MITRE list of Common Vulnerabilities and Exposures as CVE-2019-19781. Interestingly enough, Citrix disclosed this vulnerability, while they could not jet deliver a patch. The only thing victims could do is take some mitigation steps, but these proved to be unsuccessful. Still, a vulnerability does not necessarily mean an attack on the system will succeed. You need an exploit, which wasn’t available yet.
Positive Technologies scanned the global internet for vulnerable servers and published a report on 23 December stating that 80,000 companies in 158 countries are at risk. A world map was als provided showing the number of vulnerable servers for each country. DIVD Chairman Victor Gevers performed a global Nmap scan on 16 December and estimated the number of vulnerable servers to be 128,777. In order to monitor the expected decrease in vulnerable servers, he performed a fingerprint scan on 28 December. On 9 and 10 January, Matthijs Koot focussed on scanning Dutch-allocated IP ranges, validated the findings and made a list of some 600 vulnerable systems.
At that time, CERTs all over the world published warnings about the vulnerability. The National Institute of Standards and Technology set the impact score on 9.8 out of 10. It became clear that attackers were actively probing for vulnerable servers. Bad Packets, for example, reported “opportunistic mass scanning for vulnerable servers”.
On 11 January both Project Zero India and Trusted Sec published code on GitHub to demonstrate that exploiting the vulnerability is trivial. To make things worse: once an attacker used the exploit and generated an account on the vulnerable server, the acquired access sustained even after patching. The only thing left to do for the victims was to monitor and block attacks, or just take the server offline. That is, if one reads the news, performed a scan and knew already one was vulnerable. Governmental CERTs were publishing but are not allowed or even capable to scan and report as we do.
Observing these events unfold, our researchers were triggered to take action and not just scan the internet for vulnerabilities, but also warn potential victims. On 13 January Frank Breedijk announced a Security Hotline that would actively approach all Dutch IP addresses hosting a vulnerable Citrix server. We couldn’t save the whole world, but we could at least do our best in our own country, knowing the Dutch have more experience with helpful hackers. First, Frank wrote a script to automatically send an email to info@, abuse@ and security@, reporting they are vulnerable and what to do.
He also chopped up the list on AS numbers and forwarded the IP addresses to the owners of the network. (A network within the internet is known as an Autonomous System and broadcasts through the Border Gateway Protocol which IP address it contains.) KPN, the largest Dutch telco and owner of many IP addresses, took a large part to the IP list to forward warnings. Through the NBIP, Supporting Services for Internet Providers, smaller providers followed suit.
Various media caught up on the activities carried out by DIVD. Both Frank and Matthijs were quoted in national newspapers, magazine, news sites and radio programs. This triggered Z-CERT, the Computer Emergency Response Team for the Dutch healthcare sector to contact us on 14 January. Frank could immediately identify any hospital or other healthcare institution on the list and forward their IP addresses to Z-CERT so that they could inform the constituents.
On 19 January Citrix started releasing the first patches for version 11.1 and 12.0 of ADC and Gateway and on 24 January those for versions 12.1, 10.5 and 13.0 and their SD-WAN WANOP. Still, that did not immediately solve the problem, as the systems could already be compromised before the patch and remain vulnerable. Our Security Meldpunt, therefore, published some advice on our blog to either do forensic analysis or to just reinstall the system from scratch using a clean image.
In continuing our scans, we could see the numbers of vulnerable systems steadily decline. (see table) It is difficult to estimate whether this decline is due to our reporting, or that of others. By 5 February we identified that mitigations were still missing on 70 of the systems we initially identified as vulnerable. Assuming these systems would be certainly compromised, we sent them a warning, again. By the beginning of March, only five were left. Some of the new volunteers joining DIVD took the task of calling these organizations. Four were happy we informed them and took immediate action, while just one call ended with an unwilling receptionist…
Meanwhile, analyzing our scanning data more carefully, we encountered yet another problem with the Citrix servers: 450 of the 700 systems that we identified as vulnerable, turned out to use wildcard certificates. Theft of these certificates and their keys is not just a risk for the Citrix environment, but it allows an attacker to intercept TLS secured connections to any subdomain of the organization without the user being notified. This resulted in a second case for the Security Hotline of DIVD.