On the 9th of December 2021, a critical update because a vulnerability was published for the popular open source Log4j 2 library. The next day it was classified as CVE‑2021‑44228. Some people started calling it “log4shell”, as it allowed very easily to do a remote code execution attack. Days later several new vulnerability reports and patches would be published for this and related projects.
The implications are so huge that I had to break this article into several parts. In this first one, I’ll center on my personal investigation of the history of the library, its vulnerabilities, software vendor response, and who are its maintainers (support them!).
To learn more about Log4j and vulnerabilities, don’t forget to visit also my other posts:
In this post I explain in my own words how the vulnerability works, and presents details of a Kubernetes proof of concept created by me. On a third part (work in progress), I will analyze security measures, fixes and mitigations.
If you are worried about your services in production, stop reading here and update all libraries in your code, the third-party software you use, and ensure your cloud services has not been compromised.
What is Log4j 2?
Apache Log4j 2 is a very popular open source Java library used to manage logging messages.
It is included in several frameworks like Apache Druid, Apache Flink, Apache Solr, Apache Spark, Apache Struts 2, and Apache Tomcat (although that doesn’t mean directly that they are vulnerable, read more about this in part II).
The extent of use of this library was so huge that a whole snowball of updates and patches took over the Internet.
Same day vulnerability is disclosed, Minecraft demonstrates vulnerable by sending a message in the chat window
History
A 0-day refers to a vulnerability that was not publicly known before. It may be zero for you and me, but being wildly exploited in secret. To put things in context, let’s start this story at the beginning.
2009 (11 years earlier)
Dissatisfied by Log4j version 1 and other equivalent libraries, Ralph Goers starts working on Log4j version 2 for the first time among the Apache foundation. You can read details about this in his blog post “Why was Log4j 2 created?”. It will take years for the developers to consider releasing a stable version.
2013 (8 years earlier)
📅2013 September 14: Log4J 2.0-beta9 is released, the first version that years later will be reported as vulnerable.
2014 (7 years earlier)
📅2014 July 12: Log4j 2.0 is published, first version not alpha or beta, 5 years after the project started.
2015 (6 years earlier)
📅2015 October 21: A related CVE‑2015‑4902 gets published with the description
“Unspecified vulnerability in Oracle Java SE 6u101, 7u85, and 8u60 allows remote attackers to affect integrity via unknown vectors related to Deployment.”
Many “unspecified” and “unknowns” for a description. It could be argued that not specifying the description of the vulnerability shouldn’t be allowed for a complete CVE report.
Source: Alexandre Dulaunoy 2013 analysis of CVEs (GitHub repo)
The same day, you can read on ITPro website news (this is a summary):
Oracle has patched up a flaw in Java that allowed hackers to breach targets such as NATO and the White House in an operation known as Pawn Storm. The vulnerability was used in attacks on web assets belonging the military organisation as well as a number of prominent companies, according to Trend Micro threat analyst Jack Tang.
The flaw in question (CVE‑2015‑4902) managed to evade Java’s Click-to-Play protection, which requires the user to click the space where the Java app would normally be displayed before it is executed. In effect, it asks the user if they are really sure they want to run any Java code.
Bypassing click-to-play protection allows for malicious Java code to run without any alert windows being shown. To mount an attack, a hacker adds the HTML code to a malicious web site and then creates a RMI registry server which has a public IP address as well as creating another web server to hold the malicious Java code, which also has a public IP address.
2016 (5 years earlier)
📅2016 August 3: At BlackHat USA 2016, a presentation by Alvaro Muñoz and Oleksandr Mirosh from HP Enterprise Fortinet, shows how a combination of JNDI injection and LDAP entry poisoning could be used to make remote code execution possible, as was described in CVE‑2015‑4902 the previous year. These are the same kind of steps involved in exploiting the later discovered Log4j 2 vulnerability.
2017 (4 years earlier)
📅2017 January 17: Oracle updates Java to version 8u121. Adds codebase restrictions to RMI when using JNDI for remote object lookup, where classFactoryLocation field is not used for deserialization any more. But LDAP protocol still executes it, and javaFactory also is still usable to use for RCE on both RMI and LDAP, making both vulnerable to this day.
📅2017 March 7: A new vulnerability is discovered, unrelated to Log4j, but on the also very popular open source framework, Apache Struts. A situation where many companies use it and have to patch it develops, similar to what has happened with Log4j 2.
📅2017 March 10: Reports of hacking groups start looking for systems using Struts that are not updated with a fix.
📅2017 May 12: Equifax is hit very hard by a data breach, the biggest known to that date because of a vulnerable Apache Struts. Some companies learn they can’t leave security and updating as a second thought. Others… don’t learn it.
2018 (3 years earlier)
📅2018 October 16: Oracle updates Java to version 8u191. When using JNDI for remote object lookup, the same classFactoryLocation restriction that was added to RMI two years ago is also added to LDAP. But javaFactory still usable on both RMI and LDAP for remote code execution to this day, being a key element for the future Log4j 2 vulnerability to succeed.
2021
📅2021 October 10: Jfrog reserves ID CVE‑2021‑42392 that might be for a specific issue or a block or issues. It will not be until January 6th it will be used for a vulnerability they have discovered on the H2 database (more on this later).
📅2021 November 24: Chen Zhaojun of Alibaba privately reports the Log4j 2 vulnerability to the Apache team.
📅2021 November 30: A pull request by Ralph Goers (original Log4j 2 creator) is opened in the Log4j GitHub repository titled “Restrict LDAP access via JNDI” to fix the issue.
📅2021 December 1st: Log4j 2 exploit evidence as early as this day will later be found and tweeted about by Matthew Prince, CEO of CloudFlare. But it won’t be mass exploited until after the public disclosure.
📅2021 December 5: The pull request is merged five days after Ralph Goers created it. Take into consideration that he states in his bio that “I work on Log4j and other open source projects in my spare time”.
📅2021 December 6: A new version of Log4j 2 is released, 2.15.0, which includes many improvements. This update fixes the remote code execution, but 7 days later will be proven vulnerable to a possible denial of service attack. Among the changes, there is a mention to:
In version 2.12.2 Log4j 2 disables access to JNDI by default. Usage of JNDI in configuration now need to be enabled explicitly. Calls to the JndiLookup will now return a constant string. Also, Log4j 2 now limits the protocols by default to only java. The message lookups feature has been completely removed.
📅2021 December 8: Alibaba’s Chen reports to the Log4j 2 team again that someone had just revealed the details of the vulnerability on a Chinese blogging platform.
My opinion is that maybe the author was somebody that saw the pull request. With all the caution Chen demonstrated I doubt it had something to do with him or people near him.
📅2021 December 9: User tangxiaofeng7 publishes publicly in GitHub a proof of concept for exploiting the vulnerability (I saw this repository as public, but it has later been deleted or made private; some content is available on Archive.org). The vulnerability is being tweeted about (the tweet has been deleted but a copy can be seen in Archive.org). At 3:25PM the tweet has 89 retweets, 34 quotes, and 190 likes.
As related information is only in Chinese, my lack of knowledge of that language, and limitations using Google Translate prevents me to investigate further.
📅2021 December 10: The vulnerability in Log4j 2 for remote code execution is published as CVE‑2021‑44228, with a score of 10.0 (the maximum value).
The same day reports and news start appearing continuously demonstrating very popular affected software. GrayNoise network security provider tweets about the number of hosts exploiting the vulnerability increasing from 100 to 150 in several hours. Fastly reports that in 24 hours of the report, the occurrences when scanning for the “jndi:” string that is employed in the exploit has spiked and that attackers are invested in researching different ways of exploiting the vulnerability, which is not true for all 0-day vulnerabilities that become public.
Spike on queries using “jndi:” to compromise systems 24 hours after exploit published.
Source: Fastly
I’m certain this will create a big campaign for hacker groups, not only trying to exploit this vulnerability, but seeking similar 0-days “gems” that may have been also missing in similar circumstances. First on Log4j (but I assume this library is going to get a lot of scrutiny), then for other popular similar libraries.
📅2021 December 13: A new version released of Log4j 2, v2.16.0 is released, which fixes some special cases that could result in a denial of service attack. It disables JNDI by default (fixing LOG4J2-3208) and completely removes support for Message Lookups (fixing LOG4J2-3211). As explained on their website:
In version 2.16.0 Log4j disables access to JNDI by default. JNDI lookups in configuration now need to be enabled explicitly. Also, Log4j now limits the protocols by default to only java, ldap, and ldaps and limits the ldap protocols to only accessing Java primitive objects. Hosts other than the local host need to be explicitly allowed. The message lookups feature has been completely removed.
📅2021 December 14: The related vulnerability CVE‑2021‑45046 is reported that is fixed in Log4j 2, v2.16.0. It is reported with a score value of just 3.2 (low), 7 days later will be increased to 9.0 (critical).
The same day Log4j 1.2 is reported on CVE‑2021‑4104 with score 8.1 (high), as vulnerable to deserialization of untrusted data when the attacker has write access to the Log4j configuration. This project is at end of life, so no fix will be provided to this or other vulnerabilities that may arise.
📅2021 December 16: A new vulnerability is also reported affecting Logback 1.2.7 and previous versions, a project that forked from the original Log4j 1. Described at CVE‑2021‑42550 with soce 6.6 (medium), using non-standard configuration files an untrusted deserialization of data may occur.
📅2021 December 18: New release of Log4j 2.17.0 is published that “protect from infinite recursion in lookup evaluation”, the same day it’s classified as CVE‑2021‑45105.
📅2021 December 22: Chinese Ministry of Information Technology (MIIT) said it will suspend collaboration for with Alibaba Cloud for not informing the Chinese government first of the Log4j vulnerability, to be reassessed in six months. The next day Alibaba Cloud would provide an official statement towards improving its risk management and compliance, arguing that it did not fully comprehend the severity of the flaw at the time.
This sets a precedent for Chinese tech companies to always inform first the Chinese government of any security flaw discovered in open source, or be punished. It gives the government of that country a heads up to fix first their own systems, exploit or collaborate as they like on vulnerabilities like this one.
📅2021 December 28: Another release for Log4j 2, version 2.17.1 with related CVE‑2021‑44832 with score 6.6, for a possible remote code execution (RCE) what only can happen if adversaries can modify the library’s configuration. It can be argued that this is just a misconfiguration that could be handled by benchmarks, not a vulnerability. The eagerness to find new vulnerabilities in the library is showing.
2022 (next year)
📅2022 January 6: Jfrog publishes information on a vulnerability for the H2 database with CVE‑2021‑42392, where an attacker may pass a JNDI driver name and URL leading to an LDAP or RMI server, causing remote code execution. The corresponding NIST NVD database entry with extended information, score, and vector is not available 2 days later.
Who has been affected?
Not only the software you are developing may use one of those frameworks or the Log4j 2 library directly, but also many other software products you use may do so.
As the main company behind Java, many Oracle products are affected (you need to log into your Oracle account to see the full list).
5 days after publishing the first vulnerability, some of the companies/products reported as affected are GitHub Enterprise Server, cPanel, SolarWinds SAM and DPA, Splunk, Tableau, several Cisco products, ArcGIS Enterprise, F-Secure products, GoAnywhere, GrayLog, Jitsi, Liferay, New Relic Java agent, Okta Radius and MFA agents, Siemens, some Sophos services, Sumo Logic, Suse OpenStack Cloud, TP-Link Omanda Controller, some Trend Micro services, Veritas Netbackup, many VMWare products. Some specific versions of Elasticsearch and Logstash are affected. Some on-premise products of Atlassian are partially affected (they use their own fork of Log4j). Minecraft Server Java edition is affected, so now is when you know that it is real.
Other products like GitLab or Anaconda don’t use Log4j, and Apache Kafka uses Log4j 1, so they claim they are not affected by this (not true, more on this on part II). Jenkins isn’t affected, but plugins installed in it may be, like Jenkins Xray Connector.
Saas software providers like Google Cloud, AWS and MongoDB Atlas Search also are known to have been using Log4j, and has been their responsibility to update, patch, or mitigate the problem.
Azure announces that it’s not using or directly providing Log4j, but Kafka Connect for Azure Cosmos DB provided by Microsoft is affected. IBM and Salesforce seem to be still investigating the impact as December 14th, but at least IBM Sterling Order Management is affected.
Users claim that Steam was vulnerable in its chat service, but I find no evidence, and no official announcement has been made. Netflix seems to have also patched the library. Apple and iCloud have been affected and changing an iPhone name has been reported as triggering the vulnerability.
Insecure versions of Log4j 2 were available on Debian and Ubuntu, who have released updates.
Red Hat Enterprise Linux ships with Elasticsearch that uses a compromised version, but because of the use of the Java Security Manager, access to it is limited so the problem is moderate at most.
Changing an iPhone name to trigger the vulnerability on Apple’s servers.
Do not use dnslog.cn por your tests! You are reporting to a third party website in China that you are vulnerable.
Kudos to SwitHak for compiling an incredible list of links announcements from software vendors, from which most of this information has been compiled.
Another great source of information you can use to check software and cloud provider services is this repository from the Nationaal Cyber Security Centrum, that for each item lists if it is known if has been affected or not, and if a fix is available.
CVE vulnerability reports affecting Log4j 2
CVE-2021-44228
- NVD database: https://nvd.nist.gov/vuln/detail/CVE-2021-44228
- Date: 📅2021-12-10
- CVSS severity v3.1: 10.0 Critical (maximum value)
- Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H
- Summary: Log4j 2.0-beta9 through 2.12.1 and 2.13.0 through 2.15.0 JNDI features do not protect against attacker-controlled LDAP and other JNDI-related endpoints that result in an easy to execute remote code execution.
CVE-2021-45046
- NVD database: https://nvd.nist.gov/vuln/detail/CVE-2021-45046
- Date: 📅2021-12-14
- CVSS severity v3.1: 9.0 Critical (was previously reported as 3.2)
- Vector: CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H
- Summary: The previous fix in Log4j 2.15.0 was incomplete and certain JNDI patterns could still be used for remote code execution.
CVE-2021-45105
- NVD database: https://nvd.nist.gov/vuln/detail/CVE-2021-45105
- Date: 📅2021-12-18
- CVSS severity v3.1: 5.9 Medium
- Vector: CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:N/A:H
- Summary: The previous fix in Log4j versions 2.0-alpha1 through 2.16.0 (excluding 2.12.3 and 2.3.1) did not protect from uncontrolled recursion that can be exploited for a denial of service attack.
CVE-2021-44832
- NVD database: https://nvd.nist.gov/vuln/detail/CVE-2021-44832
- Date: 📅12/28/2021
- CVSS severity v3.1: 6.6 Medium
- Vector: CVSS:3.1/AV:N/AC:H/PR:H/UI:N/S:U/C:H/I:H/A:H
- Summary: Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to remote code execution when a non-default configuration is used.
Learn more about CVEs in my blog post “What is a CVE?”
Originally for the first CVE, the list of affected software was indicating just Log4j2 from 2.0-beta9 to 2.14.1. It has been updated including many more software packages, and the list of reported affected software will keep growing.
I bet it will become be the vulnerability with most software-affected references in the whole database. Also as the library is scrutinized, more previously unknown new vulnerabilities may be reported continuously.
As the list of affected software in the vulnerability database keeps growing a vulnerability scan done today may not tell you all places where you are running software with a vulnerable version of Log4j 2, so you should keep monitoring continuously your security posture to make sure you are not vulnerable.
CVE vulnerability reports affecting related projects
CVE-2021-42392
- MITRE database: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-42392
- Date: 📅10/14/2021
- NVD database (awaiting analysis): https://nvd.nist.gov/vuln/detail/CVE-2021-42392
- Global Security Database (GSD) entry from Cloud Security Alliance, published on 8th of January.
- CVSS severity v3.1: 10 (from GSD)
- Vector: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H/E:H/RL:O/RC:C (from GSD)
- Summary: On H2 database versions prior to version 2.0.206, an attacker may pass as a database parameter a JNDI driver name with a URL leading to an LDAP or RMI servers, causing remote code execution, for example through the H2 Console.
CVE-2021-4104
- NVD database: https://nvd.nist.gov/vuln/detail/CVE-2021-4104
- Date: 📅12/14/2021
- CVSS severity v3.1: 8.6 High
- Vector: CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
- Summary: Log4j 1.2 when configured to use JMSAppender (non-default setup) is vulnerable to deserialization of untrusted data. No fix will be provided because the product is at end of life (migrate to latest Log4j 2).
CVE-2021-42550
- NVD database: https://nvd.nist.gov/vuln/detail/CVE-2021-42550
- Date: 📅12/16/2021
- CVSS severity v3.1: 6.6 Medium
- Vector: CVSS:3.1/AV:N/AC:H/PR:H/UI:N/S:U/C:H/I:H/A:H
- Summary: Logback 1.2.7 and prior versions, an attacker with the required privileges to edit configurations files could craft a malicious configuration allowing to execute arbitrary code loaded from LDAP servers.
Log4j 2 maintainers
Source: XKCD comic 2347
Log4j 2 was created by Ralph Goers. The most active people committing to the GitHub repository now are him, Gary Gregory, Remko Popma, and Matt Sicker.
Just before the exploit was published, Ralph had only 7 sponsors on GitHub, on December 15 he had 78 and going up, but still, none of them are companies. The rest of the developers with sponsorship enabled had that day a combined amount of 3 sponsors.
In Ralph’s words that you can read on his GitHub bio:
I currently have a full time job as a Software Architect. I work on Log4j and other open source projects in my spare time and so I typically work on those issues that are of most interest to me. I have always dreamed of working on open source full time and would love your support to enable that to happen.
As you can read on the Apache Log4j website:
The Apache Software Foundation does not employ individuals to develop and support any of its projects. The individuals who contribute to Apache projects do it either as part of specific tasks assigned to them by their employer, on their own initiative to benefit their employer or on their own free time. While some projects at the ASF have employees who are specifically paid to work on the project, none of the committers to any of the Logging Services projects are directly paid to work on them.
I think they have done an excellent job reviewing, patching, and publishing fast updates to the vulnerabilities.
You can sponsor maintainers of Log4j with a one-time donation, or on a recurring monthly basis on their GitHub profile. If you can get your company to sponsor them, you will contribute to keep open source safer for you, your company, and everybody else. The maintainers that accept sponsorship are:
Thanks for the help
Thank you to all referenced sources for their great job sharing information very early since the beginning of the situation.
Thanks to @Quemandoacromo that after reading the article, sent me a link to Archive.org for the removed repository, and another link for the Nationaal Cyber Security Centrum list of affected software.
Conclusion
The Log4j 2 vulnerabilities discovered on the 10th of December 2021 have existed since many years ago and have caused a huge snowball of insecurities affecting your project, other open and closed source software, as well as cloud services.
The maintainers of that library work on it in their spare time without official support from any company but have been able to act very fast to publish fixes for the problems.
Despite this, the situation right now is very uncertain, and many malicious campaigns will surely go rampant.
To learn more about the technical details and a proof of concept using Kubernetes, jump to article part II: Kubernetes POC.
If there is some information I missed in this article, or you just want to chat with me, let me know.
And if you found this information extremely useful to you, why not let others know? Share it in a tweet, or invite me to coffee!
Vicente Herrera is a software engineer working on cloud native cybersecurity (cloud, containers, Kubernetes).
His latest job is cybersecurity specialist at Control Plane, and previously was Product Manager at Sysdig.
He is the published author of Building Intelligent Cloud Application (O’Reilly, 2019) about using serverless and machine learning services with Azure.
He has been involved in several research projects for using AI on healthcare and automatic code generation.
