> It’s an example of a utility box that runs various random services - maybe acts as a bastion host or testing ground - and nobody quite manages it or knows what it is used for. This server is as weak as its weakest service; and because it is not purpose-managed, it can be difficult to keep track of what is running on it and ensure all services are patched and secured. If you have one of these servers floating around somewhere, you might want to think twice about keeping it - it may very well be the chink in your armor.
True for applications AND servers.
Wasn't entirely sure how they knew where to look for the Passport credentials though, unless they'd gleaned that information earlier. From the article the author makes it sound like they had a limited window to mount the disk, pull off what they wanted and shut everything down, did they already know where to look?
Also, this Elasticsearch RCE has been patched a while ago and we still see a lot of servers hacked because of it. In fact, there is a DDoS botnet made of only ES servers that we have been tracking.
<unrelated>If you are using Elasticsearch, please patch it!</unrelated>
Always read the list of in-scope systems and rules of engagement before starting. If that information hasn't been provided, then don't start until it has.
1. Always run services (e.g. ElasticSearch) with a unique user dedicated to that service and nothing else.
2. You're never as secure as you think.
Quick tip; If you do this, you can also make your iptables rules be per user. For example, "webserverUser" can only accept inbound connections on 80/443 and only have outbound connections that are related to established inbound ones. If an attacker gains execution as this user, they cannot download new code, etc or even do DNS lookups for that matter.
Instead of having 'nc -l 8080 | bash' or whatever as your payload, an attacker can just run code instead. "pwd > /var/www/html/exfiltration.html". If they absolutely need a shell, they could e.g. alter nginx or its config files to run `bash` on POSTs to a hidden route.
This does make it a little trickier, and potentially a little easier to detect. But it certainly doesn't make it so that "they cannot download new code".
Correct, I should have said, it eliminates many easy ways to download code. Defense is depth is all about making the attackers job harder and increasing their likelihood of being detected.
Good thing I don't do dev ops except for my personal projects.
It's simply reality. I don't enjoy it. Fortunately I've never done that in any type of production environment (just development). But I don't entirely blame people. Many of the infrastructure pieces that need to be deployed can get very complex very quickly.
I used to be in the same boat as you until I had my lightbulb moment: if you're constantly coming across permission errors like this, it's a sign that you're just doing things wrong.
I don't mean this as an insult; it's more that often when you download, install and provision a new piece of kit, it's easy to do so without taking a few minutes to read the docs and find out the best way of doing things. Especially if you're glancing over e.g. Ubuntu instructions when installing on CentOS.
(in fact, this rule applies to a lot of IT. Spending hours trying to get a CI tool to do X, Y and Z? Chances are it's just not meant to do it that way. God knows how much time I've lost to this.)
> I don't mean this as an insult; it's more that often when you download, install and provision a new piece of kit, it's easy to do so without taking a few minutes to read the docs and find out the best way of doing things. Especially if you're glancing over e.g. Ubuntu instructions when installing on CentOS.
As a sysadmin, I find software documentation to often be the worst place for deployment advice. It's often written by developers with the same mindset as the GP - just get it working, best-practices can come later (which they never do).
One app per user (and the corollary, proper segmentation between system users) is something you just have to train yourself to do, screw what the developer had in mind with his deployment instructions.
Though to be fair I've also run into weird bugs with OpenShift, Mongo and a few others over the years so it's not necessarily something wrong that the developer is doing.
Moral of the story is not to trust your provider. Don't use cloud if you don't need it, don't let your host have your ipmi crdentials, run FDE (this has saved several bitcoin exchanges from losing millions).
. Here's one particularly funny thread from their support forum that perfectly captures exactly how much of a mess this setup was (is? I haven't hacked linode lately): https://forum.linode.com/viewtopic.php?t=3231%3E
Usually a bad sign by itself lol...
Nice read anyway, some important lessons there.
Here is Inversoft's account of the events leading up to the hack and the lessons learned.
HackedThat: Mind the backdoor https://news.ycombinator.com/item?id=12390936
What I do find to be a PITA is when you attempt to create a password that does not meet the minimum rules and the error message gives you no indication of what you need to change to meet the requirement.
But don't block a 6 word diceware phrase because it has not numbers (or because it is to long.... looking at you PAYPAL).
Meanwhile, I'd venture that P@ssword1 meets their requirements...
Next, they really should be using ed25519 keys if possible. Better, faster and all that good stuff.
I can almost promise you that if you could reach the external ES instance and exploit it - it was already exploited. They need to wipe that box and rebuild.
Overall, great writeup - thanks for sharing!
$$('img').forEach((i) => i.hidden = true)
(Also possibly of interest is that "pungent" derives from the same root, for much the same reason. In this case, the pricking sensation is that caused by the irritating sulfoxides released when you cut up an onion.)
Was the word you were after perhaps 'pertinent'?
I a few GIFs can definitely split up the article a bit and make it easier to read, but too many can also be distracting.