> The attacker gained access to the server — which had been active for about a month — by exploiting an insecure remote management system left by the datacenter provider, which NordVPN said it was unaware that such a system existed.
This screams for clarification and I'd love for someone more knowledgeable in the area to elaborate on it. Is this common practice for data-center providers? Do I now not only have to worry about my own infrastructure security but also worry that my IaaS provider hasn't installed some backdoor to my servers?
I work for a web hosting company in the US and at least in our case, it's quite common for remote management to be enabled on pretty much all of our dedicated hardware. However, because of the inherent dangers in opening this up to the public internet, unless explicitly requested by the customer (or Managed Colocation), the NIC used for Dell iDRAC or HP iLO is on an isolated network unique to the physical datacenter. Remote access for our techs is managed through a secured bridge that requires all sorts of security hoops on our company intranet, and remote access for general internet traffic is not available due to the firewall restrictions. While it's plausible for remote access to be gained this way, it is extremely unlikely and would require several exploits at different points along the path.
I cannot speak for the industry as a whole, but remote management systems like this are bound to be common; any large enough physical datacenter is going to need a more efficient way to access a misbehaving system than sending a tech physically running to the box to plug in a keyboard and mouse. It should be extremely uncommon to have these management interfaces open to the public though, and I'll bet that's what NordVPN is surprised by. Generally these systems should be private and isolated due to the power that an attacker can wield through them.
IPMI does not have to be open to the internet to be open to a wide audience. Many of these out of band management interfaces are hosted on an internal network, but not isolated by customer.
Cheap datacenters are favored by VPN providers for their unlimited bandwidth and lax abuse policies.
Many of them allow access to IPMI only over a VPN, but do not isolate each customer’s IPMI to a customer VLAN. I personally know at least three large budget datacenters which allow all customers access to each others’ “private” IPMI IP addresses.
Generally speaking, there are four (4) tiers of "public" data centers are on the market, ranging from essentially a big room with some alright AC and a line out, to huge, highly secure (cameras, fingerprint readers, SSAE certifications, etc.) buildings with redundant power and HVAC systems.
The higher end ones are usually newer-ish, but there are lot of older "computer rooms" that offer acceptable-level benefits for a reasonable price. Lots of legacy customers and ISPs in these rooms, for the record. You get what you pay for but a lot of older DC spaces are just fine for most users; everyone thinks they need 99.999 but most don't.
There are also re-sellers and managed services companies who take out a footprint in data centers and then lease space in their cabs, sell bandwidth, IPs, etc. Using a series of resellers you can often get around restrictions as to what you're doing -- small fry MSPs don't ask a lot of questions -- but still get a data center footprint. These are sometimes one-man shows, and their quality and professionalism are often sub-par, which is how you end up with default iDRAC creds and the like.
Check out Data Center Maps or WebHostingTalk for some examples.
Source: data center ops manager for several different companies.
Maybe it wasn't open to the public Internet, but the VPN exit is inside the datacenter and connects out to the public Internet. Is it feasible that NordVPN provided their customers with a secure tunnel into their own datacenter's management software?
User root, password calvin. That's the default. And, if I had a dime for every time I've seen one of these in a data center, I'd be a rich man. I have literally begged sys admins to change the default password, but they say, "Why... we're behind a firewall using RFC 1918 addresses. No one can get to these." The rest, as they say, is history.
This is the dumbest thing I've ever seen... unless your firewall is between your host versus every other host and there's no multi-tenancy, this will suck.
In well maintained networks the management interface (IDRAC, etc.) for each server is placed on a separate VLAN which the servers cannot access. This isn't to say that cheap providers actually do this, or that the VLAN can't be accessed by a compromised technician's workstation/laptop.
Yes, this kind of firewall is always supposed to be between the management hosts and everything else. Only the sysadmins at the data center a very limited set of applications is supposed to be able to access it. The very real risk is misconfiguration.
Depends on what kind. In case of idrac, yes; but it's weird that it was insecure by default in the first place. Usually credentials are configured and provided to the customer. Makes me think there might have been some other interface. Clarification is definitely needed.
There were many IPMI/iDRAC/etc. exploits published in the past few years. Throw a dart at a list of them, and you'll probably find one that was unpatched in most systems as of March 2018.
If you can reboot it without anyone noticing? Really, really easily: iDRAC gives you access to the local console, like a remote KVM. Reboot into single user mode, change a password, done.
Oh, IPMI and friends are a total mess. Some implementations allow one to take control of a running server remotely especially if they use a shared ethernet for management ( popular in supermicros ). I once had our security geek demonstrate it by taking over the running server, rebooting it using network emulated USB stick, adding a file into /etc and rebooting the server again.
In secure environments one pulls IPMI module from the server or only uses the modules that have their own dedicated NICs that have to be wired to their own management network.
The first time I booted a server using a virtual CD-ROM (iso on my laptop shows up as a hardware CD-ROM on the server) over IPMI I was simultaneously relieved (because I could fix the machine remotely) and absolutely totally horrified.
iDRAC is a full onboard whitehat rootkit manufactured and supported by Dell. It runs independently of any OS and has control over the system. It is intended to be a substitute for physical access.
In certain configurations, iDRAC gives you an rdp connection. If idrac is left at default, windows admin login not being changed isn't too much of a stretch.
It doesn't make much sense to me, even with iDRAC/some other console access you don't really have access to OS unless you reboot & go to single user mode etc at which point they should be noticing their servers rebooting etc. would love more info
Just set up your code as a boot-once config and wait for the owner to reboot their machine. Make your code end by booting the installed OS (or even by just rebooting again, most people will just curse about the damn slow server boot process).
You can't do that as you don't have any access until it's being rebooted. It's basically like you're standing in front of the machine so there's not really much you can do when you're just looking at a login prompt, you have to be able to stop grub from just booting with the default options and instead boot up using init=/bin/bash or maybe if the server supports iPXE you can just chain load some payload off the internet.
You can manipulate boot settings using BMC commands. No need to mess with Grub or the running system. Instead, tell the system to boot up from an emulated USB drive (image can be attached from some remote server, often including your web browser).
Now wait for the machine to get rebooted (or do it yourself using the BMC, e.g. 'racadm serveraction powercycle' for Dell/iDRAC machines).
Even SecureBoot won't help as you can just turn it off using the BMC.
See here for a bunch of examples for Dell machines using the BMC's HTTP API:
When someone gets notified by their monitoring system that a server was unavailable (because it rebooted) they might investigate and see that the IPMI logs don't mention power loss
Power failure would require both of the power feeds in the DC failing simultaneously and would be easily verified by contacting the DC and asking if they had any power outages reported at the time. Of course there are cheapskates who don't go for redundant power supplies so it's possible but would be indicated in the IPMI logs
Servers can reboot for any reason. There are tons of kernel issues, especially since Meltdown & Spectre, that cause machine reboots in Linux especially on high traffic machines.
I've worked in production environments with thousands of machines and random reboots are a completely normal event for some workloads. Combination of hardware issues & kernel issues with hundreds of thousands of lines of code makes it inevitable. I would be surprised if NordVPN even noticed and their architecture wasn't designed to automatically start everything at boot.
You can't be perfect at scale - you just need to design your work loads to be redundant and fault tolerant.
It opens an exploit chain, in a normal circumstance you are correct. In a malicious circumstance, it is always feasible irrespective of the likelihood.
""All servers we provide have the iLO or iDRAC remote access tool, and as a matter of fact this remote access tool has security problems from time to time, as almost all software in the world. We patched this tool as new firmware was released from HP or Dell.
"We have many clients, and some large VPN service providers among them, who take care of their security very strongly. They pay more attention to this than NordVPN, and ask us to put iLO or iDRAC remote-access tool inside private networks or shut down access to this tool until they need it. We bring [iLO or iDRAC] ports up when we get requests from clients, and shut them down when they are done using this tools. NordVPN seems it did not pay more attention to security by themselves, and somehow try to put this on our shoulders.""
Of course you have to worry about all of that! When you don't self host you have to assume whoever you are renting from hirers the lowest paid employees they can get to manage infrastructure for you. That's how they get profitable. You are not outsourcing expertise.
Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.
Even a lot of VM/cloud systems have some kind of virtual management console (Linode has their LISH system that lets you SSH in to console and Vultr/Digital Ocean have similar web based consoles .. AWS surprisingly doesn't. You can get console output but can't send VMs any console input).
Not only should have NordVPN been aware of this hardware KVM, they should have secured it and had version checks on its firmware as an essential part of their security. I could see this oversight with other companies, but not with one whose primary business claims to be security.
> Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.
Power on/off should be done via APIs that issue commands to a PDU, like Atlantic.net started doing in the early 200s.
And there's nearly zero reason to access "console" - configure your server to always but off PXE and fall through to disk if that intercept is not needed.
Yes, software that runs on the instance can learn instance metadata. No, that is not a problem. Running e.g. user-supplied scripts on the instance would be "pretty damn dumb", but no one is that dumb. Any widely distributed software that did something shady with instance metadata would get busted PDQ. Just like any widely distributed software that did something shady with e.g. root credentials, which is about the same threat scenario.
It's crappy design which bypasses important security mechanisms of the OS (lower privileged users) by allowing every application with network access to access such critical functionality. One sane approach would be passing this information to the OS through the hypervisor which then exposes it as a properly ACLed file system.
This is like an author of a website vulnerable to CSRF (because it relies on IP for auth) blaming browsers for allowing cross site requests instead of require proper authentication. Except that Amazon is powerful enough to get away with pushing all the effort onto developers and admins.
You can use iptables to limit metadata access to certain users but that takes effort so no-one does it.
I guess a machine-local service that takes ownership of the metadata service and implements additional restrictions (such as limiting access keys to privileged users) might be doable.
Yes, in some situations it will take effort to not allow network access to untrusted software and/or users. For those situations, EC2 is not a good fit.
How is it “obviously” the right policy? If implemented, console input would presumably be part of the AWS API, guarded with IAM permissions like everything else. If you have full IAM permissions, you can already take over any instance by temporarily attaching its disk to a different instance and modifying the data from there. (That requires rebooting, but so would takeover via console input.) Indeed, I’ve had to do exactly that on multiple occasions to fix broken config files on my personal instance; it would have been much more convenient to have console input.
FreeBSD has an interactive kernel debugger that you can use with a serial console. Super useful to track down some things, not usable on AWS -- you'd need to (somehow) do a core dump and hope you can figure it out from there.
If they're going with a bottom-dollar host, it's possible that the out-of-band server management tools were exposed. It's less likely to be a software backdoor, and more likely to be Supermicro IPMI or other baseboard management controller.
I know that public cloud providers like Rackspace and Azure insert their own accounts and services into cloud servers and VMs mostly under the guise of being able to support said servers and monitor them and their health.
True data centers where you own the hardware shouldn't... they give you an ethernet cord and everything is on you.
Very few people go to "true data centers". Those are very expensive because you are buying power, space, cooling and cross connects. Racking machines, replacing hard drives, building a PXE-boot infrastructure, building a remote access infrastructure that bypasses the customer facing network is expensive and time consuming.
Full Disk Encryption is an option, but that’ll entail needing the use the IPMI console to enter the password at every reboot, essentially turning it into a manual operation.
However, if the attacker gains RCE many IPMI implementations theoretically allow for DMA, but this is a significantly more complex attack to mount in practice with no public PoCs available.
On Supermicro hardware and maybe others, IPMI has a very dangerous default setting: if you're not connecting the dedicated IPMI port to a network (typically some closed network dedicated to management), it will use the first ethernet NIC on the motherboard (sharing it with the host), possibly making it accessible through the internet (with default, insecure credentials adding insult to injury) or at least neighbouring machines.
Leaving default creds on IPMI/iLO is not uncommon,some providers don't allow internet access,you have to login on the web console and use a java applet to access it iirc. I can't imagine a well reputed provider exposing ipmi to the internet but the nature of their business means they have to diversify server and network providers.
Not sure if it is still the case but a while ago it was standard on OVH servers for them to put some public keys in their root-like's user authorized_keys. I think they used that to perform tasks requested from the web management system.
Most (not all) servers have out-of-band management; of which IPMI is just one of many such solutions.
It's also worth noting that the hack could have been against in-band management if the Nord used an OS image provided by the DC hosts. However OOB feels more likely given their description (as vague as it was).
Typically data centers have compliance requirements like SSAE 16 specifies controls around physical access. Most any major retail data center would have that certification and others.
One presumes that because of NordVPN's business, they're colocating a server or two in many very many "POPs", presumably not all of them have tight controls on physical access. Its likely that there are none available in many areas where they seek to maintain a point of presence.
To the person / people down voting, can you share your ideas for how you can go about providing at-scale management of large estates of physical machines? I've never seen anything else that's both as practical and as affordable as IPMI / IDRAC / ILO systems that ship with servers, that doesn't introduce weird new failure conditions that can impact significantly more than a single host.
This screams for clarification and I'd love for someone more knowledgeable in the area to elaborate on it. Is this common practice for data-center providers? Do I now not only have to worry about my own infrastructure security but also worry that my IaaS provider hasn't installed some backdoor to my servers?