I think the story here (at least my intent when I submitted it) was that Ubiquiti released a "statement" in response to the Krebs article and, distressingly, they don't seem to challenge any of the article's core assertions. Their response seems to be "Oh, um, yes...that happened. But hey! We don't think they accessed any customer information! So...please stop looking at this."
What I want to know is the likelihood that customer devices were compromised. Frankly, I could care less about whether my data/password/whatever were lost in the breach. Happens all the time, who cares. The problem with the Ubiquiti case is that this compromise has the potential to have poisoned official firmwares, with the silly source code/customer data bitcoin ransom as a smokescreen.
Remember, there were VMs spun up in their cloud that they couldn't account for. That doesn't look like someone just tooling around to me, that looks like someone recreating a build process.
Is anyone monitoring their Ubiquiti devices in their homelabs to watch for any suspicious traffic? Being part of a nation-state botnet is not a very comforting idea.
Not to discount the threat, but 'vms spun up in their cloud' is ...exactly what coinminining attackers do. It's so frequent, it's even called out as a 'thing to watch your bill to notice' in amazon security talks/etc.
Gain access, launch N VM's, where N is as many as your credential and the cloud account limits in place allow, mine coins 'for free', and move on.
I agree, but APTs have gotten clever enough that I could also see them doing this as a distraction. Like burgling somebody's house to find blackmail material, but dressing it up as a common "smash & grab" theft so as not to raise their deeper suspicions.
That's fair - and I hope that is the case that it's just mining.
But I also hope Ubiquiti performs a full audit (ideally by an expert 3rd party) to ensure the integrity of their build systems and reassure customers that the worst hasn't happened. Because it's been radio silence on that part, and that they haven't even announced they're planning on it makes it seem like a blindspot.
The problem is that UBNT now requires you to cloud-enable many of their modern devices as part of the setup process. You can later disable that after setup, but I think it's safe to assume most people don't know that (they certainly don't go out of their way to tell you that), and even fewer people will go through the effort.
So, this hack also implies that a substantial number of devices out there now have their remote admin credentials leaked to an unknown, presumed malicious, third party.
I have 2 cloud keys that I only use for UniFi Protect -- mostly because they abandoned the UniFi Video that I could self host; my Network controllers are all containers that are mine (I do have 2FA set up for the UniFi Protect).
I don't see how to remove the remote login from the Cloud Key, I tried creating a new Super-admin, but there's seemingly no way to remove the user.
I guess I'll just have to factory reset them and start over with all new credentials.
> If you have 2FA enabled are you protected from this?
Very hard to say based on the limited information released. Part of what was released mentions "SSO cookie secrets" and "remote access". Best to assume 2FA isn't protecting you if the attackers were possibly able to forge cookies.
Yes it does, both on the local device, and on the Unifi cloud service. This is true as of recent firmware versions.
But 2FA in this case is useless. The hackers owned the UBNT servers. If they harvested user logins, then one can assume that they also harvested the corresponding TOTP secrets.
To make matters worse: When you setup a Gen2, Gen2+, UDM, or UDM-Pro, you are required to login to the UBNT cloud service to activate your device. Once you do this, your device now has an "Owner" account cannot be disabled.
If you have enabled 2FA on your Unifi account, then you can now login to the device locally using your Unifi credentials + 2FA, with a nice false sense of security.
So in other words, if your account creds and TOTP secret got leaked, and you did not change them, anyone who gains access to your local device's login page can still own you.
This remains true even if you disable Remote Access. (And in fact, as of recent firmwares, the ONLY way to disable remote access is through the Unifi cloud panel. You can't even do it locally.) It will still call out to Unifi's cloud service and authenticate you. There's no way to turn off this SSO account. The best you can do is block all access to Unifi at the firewall, and download all firmware updates manually. And good luck with that if you are using a Unifi device as your gateway.
When I managed a cloud key a couple years ago, login happened via SSO through ubnt, which has 2fa. (But yes, you can also login directly on the cloudkey locally which doesn't support 2fa)
The concern is whether their remote login / SSO authentication via cloudkey was compromised.
There is no way to disable cloud login to the unifi dream machine's web console. Not the network controller, but the web console itself.
Like, the screen you get if you hit up your UDM's public IP, or go to the network controller (private IP) and open settings -> controller settings -> dream machine
While my controller is on 6.0.43 and APs (uap-ac-lite and uap-ac-hd and uap-pro gen1) are on the corresponding firmware.
I believe this controller version is December 2020 and predates the January issue disclosure date(though possibly not the incident).
I have suricata taps on my net that are clean. Aps and controller are predictably trying to get to their telemetry call homes but nothing more.
That said my Ubnt controller and aps have almost no internet access and haven’t for a long time. I only enable it when performing upgrades and disable it.
I’ll pull the firewall logs and see specifically the IPs they are attempting to hit.
It's speculation as I stated. But it's really not a "leap".
The whistleblower's claim:
> They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration
And what Ubiquiti has further stated:
> we have well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure.
So someone with intricate knowledge of their infrastructure and access to their source code + signing keys, is doing what exactly with those VMs?
I'm not trying to 'prove' that the firmware is compromised. We can't do that with the information we have. I'm just highlighting strong evidence that the possibility needs to be thoroughly investigated.
Yesteday I ordered new MicroTik gear to replace the last of my Ubiquiti gear only to find there's a two week wait for it to be shipped! Looks like there's been increased demand for products made by Ubiqiti's competitors. Thanks Ubiquiti for a good few years but time to move on. You've lost my trust big time.
MikroTik is always saying delayed shipping. Amazon has been the only place I’ve seen them in stock but from 3rd party sellers.
MikroTik isn’t even in the same league as UniFi. Their interfaces and docs are terrible. The only reason I ever considered buying something from them is because they had a good price for a 10gb switch. If you want a dumb device you can’t beat their price.
Check out a few of the reviews on Amazon. Seed and Don’s reviews are very critical.
I’m not convinced that UniFi today is in the same league as UniFi two years ago. UniFi Controller 5 worked. Version 6 still wants to kick back to old mode for some tasks, changes its dashboard for no reason (ads?) every month or so, and seems far less reliable. And the actual fancy config remains in JSON, and, to the extent that anything is supported at all, JSON config is definitely not supported.
Oh yeah, UniFi APs seem to need rebooting periodically to keep working well. My old OpenWRT gear was rock solid with a year of uptime.
This is the main reason I’ve kept my Unifi stuff and cloud key despite the growing company issues.
Is there anything in this space that is good hardware with a decent UI? When I looked the options were consumer crap or very expensive enterprise hardware with no UI.
Last summer I installed a bunch of second hand Arubas for a non-profit. it was dead simple and the first AP you set up acts as the controller for the others, no cloudkey equivalent needed.
I'm not sure if the UI is 'consumer friendly' but I didn't need the commandline and I had never worked with any HP networking gear prior to that, and had no difficulties.
Are MikroTik access points still a royal pain to configure and maintain? I use their switches, but not their APs.
Decades ago, I was a network engineer working in production data centers. I’m used to setting up switches manually. But I have no professional experience with setting up meshed WiFi networks. I like that the Ubiquiti APs “just work” for the most part.
I’ve started to notice weird AP/client failures with my ubiquiti gear and debugging is a nightmare. It seems that ubiquiti has hidden a lot of useful information in an overzealous effort to streamline UX.
> Yesteday I ordered new MicroTik gear to replace the last of my Ubiquiti gear only to find there's a two week wait for it to be shipped!
I've never seen any Mikrotik product available to ship, it's always on back order. I've even had resellers call me up and offer alternatives because my order was on back order.
I manage a few of these. One I have installed at a customer location acted very strangely until I ran it through a full factory reset.
I observed the UI was hiding the ability to apply updates, SSH was enabled, but it didn't allow me to connect and the name of it was changed to what looked like the MAC address.
No permissions were changed, the UDM was sitting unattended and the other administrative user did not update their password prior to this happening.
I am very paranoid and ended up doing a full reset as I was only able to partially administrate it.
Suggestions on what to replace a Edgerouter 6p with? Ideally something with regular security updates and zero cloud.
I was considering a the Odroid-h2+ from hardkernel. It has an Intel J4115 (4 cores, 10 watt TDP), 2 Sodimms (max 32GB ram), and two 2.5Gbit ethernet ports. Looks like it would make a decent router. There's an optional board to add 4 more 2.5Gbit ethernet ports.
Wasn't sure on the OS, OpenWRT is definitely in the running. Or maybe VyOS (the open fork of vyatta), which is relatively close to what Ubiquiti ships.
Did Cisco bother to launch proper replacement for ISR C1812/892J? Looks like there are ISR 900 series but not sure they’re worthy... In my country there are Yamaha NVR/RTX lines but not sure those are sold worldwide
Just make sure the rtl8125b NICs are actually supported by what you want to run; they're relatively new . And poor driver performance will be magnified by the low power CPU at those speeds.
I bought an H2+ sometime last year to use as a router but struggled to get acceptable networking performance, so moved onto something else.
They look pretty good, managing
11Gbit/sec or so @ 40% CPU utilization with iperf3.
Did you upgrade the bios as mentioned on that page "Have to flash GLK-ESF BIOS to use H2 Net Card properly"
My use case is for a home, ideally with a NAS and 2 desktops on 2.5 Gbe with only a single client at a time using more than a gigabit for large file transfers.
Did you find anything else with 4-6 2.5 Gbe ports?
Certainly seems good enough to route 2.5gbe, although the router will be doing things other than just sending traffic (NAT, packet inspection for firewalls, etc.)
You need the drivers though. Looks like rtl8125b support landed in Linux 5.9, but as far as I can tell VyOS and OpenWRT are still on 4.x, so you'll likely be in custom land - either upgrading to 5.9, backporting the driver, or using the realtek-provided driver. Looks like people are working on it recently for openwrt: https://forum.openwrt.org/t/setup-an-odroid-h2-with-openwrt/...
I'm only using gigabit at home, so didn't need 2.5g.
My connection to Comcast is only 400mbit or so. But I need a NAS, and most have 2.5gbe ports (Qnap, Asustor, etc). Just got a ryzen b550 motherboard, with 2.5gbe ports by default. I needed a new cable modem, which came with 2.5gbe ports by default.
I do often move 20-40GB files around, so the bandwidth would actually be useful. Apparently ubuntu-20.04 has the right drives, but thanks for the info, seems like I might need to upgrade to a semi-current kernel to get the interfaces working.
I built a number of VPN endpoints uning these for PoP's / Remote Offices. They're superb, not used any RF stuff with them but for the price and performance you can't go wrong.
They will get involved as much as they have with every other breach — i.e. not at all. Even Equifax only got an extremely minor dog and pony show for the complete disaster they created.
I often tell people when they ask for company/product recommendations that I can only really recommend somebody after they've screwed up.
Any company can deliver product without hassle, but the ones you want to trust are the ones who handle mistakes properly. In this case, I just lost all my confidence in Ubiquiti because they've done the worst thing when it comes to maintaining my trust, lie and deceive.
> if only you could easily run it at home without their cloud.
Am I missing something with all these comments talking about “needing” a cloud account? I have two of their APs and a USG, and I don’t do anything in the cloud. I have a Cloud Key on my network, but that was just for convenience. I could’ve run their management software locally.
I'm in the same boat.
I bought my Edge Router Lite probably 8 years ago and it's easily been the best $99 I ever spent on networking gear and it's still working amazingly well. Before the Edge Router I couldn't imagine having a home network that didn't need a router reboot once/week.
I've grown into many of the ER features over the years. Also fun fact - the storage is literally a USB flash drive plugged into a standard USB-A port under the cover! (which is totally upgradable). I will forever be impressed with this product.
I'm hoping this spurs Ubiquiti into reconsidering its cloud-only push. I know I will be looking beyond Ubiquiti the next time I need to expand the network.
I'm hoping this spurs the board to oust the CEO who seems to be the one getting the blame for Ubiquiti's shift in recent years away from making highly reliable prosumer focused products to pushing half-baked cash grabs out the door.
The CEO is also founder and held onto a super majority share of the stock. It's his board essentially, which was a governance issue I remember seeing identified as an investment risk years ago. If it wasn't for that he might have been pushed on a long ways back, but there it is. There's risks to both full submission the shareholders and full independence of shareholders.
> At this point, we have well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure.
That's quite interesting. You can imagine a disgruntled, former employee that didn't lose Lastpass or AWS access... or someone else very close to the company?
You can also imagine an executive team trying to shift suspicion towards the whistleblower, who happens to be an individual with intricate knowledge of their cloud infrastructure.
The #1 thing Ubiquiti trades on is trust and it’s shocking how they don’t see that. They need absolute transparency directly to their security experts, which need to operate independently of management, to earn and maintain that trust. Even little hints of prevaricating won’t be seen charitably in any way. They’ll be seen as hiding something.
It’s close to importance to vaccines. Did you know vaccine safety regulators see the data from a trial before the company management? They operate independently because trust and transparency is so absolutely crucial for both the product and public health.
We rely on routers quite a bit in society to secure our identities, personal transactions, and critical infrastructure: we should be applying the same mindset that we do to vaccine safety to router security!
I have a Unifi LR AP I want to replace. I have a three level home and this AP really struggles to provide coverage. I think my mistake was getting a directional AP.
Are there any decent consumer alternatives if I'm done with Ubiquiti?
I'm in a multi-level home. Previously I had 1 centrally located Ruckus AP. Its signal was strong everywhere (solid RX signal strength), but devices could not transmit strongly enough back to the AP.
In the end I went with 3 Unifi APs + a controller. You can use wired or wireless for backhaul. The main thing is that the APs have to be able to communicate with each other.
There isn't a great way to test out the wireless backhaul without buying the equipment. I suppose you could use a scanner to test the RX signal strength with your AP in varying places. If the 2.4/5ghz signal is strong in the (AP, scanner location) pairs then you know the APs might be able to talk to each other.
If you're looking for a less work setup, the Amplifi line, or Eeros might work.
Last year, I installed an Orbi Wifi 6 system in my parent's three-storey house. And I'm using the older RBK50 version myself with an ethernet backhaul. For consumer products, both are pretty good and offer fast and stable wifi. I'm not running a complex home network so consumer-grade hardware is good enough for me. That said, I've no idea if Netgear is better than Ubiquiti at a company level. I've noticed that many people here are usually fairly critical of Netgear.
My knowledge about networking is fairly limited, but from a consumer perspective it works really well with seamless roaming, especially compared to old wifi repeaters. The Orbi system uses a dedicated backhaul to connect the nodes. Since by parents' old house doesn't really allow installing network cables, this seemed like a good solution.
In my home, I'm using a wired backhaul, which is the prefered option if possible. My only critisism is that the Orbi Wifi 6 system is quite expensive and there probably exist cheaper options to achieve the same.
I'm going to agree with ehutch on this. Don't try to cover a multi story house with a single AP. Would you attempt to cover the house with one music radio? No, it would be way to loud near the stereo and way too quite far away. Also you spill signal into neighbour's location and blast excess noise into the spectrum.
3 APs on a lower setting would give you much better performance.
1 AP is a hard requirement, sadly. I have 6 locations in my house I can plug in my cable modem and for now there shall be no hole drilling and Ethernet cable running. My current setup works, but I'm surprised I cannot just centrally locate 1 Wifi AP. We did that when I was younger in a far larger house. But I guess back then wifi was far slower and more robust?
You can't really compare things to the past for a lot of reasons.
1. In the past you were probably the only person with wifi in a mile. Now you have countless units attempting to talk in any given place on a very limited spectrum.
2. 5ghz is fast but walls and objects kill your speed quick.
3. Turning down your current 2.4ghz to wireless B speeds will increase your reliability in environments with few neighbours. Imagine a 10mbit network hub. Worked ok back in the day. Now try to increase its speed to 1gbit and add lots of uses. You wont get much done, everything gets corrupted and you retransmit a lot.
I've looked, haven't found anything clearly better. Just got a Unifi Wifi-6 LR AP. Seems reasonable to just block APs from internet traffic and run the AP management software locally (also with traffic blocked from the internet).
What was the main drive for Ubiquiti customers to buy Ubiquiti? I don’t mean this as a troll question, but as an engineer myself I always preferred my TP-Link mesh which serves my entire house and garden perfectly blazing fast internet and I never had to go to the cloud like Ubiquity customers. What do they offer for so many technical people to willingly undoubtedly compromise their security by using their cloud as well?
I don't use their cloud. I have a router, 2 switches, and 2 APs from the Unifi line. I run the controller software on one of my servers and nothing phones home out of my LAN. The APs have good coverage, everything is very reliable (as long as you don't upgrade to new firmware), and there single pane of glass management console provided by the controller software is awesomely convenient with just enough options and features for my needs. but all of this hardware is a few years old now and no longer what they want to sell.
Ubiquiti doesn't have to go to the cloud. I manage my Ubiquiti stuff locally. The reason I bought Ubiquiti is because they offered pro-ish level quality and reliability at closer to consumer prices. At a time when the only consumer options were pretty crappy, Ubiquiti offered higher quality at a small premium without having to go all the way to Cisco APs, for example. I haven't really bought any new gear for several years, so I don't know if the landscape has changed.
Yeah, this is exactly it. They used to produce enterprise-ish level hardware and at a reasonable price. Their devices, in my experience, were pretty rock solid. Setup took a little bit more work than the consumer (edited) offerings, but then you were done. They just ran. They could handle more packets, at least at the time, than the majority of consumer hardware.
They've now shifted to targeting consumers and their new stuff doesn't even have the basic features of other consumer hardware. Want DDNS? Something found in every other consumer router I've used. Nope. Want an admin console to configure or update without your smartphone (or when your wifi is down again)? Nope. Want to set a static IP for a device? Ok, but we restart the router each time you set one. Want your network to be bigger than /24? Nope. You can checkout their forums and see how long they've been aware consumers want these things and they've delivered on so little.
For me the cloud is a selling point. Though one that I'm now reconsidering after all this news.
I have been a Ubiquiti customer and proponent for a few reasons:
1. I have two houses, and the cloud feature lets me easily connect to and manage my devices at the other house when I'm not there. Not a frequent use case, but it does happen, and this just makes it easier. Sure I could setup some other remote access options, but this just works out of the box. I also have their security cams and I can easily monitor those remotely, though the video is stored locally at that location.
2. They occupy a middle between consumer gear and enterprise gear. Giving you flexibility and features that you can't really find in consumer gear.They have a fairly wide and frequently updated product line. One house has ethernet jacks in every room, and for a couple of rooms, I replaced those jacks with Unifi In-Wall HD access points. These are great because they are out of the way, you don't notice them, and they still provide wired ethernet ports. So my office has strong Wifi but I also connect my desktop PC to Gig ethernet via the in-wall AP.
At the other house I use the Flex HD access points connected to MOCA adaptors (the one part of my network not made by Ubiquiti) and just tuck them out of the way.
3. They have a fairly well integrated set of products. I have routers, gateways, switches, and APs, all discrete devices, but managed from the same interface. Managed on the web or on my phone. If I add a new AP to a location, I can just adopt it into the network using the app on my phone, the firmware gets updated and it becomes fully managed, taking on the default settings for that site.
All that said, I still really like their product lines, but this episode made me lose a lot of trust.
Ubiquiti works fine without the cloud, the cloud key/cloud management stuff seems to have a fair bit of user demand. Especially for consultants helping network trailer parks/motels/universities, etc.
I like ubiquiti because it's pretty flexible, powerful, I can keep config files in git, IPv6 support is good (I have a /64 per router port), and I can separate router and APs, and handle multiple APs well (without mesh/forwarding).
I have an Archer which stops handling non-VPN traffic multiple times a month and seems to drop telnet packets. I'm done with TP-Link.
I think part of the problem is that effectively everybody has flaws, so you're forced to choose who has the least-broken approach for your needs...and you can't do that without a lot of research.
Whistleblower: Ubiquiti Breach “Catastrophic” - https://news.ycombinator.com/item?id=26638145 - March 2021 (770 comments)
Previously:
Ubiquiti Networks Breach - https://news.ycombinator.com/item?id=25735032 - Jan 2021 (467 comments)