Given the dire security of COTS routers and Wi-Fi access points, we need alternatives which support open OS + coreboot.
$200 used: The venerable PC Engines APU2 is a fanless x86 AMD 10W TDP router with 4GB ECC RAM, TPM 2.0 and GPIO pins, open schematics and coreboot, which can run pfSense, OPNsense, OpenBSD, Linux, FreeBSD and OpenWRT, with virtualization support. mPCIe slots for WiFi, LTE & mSATA.
Constrained by supply chain at present.
$200 used: HP t730 and t620 Plus thin clients have an AMD SoC with similar TDP to PC Engines APU2.
$200+: MIPS-based Ubiquiti EdgeRouter Lite/4/6/8 can run Linux and OpenBSD (octeon/MIPS), but is also supply chain constrained.
Some SOHO networking devices made by Microtik, QNAP and Ubiquiti contain Arm SoCs made by Amazon's Annapurna Labs, who also make AWS Nitro silicon, https://en.wikipedia.org/wiki/Annapurna_Labs
We need more low end hardware chips with real mainline Linux support not more PCs with NICs doing software bridging/nat/routing. The control plane is what's creating the security problems but in order to use an open control plane we very often have to throw out what makes COTS routers and Wi-Fi access points attractive for their costs - the specialized hardware.
When I last looked it seemed like you could easily do a few 100Mbps but once your WAN was >=1Gbps it seems like you become much more limited. I ended up going for an Intel box which ends up doing everything in software - is there anywhere that compares the maximum throughput for different hardware with OpenWrt?
I'm looking forward to more XDP + eBFP, VPP, or FD.io based firewall and routing solutions as I think those are where we'll see the next big improvement in networking at. I'm hoping to get my hands on some NICs that support XDP offloading. I can see the future homelabbers leveraging SmartNICs and/or DPUs for routing/firewall applications instead of using large big iron custom ASIC switches.
I don't have any annoying router or CPE from my ISP. Just fibre to the basement and an ethernet plug in the wall in my apartment providing 1Gb Internet access.
So I have a few EdgeRouter Lite 3s doing NAT and port forwarding (IPv4) because they can do that in hardware. As they are not open and I don't really trust them, I just pretend that they are on the ISP side of my network (even though they are mine, and I "control" them). So their LAN ports count as the "outside" in my setup. They provide networks like 192.0.2.0/24 that I just pretend are my public IP's.
This is also where the AP with GF's phone and chrome-cast resides on a separate VLAN. This has the benefit of me being able to play around with "real network" behind the firewalls in the next layer without worrying about causing downtime and issues for her (big win).
After that I have my "real" firewalls (a bunch of APU4C4s) that segment my network into several parts and multiple layers. They have less work to do as they don't need to do any NAT/translation.
It also makes the firewall rules much simpler as there are no NAT/RDR rules and I don't have to think about whether a firewall rule apply pre- or post- translation (NAT/RDR).
It would of-cause be a lot easier if I just had a /24 of public IPv4 addresses, but this setup lets me sort of pretend that I do, even though I only have two static IPv4 addresses and an extra one with DHCP.
I just recently got IPv6 with a /56 routed to me from my ISP and this is what I am messing around with currently. I have some Juniper switches (EX2200/EX3300/EX4200) that can do IPv6 routing in hardware (and ACLs), meaning that I can do things a lot more like I wanted with IPv4 because I don't need any NAT.
One of the benefits for example is that I can just route a /64 to my local mirror servers over a VLAN for bulk traffic without putting any load on my firewalls. As the traffic is only to/from a few destinations and ports, ACLs in the switch are fine for me (+ local firewall on the mirror servers).
This bulk traffic is probably close to 90% of my total bandwidth usage, which is just syncing the local mirrors.
This means that this traffic is not clogging up the NIC queues on the firewall competing with "interactive traffic" like web-browsing, etc. Is this needed? No, not really. The firewalls can easily handle the traffic, but it is simple to remove 90% of the load on them for basically free.
All this may seem complicated and/or convoluted (and it probably is) but it makes (for me at least) my network much easier to reason about, and makes tinkering/experimenting easy and less likely to affect "production".
An irritating thing: the t620 used to be barely worth $40-50 until someone started snapping them up and then charging a huge (100%+) markeup with "opnsense" in the title.
You might as well just buy a Dell SFF with a low-powered i5 at that point.
Dell Wyse 5060 thin clients (similar AMD SoC to PC Engines APU and HP t620 Plus) have also jumped from their $40 eBay prices.
Ryzen Embedded SoC includes 10GbE networking, but until this Decisio device, I've not seen any OEM using it. Maybe there was a chicken-and-egg issue with OS drivers.
I have one of the upper models (DEC850) I bought last year, and been meaning to write about it. Overall, it's been great, although I haven't gone through and done as much analysis as the linked post.
I should've moved to a dedicated home firewall sooner, but wasn't super high on my priorities. But with all the problems home modems/routers seem to be having, along with an upgrade to fiber internet that included a modem with hard coded default credentials that could not be changed I made the switch.
Only glitch I've noticed is a few problems with unbound dns, that I need to spend some more time on. DNS over TLS doesn't seem to be working for me, and I've had a couple glitches now when there are power fluctuations that I have to restart DNS after boot. But that could easily be something wonky in my setup and need to do more work to isolate the issue.
Overall, I would recommend for anyone who wants something a bit better than just the consumer grade stuff provided by an ISP, and for small business, remote sites, etc.
Paying $700+ for something like this is just silly. OPNsense and pfSense are both charging massive markups on this hardware.
pfSense doubles the price over the exact same box you can get from AliExpress or ebay.
Buy a standard Ryzen mobo, a low-end recent Ryzen CPU, undervolt it, toss some ram in, an old SATA SSD, and a dual or quad port PCIe adapter off ebay. Or just buy an old Dell SFF PC; you can get your choice of how powerful a processor you want.
If you want the embedded/fanless setup, then buy one of the aliexpress/ebay boxes. Qotom is one such seller, I believe.
As someone with a ton of experience with opnsense: their release style is extremely irritating. The only way to get security updates is to conduct a full upgrade of the system. The UI is clunky, outdated, full of useless "help text", and almost seems to be purposefully designed to use confusing field names and terminology. Debugging problems is difficult at best with inconsistent logging infrastructure.
I've also never once had a config restore go properly, something I've tried to do several times because I generally get about 2-3 years out of an opnsense install before enough stuff has broken that I need to reinstall from scratch.
Last but not least: they now routinely ignore their support forum except for the most simple, common problems. Every time I've had a problem, I've found a corresponding post in their forum which has gone gnored by their employees.
> OPNsense and pfSense are both charging massive markups on this hardware.
Hardware remains one of the few viable business models for open-source software. The combination of features on this particular device have literally not been sold before, so in this particular case, the price may be somewhat justified IF they are also providing regular UEFI firmware security fixes.
With such high margins, it's surprising there are no hardware clones for AMD firewalls. Is that due to CPU shortages?
Protectli is a success story for Intel firewalls, where the open (coreboot) product earns a price premium and opaque clones expand the market.
15W idle is not bad. The 20€ eBay-Cards will eat this alone. You can get better numbers with Intel cards, but those are 70€ and up. Also, Ryzen embedded is quite a bit more efficient. Now, depending on where you live, this might not matter, but a Wattyear costs me ~3.4€. So if the thing uses 35W instead of 15W, I pay 68€ extra per year just for power. Additionally, you need to find a small formfactor passive case that supports a PCIe card and a small enough mainboard to fit it. This alone will set you back quite some money.
If space or power efficiency don't matter to you, that's totally fine, then you can easily & cheaply match those specs. If you need something small and efficient, the markup isn't that large.
The cost can still favor DIY, even using new parts. I did this twice about a year ago, when the USD prices were far worse for DIY.
Nowadays, one can get a cheap Alder Lake Pentium + ITX/mATX with a cheap case, okay PSU, RAM, and SSD. That's no more than $350. Now new Intel dual SFP+ NICs are hard to find cheap, with Nvidia cards being easier to find cheap. So a dual port CX4 or CX5 from fs.com will run from $260 to $340. This still comes out ahead of the official HW. It all gets cheaper with older Intel HW.[1]
IMO, buying the official HW is officially supporting the OPNSense developers. As you noted, the DEC700 series is also quite small in terms of footprint. Those are definitely strong factors in favor of the DEC700.
[1] I mostly went with Intel HW, moving away from using a R5 3600 in that role. It seems that AMD's Zen2 is a bit less efficient vs contemporary and current Intel desktop platforms in idle to low-mid loads. Current setup with an older used Pentium and an used dual port X520 takes ~<20W on average. Though I doubt the DEC740 takes 15W at idle, probably less.
W.r.t. older Intel HW, I am thinking of 10th/11th gen parts, which can be had on good deals new.
I actually mentioned the DEC750 the other day in another thread:
https://news.ycombinator.com/item?id=31451142
I forgot to mention the linked review that this post is around. Though it was probably the most detailed one I found at the time prior to purchasing my DEC750 from Deciso's OPNSense shop (https://shop.opnsense.com)
I picked one up a few months ago as part of my attempt to overhaul my home network so I don't have to care again for a decade. Currently complemented by a "L3" 24-port Ubiquiti PoE+ switch (twelve 1GbE, twelve 2.5GbE, two SFP+) and an "L3" 32-port Ubiquiti SFP+/SFP28 switch. The hope is as DOCSIS 4.0 rolls out I'll get more value out of using SFP+ for a WAN port.
Grabbed a Micron MTA18ADF2G72AZ (16GB DDR4-2666 ECC VLP U-DIMM) to never have to care about memory. Ever. All the Suricata and CrowdSec rules in the world.
Also grabbed an Edimax N150 WiFi 4 USB adapter. This uses a RealTek RTL8188EUS chipset, which is extremely well supported by FreeBSD's urtwn(4) driver. So I use that to automatically do WAN failover to my iPhone 11 Pro as a hotspot if the cable itself went down.
Rock. Solid. And per my PDU, really just sits at 8-9W.
If my experience with this is anything like my experience with FreeBSD (limited) over the past 25 years, I expect the only downtime will be when I move house.
We have the fairly high end DEC3850, and I don't recommend it (or their other stuff). Not because I don't love OPNsense, because I do and I've switched all my firewall/gateway/routing tasks over to it at a range of sites from UniFi gateways (which are crap). OPNsense is awesome. I have Deciso's Business Edition as well in a bunch of places. And the thing does work.
But the value is crap. Essentially it's a SuperMicro 5019D-FTN4 with a worse
performing chip, a built-in AMD SFP+ solution (which isn't as well supported as a cheap Chelsio card one can find on Ebay), and with no IPMI or normal VGA console. Which sucks. For something as critical as a gateway, it's really nice to just be able to plug it into normal rack console/screen/IPMI management for recovery and install. And originally barebones the 5019D sold for around $1k. While sadly amongst the supply shortages now they're more like $1400 despite being old, that's still ~$300 less than the DEC3850, and will end up about even with RAM and an M.2. But you still then get a faster CPU, and much better management. They throw in a single year of BE as a small sweetener but overall their offerings are straight downgrades in my opinion from just getting normal decent PC hardware. And that's part of the advantage of going to OPNsense in the first place.
So I kind of regret going for that vs just getting a normal SM system (various flavors of which I've deployed everywhere else, I got a bunch of 5018Ds for ~$600 for example). Not that setting up a special dedicated serial thing is a huge deal, but it's definitely an annoyance at that price level. Looking over the rest of their offerings it all looks similar: debatable quality for the money vs bog standard quality hardware.
And I want to be clear this isn't just a complaint about markup. I don't in principle actually mind paying more for the same thing if it comes with better support and someone standing behind it. The problem here is that the features are actively worse, and support is too! I was very surprised for example that something like Sunny Valley's Zenarmor actually has issues with the DEC3850's SFP+ that it wouldn't have with an old Chelsio card. So it's not like it runs OPNsense better.
Still, despite warts I'm very happy overall with the OPNsense, and with Deciso beyond their kit. It's also let me squeeze more life out of stuff I was feeling more iffy about (like UniFi and UISP for example, now I can just route their management VLANs via WireGuard for L3 management with zero internet exposure, and without UI's shitty ass routing/security I'm no longer feeling as pressured to leave ASAP).
Unbound was also glitchy for me. I had to switch to Dnsmasq on the firewall. Optionally you can forward to AdGuard Home for DoH/DoT and ad/tracker blocking.
Excited to see more low-power, but still powerful, networking gear. I run OPNsense on an old Dell SFF, which is awesome since I can stick a few expansion cards and have 10G networking for <$200, but it draws 60W on average. Cutting that down would be great, but the reviewed device here is still 3-4x more expensive so its hard to justify the upgrade.
I had previously tried an ARM-based "Espressobin" which was only about $50 but had no 10G and little-to-no official support, couldn't even reliably boot into Linux.
> I had previously tried an ARM-based "Espressobin" which was only about $50 but had no 10G and little-to-no official support, couldn't even reliably boot into Linux.
hmm. There apparently is a successor "macchiatobin" that has 2xSFP+, but comments of poor software support is disconcerting
I have issues with webRTC and OPNsense at home. I presume from NAT type. I end up being stuck on TURN sometimes. I've tried adding a 'static port' as an outbound hybrid NAT rule which improved things but not in every scenario.
Other than that opnsense on an old ewaste HP SFF PC has been excellent. My only upgrade would be something lower power and fanless. Or to add it virtualised on my homelab R330, but security and no Internet when I break that wouldn't be a better situation than now
With the PC Engines' line basically EOL, what is the best hardware to run a home setup < 2Gbps WAN for around or below $200? I'm not too keen on buying a largely unsupported Intel box off AliExpress that has a 40% chance of being DOA.
I suppose there isn't much profit to be made for a lower-end box, but I'd prefer to stick with *sense instead of wasting time learning Mikrotik's OS, even though they have some great price-points.
I switched from PC engines to https://protectli.com/ and I'm happy so far, although it's more expensive. I haven't tried anything else yet, so just a single data point.
Seconded. I have a protectli vault 4 port and it can handle 1Gb/s traffic fine.
If you want to use OpenBSD beware that you might not be able to push 1Gb/s with protectli devices. I had issues with OpenBSD pushing full gigabit, most likely due to all of the security mitigations.
If you want to safe some bucks and don't need coreboot or local support, you can opt for various chinese resellers like topton or qotom which also provide firewalls based on the same boards.
I'm running a J4125 based topton box and it's running fine for about 2 weeks now.
Possibly stupid question, but how do people do Wifi with these the wired-only routers? Just buy a normal full-featured Wifi router and let it do its routing, or can you get some dumb AP-only and offload all the routing on the more powerful wired box?
Almost any WiFi router can be configured as a dumb AP, in the worst case just disable it's DHCP server and plug the patch to one of the LAN ports, not WAN.
A proper AP is usually an overkill for a home usage.
I have an Intel box as a router, a managed switch and then a Unifi AP - there are certainly more parts, but each component does it's job relatively well.
Somehow I fell into rabbit hole reading into how OpenWRT apparently is in process of changing how they do switches (related to bridging?), from homebrew swconfig to upstream DSA https://forum.openwrt.org/t/mini-tutorial-for-dsa-network-co... which seems to impact the setup a bit
It would be __very__ interesting to learn how much bandwidth this beautiful beast can saturate with cake managing SQM (under Linux, esp. OpenWrt), and if proper support for ECC UDIMM is implemented. What an awesome piece of networking machinery!
$200 used: The venerable PC Engines APU2 is a fanless x86 AMD 10W TDP router with 4GB ECC RAM, TPM 2.0 and GPIO pins, open schematics and coreboot, which can run pfSense, OPNsense, OpenBSD, Linux, FreeBSD and OpenWRT, with virtualization support. mPCIe slots for WiFi, LTE & mSATA. Constrained by supply chain at present.
$200 used: HP t730 and t620 Plus thin clients have an AMD SoC with similar TDP to PC Engines APU2.
$200+: MIPS-based Ubiquiti EdgeRouter Lite/4/6/8 can run Linux and OpenBSD (octeon/MIPS), but is also supply chain constrained.
$200+: Intel-based https://protectli.com/ (coreboot) and virtualization-capable multi-NIC mini PCs s with unknown UEFI, https://news.ycombinator.com/item?id=31451142
$400 AMD/Xilinx wildcard dev board for Robotics, KR260 has Arm SoC, Xilinx FGPA and multiple 1GbE ports + 10GbE SFP, backordered 20 weeks, https://www.servethehome.com/amd-xilinx-kria-kr260-robotics-...
$400: ASRock 4x4 dual-NIC has Ryzen Embedded or 4000-series SoC with Realtek NICs, questionable BIOS and limited support focus on Linux/BSD.
$400 used: HP t740 thin client with Ryzen Embedded is Mac Mini size, with PCIe slot for low-profile quad-port or SFP NIC, https://www.servethehome.com/hp-t740-thin-client-review-tiny....
Some SOHO networking devices made by Microtik, QNAP and Ubiquiti contain Arm SoCs made by Amazon's Annapurna Labs, who also make AWS Nitro silicon, https://en.wikipedia.org/wiki/Annapurna_Labs