I want to look into adding support for sniffing and forwarding the 802.1x packets that the AT&T fiber modem/gateways use to authenticate. This is done to bypass the crappy performance and insecurity of their device. There are some hacky implementations of this but I want to do it in Go. I've gotten as far as sniffing the packets using gopacket under FreeBSD. Unfortunately, FreeBSD doesn't seem to support VLAN 0 so I was unable to implement this for PFSense. Perhaps router7 would be a better option for me.
Yes please! I love my fiber, but being stuck with this interminable box as the first hop is driving me crazy. And performance drops significantly with pass-through mode.
No, they actually use 802.1x to authenticate the connection. 802.1x doesn't actually protect or authenticate any of the data though, it just ensures that there's a valid supplicant somewhere on the other end of a port before it forwards traffic.
I didn't say anything about 802.1x in my comment, and I'm one of the folks that has published information about how to bypass these crappy AT&T routers.
Wow, that's awesome. I'm not sure if it can handle the VLAN 0 packets on FreeBSD, though. Still, this is a strong start and I want to see if I can adapt this for router7.
Definitely give it a shot! If it doesn’t work for you as-is, maybe you can hack on it until it does. Let me know if you get stuck, maybe I can give a pointer.
Why the external rebooter? Do you have stability issues with the APU? I have a couple running in production and never ever had to reboot/reset them... Also, don't they have hardware watchdogs?
The apu does have a hardware watchdog, and router7 uses it.
I don’t see any stability issues with the apu at all.
The external rebooter isn’t used as part of a watchdog setup. Instead, it is programmatically used by rtr7-recovery, with which you can recover from a faulty update, to trigger a PXE boot. This way, you can update your router each day in an unattended fashion, with automated rollback in case connectivity is lost with the new software.
it'd be cool if there was an HA mode for this so you can have two devices - and it rotates the active device while the secondary is updating... and then it'd be easy to have power redundancy etc.. .
I can see how that might be appealing in an enterprise setup, but I have no desire to build that complexity into router7 :). In case my apu breaks, I have another one and can just restore today’s backup onto that within 1 minute.
Would it be a lot of work to have a parallel partition which is updated and swaped? If something goes wrong you simply boot the old partition. Like on new Android devices : https://source.android.com/devices/tech/ota/ab/
You can prototype/test the feature by calling tc (interactively via https://github.com/gokrazy/breakglass). Then, a good long-term solution is to reimplement what tc is doing in Go.
I've always wondered, how does this compare to a dedicated hardware router (some kind of ASIC maybe. I'm not sure if what routers on the market use), in terms of performance. E.g. if the router is connected to 200 clients and they all experience heavy traffic, what kind of CPU/mem is required to keep up with the traffic?
gokrazy images currently only contain Go packages, so depending on what you want to do, the best way might be to bundle the file(s) into a Go package.
For router7 specifically, rtr7-recover’s -backup flag accepts a tar.gz archive, which will be unpacked to the persistent data partition /perm. This mechanism is used to restore state and configuration when reverting faulty updates. You can use this mechanism, or https://github.com/gokrazy/breakglass if you prefer an interactive shell, to place files in /perm.
That said, I was thinking about an -overlay flag for the gokr-packer, too, so that you could indeed place additional files in the static root file system. I’ll see if it could be done any other way, but this might be a good escape hatch for when the pure-Go model isn’t applicable (e.g. large legacy applications which cannot easily be ported to Go).
This is pretty awesome. Have you done load (pps) tests to see how efficient it is vs. other stuff?
(Oh, I guess linux is handling all the actual routing, of course. So really the question is performance of dhcp/dns/etc. at peak on the hardware, vs. alternatives.)
Thanks! The router setup achieves the full Gigabit on my line, both on IPv4 and IPv6.
Regarding DHCP, DNS and router solicitations: don’t worry about it. In my network I get barely 1 request per minute (with 31 devices). With the 1 GHz quad-core in the apu2, many orders of magnitude more traffic should easily be in the cards ;)
There are a bunch of tutorials out there on how to build your own router based on Linux or one of the BSDs. I’d recommend such a tutorial as a good starting point for a top-down view.
The lower-level is very visible through Wireshark, with which you can capture all network traffic.
Most universities with computer science have courses in networking, which usually includes programming a router. I’m sure some courses are available online
Also search for tap & tun based tutorials. Make sure to be ready to read tons of RFCs in the area of neighbor discovery, path finding in graph , and parsing
apu2: 105 € (+), no SFP (-), DB9 serial (+), GOARCH=amd64 (+)
Basically, the apu2 costs half (I now have 3), has a real serial port (now permanently attached to my workstation) and an architecture (amd64) which is closer to what gokrazy already supports (arm64) and more likely to be useful to others — I could find way more suitable (≥ 2 ethernet ports) amd64 boards/mini-PCs than 32-bit arm boards.
The apu2 has all the Omnia has (aside from the built-in SFP), and is well-supported by PC Engines and their vendors. They run coreboot and even sell you recovery SPI flash chips for a few bucks if you want to change coreboot yourself :).
The HAL is Linux in this case, but you might need to send pull requests to enable additional drivers, depending on the hardware.
The currently supported platforms are the Raspberry Pi 3 B, Raspberry Pi 3 B+, the PC Engines apu2c4, and qemu x86-64 (I’ll update gokrazy.org with an overview about this in a minute). As long as your hardware is similar enough to one of these, chances are things just work.
I guess that has to do with firmware and schematics not always being available. Current consumer routers often have no or little documentation on the internal workings and some have started to limit functionality if you can't use vendor-signed firmware. On the other hand, as soon as you can run stock linux kernels with reasonable performance, gokrazy becomes possible again.
Edit: I was wrong, the Turris has docs and firmware to play with.
I have. They are currently deploying a new DHCP platform and want to spend their efforts on that rollout instead of delaying it any longer by debugging/changing the existing infrastructure.
The first milestone of obtaining a DHCPv4 and DHCPv6 lease from my ISP and successfully pinging google from my laptop through router7 was reached on 2018-06-02.
I switched my home network to run on router7 on 2018-06-15.
Today is 2018-07-14, so it took about a month to polish before actually being able to publish it. But it was good enough to use less than a month after starting work on it. I only have a little bit of time in the mornings and evenings before/after my day job, and on some weekends, i.e. it could have been done in less time when working on it full-time.
I learnt about PXE booting, MBR boot loader code, details about DHCPv4, DHCPv6 (had previously not looked at DHCPv6 at all), how wireshark does ssh remote capture, and a bunch of other things I’m blanking on right now :). Lots of interesting details to be learnt in this space!
I find this single comment alone quite inspiring, let alone the project itself and its scope. You took a complex project from idea to working prototype in less than a month, while keeping the discipline and interest high enough to work the planning and implementation around your dayjob. And all the while, you clearly had enough knowledge to know what you didn't know, and enough planning to learn enough of it to push the project forward.
Meanwhile, some programmers (myself included) spend months dithering over the decision whether or not to start that simple web project that's been hanging around in their head for a year. Lots of respect to you. I'll be coming back to this comment again and again.
However, I had previously run dnsmasq instead of the default kresd so that the Omnia would resolve hostnames in my local network from DHCP leases, and it was such a pain! Updates would frequently break the dnsmasq setup, so I was glad when they finally integrated DHCP lease-based name resolution in the Omnia.
For me, running custom software on the Omnia is a frustrating experience. Working on router7 was a pleasant experience.
Lastly, from my experience with gokrazy.org, I expect the total maintenance/debugging time for the router to be really low, and not in C and obscure build systems, but in Go, about which you can read my thoughts at https://michael.stapelberg.de/posts/2017-08-19-golang_favori... :)
> Updates would frequently break the dnsmasq setup
That's surprising to hear. Dnsmasq has been my go-to solution for almost 10 years. Dhcp hostnames via DNS just works out of the box. I have two subnets at home, serving dhcp 4/6 on both. The only thing that was a pain (mostly because I had to understand it first) was setting up dhclient6 for prefix delegation and get hooks in place so the dnsmasq config would be updated when the prefix changes, but even that addition already survived a dist-upgrade already.
But should that stuff crash and burn next time I dist-upgrade I might take a look at router7, even though I personally really dislike go, but along as I don't have to write code in it I'm fine. Considering how happy I'm with i3 I'm pretty sure you did a good job here too. :)
I’m not saying that dnsmasq itself broke (though I have seen my fair share of weird effects with dnsmasq itself, too), but the _dnsmasq setup_. On the Omnia, you need to somehow get the default kresd resolver out of the way, or make it forward queries to dnsmasq. Either option was frequently overwritten by an update, effectively breaking the setup.
And thanks for the praise, but if Go is not your thing, maybe this is not the best project for you :).
For those unfamiliar with Netgate, they make fantastic little appliances for DIY routers (in addition to the commercial pfSense offerings). They're a great option if you are looking for an alternative to PC Engines. I don't have any affiliation with them, other than having run their 8-port model at home for the last two years without any issues.
Netgate is far, far more expensive than PC Engines (do you see anything near $100 or 100 EUR here? [1] No, you don't). It isn't even remotely in the same league. An 8 port model at home is overkill, and expensive. You're better off using a switch plus an appliance.
If we're casually mentioning alternatives to PC Engines (who provide, AFAIK, the cheapest solution and are using Coreboot):
https://opnsense.org/about/about-opnsense/ developed & supported by Deciso BV. Still supports x86-32, unlike pfSense. Their official hardware is also expensive though, just like Netgate.
If you want a cheap PC Engines alternative running Linux I can recommend Ubiquiti gear instead. The entry level products are cheap (ER-L, for example, has 3 ports and costs ~100 EUR), and the Unifi products are user-friendly, while the EdgeMAX allow more freedom/control at the price of user-friendliness. Although it isn't DIY like the other solutions, that could be an advantage. But if you're going the DIY route anyway, PC Engines gives the best bang for the buck unless you need 4G modem support.
Yeah, that's the cheapest device available. If you only need 2 ports, the apu2c0 is only $99 w/o case (~$10 for the case) [1]. An espressobin (w/o peripherals) goes for $50. EdgeRouter X (1 WAN, 4 LAN ports) goes for 50 EUR here, with SFP module it goes for 75 EUR. Those are competitive prices. Netgate's cheapest device is expensive by comparison. Doesn't say anything about the quality though!
These are full x86 machines that can handle a much different workload than an EdgeRouter X.
Try and do any kind of meaningful traffic shaping, firewalling, with an ERX. Or, just make some configuration change which knocks traffic handling out of silicon and into the CPU: you'll quickly see where the extra money would've been nice.
My only serious complaint about pfSense is that BSD only has single-threaded PPPoE, which really does not work well at anything approaching gigabit speeds.
> These are full x86 machines that can handle a much different workload than an EdgeRouter X.
These are x86-64 (AMD64) machines, running a 64-bit fork of FreeBSD. pfSense doesn't run on x86-32 anymore (even though x86-32 works perfectly fine as router). The argument is that its "ancient" hardware; yet here we are discussing single-threaded PPPoE...
EdgeRouter X is a bit weak, sure (though it also does not use a lot of power), EdgeRouter Lite (which I got) is already much better performance, sports a MIPS64 and more RAM.
> [..] you'll quickly see where the extra money would've been nice.
If I'd want the highest performance + bang of buck + got the time to maintain (including the hardware) I'd go for PC Engines plus OPNsense or Router-7. But I don't want to put a lot of effort/time in maintaining my home router.
I'm positive we'll have loads of fun with RISV-V routers in the near future. If not the big corps, perhaps smaller ones. Amazon also appears to have plans to sell switches with AWS support.
For the record: I run RouterOS in ESXi on a Supermicro SYS-5018D-FN8T (1U, Xeon D1518).
Mikrotik hardware suffers the same problems as Ubiquiti (and most other manufacturers): it's very easy to lose gigabit routing performance with even the most basic of home firewall rulesets.
I got no throughput issues with my Ubiquiti hardware whatsoever as long as I keep using hardware offloading. Which doesn't work with bridging. Either way I use a DMZ on the third port of my ER-L on which I run a WAP for guests which I (or they) fire up when they're here. Latency is very low on these MIPS machines.
There are requirements beyond that including that it's unrelated to anything your employer does. If you work at Google or a similarly broad tech giant, unfortunately that means they probably own the IP of almost anything you could do despite California's otherwise generous law.
I wish the law was worded such that it just had to be unrelated to the work that employee does for the employer, but it was not written with that forsight.
This is true, but the idea of that law is to ensure you don't steal any intellectual property and don't compete with your company, not that you may not code at all under these circumstances.
This is the default law in most states. California and Washington both have the additional law that states the company cannot contractually take away this right where is most other states they can within certain limits. Of course the legality is moot if they decide to throw enough money at it as they will just bankrupt you with legal fees.
From what I understand the burden of proof in Lab. Code, § 2872. falls on the employee. If they work on some code in their "spare" time and that code contains a license with the word "Google" in it it would seem the license is a first attempt to establish the burden of proof as opposed to the freedoms one might enjoy writing under the Expat or GPL, for example.
Edit: forgot to mention, you can use static leases, too, permanently assigning an IP/hostname to a MAC address: just remove the expiration field in /perm/dhcp4d/leases.json :)
Here's a use case that I am currently doing at home:
I have a live weather station [1] that pulls data from a Go-based service that I wrote and run on a server on my home network. When I'm outside of the house, mhkweather.com resolves to a Digital Ocean IP, where I run a proxy to forward requests back to home IP. However, when I'm at home, mhkweather.com resolves to a local 10.x.x.x IP and goes straight the the local service, no proxy. It faster this way. I want to be able to set DNS overrides for local things. I can do this with unbound but it would be nice if dnsd could do this for me.
For the time being, you’d need to modify dnsd to install your custom handler. It should be as simple as adding another server.mux.HandleFunc call in internal/dns/dns.go.
If this turns out to be a feature which many people would like, I’ll think about how to best structure the code to make this easier.
This seems to be a standard feature in many routers now. The computer name sent with the DHCP request will eventually land in the resolver's hosts table and it will respond to DNS requests as expected. It's client-controlled, which is kind of not great, but for home users it works as intended.
What I'd like to see is a home router that lets me put arbitrary rules in. Currently I use dnsmasq as a resolver and it basically does what I need, but I'd rather have it running on the router instead of needing another computer.
My current rule set is roughly:
* Resolve from local hosts file - this is where I put my statically assigned home servers/devices.
* If the domain request matches .domain.tld, forward it to a.b.c.d for resolution - this is a dnsmasq running on a remote box and covers off VMs that I run there. There is an IPSec tunnel (running on a small VM) and static routes on my router that handle getting to that box.
Finally, pass the request onto my router which will pass it onto my ISP or use its local DHCP-driven hosts table to resolve it.
I'd love it if an off-the-shelf router could do this. Currently I'm using a UniFi Security Gateway and it seems like this feature set would be a good fit. The router is professional-enough that advanced features like this make sense.
I have this set up in my house. I have a tld and internally I want the IP to resolve to the land IP and externally, to the ISP ip. I also have local caching for things like imap, but slow uploaded so I don't host my full email and web mail stack there.
I run 2 dns servers usually on the same box on different bind addresses. You'd run a caching resolver on the gw ip and forward requests for my.tld to the second resolver which is configured to be authoritative for my.tld. the hosts can then be configured either through DHCP or statically.
This is a pretty common technique to make enterprises Intranets.
Added benefit is that when you did a file transfer between Lan hosts with the globally accessable DNS, the transfer stays on the LAN. Also ssh host works in you ssh config regardless of location.
Which Ubiquiti gear? I wasn't expecting one component to be on MIPS, one component to be on ARM7 without hardfloat and one component to be ARM7-A with vfp4.
I guess Go's cross compilation makes it easier.
PS: Without hardware offload, the 8 port POE switch only manages 12 megabits / second.
I'm running a similar setup (although without IPsec). My compromise was to have name.int.my.domain resolve internally and name.my.domain resolve externally (easier with ipv6).
dnsmasq should be able to do what you want to do and it can be enabled on the USG - at least from the controller.
So if I ssh into another computer by it's name everything works without me having to maintain a static device table somewhere w/scripts to deal with inevitable versioning issues.
You don’t need split horizon dns as long as the router is running a resolver. Just configure the resolver to use some TLD (eg .local) as a local network so that it looks up any names ending in eg .local in its local /etc/hosts or equivalent.
.local isn't a recommendation any more due to zeroconf.
The "best way" is to use a real domain for your local network. But this can be annoying, costs money and has other issues (renaming/-branding, mergers/splits, domain squatting, accidental record publication and so on ---- it's a lot like renting an internet-routable IP network for LAN use, it's kinda silly and error prone).
Using non-private IP for your local network isn't "silly", it's just that we've grown accustomed to not doing so out of necessity. Reusing the same IP address for multiple hosts is problematic, even when there is an RFC that says it's okay to do so.
For instance, I've got a tinc VPN network setup between many of my hosts. What IP range should I set that to such that it will not clash with whatever access network my laptop roams onto? The proper IP-philosophy answer is to apply for a block, even if it's never announced by BGP.
The pragmatic hack is to attempt to choose the most obscure block that isn't likely to conflict with the public Internet or with how local networks are generally setup. In my case, I chose something NOT from RFC1918 (so shoot me).
It's of course less of a big deal if you do realm / split-horizon based routing. But that is not straightforward to setup, as it's getting away from how IP was designed to be used.
Ugh! It really is a situation of “we (ICANN) took money from people applying for these and don’t have the guts to officially say ‘no these are reserved for private usage, deal with it’ so we’ll call it ‘delay allocating these names indefinitely’”. At least they’re planning on refunding the applicants. And Google has the clout to push for making say .dev closed off via an RFC but instead bought it and runs it for their own (internal) development. It’s like no one has the concept of thinking about working for the public good for small groups or individuals on this modern internet. Tragedy of the commons.
I posted how to do split horizon above. I get it. You don't want to change your ssh config and doing things on lan instead of going through the router interfaces is ideal.
So can you tell me what you think of nft? Details if you feel like it please. I'm a sysadmin type who has stared at way too much iptables and decided to start going full nftables. I haven't hit any showstopper yet but I figure it would be worth asking someone who has dealt with its complexities. I've been putting off Bpf because it feels like actual programming in yet another language. Did you try that?
One last comment, you might check out Dragonfly BSDs network stack if you haven't yet for ideas, the last benchmarks I ran were surprising.
To anyone listening, does Bpf enable us to get past the 10gbs dma issues on a Linux router?
I am currently reading some source codes on tap-based IP/TCP stack and Babel. I have not wrapped my head around some of the concepts because I need to first understand some RFCs . I would be reading your code base soon, but can one fire it up to work on Raspberry Pi3 ? And can it work with something that is not fibre ?
You can fire it up on a Raspberry Pi 3, yes. That’s how I started developing it, to be sure the architecture works before I spend the effort of porting gokrazy to the apu2c4 :)
It can work with any ISP (not just fiber7) as long as you have an ethernet interface and the ISP uses DHCPv4 and DHCPv6 for configuration. If that’s not the case, you’d need to develop e.g. PPPoE support yourself.
Note that you won’t have automated recovery (rtr7-recover), as that uses PXE boot, which Raspberry Pi’s don’t do by default (and rtr7-recovery-init has some hard-coded values which won’t work on the Raspberry Pi, at least not when using an SD card for storage).
Also, 4G is a shared medium, so I prefer my dedicated 1 Gbps line. We should handle internet load on dedicated lines where possible, so that the shared medium keeps working well for people who have no other choice.
