
Terminal Server on a Budget - Jaruzel
https://blog.lasseter.org/posts/2020/07/terminal-server-on-a-budget/
======
katzgrau
> I no longer have that box of serial adaptors and my wife somehow can’t
> remember where it went since we got maried.

I used to have a glorious box of ram, spare drives, and every cable and
adapter imaginable.

My box also mysteriously disappeared some time after I got married. I feel
your pain :)

~~~
Cthulhu_
Sounds like some serious boundary-crossing behaviour; have you ever discussed
that with your spouse? Did some of their stuff "disappear" as well?

I'm just saying, this is not normal or acceptable behaviour.

~~~
halfmatthalfcat
Did I just teleport into /r/relationship_advice?

------
dxld
> One small disadvantage not addressed is that neither of the two motherboards
> support BIOS access via COM1, which is a bummer.

I've always wondered if you could burn SGABIOS
([https://code.google.com/archive/p/sgabios/](https://code.google.com/archive/p/sgabios/))
to a PCI(e) card ROM to get this working on real hardware instead of just in
QEMU.

Apparently someone did try this, so it might just work:
[https://www.flashrom.org/User:GNUtoo/Howto_flash_sgabios_on_...](https://www.flashrom.org/User:GNUtoo/Howto_flash_sgabios_on_an_Nvidia_video_card)

Maybe I should dig up some old flashrom supported NIC card or something.

------
xiconfjs
We are searching for a new serial to ethernet solution which is not as
expensive as most B2B products. Right now this [1] would be our dream product.
We are also looking into a combination of a mini-pc (like PC-Engines [2]) with
two of these miniPCIe serial cards [3].

If you have better recommendations feel free to share them :)

[1] [https://freetserv.github.io/](https://freetserv.github.io/) [2]
[https://www.pcengines.ch/apu2c2.htm](https://www.pcengines.ch/apu2c2.htm) [3]
[https://www.delock.de/produkte/G_95244/merkmale.html](https://www.delock.de/produkte/G_95244/merkmale.html)

~~~
jlgaddis
Old multi-port serial consoles can often be found used quite cheap.

Personally, I'm a big fan of the OpenGear units. They're *much^ more than just
basic serial consoles and are quite extendable. You can even write your own
scripts to interact with them, trigger them on specific events, and so on.

~~~
cure
Yeah +1 for OpenGear console servers, I've used them for years. Looks like
there are some bargains to be had on ebay, I see a 32 port one for $150...

------
nwmcsween
Interesting but why not a kvm rack unit from ebay? Cheaper and easier only
issue is most vendors stop supporting the switches and the software on them
can be complete garbage (only works with firefox version x or IE version y).

~~~
matt-attack
Yeah everyone I’ve used requires java in the browser. Feels like 1998.

~~~
nwmcsween
Later avocents have a sort of usable HTML5 implementation. It would be nice if
there was something like OpenWRT for kvm switches

------
jlgaddis
"conserver" (which the author is hsing) is really handy for quickly and easily
accessing and/or monitoring the (serial) console of not just physical machines
(like the author is doing) but also virtual machines as well.

I've used it with virtual machines running on both vSphere and QEMU/KVM to
make their serial consoles available over the network... to access the serial
console you can just connect to the corresponding TCP port. Just make sure to
lock down / restrict access appropriately and ensure you don't send sensitive
info over the network in clear-text!

It works the same way with physical devices too (Cisco devices, primarily).
It's great when you want to a "permanent record" (on disk) of any output that
is sent to the console.

As mentioned, it's really nice when you need to provide "shared" console
access and/or access to multiple devices and/or users simultaneously. It's
really an under-rated tool.

------
ed25519FUUU
> _more than a few times I have gotten into a situation where I have attempted
> to preform a system update on one of the pair remotely from cafe or sofa and
> the PC failed to come back up after a reboot._

This happened to me as well. In fact it happened so often that I took all of
the critical network software off the home server and run it on an edge
router-x.

Also, setting “power on after power loss” and running the computer through a
smart plug is a good idea. When it gets jammed, you can at least power cycle
it remotely and it _attempts_ to reboot.

------
drcross
Why not use a Raspberry Pi populated with several USB to console adaptors,
there are USB console cables with up to four serial ports per connection which
would result in 16 console ports per Pi.

------
teddyh
Note: Some server motherboards also have a setting in the BIOS to show boot-
time text, and allow BIOS configuration, on a serial port.

~~~
wokkel
Not only server. If you have a Q-series desktop motherboard with AMT you can
enable Serial-Over-Lan and access the machine remotely (and power on/off or
reset it as well). Q series chipsets are also found on Fujitsu/Dell/HP
prebuilts which can be picked up quite cheaply when companies decide to
upgrade. For servers you usually have IPMI/iDrac etc. which uses more power,
but usually has a better (web)interface.

~~~
plett
Yep. I've just picked up a few second hand HP EliteDesk 800 G2 desktops with
the intention of turning them into a homelab for k8s learning. They have AMT
and the web power control, VNC based KVM and SoL work well.

There is no BIOS serial redirection, so SoL is no use until the OS starts
booting, and grub has a bug[0][1][2] which means it can't find the AMT SoL
when booting in EFI mode. But SoL works fine as soon as the kernel starts, and
the KVM VNC works fine during BIOS and GRUB if I have an emergency.

One gotcha which would catch someone out who wasn't expecting it - the onboard
video disables itself when no monitor is connected, so the AMT KVM just shows
a black screen. This can be solved by DisplayPort or VGA "ghost" dongle in the
back of the machine to talk EDID and pretend to be a screen.

I'm a big fan of conserver, and intend to use it in this project to spawn
amtterm processes to connect to the SoL ports.

0:
[https://savannah.gnu.org/bugs/?42026](https://savannah.gnu.org/bugs/?42026)

1: [https://community.intel.com/t5/Intel-vPro-Platform/No-AMT-
se...](https://community.intel.com/t5/Intel-vPro-Platform/No-AMT-serial-port-
in-Linux-grub2/td-p/720528)

2:
[https://wiki.networksecuritytoolkit.org/index.php/HowTo_Head...](https://wiki.networksecuritytoolkit.org/index.php/HowTo_Headless_Intel_NUC_vPro_AMT#AMT_Serial_Port)

------
cl3misch
Maybe I missed it, but what device is on the other end of the COM connection?
His router?

What's the reason against SSH'ing from the gateway to the servers instead of
the physical COM connection?

~~~
jlgaddis
> _Maybe I missed it, but what device is on the other end of the COM
> connection? His router?_

The plan was to use a Raspberry Pi to access the two PCs (via USB-to-serial
adapters), though he hasn't made it to that point yet ("I have not used it for
this project yet, but it is still in my plans."). Right now, the two PCs are
just connected to each other, back-to-back.

> _What 's the reason against SSH'ing from the gateway to the servers instead
> of the physical COM connection?_

I would assume that he _does_ just use SSH -- under normal circumstances.

It's when SSH access isn't available that access via the serial console would
be used -- such as the network connection being down or b0rked, when the
kernel (or required) modules failed to boot/build/load properly, a bad
firewall change has locked you out, and so on. This method even provides
access to the bootloader.

If your BIOS supports "console redirection" (to a serial port), you also gain
access to the BIOS, can watch the startup (POST) process, or even change the
boot device -- in order to fire off a kickstart installation, for example.

He mentioned that his PCs don't have ILO/DRAC/IPMI, so this is his backup
solution. -- remember that his machines are headless.

------
twicetwice
This is super cool! I'd love to get this working for my home servers as well
one day (priority number one is making them accessible outside my LAN again).

~~~
jjice
I'm currently using a Digital Ocean droplet with wireguard and my home server
connects to that VPS via wireguard site-to-site. I have Nginx on the VPS and
do a proxy_pass to my local services. Just set it up a week ago and works
great, very simple once you understand the setup.

~~~
wahern
Wireguard requires a more complex setup than for IPSec, believe it or not.

To give my home router a static IP address, I use a Vultr VPS and pay for an
additional IP address. I then use IPSec encapsulation to forward traffic for
the second IP address directly to the router, which initiates the IPSec SA
setup. The router has a loopback device configured with the static IP address;
the routable static address is effectively local, even from the kernel's
perspective. The IPSec rules handle forwarding of everything, so I don't even
need packet filtering (e.g. OpenBSD PF in my case) rules.

No need to fiddle with private address ranges, NAT'ing, reverse proxying, or
packet filtering of any sort. The IKEv2 setup is a single line on the VPS and
a single line on the router, though OpenBSD's OpenIKED (as well as their
isakmpd fork for IKEv1) is awesome that way; StrongSwan and other Linux IKE
daemons require a more verbose configuration on account of their key-value
pair syntax, but in any event it's still relatively simple.

~~~
twicetwice
Unfortunately I can't fiddle with the router I'm behind, so I think the
wireguard route is the way to go. I also like the wireguard approach because
as far as I can tell it should be completely portable -- if I move to be
behind another router/NAT, I can just plug in my box (it's a laptop with a
broken screen), run my dynamic DNS update script (I'm using DuckDNS) and I'll
be off to the races, as I understand it.

~~~
wahern
I just happen to put my ISP's (Sonic's) router in passive mode and use a PC
Engines APU2 running OpenBSD for NAT'ing my home network, but the same IPSec
setup works even if the termination point is behind a NAT gateway. Sonic
doesn't offer static IP addresses, so the VPS could never directly connect to
my router unless I fiddled with dynamic DNS or scraped together some other
hack, which IMO is far more hassle than it's worth. Also, I've used this exact
same setup before except with the termination point inside a corporate LAN,
tucked away on someone's desk. IPSec+IKE has done NAT traversal and dead peer
detection for many, many years, notwithstanding that some IKE implementations
have sucked at it.

Wireguard is just a better wire protocol for moving packets, but IMO it's
mostly a solution in search of a problem. Most of the complexity that people
disdain about IPSec actually comes from IKE, but it's largely irreducible
complexity. IKE is what initiates and negotiates security associations based
on abstract flow rules--in this case 0.0.0.0/0 <-> W.X.Y.Z, where the latter
is the static, routable address. Wireguard doesn't diminish the need for this
layer, which is why you've been forced to hack a complete solution using a
reverse proxy. IKE is far more mature than the experimental additional
software Wireguard developers have been cooking up to automate tasks like this
(abstract flows, key management, etc), and definitely more mature than the
homebrew solutions people put together.

I bring this up because I think IPSec has gotten an undeserved bad rap. If
people make do with Wireguard + hacks, great, but this particular scenario is
a great example that shows the strengths of IPSec+IKE and the weakness of
Wireguard; a reality people don't hear about given the unabashed enthusiasm
for Wireguard.

------
tyingq
A fair amount of PC's already have remote access built in, via Intel's vPro.
Might be worth checking for that if you're on a budget.

------
Harvesterify
So, why not a KVM ?

------
xupybd
I would love be to know what the family is using the servers for.

~~~
dredmorbius
_I run a number of services in my home covering all the essentials such as
media serving, DNS, local cloud, PBX, Minecraft containers, etc._

'graph 2 of TFA.

~~~
sushshshsh
Is it really a time-savings to say that instead of "Paragraph 2 of the
article"?

~~~
xupybd
Ah missed that, thanks for pointing that out.

------
gatvol
Why not use a jumpbox /ssh ?

~~~
iso947
Because when the box doesn’t boot you can’t ssh in.

