This looks like one of those home server infrastructure setup rabbit holes I would've bailed out of within the first 10 minutes of Googling to research the problem.
I commend the author for their perseverance and the writeup - rabbit hole writeups are their own form of entertainment. That alone was worth it. I could never run anything that looks like such a conceptual pain the butt. A pain to maintain and troubleshoot to boot.
Especially since the setup requires a Raspberry Pi to run permanently. A Raspberry Pi that itself would work just as fine as the Time Machine server the author seems to need on demand?
I agree, but you could argue that the Raspberry Pi is not performant enough during restores or browsing the backups. During the backup itself performance should not have to be that big of an issue as it's mostly a background operation you will not be waiting on.
You could also use the RPi as a stepping stone which syncs it's files to the 'real' backup server which is on a daily/weekly timer instead of on-demand setup.
I'm using this concept for the problem that I want to collect metrics all day long, but not have a power hungry server running 24/7. So my RPi collects all metrics and streams then directly to my server for me to view realtime. But whenever the server is unavailable (on schedule or for maintenance) the RPi just buffers the metrics until the server is available again.
Also worth considering is just not using a single server for a single purpose but just cramming it full with utility to justify the power costs.
Nothing in this article supports this statement. It just says "high power consumption" without measuring, and declares victory without measuring the power saved. The fact is that power scaling has almost erased the difference between "suspended" and just "idle", to such an extent that there might not even be a meaningful difference, or any measurable difference.
My NUC 12 Pro is drawing, at the wall, 2.7W, as best as I can measure. The SoC reports drawing below 200mW. This server is on. The way to get low power states on these is to just make sure your Linux is up to the task and your BIOS works correctly, which is one of the reasons to choose a NUC. The first-party firmware works!
Another important thing for low-power server operation is to either use wifi, or make sure your ethernet is configured correctly for low-power operation. Unless you fix it, gigabit ethernet by default will propagate the link state all the way down to a high power state on the CPU. The NIC will say that it has tight latency requirements because the link is active, and it will configure the PCIe bus to avoid PCI link power management, and it will tell the CPU that it can't go below PC6. To avoid this you have to configure your NIC to ignore link latency requirements. Just unplugging the ethernet is the easier path, and it's faster than gigabit these days.
NOte that the NUCs with power saving and wifi have a serious problem: latency. By default in the various Linux OSen I use, wifi power management is enabled. Often times, repeated pings will jump from 5ms to 100+ms (with a very high variance) and network bandwidth will be extremely poor. Even ssh on a LAN can be unusable!
My first thought too. Reports can show the power usage for each process. So reduce the power-hungry processes. Or schedule them for lower cost times of day.
Then how about an N95/N100 (recent Celeron successor) mini PC, like the ones from Beelink? They don’t draw that much power, but are 2-3x faster than a Raspi, have HW supported video de/encoding and can even run Windows if you needed that.
The N100 has a TDP of 6W. I haven't measured power draw of the whole system yet though, I imagine that the mainboard will pull another couple watts at least.
It shines especially in scenarios where the Pi is just a tad too slow. E.g. you can't live transcode 4k videos on a Pi, while the N100 has latest UHD graphics and therefore ships with a decent HW-accelerated H265 en/decoder.
Sure there are, there are lots of NAS appliances boxes from the likes of Synology/QNAP/Aliexpress Special that can run double duty as servers built on similar low power CPUs. Some of them even have decent networking options.
The problem is that a lot of the commercial products are woefully underspecc'ed for the price and you outgrow them quickly. Spinning disks also eat a lot of power, relatively speaking, so they kind of eat into the "low power" usecase.
You should actually look at what's available. You can get a QNAP box with an ARM CPU with lots of drive slots, dual 2.5 gig Ethernet and dual 10 gig SFP+ for like $700, among many more examples. Sure saturating those is gonna eat a ton of cycles and run like shit but... yeah of course it is it's a cheap low power CPU. They suck for the sorts of stuff I wanna do, but this is inherently a market segment that involves compromise and that is probably perfect for some people.
I agree I'd much rather roll my own system (and that's what I did) and I would personally never buy one of those, but it's simply not right to say there are no options available if you want a low power server off the shelf. Some of them are even decent if you wanna pay for it.
It does if you want more than a storage appliance. I want the option of setting up any linux server software on it if i feel the need. Just a general purpose low power computer.
> This looks like one of those home server infrastructure setup rabbit holes I would've bailed out of within the first 10 minutes of Googling to research the problem.
Absolutely have felt the same often, in the past. Good news! There's ChatGPT now!
It's so much quicker than google for figuring out approaches; it's great for explaining concepts and "how to do x with tech". And it's great at writing the scripts for you too. I set up a prometheus/grafana dashboard with scripts for various temperatures from various APIs with its help.
Not trying to shill, but just sharing that this type of project is something that's a lot more feasible to implement in limited timeframes than it used to be. (and get the satisfaction of bringing the idea to fruition)
But I reckon I'll always want content like this that covers the journey of uncovering various edge-cases, too...
This seems like an example of the worst setup for ChatGPT.
There isn't much info on the internet about it. Of the info that exists most is inconsistent, outdated, and/or wrong. The specific instructions need to depend on what's supported by the exact model number of your network card. You're dealing with hardware, so there's a chance of breaking things permanently. Close has zero value, the solution needs to be exactly right. Close but incorrect solutions are hard to debug.
I didn't in-depth read the main article here. But skimmed most of it.
Not sure any of it is going to brick hardware, but I haven't played with setting flags with ethtool. I've messed with Wake-on-Lan stuff in the past, as well as "prevent device from randomly waking from sleep" issues, too (on Windows, though).
I do have the advantage of a lot of past experience with Atmegas then RPis, so know a lot of the common sense hardware considerations... and a lot of troubleshooting. For me getting 90% correct with ChatGPT, and building up incrementally, has been an improvement on doing it all myself with google.
I might take an attempt at asking chatGPT how to do what the author did. Could be informative, even if just introspective realizations like, "oh, this is a question I asked in a good way due to my existing knowledge."
So far my experience has been the opposite with ChapGPT FWIW. Overall it has cost me more time sending me down rabbit holes that it totally made up, but said with absolute confidence.
A key caveat in much of what I'm doing, is I'm implementing things I previously did for other projects in the past 15 years... so I'm using old technology, know the terminology, and can troubleshoot 90% correct stuff that chatGPT gives me.
I haven't done as much "I have no idea where to start" stuff.
I have, and while it’s not perfect I’ve found that it makes a great tutor. I’ve asked it a lot of programming questions where my starting point was 0, and my biggest takeaway is it will give you exactly what you ask for and nothing else.
This can be a bit of a problem since you can’t ask for what you don’t know, so starting broad by having it explain the general concepts to you, and then slowly getting more specific with direct pointed questions had the best outcomes.
I came at it too from a 'I've done this for more than a decade' perspective as well. I simply found that it's advice was already something I knew, or outright wrong, or wrong but very convincing that it could work.
ChatGPT can't even do sensible NGINX configs, despite feeding it documentation. Gets URL rewrites wrong et al, hallucinates things that aren't even there. Troubleshooting its output is more labour-intensive than just doing it right in the first place, but most folks' Dunning-Kruger complex prevents them from knowing when the output is wrong in the first place.
Meh, I struggled to get my pi to work as a time machine server. It would work for a few weeks and then just stop working and I would have to start the backup from scratch previous backups becoming unusable.
In the end I just plugged an external hard disk into the mac.
I have had similar issues in the past even with such simple things as SSH to a Mac. It would be easy to blame the pi or Linux, but as far as I can tell, Apple just does not play nicely with anything not Apple. It's just that little bit different. I have no such problems interfacing Linux and the pi with anything else.
Time Machine over NAS has always been unstable for me. As you experienced, my backup would be regularly corrupted and would need to restart. Alternatively, a normal <10G backup could take in excess of 24h.
I eventually gave up on the NAS and now backup to external drives at home.
Time Machine has improved a lot after the switch to APFS. Mostly because it supports full snapshots, with an easy way to get the difference between two snapshots.
The previous HFS+ version relied on directory hardlinks that were notoriously buggy and easy to corrupt. The difference computation also required basically recursively walking the entire tree and comparing the directory change times, with each lookup requiring a network round-trip.
I had the exact same problem and ended up with the same solution. I know people who have had somewhat better experiences backing up to something like a Synology, but the $Xk investment wasn't worth it for me without also needing it for big file storage or media serving or whatever.
My time machine share is backed by ZFS. I have a separate dataset for each laptop, with snapshots retained for 30 days. As long as I know roughly the last time the backup worked, I can restore that snapshot so I don't need to start fresh.
I had the same - ZFS across four drives with separate dataset and samba shares for each machine in my house. It would work and yes, I could rollback to a working snapshot. But it wasn't a set and forget system.
There are some quirks to setting up SMB so Time Machine is happy, but I have had great success with ZFS+SMB hosting my TM backups. Have been running for a little over a month and nothing is corrupt. Self hosting is still somewhat hard, I don't blame you for bailing on something so critical to uptime in the first place.
Yes but RPI 4 has usb 3.0 ports that you'd really struggle to saturate with a gigabit connection - I used to see the lack of sata as a major problem with that platform but nowadays it really doesn't matter.
On Rpi 4 the USB 3.0 ports get their own dedicated USB 3.0 controller that's connected through a PCIe 3.0 1x bus. They don't share bandwidth with anything else.
But then, why not just use an ESP32 which can replace the Pi entirely in the proposed setup. ;-)
I would have an ESP32 impersonate the MAC address of the server passively (ie. only in the Wi-Fi association), then “push the wake button” if it found traffic that is allowed to wake the server. A wake-up firewall, if you will.
Personally I like having courage to try. I don't tend to see things as requiring maintenance. This looks like an afternoon of messing around and should just work. I don't expect troubleshooting to be hard.
I really love people who can show courage & interest. I feel like computing has decayed into vapid consumerism. Products no longer even try to be semi-technical. There's a whole culture built around telling & selling ourselves the idea that computers are hard & difficult.
All it takes is one person with a good blog post to demystify & make accessible. You are smart enough. You can be that person. You can Augment Human Intellect by talking through it.
Nice to know I'm not alone. If I could reclaim time spent on rabbit hole projects that concluded with bail out, I feel I'd have a whole other lifetime. Yet somehow, each new rabbit hole project lures me in.
My solution to the "10 years old PC used as a backup target" was to just donate/sell-off-as-parts the old PC, and replace with a 10 year old laptop, with an external USB enclosure. From 180-200 Watts down to 10-25, but it was not as much fun with two weeks of a rabbit whole digging :)
> Especially since the setup requires a Raspberry Pi to run permanently.
Which they'll gladly do no?
If (poor quality) SD card corruption is a concern, you can have the RPi mount the SD card read-only or just keep a full backup of the SD card.
I've had a RPi 1 power 4 VoIP Cisco phones for years. Zero issue.
I've got another one constantly establishing a remote SSH-tunnel so that I can easily troubleshoot my brother, who lives in another country, 's LAN. The uptime is more than a year.
One is routing (well, doing SNAT really) and firewalling between two non routable LANs. I set up the thing once and it's so reliable there's neither a screen nor a monitor hooked to it: just the power adapter and two ethernet cables (including one in a USB-to-ethernet adapter).
Another one is running unbound. Same: no monitor, no keyboard, no nothing. The power adapter and one ethernet cable. Simple. Not much can go wrong. No moving parts. It's happily null routing hundreds of thousands of crappy domain names 24/7.
I've got another one in a vintage arcade cab with a Pi2JAMMA adapter. In years it never failed to boot once. As we the others, just in case, I still have a "dd" of the entire SD card. Should something happen I'd just dd the image to a new SD card.
The shortage of RPi is apparently due to companies snapping them all for industrial uses.
I haven't heard of many people complaining "my RPi died overnight" (unlike, say, the Apple M1 laptop's screens).
They're incredibly reliable pieces of equipment for their gentle price. In case something goes wrong on one of mine, I just have a dump of the entire SD card.
I don't disagree that they may be a pain in the butt to set up, just like any server. But once setup, they don't move under your feet so the maintenance is really minimal.
I hate that there's a shortage of Pi but I'm not sure that criticizing them as "non reliable" is a criticism made in good faith.
I'm not sure OP was necessarily implying that the Pi specifically wasn't suitable for the job -- perhaps just that requiring /any/ other additional computer to achieve this is not without additional administrative overhead and power costs.
It does seems likely that a Pi is going to consume a fair bit less energy than the x86 SFF PC, so it's probably a net gain there, but that admin overhead thing isn't necessarily one to ignore -- it's not entirely set and forget.
Man I loved my RPi remote SSH-tunnel for being able to access my home network from anywhere... until my kid needed a Pi for a robot he got for Christmas and I decided to donate mine to the cause :)
If an idling raspberry pi is too power hungry, a microcontroller like an ESP32 could be made to respond to ARPs. $8 for the MCU and a couple of afternoons of noodling. It's not going to run avahi for you though.
I had a somewhat similar setup and it worked pretty good as long as I needed it.
A Raspi, which would also do other stuff, like provide the DNS service, would wake up a backup server, which was a Xeon machine and consumed a lot of power, also was running Windows. The Raspi would wake it up 4 times a day, the server had a Task set up to run some Python scripts which invoked robocopy so that the server pulled the data off of another machine which was always on and required to be backed up. When ready it shut itself down and all was good.
went down that route years and years ago and... it heats up the closet.
Actually, (though not a server) I remember one place I lived, I just put my monitor, keyboard and mouse in the bedroom at a nice desk. Then I ran the cables through the wall to the computer in the next room where it could be loud and cool. (quiet, fast, cool: pick two)
i run an rpi to host stuff for cheap (the energy drain of the pi is low), my main machine goes to sleep after 15min to save costs. seemes like a neat trick to remotely wake it up
I think it’s not too bad. Of course, I knew most of the ingredients before reading this article. I think my Router OS router can also do the ARP thing.
Disk have power modes and can be spun down to save energy. A Raspberry Pi can run the OS on an SD card and mount the disk for backup storage. Leaving it in low power for a long period of time.
I'm worried about the wear on my drives spinning up and down so much? Maybe it isn't such a big deal. It is the most stressful time for spinning drives, though.
This feels like the sort of functionality that ought to just work now. Its a clearly useful thing for PCs all over the planet and it would be nice if they just went to sleep almost completely when idle and woke up on network or USB/PS2 input. The power savings would be enormous.
Yet today I still have issues with power saving features, performance is lowered by having power saving on instead of flat out and various things don't work like Keepass looses connection to my browser for my passwords and one of the USB devices that came on the motherboard crashes the entire computer on some power transitions. Its taking an awful long time for power saving features to mature.
I would love to have this for my NAS/server, it has a lot of performance for when its needed but its also idle a lot. This is not the way.
The main problem is defining what it means to be idle.
Suppose you have some browser tabs open, so there is some javascript running to switch out the ads on the page every few seconds. Is that "idle"? There's CPU and network activity. Suppose there is an open ssh connection to a server. It sends keepalives because the client is remote and behind a NAT, but there is no "real" activity. Should the server go to sleep?
It's entirely possible to be too aggressive. Samsung is proving this. They've disable "unused" apps by default. But "unused" just means you haven't opened it in a while, not that you e.g. don't have a reminder set in it to give you an alert for something which is happening in six months.
Conversely, I wish web browsers were more aggressive about denying background tabs any use of the CPU whatsoever by default, and then allowing the user to make exceptions in the rare case where that's needed.
But in general the way to do this on the hardware side is per-component, and that's largely already happening. Modern CPUs and other devices at idle shut down most of the silicon and use hardly any power. I've seen Core i7 desktops use 6 watts at idle. In that case you hardly even need suspend, what you really need is for the hardware makers to find what's using the last 6 watts and get it down below 1.
Conversely, I've seen server hardware that uses over 10 watts when it's "off".
I think Chrome added something like this recently. It "hibernates" the tabs, but you can opt out. The only downside is that I think it loses the tabs' state.
It's not really worth it, even though I'd commend the OP for wading through the WoL stack. I have a NAS that's running 8 drives and a modern Intel CPU and it runs around 40w from the wall under load, and idles <10W with the drive spun down and CPU not doing anything. The WoL setup made more sense 5-10 years ago when power save functionality was much more primitive and drives were way less efficient.
It's also a solved problem for the vast majority of the population. You pay Apple or Google a few dollars a month and they handle backups for you. Those servers aren't sitting idle 99% of the time waiting for you to backup so there is no wasted power. Self hosting is fun but not the most practical option. And now iCloud has end to end encryption, there is little reason to bother doing it yourself.
This is only because the vast majority of the population agrees that it’s a reasonable tradeoff to not bother with all the digital privacy and security stuff beyond what the defaults are and just go on with their lives. The reason behind this is that almost no one is significantly negatively affected by not being vigilant and self-hosting everything. The downside is that things like PRISM can exist and nation-states and any sufficiently powerful or skillful interested party can and do far more easily target, surveil and harm any individual they want. Again, those individuals are not your average Joe so the public doesn’t care. As long as you don’t leave the herd and stay in line, you’re safe. Personally, I didn’t give any f… about privacy until I felt there’s a good possibility US agencies can target me, due to things I would start to talk about online. Those things were not illegal neither in US nor in my country, just different ideologies from the status quo of modernity, but I knew for a fact people have been (illegally) arrested by US agencies before due to voicing similar things.[0]
I just let my computer run all the time. Disable all sleep functionality except monitor sleep. Then it always works when I need it to.
My life is way too short to fool around with stuff like this to save $5 in electricity and have to deal with the inevitiable "won't wake up" or "wakes up in a brain-damaged state" situations.
It is not as hard, but requires a device in the middle to send the magic packets from time to time. Using my home desktop like this for years, 10 years ago it was a native feature of my tomato based router back then. Today it is with a small script that sshes into the openwrt router and sends the magic packet to the PC, but very similar in terms of UX.
I'm tempted to do this as well, given the likelihood of everything working if/when my linux desktop (or laptop) actually wakes up is about 50/50 these days.
I do this too. I'm on a Mac and use an app called "Amphetamine" to keep the machine active (it's the successor to "Caffeine" IIRC).
However, I dim the screen to black when I go to sleep, since it's the only component that can't really "idle" - it's consuming power as long as it's displaying pixels, and thus it would also be the first component to die from overuse even while "idling."
This reminds me of back when Apple sold WiFi access points (they were high-quality, but a pain to administer since they required running a Windows-only or a MacOS-only executable). I tried running arpwatch for a while on a network which had one of these, and gave up, since the MAC address for any Apple machine on the network was constantly bouncing between the real machine and the access point. Apparently, whenever an Apple desktop went to sleep, that WiFi access point proxied ARP and mDNS requests for it, much like what's described on this post (but it did that by default, with no configuration required to enable it).
They were high-quality. I used one for years, even had a hard drive attached for a while for Time Machine backups, which worked well.
I had to stop using it for some reason--it wasn't getting updates, or newer WiFi cards weren't working well with it, something--and everything I've used since has been disappointing. Even at the price point of the AirPort, they all have various issues. I wish this technology was a done thing.
Another option, if your needs are relatively modest, is to choose a low power mini PC. The one I chose idles at around 5W under Linux before any tweaking. It is based on the Intel N5105. I have a few 2.5” external HDDs attached for backup which are configured to sleep after a short time.
I'm curious if you have measured power at the well (e.g. smart plug with energy metering or Kill-a-watt.) I just set up a test server with a J1900 (10W TDP.) With 8GB RAM and running from an SSD it uses 25W. Add 4 7200RPM HDDs and it's up to 65W idling.
I have another test server based on a Raspberry Pi 4B. The Pi boots from SD card and uses about 5W. Add 2 drives in a USB drive bay and power for the setup is now up to 25W.
> I just set up a test server with a J1900 (10W TDP.) With 8GB RAM and running from an SSD it uses 25W.
Did you use a regular desktop power supply? PC power supplies can consume 10-15W or more at idle. To build a low power system you need to find a good low power, high efficiency power supply.
Does anyone know a good supplier for such supplies?
For 10 years my home server used a Corsair VS350; it was fantastic, and very efficient at low power draw. Recently the fan failed. The PSU seems discontinued. The _fan_ seems discontinued. I'm going to try replacing the fan but I don't know much about electronics. As a temporary measure, I got a regular "gaming PC" PSU and it has 10% more power draw, it's not efficient at all.
I've been looking for a replacement PSU, and I can only really find very high power ones - minimum about 550W - for reasonable prices. While you can run a low wattage PC on such PSUs, none of them make very much in the way of efficiency promises; only the highest certification, 80 Plus Titanium, makes efficiency promises for 10% load, e.g. it'll be 90% efficient for a 55W load on a 550W supply. Lower certifications only promise to be e.g. 80% efficient for a 110W load on a 550W supply, and make no promises at all about a 55W or 30W load.
So, how does one buy a good low power, high efficiency PSU, without it costing a fortune?
2.5" hard disks actually use way less power than the 3.5" drives, but if you spin down the bigger drives they get down to about 1watt.
My server at home I actually use a 2.5" drive for the torrent scratch disk since I don't really need speed there I just need something that is always spun up with a moderate amount of space that doesn't use much power.
Running powertop --auto-tune and TLP makes a lot of difference on my server (60->30W). You can further tune CPU, disks and GPU to preserve energy. I suggest starting with a laptop or a phone (~postmarketos)instea d,as those are already tuned for low-energy modes, and you won't have, say, a BIOS preventing PCI devices from sleeping (looks at server).
You could also use postmarketos, mobian, etc. Have a look at the pmOS wiki for device support, most support USB OTG, and usually you don't need GPU acceleration nor screen/audio/cameras, which are usual pain points.
Don't forget to turn on option "Don't turn screen off when charging". Use a less powerful charger to power the smartphone. 5 Watt charger is enough. 2.5 Watt may work too.
8th gen intel and above cpus are quite perfect for this, they often have hyperthreading and vPRO built into an enterprise/military grade box for hundreds and have near perfect linux support out of the box, whether it's Lenovo, HP, Dell, etc.
I've just bought a Pi for some always on stuff (like Home Assistant). I wanted to buy a mini-PC but I found it really hard to identify which ones would have low idle power. Did you find a good reference for this?
A lot of times I want to build something on my homeserver and N5105 ain’t gonna cut it. I don’t build every day or every hour but if I want to I would have to turn it on and off every time.
Honestly, this is the right answer. I have a HP ProDesk 300 G3 mini (the mini version of the server the OP is using) running home assistant, syncthing, a bunch of other random 24x7 stuff, and the "server", router, and PoE switch powering two Wifi APs is drawing a grand total of 9 watts*. It just doesn't seem worth the effort to try and make this more efficient.
PCIe switch chips can be used to multiplex many devices on to a few lanes already, today.
There isn’t much market demand for an extremely low power chip with excessive PCIe IO and a ton of RAM because most applications with high RAM and IO requirements are going to be very power intensive.
It’s already possible to build an Intel system with plenty of RAM and IO where the CPU doesn’t take much power.
For building a high capacity NAS you don’t need the throughput of PCIe for all of your drives. Cheap SATA SSDs can easily saturate your network link when used in parallel.
Diagrams are just visible enough that I can tell that they're present but I cannot otherwise see them. I suppose that's because I'm using a dark theme on Firefox.
Not that it matters. There's enough text to figure out what's going on in the first place and it would not help me much in any case.
I have a file server running of an ancient server motherboard from another age. (Supermicro X8SIL w/X3460 and 16GB ECC RAM) It uses just about 95W idle and bumps to about 150W when the drives (5 HDDs) are being scrubbed. I tried suspending the server and power dropped to about 75W - not a huge saving.
Another concern is drive longevity. Starting and stopping repeatedly during the day is a lot harder on them than running continuously and could lead to premature drive failure. I could explore this if my server were SSD based but it is not.
> I tried suspending the server and power dropped to about 75W - not a huge saving.
This doesn’t sound like it’s actually suspending. Or it may have left the drives spinning. Server gear usually isn’t optimized for actually sleeping. Workstation boards are much more likely to work.
I'm not using any dark theme addons or similar, just the standard prefers-color-scheme, and I have the same problem. So they just didn't account for their website's dark mode when writing the article.
I wrote it a bit further down, but another solution is to use SVG's with stroke='currentColor' instead. This value uses the inherited text-color and would therefore always work.
Windows dark theme here and I had the same problem.
I clicked the light mode on the website and then back to dark mode and it was fixed. So their website is reading your theme and adjusting the background correctly but fails to adjust the test and diagrams.
Sometimes servers don't actually go into deep suspend modes because of proprietary management interfaces. Disabling those in the BIOS can result in huge savings.
He is inverting the embedded SVG's colors using CSS. However, that CSS is JavaScript applied and doesn't trigger on page load. SVG offers a nice 'currentColor' color value, which resolves to the inherited text-color. Replacing stroke='#000000' with stroke='currentColor' would solve this bug without requiring any JS.
That sounds like a lot. I have a HP Z440 with a Xeon e5-2650v4 (24 cores) with 128GB RAM, 2x8TB spinning rust, 1x PCIe SSD, and another SATA SSD that’s idling at 70W. When it’s scrubbing the drives, it goes to around 100W. Running proxmox.
The machine without most of the drives and 32GB RAM cost me about 250$ used.
It's his CPU (X3460). They are quite ancient and were known for being power hungry (Nehalem). Even a move up to a $15 Broadwell chip would improve energy efficiency dramatically. The motherboard probably doesn't even support sleep states like S2/S3.
> Disable IPv6: this approach relies on ARP, which IPv6 doesn't use
This makes me kind of sad. Even if it’s just for a private/local usage. IPv6 has the features of ARP built in via “neighborhood discovery protocol” (NDP) with the addition of various security to prevent spoofing.
You can but the issue is that if the caller _thinks_ it can reach it over IPv6 it'll never send the IPv4 unicast packet which means the server won't wake.
Very nice, I didn't know that you could wake computers with unicast packets.
For my use case, I run a Raspberry Pi that is on the same local network as my PC. When I connect to the Pi to forward my PC's RDP port to my laptop's loopback interface, the Pi sends a normal magic packet to wake my PC. All that was needed was an entry like this in my laptop's .ssh/config:
Host remote-desktop
HostName raspberry-pi
User pi
Port 1234
IdentityFile ~/.ssh/raspberry-pi-key
LocalForward 3389 my-pc.local:3389
SessionType default
RemoteCommand wakeonlan -i 192.168.178.255 11:22:33:44:55:66; cat
Each time the RDP port is forwarded, the wakeonlan command is run on the Raspberry Pi. The 'cat' at the end keeps the command from exiting so that the RDP port keeps being forwarded. (I also configured the NOPASSWD option for wakeonlan in /etc/sudoers so that no root privileges are needed to run it.)
There's also the "exec" keyword that lets you run a command in a "Match" block like so:
Match host my-pc.local exec "wakeonlan -i 192.168.178.255 11:22:33:44:55:66"
This doesn't cover the case when the connecting machine is in another network but it does avoid a hop. Perhaps you could also run another SSH command in the exec condition that connects to the Pi and has it send the WoL packet.
Especially when the reality is… you can almost go IPv6 only nowadays if you wanted to.
I went down the rabbit hole recently, switching my network to IPv6 primary with IPv4 as the fallback. The ultimate test was disabling IPv4 for a weekend to see what, if anything, broke.
I had set up DNS64, NAT64 and 464XLAT. The only weirdness is how Windows clients handle IPv6 literals in UNC paths, which is super ugly, and how some applications (like Discord calls) will actually embed IPv4 literals. Discord apparently does that for the relay servers for calls.
Those things — and the rare website not supporting it - aside, I could actually be IPv6 only. I have IPv4 enabled as a fallback now, but it’s no longer primary on my network.
464XLAT should work fine with ipv4 literals, no? At least on macOS, this will get routed to a local 192.0.0.2 interface, which does the CLAT, translates it to an ipv6 64:ff9b::<ipv4> address, and relays it to your nat64 server. The ipv4-only software doesn't know any different, and the only traffic going on your LAN is ipv6.
I'm not sure if windows works the same way though...
Discord calls on Windows was the only exception I ran into, because as you found yourself, it functionally had to fallback on DNS64. In general, some peer-to-peer situations are the likely one of the few cases where it will end up falling back onto DNS64 for resolution. This isn’t something I entirely realized myself — I’m apparently far from the only person to discover this behavior (Discord embedding IPv4 literals for its relay servers). Discord has had tickets open about it for years so far.
I appreciate my son being patient on that one, but he appreciated how seriously I dug into everything. Again, we have IPv4 enabled again as fallback, but the family agreed for an IPv6-only weekend as a test, and that was the only thing (outside of one website) that failed.
Everything else worked perfectly, including tons of legacy devices and software, some of which had no concept of IPv6.
For 464XLAT on clients, phones are actually the leaders here. It’s worked perfectly on at least iOS (and I assume Android) for a LONG time because of its built-in automatic tunneling. Mac OS had some recent improvements in Ventura to make things easier. Windows absolutely has some quirks, the biggest being IPv6 literals in UNC paths ending up using a domain Microsoft doesn’t actually own - a potential huge future attack vector.
Again, I disabled IPv4 purely for testing reasons, to guarantee nothing was using IPv4 without me knowing it. There’s no reason to actually disable IPv4 as a fallback.
Over 99.999% of traffic through my home network is IPv6 now. The tiny remainder that has to use IPv4 does, without issue. 5 months on, and nary a complaint. Just works.
The world is ready for IPv6 as your primary, with IPv4 as the fallback.
Not really. When I worked in a networking company, we've observed that connections between hosts via IPv6 were often worse than with IPv4. By connection, I mean between two hops on route to your destination.
I understand that. I'm just saying that the people who don't want to try IPv6 aren't motivated by that. For many the calculus is simply functionality divided by effort. They're not configuring IPv6 because it's hard or because it doesn't work. They're not configuring IPv6 because they already use IPv4 and it still works for them.
I’m behind CGNAT for the last 1.5 years without issue. What am I missing? I actually prefer my router not being bombarded by connection attempts all day.
Try connecting to an SSH server for more than a few hours without passing traffic and then have the server be the one to send a message. Oops! Your ISP tore down the NAT association and you have no idea the server isn't sending anything until you try to communicate with the server and get a timeout / RST.
NAT breaks TCP, but at least with consumer NAT you're in control of the timeouts on your router. With CGNAT you're at the mercy of an ISP that likely optimizes for HTTP and has low timeouts that you can't control.
I actually used to have that issue years ago at work. To work around that I just enabled a keepalive (ServerAliveInterval maybe?) setting in my ssh config. I don’t connect to any ssh servers outside my house for long periods of time, so I haven’t encountered that. Thanks for the heads up, good info!
OpenSSH does enable TCP keepalives by default so that it can detect and close dead connections, but the keepalive interval is far too high to work around bad NATs.
Kind of related to the OP, I spent a decent amount of time trying to have my SSH sessions survive a sleep on Windows. With keepalive disabled, proper Wi-Fi adapter sleep behavior and long enough DHCP leases, I was able to put my PC to sleep and come back the next day and still have my sessions active on resume. Unfortunately it wasn't too practical to disable keepalive as sessions that really do crash never get cleaned up.
With IPv4 you need an addres, a gateway, netmasks, DNS.
On v6, as long as you have a working router sending RADV packets, clients will self-configure via SLAAC. Granted, same with IPv4 and DHCP.
If you don't have a router, most things should work thanks to link-local+mDNS.
You can easily pop a second router on the network to bridge two LANs, no need to reconfigure the DHCP. Gateways self-advertise, etc.
The point I'm trying to make is that most people trying toconfigure their IPv4 network have a functional IPv6 network the moment they put the cables in (on Linuxes es at least, not sure about other platforms).
I hope you're right, if only so that cgNAT goes away someday. But I'm pessimistic on that front. cgNAT is too easy and works just well enough to make adopting something better too low a priority to ever happen.
The increasing prices of IPv4 address blocks will probably drive adoption of IPv6. The increased complexity will be outweighed by the elimination of scarcity that IPv6 brings. If we are still using IPv4 in 2100 that would be tragic.
IPv4 block pricing: https://ipv4marketgroup.com/ipv4-pricing/
I do (and I guess I'm not alone...) -- have IPv6 exposed machines over a HE.net tunnel. Some of the things are ONLY accessible over IPv6 (because nobody needs them over IPv4, so that's enough).
I wonder how many sleep/wake cycles would it take a day to outweigh the TCO decrease of the power bill with the TCO increase of the increased stress/amortization of the hardware, especially HDDs.
Probably a lot. My anecdata: I have had four Seagate HDDs in my desktop PC since 2007 and despite a couple of remapped sectors, they're still working fine.
I often power it on and off 1-2 times a day, and the drives are set to spin down when not in use. The SMART data for these drives shows approximately 37,000 power on hours and 17,000 start/stop cycles. That's about 4 years 3 months of actual power on time, and about 3 power cycles per year over the past 16 years.
Assuming 15 cents per kwh, that comes to $494 saved by shutting the drives down over the past 16 years. And I paid $232 total for the drives back in 2007.
Obviously YMMV, just thought it would be fun to calculate :)
That might possibly be close where you live, but not where I live. I need to heat my home at least 8 months a year and I heat using electricity (which isn't the smartest thing to do, but that's the way the house was built 30 years ago). It is too hot inside less than 4 weeks a year. So saving energy is done by trying to live in cooler rooms, but not by searching for small consumers.
I'm curious if you mean the 15 cents per kwh estimate is too high or too low? It's a conservative average for me in upstate NY over the past ~15 years.
If you're using electric resistance heat, then I suppose a running computer is kind of like processing power plus "free" heat :)
Prices obviosly differ a whole lot depending on the country. My point was not absolute monetary savings, but energy saving.
My school physics tells me that nearly everything you do with electricity at home will heat your home. Doesn't matter whether it's the baking oven, the computer or the fridge. So if you heat using electricity and live in cold climate saving energy for computers or household appliances just moves the same consumption to heating.
The big exception is heating water. That energy is typically lost to the sewer.
As long as you pay the same electricity price for your computer as for your heating, it does not make a differencd whether that's 7 or 50 cent or the 15 you estimated.
Oops. Yes, an average of 3 cycles per day, not per year!
I have the spin down timeout set to 20 minutes. But this is a desktop PC with intermittent usage (and different usage patterns over the years). They used to be in a RAID1+0 but now they're JBOD and I use them as scratch drives for random stuff.
Somehow "the simple way" being a persistent Pi with a custom script seems like it's not exactly that simple. What you're trying to do is something so basic and yet I've been unable to get this setup on my NAS either.
Would a static mapping of the ARP work? I'd want the server to WOL and work with timemachine without external dependancies.
Makes for a great little load-balancer setup on bare metal k8s. I have to lose a LOT of machines all at once for anything to go down and stay down, and my router configs use a known ipv4 address for everything, without having to care at all about which machine is actually handling it.
Proxy ARP would have the raspberry pi provide its own MAC address as the MAC for the server IP. All the traffic from the clients to the backup server would have to pass through the Pi's network interface for the duration of the session.
With this ARP stand-in, it provides the server's MAC to the client so subsequent traffic to the server doesn't need to pass through the Pi.
I'm using a smart plug that I can turn on via curl from a script of from its web server from my phone. However sleep to RAM has some advantage. I already entered the wake on demand rabbit hole and stopped at the first hurdles. Maybe I'll try again with the experience of this post.
> I discovered that if the server gets hard shutdown by killing the power it will automatically start back up when the power turns on.
That's usually a configuration option in the BIOS setup screen, what to do on power loss: "Last state" or "Off", and your server probably has it configured as "Last state".
Saying Windows has had it since Windows 98 and then, acknowledging that the issue is wake-on-demand is a bit confusing to say the least. Obviously Linux can sleep easily since decades ago. The issue is wake on demand (and I don't know if Windows has that even)
Edit: I don't why everyone's having issues. I run Fedora on a cheap Gigabyte G5KD and never had any problems. Never did with either of my thinkpads or my Xiaomi Mi Notebook...
Low-level power savings (s3 sleep) and magic packet (Wake-on-LAN) have been a standard since the late 90s. Getting a Linux OS that actually delivers these standardized power features is now a 20+ year old bug. I can only imagine the amount of energy that would have been saved if this bug was properly fixed 25 years ago. With solid-state tech becoming cheaper every day, I now expect nobody will make an "it just works" OS that has both sleep and wake working on-demand. Globally, the HDDs will simply continue to spin until there are none.
Who pays for Linux development? Either companies for servers or embedded Linux. Servers don't sleep. Embedded might sleep, but it's not PC hardware, so it needs completely different power management features.
Nobody is paying for Linux on average PC hardware, that's why 20 year old bugs are not fixed.
I have no idea how it looks on the dev/setup side, but suspend and resume is definitely a solved problem, seeing that I've never had any problems with it on my Steam Deck
> Obviously Linux can sleep easily since decades ago.
Over the last 5 years, I had about 4 laptops that I ran Linux on. Not a single one was able to survive this:
- Do some work on it
- Close the lid
- Drive somewhere else
- Resume work
At best, the laptop would freeze and require a hard reset (extra annoying on Dell XPS for some reason). At worst, it will be completely draining of power. All of my linux-loving colleges had the same issues. Windows worked just fine on those laptops, btw.
Classic "works on my machine". Operating systems are complex beasties. Bugs happen. I have an Asus laptop with Ubuntu 22.04 and it worked fine. Then one day, any time it comes back from sleep, the screen stays black. Can't find and workarounds and probably need to reimage. I have to do this about once a year cause something inevitably borks.
I had similar idea, but in the end I've gone a different path:
I've got a USFF PC with 2 external HDDs as NAS, and raspberry pi to keep always on stuff. I used to keep the PC up most of the day, and get it to sleep on schedule during night with `rtcwake`, and it was pretty reliable.
I've removed the scheduled sleep from the PC, as I prefer backups to be done at night, and my "minilab" of USFF PC with SSD & 2 external HDDs, rpi 4 with SSD and a 1Gb switch uses 18W idle, according to cheap power mater.
USFF PC is ThinkCentre tiny m93p with 2-core 4th-gen i5, and I guess newer models should fare better with idle power use.
I've got 2 external HDDs in ZFS mirror, cheap consumer disks. One of them is perfectly fine, second one started having occasional read or write errors, which is not great. With regular weekly scrub I hadn't had any data issues over last 3 years, and I'm pretty happy with the setup.
But it's a hit or miss with consumer external HDDs - before that, I used a single external HDD for backups at a time (100-200GB writes a week) and I lost data from 2 cheap external HDDs over span of 3 years.
Thank you for posting this, I saw something similar but about having a homeserver to run Jellyfin and lost it. Bookmarked this and will implement this in the future.
I'm doing something similar but in a less fancy way: I've got a Hydra server (the NixOS CI thing) set up on a potato-grade SBC with a more powerful desktop computer acting as a builder. The desktop runs a small Go program that I've written that shuts it down after ten minutes of inactivity. Wake-on-unicast for some reason always caused the desktop to start up after just a few seconds of being shut down, so I'm using magic packets. (I'm not sure what's causing this. I suspect Tailscale but I haven't really investigated it.) Hydra connects via SSH so I've added a Match block to the potato's SSH config that sends the WoL packet first:
Match host desktop.local exec /bla/bla/send-wol
I'm also doing full shutdowns instead of sleeping. The difference is only about ten seconds for a CI job. If the power draw during sleep turns out to be negligible, I might switch to that at some point but first, there are bigger bottlenecks to kill (like Hydra still running expensive evaluations on the potato).
Tangentially, I remember MythTV had support for the self-scheduled variant of this. I used to have a PC with multiple HDDs hooked to the TV and antenna which would look at its own TV recording schedule and power off when it was idle and wouldn't be recording in the next hour or less. Before powering off, it would set an RTC alarm to wake 10 minutes before the next scheduled recording. Meanwhile, if I walked into the room and wanted to watch recordings while it was off, I would just press the power button let it boot up to use the onscreen UI with IR remote control. I never bothered to figure out if I could make the IR receiver behave in some standby mode to wake the PC.
The next time we moved, I couldn't be bothered to set all this up again. Now we just use an old laptop and browser for streaming and a bluetooth keyboard with pointing device for "remote" control. It idles at low enough power to run 24/7 and not care, but we can also just open/close the lid as if it is a sleep switch for the media function.
> To prevent unwanted wake-ups, you'll need to ensure no device on the network is sending extraneous packets to the server
Assume the home network is using 10.0.x addresses.
I wonder if it would be useful to put the server on 10.1.x, add a virtual ethernet interface to the RPi that is also on 10.1.x, and configure the RPi to route between 10.0.x and 10.1.x. Configure the RPi to drop any extraneous packets directed to 10.1.x from 10.0.x.
On the Mac "route add -net 10.1.0.0/16 Rpi" and add a name and the server's IP to /etc/hosts.
With that approach you might also make it so the RPi runs a script whenever something tried to make a TCP connection to the server, and that script could deal with telling the server to wake. That way you can choose a wake method for the server without having to worry about whether or not other things on the network might accidentally trigger it.
What would happen if after long period of inactivity the server become a “silent” device i.e. device that does not transmit traffic for extended periods of time because can result in a loss of service binding? These periods of inactivity, thus making the device effectively unreachable (for example for a WAKE-ON-LAN packet) [1].
The diagrams in the article are just gorgeous, can you share the drawing tools being used here?
> You'll need: An always-on Linux device on the same network as your server, e.g. a Raspberry Pi
> First, a bit about my hardware, as this solution is somewhat hardware-dependent: HP ProDesk 600 G3 SFF
I seriously doubt any of this can be gotten to work on a Pi in a way that actually reduces power usage in any notable way. Hell the Pi 4 uses almost full idle current when completely powered off unless you turn off just about everything that would make it possible to wake up again without a full power cycle. And even then the draw remains significant.
Many people who would do this sort of thing likely already have an always-on pi (pi-hole, dns, ssh jump box, etc). And even at full bore, pi 4 is about 6.5W, so 90% savings over the HP.
Interesting. I believe with runtime (opportunistic) S0ix on x86 we'd be able to get similarly low power consumption at idle, but that would require cooperation from all devices and system firmware. It's already a thing on laptops, where if certain conditions are met, the laptop can technically be "sleeping" when the screen is displaying static content, like a document.
I have servers that I don't want to run 24/7 but I still need to access them from time to time. My current solution was to schedule shutdown of all the servers between 12am - 6am (usingcron) and start them up at 6am (using BIOS scheduled power-on), to save on electricity cost since no one uses them during those hours. However, this is much elegant solution.
If you're involving an RPi in this, in order to save power, might as well just have the RPi be the main driving force behind everything. Flip a switch to power on the server, command the backup once a custom app on server informs the RPi that server is ready, shutdown the server again by setting the above switch to OFF. This way your server will not even consume those idle watts when waiting for wake packets.
While it is easy to criticize the approach the author took to solve this particular problem (as others have done here), I must say that I appreciate well organized posts that detail a technology problem and solution. While I may not implement a similar solution, I always manage to learn something from reading them that I'll likely use when solving another problem in the future.
Nice writeup. It was enjoyable watching him re-invent the Apple ARP proxy solution. The first time I tried to figure out how an ssh request could wake a Mac, I was baffled.
While I applaud the effort, I'm really wondering about leveraging something like a ODROID HC4 and don't worry too much about the standby power consumption.
I mean, I even recently purchased an old refurbished i5 enterprise mini PC that consumes about 5W in idle mode, not even standby.
It makes sense as it is leveraging laptop components inside.
There's not really such a thing as a "WOL packet" exactly. Basically the network interface accepts any old packet, and then maybe wakes up depending on the content of the packet.
You can configure it to wake up if it sees any packet (what he did). The problem is networks are rarely completely silent and your machine will just wake up randomly.
Which is why they invented "MagicPacket(tm)" which is just any old packet that happens to contain a specific repeated byte pattern. The network card will wake up on any packet that happens to contain that pattern (which shouldn't happen randomly).
I don't know why he didn't use MagicPacket, even though he mentioned the problem it solves.
The simple way is paying $9.99/month for 2TB of iCloud space and put your Time Machine backups there.
Home labbing can be fun and a good way to learn, but if you are just trying to back up your files you cannot beat the economies of scale and reliability of cloud providers.
In the cloud you would just call the start_instance() API method to wake up the instance on demand.
Is there a way we will have mass-produced personal cloud devices that use hypervisors with a low-power consumption control plane and use standard APIs and Terraform
I want something like this but on a smarter router/switch that can wake my device up on certain port/packet requests going to the sleeping device. Having to setup a pi for this sort of thing seems like yet one more thing to manage
Once again I get the distinct feeling that networks talk behind our back in a phylogenic tree of patois obfuscated by deep time. (Where deep time for computers is like 30 years).
Hello I greatly recommend you Via our ongoing Invitation to join our Great Illuminati and become Rich and Famous.
With help of Bryan George Steil made me believe that illuminati is real, when you come across wrong person’s you will think that life is not real but when you are with the real one you will experience the goodness of your life, Mr Bryan George Steil has made me discover my purpose of life. Welcome to the great brotherhood of Illuminati and everyone in other Countries can join the Illuminati brotherhood to get rich and famous, Are you a Business man/woman, retirement pensioner, musical artist, Actor, actress, student, footballer Pastor, pianist, engineer, scientist, lawyer, Politician, Model, Music Teacher, Photographer, Musician, Doctor, fashion artist, pilot, Swimmer, social media influencerS, philanthropist, filmmaker, podcast host, producer, tv host, comedian, Chief Executive Officer (CEO) etc. Do you want to be rich whatever occupation you do, you want to be rich, powerful and famous in life. Illuminati can grant all your heart desires, Illuminati will make you achieve all your dreams to become rich and protected all the days of your life. BENEFIT GIVEN TO A NEW MEMBER, a brand new dream car valued at $200,000.00 USD a dream house to be built in any country of your own choice one month holiday (fully paid) to your dream tourist destination Countries. V I P treatment in all airports in the world is a total lifestyle change, a membership benefit funds $1,500,0000 USD to start a profitable business, YOUR FINANCIAL DIFFICULTIES WILL BROUGHT TO AN END. WE SUPPORT YOU BOTH FINANCIALLY AND PHYSICALLY SPIRITUALLY TO ENSURE YOU LIVE A COMFORTABLE LIFE. Email: illuminatisecretauthorities@gmail.com Note: There aren't any Human sacrifices and no bloody involvement.
The Club of the Rich and Famous; the world oldest and largest fraternity. We are one Family under one father who is the Supreme Being. In Illuminati we believe that we were born in paradise and no member should struggle in this world. Hence all our new members are given Money Rewards once they join in order to upgrade their lifestyle.; interested viewers should Get instant initiated to the Illuminati Brotherhood and receive a membership profit a sum of $1,500,000.00
After Illuminati membership profit funds a sum of $1,500,000.00 USD was given to me to start a profitable business, I now earn a daily profit of $50,000 USD for just inviting new customers to The Club of the Rich and Famous and being part of this life changing opportunity.
Invite customers Get reward 100% guaranteed and legitimacy
Got a link or something for me to reference? I have built a few micro itx home servers and have always wondered what a vertically integrated home solution would look like. I've been working on the custom software control plane for it for about 2 years now and gone through several permutations. I run a private cloud with these over mesh vpn and for personal use the cost cannot be beat.
I commend the author for their perseverance and the writeup - rabbit hole writeups are their own form of entertainment. That alone was worth it. I could never run anything that looks like such a conceptual pain the butt. A pain to maintain and troubleshoot to boot.
Especially since the setup requires a Raspberry Pi to run permanently. A Raspberry Pi that itself would work just as fine as the Time Machine server the author seems to need on demand?