Ask HN: Homelab projects to learn Linux admin and DevOps to get hired as one?

[shivajikobardan]

I'm purchasing a mini PC soon(3-4 months) as I don't have any personal device as of now for labbing. Till then I want to make a list of project ideas to work on that will enhance my resume as well as give myself a point to talk about in interviews regarding system administration or devops engineering.

Start With A Static Website

Host A Dynamic Website

HTTPS

Implement Basic Auth.

implement monitoring solutions like naemon, graylog

implement alerting solutions(Learn email server as a bonus)

glusterfs, nfs/efs, ceph, borg

Do all these with

manual instructions from command line

automated with bash scripting

Configuration Management Tool(Ansible)

Docker

Kubernetes

[aos]

This is what I used when I first learning: https://serversforhackers.com/

For a really good practical project: build a website, then host it on your mini-PC and find a way to expose it to the internet. This will teach you a lot about DNS, proxying, building websites, managing the server (via some config management or container), monitoring, etc.

[rochak → aos]

Thanks for sharing this. It is incredibly useful!

[Vampiero → aos]

Then remember that we live in 2024 and that you need to learn how to do DevOps, not sysadmin

[herbst]

Sounds fun. What kind of mini PC are looking at? I am a big fan of used Dell wyse 5070 devices you find from $70, native Linux support, rather big community (easy to Google). Running one as a server and one as an mobile PC even.

One thing I would consider earlier is virtual machine hosting. Making it easy to setup, test, build and destroy a system without consequences. Virtual machines (and docker container, and more) are easy to setup and manage trough 'cockpit' and you can learn the command lines later when you need it.

Imo there is no point going directly into Industrie tools (kubernetes, docker, anisble) it may is better to first learn why they even exist by building systems the traditional way.

[shivajikobardan → herbst]

https://exortstore.com/product/gmktec-nucbox-9-mini-pc-cpu-a...

This is the one I am looking into.

Availability in Nepal is a tough call. I can import from India given they're available in amazon.in and aren't too big weight. But that's about it. Yeah, I am going to do VMs hosting. Everything on bare metal, easy to make and break.

[herbst → shivajikobardan]

Nice specs. Looks like more than you need (mostly on the CPU side) but if that's the thing you can get I am sure you aren't going wrong.

[js4]

This is a great way to learn!

Look into Proxmox(https://www.proxmox.com/) for setting up your own ec2 like VM platform. This makes it pretty easy to experiment with setting up and running different services on the same box in an isolated way. Practice things like setting up a VM with a GPU passthrough and then running gpu enabled docker containers on the VM.

Also look into getting a Hetzner server and setting up site to site Wireguard.

FWIW you don't really need to buy a mini pc. You will learn more by doing a build. The great thing about doing something like this is that you can get old data center parts for cheap. Ex: pickup a server motherboard (maybe a SUPERMICRO X9DRI-F), CPUs, and RAM on ebay for < $300. Then put everything into a used server case with a new PSU and you can do everything for $500 or $600. This setup will have more resources then any off the shelf thing you can buy.

[sandreas → js4]

While I absolutely support the idea of Proxmox (especially with ZFS), I think to start a learning project it is probably overkill to spend $600. I would maybe start with a Dell T30 or Fujitsu Celsius W550 for < $100 used or using a Gigabyte MC12-LE0 Mainboard with a Ryzen Pro 5600. Stacking one of these with a modern NVMe and some ECC RAM should go for additional $200, the Fujitsu Machine is OLD, but has intel AMT Remote Management and draws <10W Idle.

[shivajikobardan → js4]

I can get a mini pc for similar price range. I don't think that'd be true in Nepal(more resources than...)

[3np]

General advice not necessarily optimized for hirability:

First thing you do: Get another one of what you're already getting.

That way you can actually run "production" and host stuff while you can experiment and play around fearlessly without breaking prod. You'll also have a spare on hand in case of hardware failure etc.

Second, do at least basic separation of networking right away. Get a $10 switch and separate subnet instead of piggybacking on your home LAN.

Then, get comfortable with virtualization (qemu+kvm) and figure out a way to automate image your image builds and deployments.

Automate backups early. Start simple and iterate over time.

[abhiyerra]

I found the way I learned was to install these technologies but also write my own apps that take advantage of them. This way you aren’t just playing with the technology but can also make insightful recommendations to others on that technology’s limitations.

[JSDevOps]

https://www.reddit.com/r/homelab/

[lakomen]

I find the basic Linux stuff is learning by doing and reading docs, using search and asking around.

Docker has docs and also searchable examples and blogs.

But k8s is an IMHO artificially fenced off area. No matter where I ask, I receive no response or am met with arrogance and elitism.

I can't even get the simplest thing answered, which is "what is the minimum required setup for k8s, if not using a hosted solution? I plan to use 3cp 3w nodes and utilize all resources available. Do I NEED external storage, do I NEED an external load balancer or can I use DNS-LB?"

Official k8s forum - zero response. Reddit k8s - zero response and downvotes Home operations Discord - zero response

It's like this little well kept secret only select few have access to.

The documentation is also not clear on that subject, because I believe the big companies like Google and Amazon want to sell you their k8s offerings, which are super expensive, over 2k per month for a 5 node cluster are you kidding me

[moondev → lakomen]

I can't even get the simplest thing answered, which is "what is the minimum required setup for k8s, if not using a hosted solution?

Most mininal is to run a cluster inside a docker container with KIND. 2 cores and 4G of memory should allow you to run some small workloads

I plan to use 3cp 3w nodes and utilize all resources available. Do I NEED external storage,

No. Storage is optional AKA stateless workloads.

do I NEED an external load balancer or can I use DNS-LB?"

I'm assuming you want to bootstrap an upstream k8s cluster with kubeadm. In that case yes you need load balancer to sit in front of your control plane nodes. You can use a project like kube-vip to function as the load balancer without introducing an extra machine. If you want to use an extra machine you use something like haproxy.

You can use DNS but it's not ideal for production without healthcheck - but for a lab go for it

[lakomen → moondev]

Thanks, yes rent 3 small (well 4 dedicated cores, 8GB RAM, 256GB storage) nodes for CP and double the size for workers, which should be enough for a test cluster, well test and actual real life prod cluster, external of course. Costs around 80€/m. My current server costs are about the same price.

I noticed when the master or 1st cp node goes down, the other 2 are not sufficient to keep the cluster running. I wonder why do I then even have (and am paying for) 3 CP nodes when failover isn't working.

I had the most success with microk8s. I tried many other solutions, and I'm not going down the manual kubeadm path. I can't dedicate all my time into maintaining that cluster. I want to focus on developing my services, writing a configuration file, or rather with k8s that's a collection of files... Everything screams "don't do it". "It's too complicated, too time intensive, don't do it, you will regret it"

I'm not even sure k8s will be able to recover the loss of 1 worker node. All my experiments with a 3 worker postgres cluster showed that if the node count is less than 3, it goes into an endless loop of trying to bring the instances up.

I mean the most important thing is, how good is the solution when disaster happens? K8s tries to reallocate the resources on other nodes. But if there are less nodes than required by the helmet chart, that fails. So that means in conclusion, I need a fallback node, aka 4 worker nodes, and the cluster now costs about 97€/m. But that also means that I can't say with certainty which IPs are being used, if I scale the service up. I'm trying to be cost effective, my resources are limited. And I'd like to learn about this, but a course costs over 10k€ and takes 6 months.

I also noticed, when I have 3 CP nodes, only the 1st node has high CPU and RAM usage, the other 2 are pretty much dead, except for the 10% resource usage of etcd, no matter if they're doing anything or not.

Stateless workloads are a myth. Or rather you can't provide a service completely without state. The units can of course be stateless, but they will access stateful containers, like postgres, which also run on k8s.

I don't think I need k8s for now and when I will, other solutions are more likely. The whole effort of maintaining and upgrading the cluster, and right now there's a transition to gateways instead of ingress controllers, so you need to bother with infra instead of solving actual problems.

Thanks for feedback anyhow.

[d3Xt3r → lakomen]

That's what I've been seeing in the industry as well, and it's honestly pretty disgusting.

We were evaluating this one particular I*M product for infrastructure management, which needed to be installed on top of this other I*M platform, which specifically required to be run on top of a particular version of RHEL and a particular version of OpenShift (no other k8s flavor would do), and they wanted it as dedicated instance on a dedicated physical server (so we couldn't use our existing OCP instance).

The overal system requirements to run the damn thing (which was ultimately just a Ruby app) was enormous, and the bill quoted was astronomical. Customising, upgrading and maintaining it would've been a massive PITA too.

Thankfully our C-suits had a rare lightbulb moment and the whole project was canned. But man, what a massive waste of time and effort it was, we spent hundreds of hours doing the discovery, design and coding for it. Heck, I even ended up having to learn Ruby just so we could customise the app, because I*M were incapable of delivering a decent product.

We then decided to go for another white labeled product from a competing vendor - which was admittedly a lot better than that I*M crap, but during our consultation with them it was decided that we'd need to port all our existing code and tooling to their proprietary platform - a lengthy process that would take several hundreds of hours of effort, with a migration plan set over the course of an year.

Personally, none of this makes any sense to me. I can understand why we wanted to go for a commercial solution, but with the amount of time, money, effort we'd be putting in, we could've just continued to develop our in-house product which leveraged existing opensource solutions like Ansible. The whole insurance for "getting hit by a bus" scenario and "vendor support" is so blown out of proportion these days.

I just want to go back to the good ol' days where being a sysadmin meant you were the one in control.

[ml_more]

I'd buy a mini PC on craigslist for $100-200. You can usually get one with 16gb of ram and i5, a 500gb ssd for that. Install a debian based linux on it (ubuntu is very common and common is good for training) Be sure to get one with the power adapter if you go ebay.

Configure it fresh from install using ansible. Prefer ansible built-ins when possible. If you're calling shell scripts you're probably doing it wrong. Create named user accounts and create keys for that user. grant your named user sudo nopasswd and disable password login. Named users are preferable for security audit purposes. ("Shit who logged in at 2PM on tuesday and deleted critical resources?", "dunno it was the 'ubuntu' user", "great I wish we logged in as ourselves so we at least had some way of knowing who's account got compromised") Learn how to troubleshoot ssh login problems (-v), did you know that you can't log in if the permissions on the authorized keys file are wrong?

Enable and configure unattended-upgrades. Learn how UFW works and learn how to show iptables rules. Learn how to check which services are running, which are listening, which tcp connections are active. Learn how to write a systemd start/stop script (it's like 5 lines, super easy) Learn how to tail and grep logs to diagnose problems. Learn how to use find and grep.

When using ssl you'll probably use let's encrypt and you'll probably want to renew automagically with certbot and dns verification.

Create an elk stack, ship some logs to it. (maybe loki too) Hold off on k8s for now. It is advanced and you're wasting your time and your most precious resource (your ability to work through frustration) if you beat your head against it too early. When you learn k8s assume the cluster is already set up and learn to deploy a single service (there's minikube and k3s for this sort of thing). Anyone running k8s already has a lot of k8s skills (or they shouldn't be using k8s to begin with). Contribute in other ways. If they don't have a lot of k8s skills and they're using it 1) they don't need a junior, they need a senior and 2) they need to get onto something simple and easy like ecs. And if they already screwed up that bad they probably need to move to heroku. (but i digress)

Run through some command line practice, learn awk, sed, the general way commands are structured (command, options, arguments). Learn to chain commands. Learn vi (did you know you can call arbitrary shell commands?). Learn how to replace lines in config files with sed, learn how to concatenate content to the end of a file, learn how to do that for a file owned by root (hint, look into tee)

Learn how to deploy code with github actions

Learn Terraform Take the free online AWS solutions architect training courses and begin taking practice exams. The other clouds are clones. Azure with a preference for active directory, ghoogle with a preference for unnecessary complexity and pedantic bullshit.

Skip configuring a mail server, if you're doing that you're probably doing it wrong. (hint, you'll be hitting a mail sending service. Several of the small ones have a free tier which is great for learning). You'll need some more info here but you can search it up.

When learning bash scripting focus on readability and maintainability. The Google style guide is a great reference. https://google.github.io/styleguide/shellguide.html In bash there are 5 ways to do something, the best way is the most readable way. The person who thanks you may very well be yourself.

While we're on the topic, Google's SRE book is free online and epic. Learn the section on postmortems backwards and forwards. If you got a job and only brought the google postmortem template and culture with you you could improve nearly every company in the world. Same with improving their code testing and deployment. Same with doing simple cloud security and cost accountability tasks (like reviewing network perimeter and cost control suggestions in trusted advisor, finding unpatched servers and patching them, resolving dependbot vulnerability notices, etc.)