Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: What is the fastest way to ramp up on DevOps, k8 and GCP?
165 points by epimetheus2 on Nov 17, 2021 | hide | past | favorite | 75 comments
I have a limited time and need to quickly ramp up on these skills. I need hands-on experience as that's how I learn best. What would be your go-to resource/workshop for that?

Is there some standard "here are existing 10 services and in this guide we'll go through deploying them, securing them, installing service mesh, provisioning database..." kind of thing?

I would look at these topics and in this order:

1. Containerisation - can you build a hello world web app in any language then Dockerise it.

2. Now break it into two containers - one is the original hello world but now it calls an API on a 2nd container that responds with hello world in one of 10 different languages. Just hard code this the point is that it’s now 2 containers for your “app”.

3. Create a Docker repository in GCP’s Artefact Registry and upload your images. Now remove them from your local Docker registry and run them again but this time pulling them from your registry on GCP.

3. Use Cloudbuild to build them.

4. Spin up a local Kubernetes cluster such as Minikube.

5. Read docos about K8s deployments and service types. Special attention to ClusterService, NodePort, LoadBalancer.

6. Deploy the first version of your hello world. Maybe try to point your YAML to the local registry then the one on GCP you created.

7. Create a service to access your app.

8. Figure out how to turn in and access (via proxy) the kubernetes dashboard.

9. Now deploy the 2nd version of your app. Learn a bit more about K8s networking, pod horizontal scaling, pod resource claims, kill some pods, etc.

10. Learn Skaffold.

11. Create GKE cluster.

12. Deploy same app.

13. Learn about K8s Ingress.

14. Get familiar with GKEs console.

15. Use knowledge of Skaffold to understand and use the brand new Google Cloud Deploy.

Edit: autocorrect “fixed” several things

I've done all this in past and am familiar with it, but I'd be interested in one-step-after-this thing :)

Try Cloud Deploy - it’s a new service and significantly easier than using a self managed CD tool like Spinnaker or Argon.

If you know all of the points I listed before, have a look at GKE benefits like horizontal and vertical pod autoscaling, cluster autoscaling, istio, config connector, anthos

Build another app, repeat :)

16. Avoid helm.

While I agree generally, this isn't helpful for someone trying to expand their knowledge. Helm is a popular tool, and it's important to understand it, when it's useful, and when it's not.

Care to elaborate on the three points you mentioned with more specificity?

I can, I suppose.

With regards to "understanding it," I mean knowing what it's doing under the hood. It's all well and good to know that you can install something by doing "helm install," but you should really know what helm install actually does.

With regards to "when it's useful," I mean when you have a very complex system that needs a high level of abstraction to make manageable.

With regards to "when it's not [useful]," I mean various scenarios. For example, when you really only need a single pod running with no configuration and no ingress, Helm is not only overkill, but also a negative, as it abstracts without adding any value.

To my experience helm is great at some things but the fact that does too many things leads to engineers shooting their foot and then you have two problems: K8s and helm problems to solve ;-)

The fact you add complex logic in the chart makes deployment logic problematic when debugging complex workflows because team1 instilled all the logic in the CI while team2 in the helm chart(!) and team3 50-50.

I prefer using bash + gnu utils (envsubst & see) and kustomize than playing around with a template language running on top of another … the funny part was that we run all this thing via Jenkins & bash. I recall having to use an inordinate number of escapes in some cases… anyway.

> when it's useful: 0% if the time, or possibly more if Self-flagellation is your style

> when it's not: 100% of the time.

I'm sorry, why is helm so bad exactly?

Why? What's the alternative?

I see a lot of things that has to be solved on your own even when using Helm. I find their templates repulsive. But "avoid helm" is not an advise, really. How do you suggest to structure, version, deploy, encapsulate with important metadata your deployment descriptions? How do you package it? Is there anything else that integrate with larger tools like Rancher so well?

Kustomize look it up, much more native for k8s apps

17. Avoid kustomize even more than Helm.

Lol. Helm is like the ex that still comes around to key or graffiti your car. Kustomize is the girl that makes you realize you want to get married.

Nice list! This is really good on ramp to get functional knowledge fast.

- Go to https://tanzu.vmware.com/developer/workshops/ and create a free account (the workshops are run inside of your own kubernetes)

- Do the container 101 hands-on workshop (bottom of the page)

- Do the Kubernetes 101 hands-on workshop

- ???

- Profit! build and deploy applications on ANY kubernetes (not just GCP)

And then look into the GCP specifics for k8s (GKE), networking, DNS, load balancing, storage, monitoring, etc...

I highly recommend the book Kubernetes In Action by Lukša [1]. It's a fantastic walkthrough of the whole containerized stack -- from Docker containers through the Kubernetes API and some info on Kubernetes internals towards the end. This is one of those rare books that takes an intimidating topic (k8s) and makes it all make perfect sense. It's heavily examples-driven with lots of annotated code and diagrams. The first edition uses both minikube and GKE.

[1]: https://www.manning.com/books/kubernetes-in-action-second-ed...

I would say there is no fast way to do this. If you specifically want to learn Kubernetes on GCP as opposed to DevOps generally, there are many things to learn.

Sure, you can get something basic running very quickly but if you are talking about production systems, "ramp up"? then there are many things to learn, understand and avoid and this takes an amount of time.

For an example, as a very experienced Dev and Ops person, after 18 months of Kubernetes, I am still learning things that I should have ideally done from day 1 and probably only avoided problems because I am not using it with very high traffic.

You can read lists and follow tutorials but engineering is trade-offs and even in the world of nice tooling, there are still many decisions to make and don't end up in a position where, for example, you cannot upgrade an ingress without the system becoming unavailable because you didn't consider this early on.

Don't get me wrong, it is worth learning but it is not fast.

As someone 6 months into Kubernetes can you elaborate on some of those things you would've preferred to have done from day 1?

* The hardening of images is pretty important: https://blog.gitguardian.com/how-to-improve-your-docker-cont...

* Working out/testing how upgrades can be made to ingresses without downtime

* Designing and testing SSL cert rotation

* Having a strategy for image version updates (e.g. not using "latest" but how to decide when an upgrade to a newer image is or isn't acceptable)

* How to super-optimise layering or image builds including ADD vs COPY, realising that copying everything at once usually breaks this, working out where/how to perform things like OS updates

* How to correctly inject build versions from CI into the docker executable

* How to use databases inside containers with e.g. schema and data updates

* The various kubernetes network options

* How DNS works in kubernetes (and when it doesn't)

* Debugging containers that fail to start or run

* Using probes correctly and understanding ready vs liveness vs startup

* How to check for node health dynamically to control external load balancers

* Using pod anti-affinity to balance pods across all nodes

* Understanding how to lof effectively from the container

Those are a few I can think of off the top of my head.

Im very much a bottom up sort of learner so I wanted to post something I haven’t seen yet in any of these comments. Go through and build Linux from scratch https://www.linuxfromscratch.org/

You don’t have to do all of it, just enough to feel comfortable with the problem space that containers in part are trying to solve for. In doing so you will learn about the fundamentals of the stacks you are working with. Then go learn about cgroups at the lowest level. Again you don’t need to master these things but setting up a container from scratch will then give you a more intuitive understanding of what container apis like docker are trying to solve for. Install a hypervisor on your local workstation and set up a few vms networked to each other. Use a tool like netcat to talk across them. Once you have these basics, and yes it is a long and round about way, I think the cloud stuff becomes a lot easier and more approachable since you will have a more in depth understanding of what cloud providers are actually giving you in terms of configurations and managed services. I came into software development from doing sysadmin work at a NOC during a graveyard shift. So I had a lot of time on my hands to mess around with this stuff. I don’t think that tools like docker should be a substitution for the fundementals, but rather something you add to your tool belt later to make these sorts of configurations easier to set up.

Yeah this isn’t going to be a 1-2 day crash course but if you are diligent you can get a basic understanding of what these tools do in under a month. Only after you have a complete understanding of single node use case (e.g. multiple containers running on a single host) would i recommend starting to learn k8. Good luck!

* edit to fix small typos

I don't think there's going to be a shortcut to learning Kubernetes to the point where you'll be comfortable setting up a production-grade cluster.

I recently went down the path of learning it. Even with 6 years of experience using Docker and ~10 years of general deployment / Linux / sysadmin'y experience it took something like 180 hours of no screwing around time to get to the point where I was comfortable putting up a proof of concept production cluster.

Proof of concept in this case means having the cluster up and running to manage multiple services, hooking up multiple external hostnames with external-dns and an AWS Load Balancer controller (I am using EKS). It also involved handling things like running database migrations in a robust way and toying around with nice to haves like preview environments for all PRs. It didn't serve any live customer facing traffic but all of the pieces were there to do it.

The provisioning and managing of the cluster was all adhoc with a mish mash of Bash scripts, eksctl, kubectl and Helm commands. Now I'm transitioning all of that to use Terraform to set up the cluster and every change to the cluster happens through git using ArgoCD to monitor the repo. I'm really happy with the direction things are going. There's no way I could have started here tho, it would have been too many new things to take in at once without knowing the fundamentals.

Long story short, don't try to rush things and skip ahead. It's going to be really important to learn the basics of Kubernetes before trying to automate everything with best practices. I spent a ton of time reading the docs while experimenting with a cluster along with reading hundreds of blog posts and skimming YouTube videos. Every step of the way involved applying what I was learning to a real project and interacting with the cluster in some way.

90% of it was able to be learned with a local cluster running on your dev box.

This entirely.

And even then, I think it took our org a solid year of work from 3 dedicated engineering teams to be comfortably deploying on GKE & EKS.

And still after that investment, we ditched K8s in favor of Nomad.

Kubernetes is kind of like a steamroller where most of us are looking for a hammer.

People joke about Google releasing Kubernetes into the wild to slow down their competition. It's really not that far off from reality though.

Would you say, for someone starting in the same place as OP, that it's better to skip K8s entirely and just go with Nomad?

If you don't need the things that K8s offers, try it and see if it works for you. It's pretty minimal. Most folks only need those features though.

We put together the Kubernetes Developer Learning Center for folks looking to bootstrap their k8s hands-on experience: https://www.getambassador.io/kubernetes-learning-center/

Combined with the K8s official docs https://kubernetes.io/docs/home/ and GCP getting started guide, this will give you a good tour of the tech: https://cloud.google.com/gcp/getting-started

For a general overview of DevOps and the principles behind this, you can't go wrong with reading "The DevOps Handbook" and researching the "Accelerate" metrics

Looks a useful resource thanks!

Heya, you have the exact same issue as I am. I needed to deploy our apps back then (we're in a team of 2) after developing for 1.5 years in small company. Boss decided to suddenly go with GCP, then I decided that I want to use K8s (which paid off finally!).

Things I started doing was to follow tutorial in k8s, it is a LOT to take in. Install minikube, start with simple Deployment sets and deploy with apply instead of helm. Then you can go from there.

I reckon after you learn k8s all you need to do is to spin up GKE which you should learn after. Depends on what you need, you might need to setup external LB from GCP. I started with nginx-ingress, works amazing, setup the LB automatically and everything.

In summary, k8s docs is your best shot, start slow from tutorial, practice practice and you have to read what's the problem when something fails

Rather than looking how to debug, these 2 commands cover my needs most of the time: kubectl logs <resource-name> kubectl describe <resource-type> <resource-name>

A DevOps guy I worked with at a previous job spoke very highly of the courses at https://acloudguru.com/

I also recommend acloudguru, they have very nice courses about Kubernetes and GCP, which provides a very good basis. I especially liked the certification courses - even if you do not pursue for a certification. For example, "Google Professional Cloud Architect" course helps you dive into GCP ecosystem and get very short overview about each of the service (purpose, limitations, specifics).

For anyone, who is very fresh in k8s topic and don't won't want to spent a cent for learning I recommend starting from edX course [0]. It provides a nice introduction and great notes, which you can copy as some cheatsheet. Such cheatsheet will be a definitely helpful hand in first weeks with k8s.

[0]: https://www.edx.org/course/introduction-to-kubernetes

solid. devops person here - these guys know their stuff.

As I was going through the same process of disentangling the learning-process roadmap, a friend of mine recommend I follow Jérôme Petazzoni's [0] "Deploying and Scaling Microservices with Docker and Kubernetes" workshop freely available as slides [1]. It's less shiny than those web courses with an integrated sandbox, but it's also pretty hands on and high quality. I had started the free courses available at KubeAcademy by VmWare, but I've retained very little from that.

[0]: https://github.com/jpetazzo

[1]: https://container.training/kube-selfpaced.yml.html

As the former owner of GKE at our company, read the official k8s docs. They are good. Read them all.

Here's a quick question: has anyone made a single page reference for the entirety of the Kubernetes manifest spec? E.g. everything that should come out of the box, except CRDs?

For example, for Docker Swarm and the Compose format, you have this: https://docs.docker.com/compose/compose-file/compose-file-v3...

Yes! The API reference is automatically generated, consistent and complete for each version. It’s a shame it’s not better known.


I'm curious of other answers as well, but one place that kind of does what I think you ask for is [0]. Also you can get nice info from kubectl [1]

[0] https://kubernetes.io/docs/reference/generated/kubernetes-ap...

[1] kubectl explain deploy --recursive

That's a really nice documentation page, however what's missing in my eyes is an example or two for each parameter in YAML format. Personally, when attempting to use Docker Swarm, things like that were immensely useful, even more so for Hashicorp Nomad and its somewhat niche HCL.

I know that people like to use something like Kustomize (https://kustomize.io/) and Helm (https://helm.sh/) at least last i checked, but it's also really nice to be able to read up on everything without necessarily following tutorials and such, or using specialized tools.

For example, with Docker Swarm, if i want to change how restarts would be handled, i have a really easy reference available: https://docs.docker.com/compose/compose-file/compose-file-v3...

But with Kubernetes, in the first linked page i find a short description (just scroll down a bit or CTRL+F "restartPolicy") which doesn't offer further parameters: https://kubernetes.io/docs/reference/generated/kubernetes-ap...

To its credit, there is a link to another page as well which contains more information: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifec...

That said, usability wise i'd have to spend more time to figure out how to get my containers/pods restarting just once every 1 minute as opposed to whatever the default is (say, low end cluster with limited resources) with Kubernetes instead of other technologies, due to this difference.

I can relate to all your experiences. Swarm has such a nice learning curve and I wish it would still be used widely.

With K8s I guess we are between a really decent (!) documentation that doesn't work very well as a "tutorial", and third party resources that does that.

It does make things harder at the beginning, but is still much better then, say, Hashicorps docs: "here is a cool starting tutorial that covers basics you'll need for the first 30 minutes. That is all, look other things up in the code". For a longer while I felt this was the case with Vault or Consul.

There is a series of Google-created Kubernetes/GKE courses on Coursera that include hands-on labs where you get to try stuff on GKE without paying usage fees.

The course content is a bit wanting (it sometimes feels more like an advert for GCP products than a course), but if you watch the videos that look relevant and do the labs, I think you’ll get something out of it. Coursera does cost money but has a free trial.

I can second this suggestion.

I've been doing these GCP and GKE related courses the last few months as an experienced full-stack developer who is expanding their skill set be to fullerer-stack(?).

The hands-on "labs" might feel like you are mindlessly "following along" with the tutorial but you actually learn quite a lot. The "labs" are also safe in the sense that you can't shoot yourself in the credit card.

Yes, there are clearly some small parts of the courses which were later inserted as ads for new services, but it is not a big deal.

I do recommend learning the basics of what GCP has to offer though and not just concentrating 100% on k8s/GKE.

The same way we ramp up on anything: by building a system that (actually) needs it. The XYZ (service mesh, provisioning, etc) will fall into place, from there. Or not -- if it isn't really needed.

But in general: learning by building stuff based on a real need -- even if that "need" is just to scratch the itch of some personal project you've always been wanting to do -- has about 10x the staying power (and "interview cred") as what one gets when starting from the mindset of "oh shit, I'm falling behind, I gotta cram on this stuff".

And on top of that: knowing when you really need X or Y, and when you don't -- is often as valuable as, or even more valuable than simply knowing how to do X or Y.

Why you need these skills? Thinking out of the box and complementing others: to hire a specialist is the fastest way to do it.

To get the skills for yourself is a different matter: hire a specialist for a day and go through a basic app deployment, workflow and notes for a day. Complement with the basics tutorial to avoid gaps, try your skills on a real environment.

But such infrastructure skills are no joke and mistakes are expensive so I would recommend to take your time learning about it.

You may also consider using The HashiCorp tools (nomad, consul, vault) rather than K8s. They are a bit more approachable (single binaries) and do a similar job.

I'm putting together devops content for the B2B arm of $FAMOUS_BOOTCAMP.

Here's my curriculum so far (based on AWS, though):

- Why DevOps (basically, a distilled version of "Accelerate")

- Terraform

- Ansible

- Docker

- Kubernetes

- ELK, Grafana, Prometheus

- Jenkins

- Automated testing

- MLops

- Secret management, databases, database upgrades

(The nature of the $FAMOUS_BOOTCAMP and the requirements from the initial customer makes it more about "here are the specific technologies you might use in your job today" and less about "here's the background principles").

Any suggestions or thoughts?

If you're looking to add secrets management and scanning to the curriculum, take a look at GitGuardian–https://www.gitguardian.com/monitor-internal-repositories-fo....

"I have a limited time"

You blagged your way through a job interview, didn't you?

not at all, but our devops guy ragequit the job and I want to be able to help piece things together if fire emerges, or help the team. Every engineer has some knowledge so together we're able to handle most of it. I want to use this opportunity to learn more.

It's sad that you assume the worst in people.

This is almost exactly how I became devops in the first place - the existing guy quit suddenly and I had to pick up the pieces in a hurry.

If you're already comfortable with the basics, then focus on one particular element and master it. The other bits will come over time. I recommend Kubernetes as a focus, given your list.

Keep in mind at all times that the infrastructure serves the business.

Hey, no judgement, and good for you for turning up for your team!

Hey, we have all been there ;)

At least he is trying to learn now instead of pulling a brillant paula.

I am also trying to learn modern DevOps techniques. While researching i found that many people recommended the courses from KodeKloud (https://kodekloud.com/). I have not taken them myself already, but i think they also include labs so you can practice the techniques.

The Linux Foundation offers free online intro courses covering these topics and dozens more at https://training.linuxfoundation.org/resources/?_sft_content...


That course got me up to speed pretty fast (3 weeks) with docker and k8s. I highly recommend it.

It sort of depends on what you are building. I work on web apps and services and there are so many things in k8s and GCP to use that its like rabbit-holes all the way down. You can get overwhelmed easily with stuff you probably don't need.

You can work on what is running on your container(s) locally using docker. I would suggest reading thru k8s docs and using minikube (https://kubernetes.io/docs/tutorials/hello-minikube/) to understand things.

Start small and get basics working, then iterate. You will likely experience aha moments later on and want to re-do things. The good thing is you are defining infrastructure via software.

Most people learn these skills on the job which is in my opinion the right approach for fast-changing tech stacks like k8 that can be out of fashion tomorrow. It is easier to learn something when you have a concrete need for it and know what you concretely need.

K8s is just a big, structured way to make Linux do various things, so my approach would be from the bottom up: learn about the Linux features that you are trying to control, then learn to control them automatically with k8s. Learn how to manually control memory and cpu quotas with control groups, how to mount overlay file systems, how to enter a chroot, how to create and use network or pid namespaces, etc.

Most people I have encountered who were afraid of k8s also simply did not understand how their process was executed by Linux at all, and I think that was the root of their fears. If you know how it all works, you won’t be afraid of the automation.

As someone who also learns best by getting my hands dirty, I always liked the ability to pick up an application development framework and get started building a real app while I'm still learning. By accepting the frameworks opinions, I can keep going and decide where I need to diverge once I learned more about the technology and what I really need.

With my Terraform framework for AKS, EKS and GKE I'm trying to bring the same framework benefits to the DevOps world, more specifically to Terraform and Kubernetes.

Check it out: https://www.kubestack.com/

You can grab few books on the topic or attend an online course. Both will give you an idea, or a taste of the different issues at play. But to really learn devops and Kubernetes, you need to work at a company using those tools. Then you start to understand what are the challenges, strengths and weaknesses ...

I would suggest doing this excellent course that also includes sandbox hands-on exercises.


For k8s, look into https://github.com/kelseyhightower/kubernetes-the-hard-way - I did it using GCP, which would expose you to a little bit of that, too.

The CIS has a benchmark for evaluating K8s security. You may want to have a look at that: https://www.cisecurity.org/benchmark/kubernetes/

Get a job which requires it, study a bunch and try to stand up a cluster. Learn the basic primitives of containers and why they are useful. Fake it till you make it, this tech is brand new and a lot of it isn't needed and is over engineered.

Kubernetes is like the Javascript of NPM, a new tool every week, very good at complicated things, but simple things are complicated. It seems to be dominant docker server management atm, but not sure if it will stay like that.

I'm sorry but what you wrote I find very untrue. K8s is bloat, but comparing it to NPM is very unfair towards K8s.

I often see developers or admins who ask questions how to setup their "simple" setup in plain docker on dev machine, while all they have trouble with - networking, volumes, exposing services - is actually very simple with minikube. There is more cool learning resources as well, IMO. It's even easier if you use cloud providers. Creating a working cluster on Scaleway takes a minute and you can't really screw anything up. You'll waste more time looking for a proper credit card, or figuring how basic user auth and privileges work on AWS, etc..

Yes, there is a lot of "magic in the background", but I have not yet seen any developer who reviewed all 1000 dependencies in their Angular Hello World application...

The K8s itself is quite stable from simple user's perspective and does not follow very rapid changes. I am able to easily follow old tutorials, books and documentation that was not updated in a long time without much issues. If something was broken I usually knew how to fix that after first basic k8s training. Core concepts did not change much, or at all. Some things got tidied up, like storage abstraction, but it makes sense and it's nothing new. A lot of cloud integration is build into K8s for years, some integrations like ... ingresses that implement certain external products are delivered mostly by vendors and it's quite easy to find options available.

There is a lot of mess into implementing proper production or large scale setup, misconceptions on how storage is handled and "why plain k8s does not provide Read Write Many storage out of the box?" kind of questions at the start. But a lot of that are issues that exist in any on premise infrastructure and has to be handled with a personal touch - taking in to account what you already have.

Yeah true, NPM is not the right comparison, I typed too fast, meant the frontend world and k8s world remind me of each other in certain ways. And I really find k8s and HELM im using not so stable, or ready, that's why everyone is using lots of different tools. I found old school Apache/Nginx a lot more stable. True Kubernetes solves a lot more problems, but as a result simple things, like a simple filesystem mount, causes all kind of issues.

I find the excessive feature-richness to be more akin to C++ : you don't where to start, you've got the feeling it's never going to end and that it can crumble under its own weight, but in the end every feature at least has some point that's worth thinking about.

Question is what you want to learn exactly?

Setting up only the cluster? - Start by porting your local setup to a local k8 (minikube etc.)

And i would suggest to invest some time into terraform to setup everything the way you want it.

Having a credit card is the hardest part to me.

All of setup step is simple, there're many resources that you can follow and learn all in 1,2 days.

Wish I had your confidence and/or your learning abilities.

It's simple, but not easy.

Just think how things stick together first.

you can hand me your credit card if it helps you no problem

Install OpenFaas

Kube academy

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact