Use Terraform for bringing up the Kubernetes cluster and other related cloud resources (storage, load balancers, DNS, etc).
Once you have a cluster up and running you can just use kubernetes yml to manage the cluster.
It is possible to also use Terraform to manipulate kubernetes resources (or AWS ECS or another container platform) but I personally like a clear separation between infrastructure (bringing up the environment using Terraform) and scheduling work on a cluster (using kubectl)
So TF would come where you would otherwise use gcloud commands, if I get it.
One part of the cluster building that caught me off-guard with GCP is that cluster options effectively change over time, as the k8s versions march forward.
For instance some feature go out of beta and become the default, and trying to rebuild a cluster exactly similar to one built year ago would require specific “don’t use that new feature” flags added to it. Or using these new features require adapting the other resources to the new configurations.
I guess TF is still useful in that it helps audit and reproduce the same infra inbetween changes, and will be nicer as an interface than bash scripts, even if it won’t save from dealing with nitty gritty of the platform.
Indeed, I see TF mostly as an automation/idempotency tool for setting up and maintaining infrastructure components like Kubernetes clusters, ECS clusters, load balancers and all that. Something you would normally do with gcloud or aws-cli or by clicking around in GCP/AWS web console.
Yes, the "provider churn" is real: defaults on cloud platforms can change and platform-specific TF providers can change as well. The way I usually deal with that is to make sure my TF repo's stays in shape by applying them regularly. If you haven't applied a TF repo in over a year chances are real that your repo has rotted. In a similar way to an old Ansible playbook that has rotted because some apt packages have new dependencies. Keep applying them regularly and manage changes in small chunks instead of once a year.
Auditing is really useful. I would recommend preventing developers from applying TF to production directly, and have it all managed by a CI pipeline.
Also with some careful planning and structuring of your TF repository. It's pretty straightforward to duplicate environments: for example: spin up an extra development environment to experiment with a newer K8s version, or to validate some infrastructure changes.
Once you have a cluster up and running you can just use kubernetes yml to manage the cluster.
It is possible to also use Terraform to manipulate kubernetes resources (or AWS ECS or another container platform) but I personally like a clear separation between infrastructure (bringing up the environment using Terraform) and scheduling work on a cluster (using kubectl)