Anyone like the idea to use AWS SDK to build deployment automation?

[hiboyang]

Hi Folks, just saw some post about Ansible. Also lightly tried other Infrastructure as Code like Terraform. Feel getting pretty large/complex yaml files when deploying decent size infrastructure.

How do people feel just using AWS SDK or other cloud SDK to write your own deployment tools? Since it is pure code, it is easy to refactor and test.

I did some prototype building DataPunch project (https://github.com/datapunchorg/punch) to deploy Data Infrastructure using AWS Go SDK. It provides one click command like “punch install SparkOnEKS” which will automatically deploy a ready-to-use Apache Spark environment on AWS.

Want to seek feedbacks and see whether other people have similar ideas?

Thanks, Bo

[gherlein]

Before designing something new, you might consider going to look at the AWS CDK - https://aws.amazon.com/cdk/ - which is exactly what you are describing.

[hiboyang → gherlein]

Yes, there is AWS CDK! It is similar to what I describe, but it is still too complicated for end user. The "Concepts" section in CDK web site lists 16 concepts!

I am thinking for this scenario which people want the simplest way to deploy an Apache Spark environment on Kubernetes (Data Infrastructure is complicated, thus use Spark as example here). I prefer a one command to automate everything, instead of writing all sorts of infrastructure-as-code files. Right now, do not see that or similar tools. Thus I start to build a one-click deployment tool like https://github.com/datapunchorg/punch.

[PaulHoule]

That's what I've done.

The way I see it is this. Tools like ansible, terraform, etc. don't eliminate the need to understand about shell scripting, the AWS API, etc. All they do is give you something additional to learn on top of all the stuff you already need to know. It's like

https://en.wikipedia.org/wiki/There_Was_an_Old_Lady_Who_Swal...

Maybe I am cynical but once you get into the ecosystem of tools like that then you are stuck investing more resources into those tools, paying for support, adopting more tools to deal with the complexity, you never win.

Years ago I was building virtual machines that would run a web service or populate a database full of content and publish it to an AWS image that I sold in the AWS marketplace.

The architecture was: a Java program sets up the resources required to do the job with the AWS API, then creates an init script and installs it on a new VM, that init script does all the work that has to be done on that machine, then it sends a message to SQS that tells the program that started it all it is done. Very simple, and if my DSL connection flakes out momentarily during the hour or so the init script takes to run it keeps running and I don't have to do it all over again the way I might if there was excessive communication going on between the master program and the host.

[hiboyang → PaulHoule]

Yeah, I feel exactly same. There are tons of tools like Terraform, and sometime people need to wait new AWS features being added to Terraform. All these tools use SDKs from cloud providers. I prefer simple and "just work", thus thinking to directly use those SDKs without additional layer.

[thayne]

I really like the idea of using a "real" programming language for this sort of thing. That's one reason why I prefer chef to Ansible.

That said, using the AWS sdk directly is probably fine for creating resources, or tearing them down, but if you need to modify something, you'll need to have some mechanism to find the difference between the current state and the expected state, and that can get complicated fast, and is something that terraform takes care of for you.

CDK basically let's you generate cloudformation (or terraform) from other languages

Pulumi might be worth you looking into, but it still has some rough edges, and imo isn't as mature and reliable as terraform.

[hiboyang]

Thanks folks for the input! Terraform or AWS CDK or Pulumi are good tools. Kind of feel this is a very diverse area, and different people have different favorite tools :)

Curious anyone from SaaS companies are using Terraform/Pulumi to build their SaaS platform to serve their customers?

[speedgoose]

You may be reinvented a wheel that is only compatible with AWS. It’s not a problem if you enjoy that, but not something I would recommend if you need to deploy infrastructures at work.

[satya71]

Why wouldn't you use Terraform or AWS CDK or Pulumi for this?