Ask HN: How to write a configurator for deploying a multi-service system?

[sriram_malhar]

Say you have a working service that has compute servers, a bunch of queues (Kafka, say), a distributed database, Redis etc. Most of the computation is fairly parallelizable.

Suppose your product manager says, package this system in such a way that we can sell our system to another customer. It will involve scaling down to varying levels (from demo version all they way up to a smaller multi-DC version for fault-tolerance), but significantly less scale than yours.

Now, although you have the experience to advise your customers about a system to achieve a specific scale, you don't want to be stuck helping them through their deployment. So you start writing a configurator that takes in a desired input transaction rate, and outputs a smaller clone of your architecture that can handle that load.

I'm looking for ideas/papers/advice on how to write such a configurator. The problem is that there are still a lot of variables, such as how many hardware servers, how many of each software server (keeping fault-tolerance in mind), their placement etc. Because each of these systems have different demands on cpu/memory/disk/network, they are inherently unequal in their effect. It is analogous to a recipe generator that spits out a recipe for a given number of people, where different components scale differently in terms of amounts and cooking time.

It is easy to describe a minimal system. Starting with the fault-tolerance criteria, one will need a minimum of a three-box cluster. One can start all the required services manually and arrive at a configuration that maximizes the TPS. Is there a nice way to compute how many of each will be required to handle double that base workload? I am thinking I'd put in as many hardware/software servers as the number the most performance bottleneck components require, and then do some elementary bin-packing of the rest of the components depending on their resource usage. If you have some guidance, pointers for me, I'd be most grateful. Thanks in advance.

[thesuperbigfrog]

Depending on your requirements, Ansible might be a useful tool:

https://en.wikipedia.org/wiki/Ansible_(software)

[sriram_malhar → thesuperbigfrog]

Ansible, Pulumi etc allow you to script an infrastructure once the plan is in place. I'm looking at one step earlier; how to come up with an infrastructure plan automatically, given the requirements. Configurator is the wrong name, I realize now.

[thesuperbigfrog → sriram_malhar]

The plans generated by the system you describe would be very workload and requirement-dependent.

As you stated:

"Because each of these systems have different demands on cpu/memory/disk/network, they are inherently unequal in their effect. It is analogous to a recipe generator that spits out a recipe for a given number of people, where different components scale differently in terms of amounts and cooking time."

I do not think there is a good way to design such a system for a general use case, but it might be possible to build an expert system (https://en.wikipedia.org/wiki/Expert_system) for some specific use cases using well understood infrastructure patterns.

For example, you might have a recipe for a database-backed website that can handle given amounts of throughput and has best practice configurations for scaling, data backup, failover, etc.

It seems like a tough problem for the general case, but doable for "typical" business use cases.

[sriram_malhar → thesuperbigfrog]

All true. I guess I am looking for a first approximation, so that one can have a starting point from which one can fine-tune. The problem is to somewhat mimic what an expert might do given the requirements, to say, for example, "We'll have 5 48-core boxes, three of which will have Kafka, two will have the database, the in-memory grid will be spread over all 5 etc, and here are the relevant config files". Maybe it spits out ansible code.

Thanks for engaging.