Ask HN: How do you write integration tests for complex pipelines?

[dondraper36]

Apart from obvious unit tests and some mocks (no love for mocks, honestly speaking), there comes a point when you need to make sure that the end-to-end logic is correct in your application while it interacts with other services, the database, etc.

To be more specific, let's say there is a service that has a few endpoints with tricky behaviors.

For example, creating a resource might require:

* Making a call to service A * Creating new records in table X * Creating new records in table Y ... * Making a call to service B

You get the idea. Even writing mocks for such cases is tiresome, not to mention that it might say nothing about the real interactions in the wild.

My current approach is using dockertest (I use Go for this project and Postgres), spinning PG in a Docker container and, well, testing each endpoint method completely.

Having said that, I would be interested in knowing how other people write such tests, how granular they are, what exactly is tested.

Probably, you can recommend some nice resources on integration testing you know of and find illuminating.

[brodouevencode]

The first step is to really question if you need this level of complexity. Remove the things that provide no value. Break out data transformations into their own processes or microservices. Isolate them and really treat them like a black box. Unit test heavily here. Go is built for microservices. I've got a go application that I've inherited that's very light on the unit testing and heavy on the end to end integration testing. Golang doesn't make this sort of thing easy or enjoyable. I'll probably rip a lot of that out if we wind up changing major parts of it.

Remember, testing is an insurance policy. It provides no direct value to the business. That's not to say it shouldn't be done, it most certainly should. But when testing becomes the majority of your coding effort then it's time to start re-evaluating your testing approach.