
Ask HN: Legacy code – tests first or setup CI? - ZanrielJames
I&#x27;ve been working on a project for a large IT department to automate all the things. It started out small but more people are starting to use it, and also starting to contribute, adding features; it&#x27;s starting to get some traction. This is a web based app (Python) that runs scripts to automate various infrastructure tasks, like building and patching servers. It&#x27;s backed by a database and is very tightly coupled (although it&#x27;s fairly modular, organized into packages, kind of like MVC). We had to move fast and there were just 2-4 of us, server admins learning to be developers, so we&#x27;ve never had a formal testing or CI&#x2F;CD process. We do use version control. We have a dev branch, which is everyone&#x27;s upstream, and once we&#x27;re happy we merge that to a master branch and that&#x27;s considered &quot;production&quot;.<p>We know we need to get to CI&#x2F;CD, unit tests, all the DevOps&#x2F;Agile&#x2F;XP stuff. I&#x27;m thinking since we don&#x27;t have any tests at all, before we refactor the code to support unit tests or start integrating with Jenkins (CI), we should start with at least adding some high level tests. Maybe integration or UI tests... just something to make sure we&#x27;re not breaking things as we clean up the code base.<p>Some people have suggested we start with getting it into Jenkins and set up an automated build (CI) system first, then start adding tests. I&#x27;m thinking start with tests first.<p>People are using it now, but there are a few bugs. Most of them are integration related - race conditions, DB sessions overlapping, things that come up when it&#x27;s running under Gunicorn and handling a lot of requests. What&#x27;s the best roadmap to get a legacy code base up to snuff to be considered a real product?<p>[edit] I should mention, everyone who develops on this runs it on systems that are almost identical to production. We use a requirements.txt, and the OS is at the same patch level and everything. Giving contributors a way to run tests locally against their changes on their dev boxes, with a testing framework, would be almost as good as running the app in production. Nobody&#x27;s running this on their laptop or anything like that.
======
nik736
After using most CI/CD tools I can tell you that for your tech stack there is
no reason to use Jenkins. It sucks, typical Java software that looks like
shit, has a buggy interface but somehow runs well enough to be "Enterprise
ready".

Go with Drone, you will thank me later.

I would implement the CI pipeline first, write tests afterwards and make sure
to integrate with Mattermost/Slack, which will make you aware if something is
broken.

I am currently on a Gogs, Drone, Mattermost stack and am loving it. Everything
was super easy to setup and runs just perfectly fine for months now without me
doing anything other than the initial setup. Jenkins was a pain in the ass and
it was the best decision for me to get rid of it.

The beauty of Go is also that it isn't using any resources, this is why I run
Gogs (runs perfectly fine with 512MB RAM) instead of GitLab (you need 8GB+
RAM).

------
laurence-myers
They go hand in hand. If you write tests, but they're not run after each dev
commits changes, you're not testing those changes for regressions. If you have
CI without tests, what is the CI going to do? It could still be used for
packaging executables/distributions, and possibly orchestrating deployments,
but it sounds like that's not your goal.

Save yourself some grief and use a service like CircleCI, it provides isolated
(Docker) environments that are spun up per test run. It supports many
languages and background services (like databases).

As for writing tests, you would be best to start writing them at a "system
boundary". This might take the form of sending HTTP requests to your endpoints
and parsing the response.

~~~
ajeet_dhaliwal
I agree that they go hand in hand. In practical terms however it can take a
long time to write comprehensive test cases for legacy code, even if only
writing integration and system tests and ignoring unit tests.

A pragmatic approach would be to write a few tests quickly first. Then
implement CI as quickly as possible. After the CI is working end-to-end
running the few tests you quickly setup before then you can focus on expanding
coverage of testing. Basically, I'd suggest writing only enough tests to test
the CI first :-) Then once that's in place you can add tests.

One other thing that is important to get right is reporting (at the end of
your CI flow, once each build and test run has completed). This will help you
keep on top of regression but most notably demonstrate the real value of
what's going on with all the work you're putting in now.

You don't have contact information in your profile but I wanted to add that
I've worked in the role of a software engineer focused on automation (CI and
testing) at large tech companies and I felt so strongly about how important
the reporting aspect of this is that I started a web-based reporting service
that is growing in usage. Check my profile for details, I'm the founder,
contact me with any questions or feedback.

------
shoo
> before we refactor the code to support unit tests or start integrating with
> Jenkins (CI), we should start with at least adding some high level tests.

Yes, this is a good idea. before trying to make dramatic changes to internals
(refactoring etc) it is a good idea to get some kind of automated tests around
the system.

you'll get quite a lot of value from a small number of basic smoke tests /
end-to-end integration tests.

> Giving contributors a way to run tests locally against their changes on
> their dev boxes, with a testing framework, would be almost as good as
> running the app in production

a nice way i've seen this done is to focus on automating a build script for
launching the app and any dependencies it needs, running automated tests
against it, then tearing everything down.

figure out how to automate into a script that developers can use to run
automated tests

figure out how to automate the environmental set up (e.g. installing package
dependencies)

once you know how to do those things reasonably reliably for dev machines, set
up a jenkins CI build to do the same thing: check out the code, set up
dependencies, then run the automated test script

Since you've got a web app, it may already be difficult to get automated
reliable set up and tear down of dependencies under control. E.g. you use a DB
- can you automate setting that up and populating it with test data before a
test suite runs? Your app automates infrastructure tasks -- how can you test
that without actually triggering the changes, etc?

~~~
ZanrielJames
> figure out how to automate into a script that developers can use to run
> automated tests

That was my thought exactly. A test suite that starts with a setup, runs a few
e2e "smoke" tests to make sure basic functionality is there, then a teardown.

Since the real action is handled by Ansible, I think it should be relatively
easy to set up a dummy repo that only simulates playbook runs without actually
doing anything. That way you can hit "build" or "patch", get a valid response
back, output even, but it's not actually going out and hitting anything.

My main thing is, now that we've nearly tripled the number of devs working on
it, and it's now being used by another team (was 2 teams using it, now 3),
we'd be irresponsible to add any new features or do further development
without a way for contributors to, at the very least, type "python
runtests.py" or just type "pytest", and see that the site still has basic
functionality.

Here's an example: There's a system in there that allows admins to assign
roles to users. Some JS was added for a totally unrelated feature, but it
ended up intercepting form submits on that page. I didn't find out there was a
problem until a couple days later.

Another time, I was giving a demo to 4 different managers, and I went into
this one section to do something, got a 500 error. Turns out I had a couple
lines at the end of a view function at the wrong indent level.

It's embarrassing. Before trying to set up Jenkins, Docker, whatever else, a
few basic high level tests that cover 5-10 areas of the UI would surely keep
us from embarrassing ourselves at the very least.

Thanks for your input.

------
twobyfour
Your situation sounds remarkably like the state of our (also Python) codebase
18 months ago.

Start with CI and a couple of trivial tests that don't require refactoring - a
smoke test or an end to end test that verifies your home page returns a 200
status code. Expand from there.

I say this because tests are useless if they don't get run. When I finally got
CI set up for our code base, my next three weeks were spent updating abandoned
tests that had been written to verify a dependency upgrade and then hadn't
been run in months.

The real win, though, was integrating CI with Slack. Before that nobody
noticed if a build broke, and I would have to point it out to them. Displaying
an alert publicly in the middle of our primary channel of communication made
it something that couldn't be ignored.

Once you have CI running, developers will start to feel the pleasure of
expanding test coverage and using tests instead of clicks to verify their
work... and the pain of breaking the test suite.

Before we set up CI, every member of our dev team was somewhere between
unenthusiastic and mildly resentful of requests for test coverage. That
attitude has changed - most of our engineers now take pride in writing tests
that pass, and one formerly resentful developer voluntarily chose to target
100% coverage (something we don't yet require) for the most recent feature he
built.

Also, I want to echo another commmenter's suggestion about using a hosted CI
service - assuming you're a small team. It'll get you up and running faster
and require less maintenance overhead. Since we don't use Docker and have a
fairly complex stack that won't run on typical services like CircleCI, we use
a service that handles the queue, displays pass/fail, supports notifications,
etc. but lets us run the test suite itself on our own boxes.

