
The path to production: how and where to segregate test environments - Bary0n1cMatt3r
https://circleci.com/blog/path-to-production-how-and-where-to-segregate-test-environments/
======
bunderbunder
> ...you should be using mock external services and databases to keep things
> running fast.

I've become skeptical of this advice, of late. If the database is just a dumb
object store, then I suppose it's fine. But I've also seen a fair bit of
rework resulting from tests not catching a bug that was only reproducible when
working against a real database. You then end up needing additional tests to
cover that sort of stuff, which takes extra time to develop and maintain.

Between that and the whole "test-induced design damage" thing, there's some
amount of test slowness I'll accept in return for reducing the development
effort needed to maintain the tests. On the general principle that computers'
time is way, way cheaper than humans' time.

As far as keeping tests that rely on stuff like the database running quickly
goes, my new preferred plan of attack is configure those components to use RAM
disk in the local dev and CI environments.

~~~
bluGill
In my experience a real database is not slow.

The real database is slow only if the only one you can use is the production
one (it should be obvious why this is a bad idea) If you copy the schema and
import it into sqlite with just the data you need for the test it is fast.

Mocks do have their place, but people get confused about the place. The
purpose of a test is to assert "This will never change". That is a very strong
statement. With a mock you are asserting that you will always call that
function with those parameters - this is rarely what you want.

~~~
bunderbunder
Substituting a different database for testing purposes is right up there with
trying to mock the database, in my book.

It places big constraints on your development, because you get stuck limiting
yourself to the subset of SQL that is supported by both databases. That
usually means trapping yourself in the 1990s, which, given everything that
been implemented in the major RDBMSes over the past 20-25 years, implies
accepting an artificially low ceiling on the performance, and possibly the
semantics, that you can get out of your data store.

Second, even if you're fine with sticking to a small subset of your database's
features, you're still not guaranteed that the two DBMSes will behave the same
way given the same input. This is particularly true if one of them is SQLite,
which is dynamically typed.

With the possible exception of key/value stores, databases generally exhibit
such complex behavior that, if you're looking to conclusively verify that
logic involving your database actually works, you probably shouldn't accept
any substitutes.

~~~
bluGill
I would argue that sticking to 1990's sql is good anyway as it means the
business can switch to a different engine. There are several great databases
out there with support, if you limit yourself to standard sql you can throw
our your current vendor if they don't give you enough value (which could be
support, speed, or even the salesman beats you in golf)

I understand sometimes the extensions are worth using, but good engineering is
to limit those uses to where it actually is significant.

~~~
bunderbunder
If you're looking for the ability to switch data storage technologies, you'd
do better to just make sure your data storage layer is cleanly isolated.

Sticking to 1990s SQL gives you the ability to swap among 5 or 6 different
databases, all of which have been around for a day or two, none of which have
seen a whole lot of change in their relative strengths and weaknesses in a
good long while. It's just not super likely that you'll find a compelling
reason to switch among them, especially if the people who made the original
choice did their homework first.

Keeping a well-isolated data layer, on the other hand, gives you a much wider
universe of things you can attainably switch to - document stores,
triplestores, graph DBs, Cassandra, Hadoop/Hive, stuff that hasn't been
invented yet, etc. I've lived through my fair share of database technology
migrations.

So far, every single one of them has been this kind, not a SQL -> SQL
migration. So, while you're at it, might as well get your money's worth out of
the SQL DB.

------
GordonS
I typically use something similar for web apps:

1\. Local Development 2\. Development 3\. Test 4\. QA 5\. Staging 6\.
Production

Local Development is the dev's local development environment. For performance
reasons, and so we can work offline when required, we usually try not to touch
any 'real' cloud-hosted components here, and run a local database and blob
storage emulator in docker containers.

Development is a cloud-hosted environment where devs can deploy to at any
time, for example to test that some code they've written works with real PaaS
components.

Test is where our CI pipeline automatically deploys code to after every
succsessful build, unit test run and integration test run, so as well as
testing the code, this stage also tests our database migrations and
deployment. Despite hitting a real database, we usually manange to keep our
integration tests fast.

QA is where our testers and pilot users can test new features. We only deploy
stuff here when we have something worth showing - the point is to have a
relatively stable environment to test in.

Staging isn't a 'real' environment for us - when we reach here, database
migrations are executed against the production database, and a Staging version
of the web app is deployed that runs against it _. This means we can do smoke
tests before the final deployment to Production.

As an aside, our web apps are all hosted on Azure, using their App Service
PaaS. When we move from Staging to Production, what we are actually doing is
just swapping the slots around, so we get zero-downtime deployments to
Production.

As another aside, we generally don't run a replica of all cloud services for
each environment - mainly for cost reasons. So, all the test/non-critical
environments (Development, Test, QA) are hosted on a single instance of each
cloud component: we use a single App Service, with a slot for each
environment, we use a single database, with a separate schema for each
environment, etc.

_ This takes a lot of discipline to work, but in essence you need to make sure
that any database changes are backwards compatible, since the Production web
app must be able to run using the newly migrated database - so no
columns/tables can be dropped, new columns must have default values etc.

------
beaker52
How about testing in production? I'm heading in the direction of doing most of
my testing there since all to often qa/test/dev environments don't actually
represent the production system very well and it feels like we're doing it for
the sake of it.

Deploy your new service alongside and mirror traffic. Treat your production
system as a living organism that you're running tests inside of. Assess the
impact/risk of your changes and decide your test strategy. Make large changes
in small increments. And most importantly monitor all your changes with
metrics, do canary releases when you're rolling out the real deal.

Spend more time testing and validating in production and less time building
extensive and usually flaky e2e suites and focus on well pointed smoketests
that you run in production.

Reduce the time it takes from writing code to seeing it in production to close
to zero and painfully waiting for thousands of unit, integration and e2e tests
and getting signoff in n different and yet never quite like production
environments suddenly seems like an ancient practice. "How do you know it
works?" "The metrics, logs and traces tell me it works"

Of course, it's a living organism, you don't want to screw it up, but there
are plenty of ways to isolate your services under test. It definitely isn't
something you can just start doing overnight because you'll need tooling and
infrastructure to make it happen.

~~~
pytester
I've had success making an effort to build more realistic and isolated
integration tests.

One technique in particular that has worked well is to identify small
infrastructural problems in my integration tests that prevent me from writing
integration tests that replicate real production/QA bugs and, when cost
effective, building out infrastructure to enable me to write a test that
mimics those conditions.

E.g. take the selenium tests that just test with chrome and get them to run on
firefox to catch those firefox only bugs.

This often allows me to uncover whole classes of bug before the code even
leaves the testing environment, never mind hitting prod.

