I thought this was common knowledge and that it became even easier after Docker became a thing?
Mocks are wishful thinking incarnate most of the time, though here and there they are absolutely needed (like 3rd party APIs without sandbox environments, or quite expensive API, or most of the time: both).
Just pick a task runner -- I use just[0] -- and make a task that brings up both Docker and your containers, then run your test task, done. Sure it's a bit fiddly the first time around but I've seen juniors conquer that in a day maximum and then your tests actually work with the real world 99% of the time.
Mocks in general are rarely worth it, the DB ones: 10x so.
testcontainers is great. I struggled a bit with testcontainers due to the nature of one container per test which just felt too slow for writing gray/blackbox tests. The startup time for postgres was > 10 seconds. After a bit of experimenting, I am now quite happy with my configuration which allows me to have a snappy, almost instant testing experience.
My current setup:
- generate a new psql testcontainer _or_ reuse an existing one by using a fixed name for the container
- connect to the psql container with no database selected
- create a new database using a random database name
- connect to the randomly generated database
- initialize the project's tables
- run a test
- drop the database
- keep the testcontainer up and running and reuse with next test
If your table setup process starts to get slow like ours, checkout psql TEMPLATE (https://www.postgresql.org/docs/current/manage-ag-templatedb...). Do the setup once to a db with known name, then use it as the template when creating the db for each test.
Yes it’s a shame that this is not leading the discussion, works very well and surprisingly efficient in our experience compared to manually setting up docker containers or docker compose
I've had a nice experience using Docker Compose for this, although you might still want a task runner wrapper around it. Podman Compose should work fine as well.
> Also keep your lynch pin invariants in the database and not in your code.
I do both. That is how you protect against stressed devs at 2:00 AM trying to insert crap in the production database and how you also make good apps that don't blow up with database exceptions and give you neat validation errors in forms and/or JSON `errors` keys (when it's an API).
I am definitely not talking about a full copy of the data here. I more head in mind a small but representative sample, enough to run and exercise units of code with confidence.
Exactly what I do. Having a local `docker-compose.yml` that helps bring up a dev and test environments is one of my first action items when hired. Paid huge dividends, many times.
I've beconeea fan of good fakes. Good fakes closely resemble the real service but let me query useful things in tests. Mocks just tell me I called a function without concern for the 10 different ways to doethe thing.
If your mocks are hard to write you may have your fanout too wide and would do better with squaring up your dependency tree a bit. I have a rule of 5 for DI frameworks. If any module needs more than 5 other modules you should start looking to refactor to push dependencies down the stack.
What I see with Fakes too often is that they are difficult to write so people reuse them, end up writing too many tests that only test the fake, and couple a bunch of tests to each other which makes refactoring fraught. Anyone who makes refactoring difficult is an enemy to your project, not an asset.
Mocks make DAMP easier. You don’t care how the other function turns “foo” into “bar”, you just care what happens to this function when it gets “bar” as a response. That’s the surface area.
There’s a lot of people in this response chain who are very obviously conflating “unit tests” with all tests. Unit tests don’t care about integration. That’s integration tests. I liken testing to plumbing. Unit tests are similar to quality control at a pipe factory. You verify that the pipes aren’t damaged prior to assembly. The plumber inspects them before installing, and inspects their work as they bond or solder them together. But at the end of the day your plumber still has to turn on the faucet and make sure water 1) comes out the taps and 2) stays in the pipes everywhere but the sink, otherwise they haven’t done their job.
Unit testing the individual pipes is necessary but insufficient. You have to verify the connections and the end to end system still, but more work up front saves you some embarrassing rework at the end.
NOBODY HAS DEFINED WHAT A UNIT IS! This needs to be shouted because all your arguments fall apart when you realize this. Until you get down to atoms (or something in quantum mechanics) you can go a level deeper. String.length is a integration test if you are the hardware engineer building the microcode on the CPU (do CPUs use microcode anymore?). If you are laying out the traces tests for the microcode are an integration tests for you. When I write a simple function that happens to use string.length I'm an integration test for that. Note that I skipped many many levels in both directions.
What people care about is that things work. The term unit test and integration test needs to die as they are not useful. Test at the right level. If you can write all your tests as full system tests (that is via UI input) that runs fast and are reliable then do so. We generally say don't test via UI input because painful experience is UI changes often in ways that break 1000s of tests all the time, which is good reason to inject tests below the UI. There are often other reasons to test break up tests into smaller areas, but that is trade offs.
Odds are you are not writing basic containers (ie string.length) and so your API is something that should be easy to refactor (you have no reason to think it won't change in the future as requirements change) and thus you should not test that API. Tests are an assertion that in the future this will not change, so if you are testing something that should change you are doing it wrong! (of course you will never get this 100% right, but that should be your aim!)
---
If your mocks are hard to write you may have your fanout too wide and would do better with squaring up your dependency tree a bit.
Mocks are not hard to write. They are the wrong solution to most testing problems though. Mocks assert that you call a specific function as expected - it is really easy (in most languages) to automate writing with a framework and so they are easy to write. The problem is calling a function is not what you want to test in most cases. I used write as an example because file IO APIs generally have many different forms of write. You have WriteWithBufferAndLength you have WriteAVector (or whatever your language calls the basic list container), then WriteWithSomeWeirdOptionsWeForgetToPutInOriginalAPIAndDidNotWantToRefactorEverythingSoWeAddedANewAPI... Some languages these are all named different, some allow overloading - it doesn't matter, the important part is mocks don't let you switch which one you use as 100 tests will break if you try even though switching might be better it isn't enough better to be worth trying to fix all the tests that happen to use that mock. Making the above worse, write often can be called with one large buffer or many smaller buffers and the end result is the same yet mocks assert how many times write is called which is wrong.
> What I see with Fakes too often is that they are difficult to write so people reuse them, end up writing too many tests that only test the fake
When I write a fake I write one fake that close for everyone to use that fake in their tests - I know how the thing it is faking works in production and model that behavior except for whatever it is that makes it unsuitable to use in real tests. I often do add verification helpers - you can ask what data was written at any point (including concatenating several writes) for example.
Guilty as charged, I mix them both freely; I use fakes but also assert that a function has (not) been called [N times]. The latter is mostly testing implementation but there are rare cases when it's too much work to mock a quality 3rd party library (say, an HTTP client) but you still want to mock parts of it and ensure they get called.
Not sure why mixing up both things makes the conversations tortured though? Can you clarify?
Mocks are great for testing specific code paths, but if you need to do that, there is usually a better way of doing it. Mocks hide contractual incongruities. For example, a function that returns either "fizz" or "buzz" that gets mocked out. If it gets changed to also return "bar", and you forget to update the mock, you've got a time-bomb in your code.
> If it gets changed to also return "bar", and you forget to update the mock, you've got a time-bomb in your code.
In that case the problem isn't so much that you forgot to update your mock as much as the changed function no longer satisfies the interface. Ensuring that an implementation properly satisfies the interface is not within the scope of tests where mocking would be useful.
> Ensuring that an implementation properly satisfies the interface is not within the scope of tests where mocking would be useful.
As I mentioned, if you need mocks to test a code path, you are probably "doing it wrong" as there are much better ways to do that. Such as refactoring the code so you can test just that code path. Mocks are a code smell, IMHO. In other words, if you need mocks, then the interface/contract is a part of your code -- and you should test it.
> if you need mocks to test a code path, you are probably "doing it wrong" as there are much better ways to do that.
Most code paths you want to test are error branches, and creating real error conditions is hard to completely unrealistic (e.g. causing hardware malfunction on each test run). What is the better way in your mind?
Mocks can only exist at the integration points, fundamentally. There is a natural interface there, an interface that you are not supposed to be testing. Those tests already exist around the module that provides the integration. You don't need to test it again.
If a mock standing in for the real thing causes ill-effects, you've done something wrong. You're not supposed to be testing the mock while using the mock. If your mocks need tests (they probably do!), they should be tested outside its use.
Heh, when you are working on hardware, you usually build a real test device and yes, actually cause real hardware faults. Mocks or tests will not prepare you for the real thing; as the hardware fault you detect is usually just the surface of the problem. Let's examine a practical example where a disk becomes full. Suddenly, file creation will fail, as will writes, yet, how do you handle that? In isolation, you may mock that condition out so you can test it. You handle the error in isolation, but this is actually quite dangerous. Your application should fail explosively -- don't be like some popular databases that just continue as though nothing happened at all, corrupting its state so it will never start again.
Generally, if you can detect a hardware fault in your code, crash unless you know for certain that you can actually recover from it -- meaning you know what the problem is (can somehow detect the difference between a missing disk or a missing file). 99.9% of the time, you cannot recover from hardware issues with software, so it is pointless to actually test that case. Please, for the love of working software, just crash so the state doesn't get corrupted instead of trying to overengineer a solution.
> Heh, when you are working on hardware, you usually build a real test device
All software works on hardware. Your web application doesn't need a test device, though. The hardware is already tested. You can treat it as an integration point. But even if you did create a test device for whatever reason, that's a mock! Which you say is to be avoided, and that there are better ways, without sharing what those better ways are...
> Please, for the love of working software, just crash so the state doesn't get corrupted instead of trying to overengineer a solution.
While you're not wrong, you need to test to ensure that it actually crashes. All defined behaviour needs to be tested, and you have defined behaviour here.
> Which you say is to be avoided, and that there are better ways, without sharing what those better ways are...
That's because it would better fit in a book than an HN comment, not because I don't want to answer. Basically the gist is to write "obviously correct" code that doesn't need to be tested, along with an architecture that lends itself to being testable without mocks.
Most people tend to write an interface and then inject a concrete type that could also be a mock. I've seen tests written this way that need to mock out 20-60 things just to test the one thing they want to test.
In most web frameworks I've worked with, this is mostly unavoidable as most frameworks provide dependency injection that is natural to mock.
If you aren't using a framework that has an opinion on how things should work, mocks can be avoided completely through different techniques, such as test harnesses, replacing dependencies with alternative implementations (such as in-memory queues instead of cloud services, sqlite instead of heavy databases, etc), etc. Sometimes you don't have any choice but to not use mocks, for example, distributed systems usually avoid mocks for certain kinds of tests because they simply can't be emulated very well -- either due to latency, network partitioning, or network failures that are just too numerous to mock out (similar to the disk issue I was referring too earlier). You don't know if a node is down, or the cable got cut, and need to behave appropriately to avoid split-brain scenarios. In these cases, test harnesses that can emulate specific scenarios are much better.
> Basically the gist is to write "obviously correct" code that doesn't need to be tested
I don't see how that follows. The purpose of testing is to document the code for the understanding of future developers, not to prove correctness. The only 'correctness' a test proves is that the documentation is true. Which is still incredibly useful, as I am sure you are painfully aware if you have ever dealt with legacy forms of documentation (e.g. plain text files, Word documents, HTML, etc.) that quickly become out of date, but is not a statement about the code itself.
> such as test harnesses, replacing dependencies with alternative implementations (such as in-memory queues instead of cloud services, sqlite instead of heavy databases, etc), etc.
These are all mocks, ultimately. Some desperately try to give them different names, but it is all the same at the end of the day.
> The purpose of testing is to document the code for the understanding of future developers, not to prove correctness.
Hmm. I've never seen tests with that goal in mind, except for behavioral tests that test the acceptance critera.
> as I am sure you are painfully aware if you have ever dealt with legacy forms of documentation [...] that quickly become out of date
I have, but allowing that to happen is a culture-issue, not something that is guaranteed to happen. When I open PRs to open source software, I always include a PR to the docs if it changes anything. At work, updating the docs is part of the default acceptance criteria and is usually the thing we do before writing any code, and goes through a PR process just like the code. But, we service enterprise customers, so we aren't going to be giving them code or tests to understand how to use our product.
> These are all mocks, ultimately.
This is a software field and there are specific words with specific meaning; trying to shoehorn things that aren't those things to generalize a meaning is acceptable when teaching. It isn't acceptable when working on those things. In other words, I would accept this if trying to explain the concept to a junior engineer, but not from a senior engineer to a senior engineer.
Then you've never seen a test, I guess. That is the only goal they can serve, fundamentally.
> I have, but allowing that to happen is a culture-issue, not something that is guaranteed to happen.
Mistakes are guaranteed to happen given enough output/time. No matter how hard you try, you are going to make a mistake at some point. It is the human condition. In the olden days one might use a proofreader to try and catch the mistakes, but with the advent of testing a computer can do the "proofreading" automatically, leaving the human effort to be pointless.
Maybe in the age of LLMs we can go back to writing documentation in "natural" language while still using machines to do the validation work, but then again if you write code you probably would prefer to read code. I know I would! The best language is the one you are already using. Having to read code documentation in English is a horrible user experience.
> This is a software field and there are specific words with specific meaning
Sure, but in this case you won't find any real difference in meaning across the vast array of words we try to use here. The desperate attempts to try and find new words is to broach silly soundbites like "mocks are a code smell", so that one can say "I'm not mocking, I'm flabbergasting!", even though it is the same thing...
> Then you've never seen a test, I guess. That [,not to prove correctness,] is the only goal they can serve, fundamentally.
I cannot wrap my head around this statement. It's literally in the name: "test" as in to prove something works... hopefully as designed.
> Mistakes are guaranteed to happen given enough output/time. No matter how hard you try, you are going to make a mistake at some point.
Yep, and they do. Its really easy to figure out which one is right: if the docs say that something happens, it happens. If the code doesn't do what the docs say, the code (and the tests) are wrong; and not the other way around.
> Having to read code documentation in English is a horrible user experience.
It's the difference between intention and action! I worked with a guy who opened PRs with totally empty descriptions. It was annoying. When I was reviewing his code, I had to first figure out his intention before I could understand why there was a PR in the first place. Was he fixing a bug, adding a new feature, or just writing code for the hell of it? ... nobody knew. Then, when you spotted a bug, you had to ask if it was a bug or on purpose, because you didn't know why the code was there in the first place.
Documentation is that living PR description. It doesn't just tell you WHAT exists, but WHY it exists, what purpose it serves, why that weird little line is the way it is, etc., etc.
> It's literally in the name: "test" as in to prove something works...
The documentation is what is under test. It proves that what is documented is true. It does not prove that the implementation works. This should be obvious. Consider the simplest case: A passing test may not even call upon the implementation.
I have most definitely seen that in the wild before! More times than I wish I had. This is why TDD urges you to write tests first, so that you can be sure that the test fails without implementation support. But TDD and testing are definitely not synonymous.
> Its really easy to figure out which one is right: if the docs say that something happens, it happens.
Under traditional forms of documentation, you don't have much choice but to defer to the implementation. With modern documentation that is tested, typically the documentation is placed above the implementation. Most organizations won't deploy their code until the documentation is proven to be true. The implementation may not work, but the documentation will hold.
> I worked with a guy who opened PRs with totally empty descriptions.
I'm not sure PRs fit the discussion. PRs document human processes, not code. Human processes will already typically be in English (or similar natural language), so in the same vein the best language is the one you are already using. That is not what we were talking about earlier; but, granted, does do a good job of solidifying the premise.
Completely agreed, that's why I have advocated for periodically (once a month) revisiting the test suites with them by doing re-captures of real data. There are frameworks that help with this.
Is it a chore? Absolutely. But peace of mind is important.
The person who changes it should write a test to verify his change, right? Ultimately the person should write the test in the same file as the previous tests, chances are high it will be seen there. And even if not, the person doing the change should write a test, so no time bomb there?
And yes, people forget to write tests, sure, but even then it would be a time bomb without mocks.
I‘m not a friend of mocks either, but most examples here are not really an issue with mocks.
Generally, mocks are quite far away from what they are mocking. You might mock out a library, for example. I can guarantee you that the library author will not show up at your workplace to update your mocks.
Mocks are wishful thinking incarnate most of the time, though here and there they are absolutely needed (like 3rd party APIs without sandbox environments, or quite expensive API, or most of the time: both).
Just pick a task runner -- I use just[0] -- and make a task that brings up both Docker and your containers, then run your test task, done. Sure it's a bit fiddly the first time around but I've seen juniors conquer that in a day maximum and then your tests actually work with the real world 99% of the time.
Mocks in general are rarely worth it, the DB ones: 10x so.
[0] https://github.com/casey/just