
What Test Engineers Do at Google: Building Test Infrastructure - kungfudoi
https://testing.googleblog.com/2016/11/what-test-engineers-do-at-google.html
======
bit_logic
Summary of the article: At google-scale, integration tests take too long
because it triggers so many service calls. So the solution is to use static
hard-coded mock data. But also create tests to verify the mock data with real
service calls. So previously, if you had N tests calling a rest service A, it
would result in N calls to service A. But now, those N tests go to a hardcoded
mock data, and there is a single test verifying the mock data by calling A. So
N tests can run fast and there will be only a single call to A to verify the
mock.

However, that would be the best case, which is the mock data is sufficient for
all N tests. That is not likely to be true, but probably a M number of mocks
can still cover all N tests. And M < N in most cases so there's still savings
in number of service calls. In the worst case, N tests will need N mocks and
it will be the same as before.

~~~
sly010
In a large graph you would re-test the same part of the code over and over,
because many top level inputs is going to do trigger the same code-path deep
down in the graph, which is wasteful. In a sense they memoize service call
answers, so any tested code-path only needs to run once.

If they actually do this across their code they will never have to run "real
service calls", because those services are also tested the same way. It's
mocking all the way down.

~~~
bit_logic
Paraphrasing the famous quote: There are only two hard problems in CS: naming
things and cache expiration.

An issue I see with this is that there's a potential window when tests pass
with bad data. For example, tests using a mock and the mock is periodically
verified with a service call. The mock could become bad data, but won't be
marked as bad until the next service call verification. Until that happens,
the tests using the mock will all pass. It's not clear from the article how
they address this.

~~~
dmoy
... and off-by-one errors.

------
lifeisstillgood
I am writing a book on software development (current title "The Software
Mind") and I am more and more concluding that the vast majority of modern
software development is scaffolding around the actual thing you want to do.

Writing tests is not something you knock up during coding, it needs
infrastructure - a test suite, some test data, a Jenkins server and then
whoops we need better test data and ...

This touches that feeling - I guess I need to write it down a few more times
...

~~~
ebiester
This is absolutely true. It's an observation many of us have had. There are
people that are trying to develop startups to make it easier, but the
combination is hard.

I think the next successful framework will focus on the entire lifecycle and
aim to solve the scaffolding problem.

~~~
UweSchmidt
Maybe the language/framework/GUI could generate or require
unit/integration/system tests (with just a few clicks to confirm or correct an
assumption), and visualize in broad strokes which scenarios or input ranges
are "green" yet.

That would mean a lot more infrastructure (sorry if that wasn't what you
meant), but automatically provided or accepted by everyone involved.

------
dm03514
In my experience end to end tests are often difficult to write tests for. It's
been difficult to maintain frameworks that model arbitrary asynchronous end to
end tests, the test frameworks often tightly mirror the use cases. I haven't
really seen any good tools that allow non-technical testers to write complex
end-to-end tests. This is a shameless plug for a project that is hardly more
than an idea, but to address this, I am trying to create a flexible end to end
test framework that deftly handles asynchronous state transitions and allows
non technical people to write tests in a declarative way. It has not even
reached functional proof of concept state yet :(

[https://github.com/dm03514/async-states-task-
engine](https://github.com/dm03514/async-states-task-engine)

The goal is to model end to end tests as state machines and define conditions
to trigger transitions. There should be a code based interface allowing for
arbitrary transitions using any go library, and a higher level interface,
allowing a handful of operations to allow non-technical testers to model end
to end tests.

~~~
ben_jones
I wonder if such a process could be done without a "non-technical friendly"
DSL. Teaching people to speak to computers through declarative languages is
programming in one shape or form no matter what (IMO).

I wonder if you could record a user action, and add variance to the recording
as a test? Break a recording into ~5 actions and then perform those actions at
various speeds and points of repetition.

A recording of

1) Move mouse from A to B

2) Click button

Produces a range of tests including

1) Move mouse from A to B

2) Click button 3 times

~~~
crdoconnor
This is basically what I did with
[http://hitchtest.com/](http://hitchtest.com/)

The declarative language is a readable YAML, which translates each named step
into a python function call with arguments (e.g. - click: submit button).

The asynchronous part is 'under the hood' where it runs services which log
lines of JSON. Those are watched using epoll triggers and parsed into objects
which can be verified by one of the steps (e.g. check email containing name of
user is sent).

I wouldn't say that it's necessarily possible nor desirable for a non-
programmer to write the stories using YAML, however. Even with a perfect
declarative language you need a mindset that requires precise thinking in
order to write executable stories and that usually means programming ability.
Stories also need refactoring as much as code does, alongside engine code.

However, the YAML _ought_ to be useful as a way programmers can
collaboratively write and refine stories with product owners/managers and
should serve as a useful way to communicate back changes to the product that a
non-programmer can understand.

------
marknadal
End-to-end testing is super important, but still hard for a lot of
people/companies/startups.

Using stubbed or mocking based approach can only go so far, despite it being
less "brittle" than end-to-end tests. That is why our team built PANIC, which
is a distributed testing framework for the every-programmer (
[https://github.com/gundb/panic-server](https://github.com/gundb/panic-server)
). Everybody should be able to do tests like this, not just the Googles.

~~~
gertef
end-to-end tests are only brittle if your end-to-end-system is brittle. That's
not a problem with the tests.

~~~
0xCMP
It's the natural thing for services to be brittle. We're all human after all:
a mistake in recruitment, hiring, training, scoping, managing, documenting, or
etc. can cause an engineer to make the mistakes needed to make a brittle
service you might rely on and that's something I think we can all agree is
something you just need to accept as "going to happen."

In that case it's worth dealing with it as it is not how it should be.

------
djfm
At our company developers rarely write tests. They rely on the QA for testing.
Obviously it doesn't work, like, at all. I think a key aspect of this post is
to highlight the fact that testing is in fine the developers' responsibility.
Test engineers are here to make it easier for them to write tests.

~~~
Afton
There are levels and levels at which companies engage in Q.A.

1\. There is none, and it's awful, once there are more than two developers.

2\. There are dedicated QA people, who click on things in a more or less
formalized fashion depending on the company, and sign off on releases. This is
"pretty bad" at most places once you have a product that can't be easily
verified by a couple of people in a few hours

3\. You have dedicated Test Engineers, who are just software developers who
write test automation. This can be good, but usually ends up with mid-low
quality engineers filling this role for a variety of reasons.

4\. Test Engineers are just Software Engineers who specialize in test-
automation work. They write automation in high bug areas, and keep a larger
overview of the product (set) to identify problematic areas and work with the
product developers to solve those problems.

5\. The article. Test Engineers write infrastructure for testing and
reporting, (and possibly proof-of-concepts). Product developers are
responsible for their own quality. Test Engineers may also act as consultants
for product teams.

It's probably worth pointing out that 2-5 are all reasonable places to be for
different sized companies, at different stages. We shouldn't assume that
everyone should be at a company that is 'like Google', since things that make
sense at Google scale may not make sense for your 20 developer startup.

edited for formatting only

~~~
adrianratnapala
#5 is all very well, but I don't think helping devs write test is a substitute
for actually having QA people. Testers should have a mentality of "I want to
break this", and like it or not, the developers just won't think like good
testers.

Do Test Engineers at Google do this kind of work as well?

~~~
yaaaq
To add another question from an sre (swe), tools and infrastructure is a
subset of my daily responsibilities (60%). Reading the couple comments it's
hard to see where the line falls between these two roles. I'm on a smaller
team so perhaps I just find myself handling both roles or.. ?

------
zupancik
Is Test Engineer different from SETI (Tools and Infrastructure)? What does
SETI do?

~~~
curiousDog
Search for great tools and engineers on far away, exotic planets :-)

~~~
junke
The begs the question: can we SETI at home?

------
lifeisstillgood
While it took a day to build a proof-of-concept for this approach, it took me
and another engineer a year to implement a finished tool developers could use

This!

~~~
carsonreinke
I believe you are missing a link to "This!"

------
aslakhellesoy
I gave a presentation called Testable Software Architecture [1] a week ago
with very similar recommendations to this article.

Decoupling is essential in order to have fast, maintainable tests that give
you confidence to deploy continuously.

My two favourite techniques for this is a ports&adapters architecture where we
plug in fake adapters for the majority of the tests. We then use contract
tests to be confident that the fakes behave the same as the real services.

[1] [https://skillsmatter.com/skillscasts/8567-testable-
software-...](https://skillsmatter.com/skillscasts/8567-testable-software-
architecture)

------
sheraz
Funny this should appear because I just started at a company today where a big
part of my job will be to help the team work with big messy monolith of code
that is difficult to test.

Can anyone here recommend further reading on this particular problem?

~~~
dockd
The book "Working Effectively with Legacy Code" by Michael Feathers.

~~~
greendragon
Seconded. Some of it feels a bit dated, and it doesn't cover tools you can use
like PowerMock if your codebase is Java and you really just need to get a
piece of hairy code under coverage as quick as you can without having to
refactor it all to be testable the 'normal' or 'clean' way. Nor does it cover
refactoring for a functional and immutable approach (its approach is much more
OO). Still a great resource to have on hand and be inspired by.

The most important thing is to follow its advice and just start doing it
little by little, even small refactors of other people's code. A friend's
company had a Book Club and some testing book got on the list at one point,
and as they were discussing it they kept having arguments over how effective
various things were, so they resolved the arguments by starting a Testing Club
where every week everyone would put in at least an hour into getting some part
of the system under test, or trying out a testing technique, and discussing
it. Over time they got most of the product under test and fixed a lot of
previously unknown bugs.

~~~
u14408885
This might not exist, but could you recommend a text that _does_ cover
refactoring for a functional and immutable approach? I would be interested in
reading something like that.

------
kleigenfreude
Some feedback for the OP:

1\. The tester and another team member spent a year developing something that
would intercept calls and relay them. Two problems with that: (1) two person
years spent, (2) and that sounds like serious NIH (not invented here)
syndrome. The problem that should have been solved was everyone spending the
time to write better tests and changing code as needed. Instead, they spent a
year on a workaround, invented in-house. Was there not anything else out there
that did this?

2\. The word is "focused", not "focussed".

3\. Lack of detail: how exactly does it work beyond that basic diagram?

4\. Where's the code for the project? Would it be useful to others?

However, I admire that the OP posted their experience, and it is useful
information.

~~~
sampullman
I believe "focussed" is acceptable, although "focused" is preferred in the
US.[1][2]

Can you explain why this sounds like serious NIH syndrome? It looks like they
built a system to cache service requests on top of an existing test framework.
It seems specific enough that there might not be an existing method that fit
well enough. The article is a bit light on details though, so I suppose it's
hard to tell.

[1]
[https://en.wiktionary.org/wiki/focussed](https://en.wiktionary.org/wiki/focussed)
[2]
[http://dictionary.cambridge.org/dictionary/english/focused](http://dictionary.cambridge.org/dictionary/english/focused)

------
whenwillitstop
Could this be one of the most meaningless and trite jobs whilst still
maximizing prestige? How many people will work this job in despair? How many
intelligent people will spend years coding the obscurity that is test infra?

~~~
dang
Please stop posting unsubstantive comments to HN.

~~~
johansch
He/she did address an interesting and touchy subject of software development.
It could have been more diplomatic, but then also, I'd say, needlessly watered
down.

A challenge in software development has long been the division between test
and development.

You could do as Microsoft recently did (more or less abolished "test").. the
jury is still out on whether that is what has caused the recent Windows 10
quality issues.

Or you could keep a tester/developer separation. Good luck trying to recruit
top (e.g. your testers should on average be as smart as your developers)
people for the tester positions unless you are Google/Facebook.

Either way, I think this is a really interesting issue.

It should be noted: some pieces of software are a lot easier to test by its
developer (say, a compiler) than others (say a GUI).

~~~
enigmo
MS abolished manual testing (STE) positions prior to Windows Vista being
released, somewhere around beta 2 or rc1. Vista probably suffered from this
but Windows 7 didn't.

~~~
jacalata
No they didn't. Source: I joined MS after Vista was released and worked with
manual testers.

~~~
enigmo
Did they have the STE title or SDET? When this happened it wasnt like there
was 100% automation in place so developers, architects, SDETs, had to pick up
the slack. I personally got stuck verifying bug fixes for our components
(kernel stuff and supporting user mode services) in various languages I don't
speak (particularly fun for left to right languages) just to get the bug count
down. That team had something like 20 STEs that got the axe post beta 2.

~~~
jacalata
No idea what their titles were but they worked entirely running manual tests.
I ran some manual tests myself as an SDET but this was different.

