
How Secret Used Automated Testing for Their Android Launch - weitingliu
http://blog.testmunk.com/how-secret-used-automated-testing-for-their-android-launch/
======
swanson
This is not terribly difficult to get running locally (at a smaller scale).
Square has a tool called Spoon[0] that makes it really easy to run a test
suite across a bunch of devices.

On a recent project, we had 5-7 devices plugged into a spare box that would
run a test suite on every push. It worked out great and the ability to capture
screenshots was really helpful for the design team to help spot strange UI
issues. More details on the setup here: [http://www.sep.com/sep-
blog/2014/01/06/android-robotium-and-...](http://www.sep.com/sep-
blog/2014/01/06/android-robotium-and-spoon-a-practical-guide-for-testing-on-
android-devices/)

But if you need larger device coverage, Testmunk looks pretty awesome :)

[0]: [https://github.com/square/spoon](https://github.com/square/spoon)

------
saclark11
I appreciate TestMunk's goal to make automated mobile testing easier and more
accessible, and I truly hope they succeed in that, but for the love of
testing, please do not view that example feature file at the bottom as an
example of proper Gherkin.

It is riddled with implementation details and brittle explicit waits. To be
fair, they did clearly state: "we generally recommend against fixed waits" but
then please do not put this forth as an example to be copied. Scenarios should
describe the behavior from a user perspective and leave out implementation
details. You should not have to change the your Gherkin no matter how much the
underlying implementation changes so long as the expected behavior remains the
same. Not to mention the misuse of the Given/When/Then keywords and the "I
logout" steps -- that is what before/after hooks are for.

I also recognize TestMunk points out that this feature utilizes standard steps
to get your first test cases going and to get some screenshots. They also
advocate for the page object pattern, which is good, but it might be helpful
to be clear that this is not how you'd actually want to write your Gherkin,
whose true value comes when written with/for product owners, free of
implementation details.

~~~
StavrosK
Can someone explain to me the appeal of Gherkin and other such tools? I don't
see why the example on that page are anything but code that wants to "look
better", and just ends up introducing another thing to maintain in your
application.

How is writing a step in English and writing the English to translate it to
code better than just writing the exact same steps in code to begin with?

~~~
Karunamon
Gherkin/Cucumber's big thing is that it lets the people who can't read code
(i.e. project managers, suits, and customers) understand what's being tested.
Usually along the lines of:

"As a user, I want to buy things, so I should be able to put things in the
cart and get an accurate total price"

or

"As a user, I want to view the site in my own language, given I have come from
a German network address, I should get the German language site"

This frees you from having to explain the finer points of cookies, or geoip,
or any number of concepts to people who do not operate on a level that close
to the bare metal.

The translation isn't really error prone either and isn't hard to implement
from the programmer's side. Say that language example above? The actual test
would be written like:

    
    
        As a user,
        I want to vew the site in my own language
        Given I have come from a German network address,
        I should get the german language site.
    

Those last two things are recognized via a verbatim regex (basically /^I have
come from a German network address$/),

and the actual implementation code does something based on that, say, setting
a test IP to a known German netblock, and the last line would actually load
the site and look for the right localized strings.

So they're rather closely coupled together. If you change the wording of the
test for whatever reason, the old code won't work anymore (actually, I think
Cucumber will tell you that there was no test found), necessitating someone to
actually verify that the test as written actually tests what the new
requirements are.

~~~
thaumasiotes
> "As a user, I want to view the site in my own language, given I have come
> from a German network address, I should get the German language site"

I hate this philosophy so so much. If I wanted the German-language site I'd
ask for it.

~~~
Karunamon
It's somewhat difficult to "ask for it" in a way that's accessible for a large
number of people. Either way, you're presuming something that may or may not
be true.

It's safest to handle the main use case, that is, people using the most
prevalent language of the region, and implement a language chooser for
everyone else.

~~~
thaumasiotes
> you're presuming something that may or may not be true

What's that? I'm presuming that there's more than one language available, but
so does the initial example.

~~~
Karunamon
I mean no matter what you do, you're defaulting to a language that the user of
your site may not speak.

~~~
thaumasiotes
I don't consider it "unsafe", or problematic in any way, that
[http://en.wikipedia.org/](http://en.wikipedia.org/) gets you English
wikipedia _even if you access it from a German IP_ , whereas
[http://de.wikipedia.org/](http://de.wikipedia.org/) gets you German wikipedia
_even if you access it from an Australian IP_. There are plenty of websites
that will not give you the page you request no matter what you ask for, if
they think you _should_ be going to a different page. That's idiotic.

------
tomasien
Secret did NOT get $35M in a Kickstarter campaign, it was venture capital. Or
was that a joke I didn't get?

~~~
herbig
Looks like it was edited on the sly:

[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://blog.testmunk.com/how-
secret-used-automated-testing-for-their-android-launch/)

~~~
chimeracoder
Better link, since Google will eventually update their cache:
[https://archive.is/RAKfE](https://archive.is/RAKfE)

(Yes, I get the recursive irony of archiving a Google cache result).

------
tdumitrescu
Given I wait for 5 seconds

Then I press view with id "splash_signin_text"

Then I wait for 2 seconds

Then I clear input field number 1

Then I wait

...

Looks like someone never bothered understanding what Given/When/Then are for.

~~~
recursive
I think that describes most people. What's wrong with this?

------
matthewcanty
I find the text hard to read on my screen. Does it have to be so gray?

------
CodeSheikh
It was like a mobile phones orchestra. Amazeballs.

