
Ask HN: How does everyone handle mobile browser testing? - trevelyan
Hi Everyone,<p>So I&#x27;ve been working on a mobile game engine for Saito (https:&#x2F;&#x2F;apps.saito.network&#x2F;arcade) and am running into testing issues.<p>Right now I&#x27;m having to manually run the applications on whatever cellphones and tablets we can find. Is there a better way to do this? Asking as I figure if anyone has solved this problem with an automatic testing suite or something similar it will be someone who reads HN....<p>Goal is to test mobile browsers to see which ones work and eliminate avoidable failures. Most of the problems we are flagging to date are javascript-implementation issues or things like auto-transaction &quot;features&quot; on foreign-language browsers that strip events from the page and&#x2F;or stop them from running very quickly.<p>Any suggestions? Anyone have a startup that tackles this sort of problem?
======
blueimp
For local automated testing of mobile browsers on both iOS and Android you
probably want to have a look at [https://appium.io/](https://appium.io/),
which uses the Webdriver protocol.

If you want to run the same tests against both mobile and desktop browsers, I
recommend [https://webdriver.io/](https://webdriver.io/), which uses Appium
for mobile testing.

If you're interested in a Docker setup as a boilerplate to test against all
major browsers (both mobile and desktop) with best practices for e.g. test
video recordings, you might also wanna take a look at this repo of mine:
[https://github.com/blueimp/wdio](https://github.com/blueimp/wdio)

~~~
trevelyan
This is fantastic advice. Thank you so much for sharing!

Multiple upvotes if I could.

~~~
blueimp
You're welcome! :)

------
ajeet_dhaliwal
Are you manually testing, or programmatically/automated? If it’s the latter
then Browserstack ([https://www.browserstack](https://www.browserstack))
provides mobile devices for testing on mobile browsers in a cloud device farm,
there are others too like SauceLabs and AWS Device farm. I’ve had good success
with BrowserStack but you should check whether they support the browsers you
need, they also provide an easy way to tunnel through to your staging/qa
environment if you need to do that. For reporting, especially if it’s not just
you and you’re in a team, consider Tesults
([https://www.tesults.com](https://www.tesults.com)), I’m the founder so hit
me up with a message and I’ll do what I can to get you setup.

~~~
trevelyan
Right now it's been manual stuff. We get reports of users have trouble on
bizarre mobile phone combos. Midnight here and just back from the gym, but
I'll make sure we check out BrowserStack and Tesults tomorrow and let you know
if we hae any questions.

------
mariefred
Appium with Selenium (or XCUITest for iOS) will shorten your life expectancy
and grow you some grey hairs, but there is no other better alternative for
testing at user level. It is highly unreliable and each new phone OS version
will tend to break your automation

As for finding devices there are a few cloud providers like xamarin or
perfecto (never really worked with them in production) that gives you remote
access to a lot of real devices and software versions

Having said that try to divert more testing towards lower levels- unit tests,
testing on simulators, integration tests with a full phone and browser etc.

A good place to also ask is:
[https://testersio.slack.com](https://testersio.slack.com)

~~~
trevelyan
thank you

------
pulkitsh1234
The first task to write some sort of test script to have assertions for the
core features of your game.

For testing specific UI features, I would recommend looking into
Selenium/Appium. Selenium for desktop browser testing, Appium for mobile
testing (using Appium you can do mobile specific things like touch actions for
instance).

For testing JavaScript, you can take a look at libraries like
[https://jasmine.github.io/](https://jasmine.github.io/).

Once you have a test script(s) ready, you can point it to cloud based testing
services like BrowserStack (I work there).

We provide a large number of actual physical mobile devices to test on (no
emulators), For complete list see: [https://www.browserstack.com/list-of-
browsers-and-platforms/...](https://www.browserstack.com/list-of-browsers-and-
platforms/automate)

Our service is completely free for open source projects !

If you want to go the manual way, you can also do that with BrowserStack, we
provide the 'Live' product where you can control and test your website on
actual mobile devices straight from your browser.

~~~
trevelyan
Thank you Pulkitsh, late at night now, but I'll have us look at this tomorrow.

------
pezo1919
Let me add visual diffing constraint to the discussion, otherwise I need the
same as OP. percy.io is a cool visual diffing tool I used and liked a lot.

Last time when I used it it supported only chrome/firefox, but both mobile and
desktop.

Please let me ask how is visual testing supported in other solutions like:
selenium, appium, browserstack, webdriver.io, etc aka what do you use for
crossbrowser visual testing?

------
GrumpyNl
They let you test with real hardware.
[https://www.perfecto.io/](https://www.perfecto.io/) Used them for a long time
and everytime amazed by the amount of hardware they have. The whole setup is
kind of need.

------
headspin1
HeadSpin [https://www.headspin.io/](https://www.headspin.io/) provides a large
set of real Android and iOS devices distributed around the globe. Testing on
real hardware is always better than testing in an emulator, and testing in a
real location provides additional data points that help you eliminate
performance issues with your application.

Testing just mobile web browsers is of course supported, but if you ever have
the need you can test native game engines and get video and network
performance statistics at the same time. Automated testing via Appium is
natively supported, or you can bring your own framework.

------
benbristow
We mainly use Browserstack. It's a very powerful tool, especially with their
very impressive Chrome dev-tools integration, though it can be a bit laggy at
times. Quite unbeatable for the huge array of devices they have available
though.

We also have a device table with some common devices that we can use whenever
needed. A recent Samsung Galaxy phone (not sure which model), iPhone SE,
Kindle Fire and an iPad etc.

------
sandyfjord
You mention foreign-language browsers. Interested to know which ones in
particular you are dealing with.

~~~
trevelyan
From our manual testing the worst seem to be 360, Sogou, and Liebao.

------
connorcodes
Most devtools on browsers have "responsive tools" \- letting you change your
screen to mobile or adjusting pixels. I use it sometimes for my web apps.

~~~
midnightmonster
Responsive design mode helps with a subset of CSS layout issues. It's much
faster and easier than hooking up debugging on a live device, so it's
invaluable during CSS development, but since it's still the desktop browser
with just a few parameters tweaked, it's not going to be adequate for tracking
down real-mobile-browser-real-device issues in a game engine.

------
seonsakke
I'm interested in this, too.

------
otabdeveloper3
Judging by the so-called quote-unquote mobile browsing "experience", they
don't.

