
The State of the Art in iOS 10 Test Automation - carlsbaddev
http://bitbar.com/the-state-of-the-art-in-ios-10-test-automation-frameworks/
======
joeblau
I've had the worst experience in testing with Appium while working at Amazon.
The test suite was extremely slow, unreliable, and tests are written in a
language that's not native to the platform. The whole Appium testing interface
is a hodgepodge UI that had a lot of broken UI interactions.

Our team members who were proponents of Appium made arguments like:

\- Business people can write test cases - That never happened

\- You can write cross platform tests - Yes, but we end up with all of these
one of cases if iOS and Android aren't written with the exact same naming and
design conventions which forced multiple testing paths anyways.

KIF is also not necessary anymore since Xcode 7+ support UI Test Case.

UI Test Case(for interaction), Unit Test Case(For Unit/Performance Tests), and
Snapshot Testing[1] is way better combination. You write statically compiled
Obj-C/Swift, tests run a lot faster, the build on the native xcodebuild
toolchain, and your tests are maintained as a first class citizen.

[1] - [https://github.com/facebook/ios-snapshot-test-
case](https://github.com/facebook/ios-snapshot-test-case)

~~~
mikestew
Sadly, when I clicked the "up vote" button it disappeared so that I couldn't
click it again...and again.

On my current project, the client was apparently standardizing on Appium.
Ooookay, don't know why you're not using the included tools, but I'll give it
a whirl. Sounds like something management, who will never use the tool in real
life, got sold on because "cross-platform". But let's put my cynicism aside
and give it a fair shot.

It took me several tries to get it installed. The documentation even says that
if installation fails, try it a few more times. No, I'm not making that up.
Once installed, configuration is a poorly-documented nightmare. But after half
a day I have it working on iOS and Android.

As you point out, O...M...G is it slow. Now, I understand why and I don't
necessarily blames the dev for the sixteen levels of abstractions this thing
has to go through. But slow it is. Tests take a long time to run as they wind
their way through the Xpath to get to the UI elements.

Writing test cases is even worse. No UI recorder to speak of to at least take
a first whack at it. No good accessibility viewer tool to help with control
names and characteristics. And since you kind of have to just guess-compile-
run, combined with the slow runtime performance, even simple tests can take an
hour to write.

"Business people can write test cases"? Did I miss this outstanding feature? I
can barely get a test running, and I've been writing code for decades. And
you're telling me some woman with a business degree is going to come in and
start knocking out UI tests?

So after assuming that I must be the idiot in this equation given that I do
not see the Highest Holiness of Appium that others must see, I started asking
around. Hey, you, full-time SDET: you folks are using Appium, right? How's
that going for you?

Turns out they've written about 20 tests, with an average of 60 seconds to run
each individual tests. 20 minutes to run 20 tests. I'd planned on writing
_hundreds_. Meaning it is entirely possible that the UI test suite I hoped to
write would not be able to run on a single platform in a single work day. So
much for running them at build time.

EDIT: Oh, yeah, and cross-platform: I can barely get dev to put accessibility
labels on stuff for a _single_ platform, let alone co-ordinate between the
two. And unless the UI is mostly the same, using the same classes of controls
with exactly the same accessibility labels (IOW, will never happen in the real
world because we have other stuff to do), cross-platform is a pipe dream.

I realized that the Emperor has no clothes, and went back to using XCTest and
Espresso.

~~~
joeblau
Yeah, I didn't detail the trails and tribulations of our experience but it was
terrible. Our Appium test suite took 6+ hours to run and it falsely failed
100% of the time. We would finish a feature, check our code in run the test
suite over night and things would be broken and then we'd have to manually
verify the fixes were correct in the A.M. Then go back in and fix Appium.

I honestly wish I still had the link to the Wiki where I put my full Appium
deep dive. Appium is great if you want to work on Appium and not actually
build your own iOS or Android product.

~~~
mikestew
_Appium is great if you want to work on Appium and not actually build your own
iOS or Android product._

Reminds me of what I told our PM: "do want me to make a career of writing
Appium tests, or would you like me to use the tooling of my choice and get
other things done, too?" And my current role is SDET, meaning I have nothing
better to do than write tests. I can't imagine making dev write their own
tests in Appium. I mean, I guess it wouldn't be _that_ bad since you'd have no
time for implementation, therefore making test writing a doddle. :-)

------
goku7
iOS 10 is the buggiest iOS I have seen so far. youtube starts playing in
background when on a call mixing the audio. voicemails keep getting deleted
when they reach 20 second mark. the screen freezes very frequently. swiping is
confusing delete lock screen notifications and switch on the camera. is it bad
product design or bad testing

~~~
carlsbaddev
All that, plus it is damn hard to set up everything for test automation.

