
Karate: Test Automation Made Simple - based2
https://github.com/intuit/karate
======
tnolet
“Made simple”. Well...I tried it and it’s by far the most complex, sprawling,
cluttered, “everything under the sun” testing framework I’ve ever encountered.

I kinda liked the premise, but take “simple” with a truck load of salt.

~~~
cueo
Further it's not helped by the fact that author's attitude is that of a whiny
baby[1]. His entire strategy of publicising his work is taking a huge shit on
the current industry wide standards[2][3]. Also see his reaction to this post
itself[4].

[1]:
[https://twitter.com/techgirl1908/status/1136645315090456576](https://twitter.com/techgirl1908/status/1136645315090456576)

[2]: [https://hackernoon.com/the-world-needs-an-alternative-to-
sel...](https://hackernoon.com/the-world-needs-an-alternative-to-selenium-so-
we-built-one-zrk3j3nyr)

[3]:
[https://twitter.com/ptrthomas/status/1034794832571510784](https://twitter.com/ptrthomas/status/1034794832571510784)

[4]:
[https://twitter.com/ptrthomas/status/1295009968425336832](https://twitter.com/ptrthomas/status/1295009968425336832)

------
Kednicma
I can't help but note that Intuit, who profits off of complexifying the
"simple" tax code, cannot see the complexity in their "simple" test automation
framework. Conway's Law strikes again.

------
jacekm
I've recently used Karate for contract testing and I have mixed feelings about
it. In my case it served me reasonably well. Usually my go-to choice for
contracts is "REST Assured", but in that recent project I had more complex
JSONs and I had to write rather lengthy validation rules manually. I felt that
rest assured's java-based syntax could quickly turn against me so I was
looking for an alternative and someone suggested Karate.

Pros:

* the resulting code was relatively short

* once we got used to Karate the relatively complex tests were quick to write

* the whole framework is really flexible, you can describe complex json structures using just DSL and you can always call a Java method if you are missing something.

Cons

* some of the commenters here already noticed a lengthy README.md - well, you need to read this whole file before starting any work. The framework is reasonably well documented, but you need to ingest the documentation as a whole before attempting to write a test. You cannot just search for "how can I do ___ in Karate", I mean you can try, but you are unlikely to find an answer this way.

* entry point while short is rather steep. You need about 2-3 days before you begin to understand what's going on in the code

* the code is unreadable for bystanders. Good luck with understanding "#[] #(^SOMEVAR)" or "#[_ > 0]"

* the architecture: the whole thing runs on Maven/Java. Java interprets Karate DSL, which can contain Javascript(!) which can call Java again! Yes, you can make a chain of calls Java->DSL->JS->Java! To be fair, it makes the whole framework really flexible, but still, wtf.

Will I ever use it again? Maybe. I'd say it's specific tool for rather
specific tasks. It can serve certain purposes quite well. But if you are
wondering what framework you should use in your next project - there are much
safer choices out there.

------
aldanor
Readme should probably have word "Java" somewhere in the first line.

------
regularfry
Gherkin is supposed to be for expressing human problems in a human-readable
form, that also happens to be useful for computers. Using it to specify
technical details of APIs (as opposed to, I dunno, writing some code to hit
those APIs) seems like a really weird choice.

~~~
ptrthomas
Common question, addressed here: [https://hackernoon.com/yes-karate-is-not-
true-bdd-698bf4a9be...](https://hackernoon.com/yes-karate-is-not-true-
bdd-698bf4a9be39)

------
chriswarbo
Doesn't seem to support property-based testing, which is my default go-to for
automated testing, so I'll give it a miss.

(I've had good experiences with QuickCheck for Haskell, Hypothesis for Python,
ScalaCheck for Scala, JSVerify for Javascript)

~~~
AzzieElbab
That is pretty easy to integrate as they claim to support "data driven"
testing.

------
fabbari

      Their first example for BDD is just painful to look at.
    
      Transforming the "business" language into a "developer" language so that "there is no glue code" is missing the point of BDD.
    
      Imagine putting that example in front of a user and asking: "Is this what you want?". Even if the end user is the developer using your API it makes little sense.
    
      As usual, I mat be missing something here - but it seems it's basically moving the "glue code" to the Gherkin level, making using the Gherkin useless.

~~~
ptrthomas
see my other comment:
[https://news.ycombinator.com/item?id=24183517](https://news.ycombinator.com/item?id=24183517)

------
gegtik
given that some test people might be reading these comments - is anyone aware
of a configuration for test frameworks that: 1\. lets users opt code out of
the testing footprint (I know this is a given), but also 2\. monitors opted-
out code for being called a certain number of times, therefore meaning it's
depended upon, therefore meaning it SHOULD be tested, and gets flagged?

I feel like this would be the way to go to minimize non-valuable testing while
allowing you to discover code thats becoming more leveraged and therefore more
critical to test.

~~~
hansvm
1\. In any testing framework I've used, this is (as you said) a given. The
exact mechanism you have for slicing and dicing your tests varies, but this is
usually well supported IMO.

2\. This feels like the job of a code coverage tool to me (and it's something
I would like as well), but most coverage tools I've seen don't quite have the
granularity you're looking for. In certain ecosystems you could hack something
together pretty quickly (e.g. the `trace` module in python).

Correct me if I'm wrong, but this idea sounds like you'd like to bias your
tests away from small units and toward integration tests (and a similar tool
could be useful for dead code elimination and a few other tasks). I know
they're not a panacea in general, but in your personal experience what
problems have you found in fine-grained tests that outweigh their benefits?

~~~
gegtik
Thanks for the reply;

I'm a believer in "right-sized" testing, meaning balancing the cost of test
creation and updating against the value the tests provide. I believe tests
provide the greatest value when they are testing: * public methods * critical
business logic * highly reused core logic

in other words, "code for others".

For code that's used for one specific page render or something similarly
isolated, I think it's more questionable whether to bother going for 100% code
coverage.

The latter category is where I'd like to "trust but verify" with devs
permitting them to opt out of unit tests where they believe the code isn't
critical, but have some confidence that we would catch code if it became more
central and then enforce testing requirements.

------
serbrech
Choosing java as the runtime for the framework is laughable when you want
people not to care about the technology that the binary is running on. It
requires you to install the java runtime, or run a bloated container, just to
run the api testing. If you want the runtime to be invisible, don't ask me to
install a runtime.

------
coderdd
Having done some chidokai, first thought on seeing the site is "wow, that's a
poor stance". YMMV

