The biggest problem with Cucumber is that most people trying it out don't understand what it is.
Cucumber is not a tool for testing software. It is a tool for testing people's understanding of how software (yet to be written) should behave.
Most bugs and delays caused by rework arise from misunderstandings, and this is the problem Cucumber aims to solve.
Cucumber is a tool that facilitates collaboration and software design (especially domain-driven design).
Here is how it works: You pop a story off your backlog and run a 20 min. meeting (Discovery Workshop) with business folks (BAs, POs, domain experts) and IT folks (developers, UX, testers if you have them).
You have a conversation about the story and come up with some concrete examples to describe the various acceptance criteria for your stories. Not in Cucumber's Gherkin language - just in plain conversational language.
For example: "The one where I upload a picture that is too big". Or: "The one where there are five taxis in range". These conversations act as catalysts to uncover subtle details where business and IT might have a different understanding.
Two things can happen at the end of this short meeting. You ask people to do a thumbs-up or thumbs-down vote on whether they understand everything that needs to be done, and whether the story is small enough. If enough people give a thumbs down, you send the story back for further analysis, maybe breaking it up into something smaller. If it's mostly thumbs-up, you're good to go.
After the 20 min. meeting you have 2-5 concrete examples that a developer (and perhaps a tester) can flesh out in more detail using Gherkin (Given-When-Then) to make it even more concrete. For example:
Scenario: Close taxis with higher rating win
Given taxi A with rating 0.8 is 1400m from the customer
And taxi B with 0.9 is 1500m from the customer
When the customer requests a taxi
Then taxi B should be assigned
The dev shows the example to the business person, who confirms that this is right (or wrong).
Now, the developer follows the regular TDD workflow, using the Scenario to guide the development of the core domain logic. The Cucumber scenario doesn't go through a UI using Selenium WebDriver or similar. The domain logic is implemented in such a way that external services, message queues and databases are stubbed out.
Lower level unit tests are still written, and there are far more of those than Cucumber Scenarios.
Cucumber is there to make sure you write the right code.
Unit testing tools are there to make sure you write the code right.
Using UI testing tools together with Cucumber? Please don't - or at least do it very sparingly. UI tests are expensive to maintain (the UI is more volatile than your core domain). They are slow (2-3 orders of magnitude slower than test talking directly to the domain logic). And finally - when they fail they don't tell you where the bug is.
The purpose of Cucumber is to bridge the communication gap between business and IT by providing a small set of essential scenarios to illustrate core behaviour of unwritten software. These scenarios do become regression tests, but their real value is to prevent defects by uncovering bad assumptions up-front. You end up with executable, living documentation accessible to everyone on the team. -Documentation of how the software should behave - and how it actually behaves.
Cucumber is a testing tool, depending how you use it it may facilitate BDD. It may not be your intention as the author for it to be a testing tool but it is, lets look at it some more.
Gherkin (Given-When-The)
Scenario: Close taxis with higher rating win
Given taxi A with rating 0.8 is 1400m from the customer
And taxi B with 0.9 is 1500m from the customer
When the customer requests a taxi
Then taxi B should be assigned
Lets look at the Gherkin syntax. It follows a set format with forced English language. It gives a context (Given), some input data (When) and an expectation (Then). This syntax is an example based specification and lives in a plain text file generally with the file extension *.feature. We'll call the above example a feature based on the file extension, we'll also call this Gherkin format an external DSL (domain specific language).
Now on it's own this feature file is useless, it's a plain text file. Why do I need Cucumber for this it's a plain text file? Why can't I store it in a shared wiki where people can collaborative edit it and track changes? If it's a plain text file to share knowledge why do I need to use forced English with the Gherkin syntax instead of a more natural form for the intended audience? Why does it need to be text if it's demonstrating shared knowledge, maybe a comic strip may be more appropriate for the domain? Why would I need Cucumber for a team to sit down to create this collaborative knowledge?
We need to follow this strict syntax because it's an external DSL which is used by an interpreter. This interpreter parses a feature file and asserts a given input is equal to the expected output specified in the feature file Gherkin DSL. Lets think about this some more, we give some context, we provide some data, we run a computation and we assert the output, this sounds very much like an automated test. If I was to write a definition of an automated test this would be it.
This interpreter is fairly fragile. We have a miss match between the plain text feature files in our Gherkin DSL format and our test (not sure what else to call it?) execution tightly coupled with fragile regex. The implementation also promotes heavy mutation in the test (sorry, again it's not a test?) implementation which ultimately leads to fragile assertions (aka tests).
@Given("^I am on the front page$") will mutate sending the browser to a page. It doesn't give an indication if it's successful or not. The function is marked with throws InterruptedException so I can only guess it blows up if it fails, or maybe not?
Then we have in TemperatureStepdefs @When("^I enter (.+) (celcius|fahrenheit)$") which finds an element by id and sends some key events. Again how does the subsequent step know this is successful, how does this step know the previous step was successful? We don't, we assume, our test may work or we may get silent errors which cascade down.
Then you see people use things like public String currentPage = "" and update it as you progress through the workflow and assert on it. Mutation, race conditions, silent failures if you've used Cucumber for any amount of time you've been deep in these trenches.
I digress
So, if Cucumber is not about the test part why do we need the interpreter which runs a computation with given input and tests it matches the expected output. Without this part Cucumber is a set of flat text files. What do we get? flat files? Why are these better than a shared wiki, google doc, spreadsheet? They achieve the same, canonical source of knowledge.
But Cucumber promotes a conversation. No, used in an agile productive organization features will be written collaboratively and Cucumber may be a facilitator to reach this goal. Cucumber does not enforce this or ensures this happens, it's promoted but in no way is this a requirement to use Cucumber.
If you are already an efficient team delivering software you'll already be having this communication part. Cucumber isn't a tool for test so what does it give us if we are already talking and delivering? We have other better test tools (cucumber is not a test tool right?) and more efficient ways of collaboratively writing, sharing and tracking knowledge.
Cucumber is a facilitator to help organisations to start having conversations and collaboratively share knowledge. I accept this, people who are looking at ways to improve things see this as a valid usecase, it's a trojan horse to get a more agile workflow in through the backdoor. My issue is when Cucumber is used in this way it's used when you are knee deep in mud, it's a technical solution to mostly a non technical issue. Your organisation is dysfunctional, likely not delivering and a command and control structure. Your team looking at Cucumber want change but the issues lie far deeper and Cucumber will not save you. Open up a Google Doc, Wiki Page, sit down with your team and stakeholders and first talk.
So really, what is Cucumber?
As a test tool it sucks. There far better automated test tools
As a shared knowledge base, it's easier to collaborate in something accessible to everyone where change is easier and can be audited. A wiki, a Google Doc, a spreadsheet, it really doesn't matter they all achieve the same goal.
As a facilitator, you have a non technical problem deeply rooted in how your organisation works. Sit down, have the talks, create the shared knowledgebase, solve the core issues then and only then look at technical facilitators.
The biggest problem with Cucumber is that most people trying it out don't understand what it is.
Cucumber is not a tool for testing software. It is a tool for testing people's understanding of how software (yet to be written) should behave.
Most bugs and delays caused by rework arise from misunderstandings, and this is the problem Cucumber aims to solve.
Cucumber is a tool that facilitates collaboration and software design (especially domain-driven design).
Here is how it works: You pop a story off your backlog and run a 20 min. meeting (Discovery Workshop) with business folks (BAs, POs, domain experts) and IT folks (developers, UX, testers if you have them).
You have a conversation about the story and come up with some concrete examples to describe the various acceptance criteria for your stories. Not in Cucumber's Gherkin language - just in plain conversational language.
For example: "The one where I upload a picture that is too big". Or: "The one where there are five taxis in range". These conversations act as catalysts to uncover subtle details where business and IT might have a different understanding.
Two things can happen at the end of this short meeting. You ask people to do a thumbs-up or thumbs-down vote on whether they understand everything that needs to be done, and whether the story is small enough. If enough people give a thumbs down, you send the story back for further analysis, maybe breaking it up into something smaller. If it's mostly thumbs-up, you're good to go.
After the 20 min. meeting you have 2-5 concrete examples that a developer (and perhaps a tester) can flesh out in more detail using Gherkin (Given-When-Then) to make it even more concrete. For example:
The dev shows the example to the business person, who confirms that this is right (or wrong).Now, the developer follows the regular TDD workflow, using the Scenario to guide the development of the core domain logic. The Cucumber scenario doesn't go through a UI using Selenium WebDriver or similar. The domain logic is implemented in such a way that external services, message queues and databases are stubbed out.
Lower level unit tests are still written, and there are far more of those than Cucumber Scenarios.
Cucumber is there to make sure you write the right code.
Unit testing tools are there to make sure you write the code right.
Using UI testing tools together with Cucumber? Please don't - or at least do it very sparingly. UI tests are expensive to maintain (the UI is more volatile than your core domain). They are slow (2-3 orders of magnitude slower than test talking directly to the domain logic). And finally - when they fail they don't tell you where the bug is.
The purpose of Cucumber is to bridge the communication gap between business and IT by providing a small set of essential scenarios to illustrate core behaviour of unwritten software. These scenarios do become regression tests, but their real value is to prevent defects by uncovering bad assumptions up-front. You end up with executable, living documentation accessible to everyone on the team. -Documentation of how the software should behave - and how it actually behaves.