BDD Cucumber used with Selenium seems to be quite popular now a days for Automation framework. We have been using a BDD framework with Selenium for our Automation including Integration Tests. The tests are being added by the test team and dev team and they are run as part of build process(CI).
Though it is good that everybody, starting from the customer to the higher executives can read the tests, as the steps are written in (simple?) human understandable language - Gherkin, but BDD has its own limitations. Here are a few of the drawbacks that we encountered during our journey:
1. Steep Learning curve: A QA member is not a programming expert, else he should be in development. In our test team we had a mix of Java programmer(basic) and non-programmers. We really had a tough time explaining about Cucumber BDD to the non-programmers, and they still are not able to write tests properly. Imagine a non-programmer, who never has written a piece of code has to learn Java, Selenium, Cucumber and the BDD style of writing tests. Instead he could have simply learnt Java, Selenium and start adding tests with TestNG. In TestNG or Junit you don't need to worry about Given/When/Then. As for the programmer they still make mistakes writing proper Given/When/Then.
2. Translating test cases to BDD "scenario" is not fun: When we started automating, we already had more than 2000 odd testcases for our product, and those are not written thinking of BDD. We had to update/edit the test steps, split testcases, carve out new steps to be inline with BDD. This is a huge overhead, we had to spend a considerable amount of time on this. Also when we write a new testcase we might actually think in plain English and not in BDD style. Writing testcases with BDD mindset is something which will come with good experience.
Again convention says, when we create a new step in Cucumber, we should make it generic in order to to be re-usable. With this in mind, we try to group similar tests in one "Feature" file. Again an overhead, scanning though different testcases to pick the similar ones. We have to do this because the less the files the better, and then people don't expect to write one "Feature" file for each testcase.
3. Limited Annotations as compared to other frameworks(TestNg, Junit): Cucumber has "Hooks" like @Before, @After. These are called for each "Scenario" from the "Feature" file. Annotations like @BeforeClass, @AfterClass, @BeforeSuite, etc from TestNg are not available in Cucumber. We know this could be achieved in Cucumber with "Runtime.getRuntime().addShutdownHook...", but wouldn't it be better if something is available out of the box? Also what about parallel test execution, of-course we can do this with Selenium, but TestNG and Junit already has this option compared to Cucumber.
4. The Gherkin is not-so-simple to be understood by stakeholder/customers: The Given-When-Then are usually written by technical person either test or dev member. Now our customer is a non-technical, business person who might not think the way as a technical guy. This itself defeats the main purpose of Cucumber that the stakeholder can read and understand Cucumber language. Also how many people other than a QA guy, cares about test steps. Unless there is some critical customer issue, nobody wants to go through the test steps in detail, even a developer have to be reminded multiple times to review a testcase.
Now for argument sake, lets say our customer is a technical person. How likely is it that he knows BDD Steps? Is he going to learn cucumber-BDD for reviewing your tests? Not yet, until he is forced upon :).
5. DuplicateStepDefinitionException seems to occur often: We have been seeing this often times. This occurs when we have duplicate methods in different class files and its likely to occur as different testcases have similar steps. Best solution to avoid this is to checkout all tests and run it locally before committing yours. The problem is people doesn't follow the solution, out of laziness or are in a hurry to automate more :).
Finally, saying all this drawbacks does-not mean that we are stopping with Cucumber. We are loving every bit of it and with each and every issue we are happy we learnt something new today. After all we are engineers and we keep learning!!!
Though it is good that everybody, starting from the customer to the higher executives can read the tests, as the steps are written in (simple?) human understandable language - Gherkin, but BDD has its own limitations. Here are a few of the drawbacks that we encountered during our journey:
1. Steep Learning curve: A QA member is not a programming expert, else he should be in development. In our test team we had a mix of Java programmer(basic) and non-programmers. We really had a tough time explaining about Cucumber BDD to the non-programmers, and they still are not able to write tests properly. Imagine a non-programmer, who never has written a piece of code has to learn Java, Selenium, Cucumber and the BDD style of writing tests. Instead he could have simply learnt Java, Selenium and start adding tests with TestNG. In TestNG or Junit you don't need to worry about Given/When/Then. As for the programmer they still make mistakes writing proper Given/When/Then.
2. Translating test cases to BDD "scenario" is not fun: When we started automating, we already had more than 2000 odd testcases for our product, and those are not written thinking of BDD. We had to update/edit the test steps, split testcases, carve out new steps to be inline with BDD. This is a huge overhead, we had to spend a considerable amount of time on this. Also when we write a new testcase we might actually think in plain English and not in BDD style. Writing testcases with BDD mindset is something which will come with good experience.
Again convention says, when we create a new step in Cucumber, we should make it generic in order to to be re-usable. With this in mind, we try to group similar tests in one "Feature" file. Again an overhead, scanning though different testcases to pick the similar ones. We have to do this because the less the files the better, and then people don't expect to write one "Feature" file for each testcase.
3. Limited Annotations as compared to other frameworks(TestNg, Junit): Cucumber has "Hooks" like @Before, @After. These are called for each "Scenario" from the "Feature" file. Annotations like @BeforeClass, @AfterClass, @BeforeSuite, etc from TestNg are not available in Cucumber. We know this could be achieved in Cucumber with "Runtime.getRuntime().addShutdownHook...", but wouldn't it be better if something is available out of the box? Also what about parallel test execution, of-course we can do this with Selenium, but TestNG and Junit already has this option compared to Cucumber.
4. The Gherkin is not-so-simple to be understood by stakeholder/customers: The Given-When-Then are usually written by technical person either test or dev member. Now our customer is a non-technical, business person who might not think the way as a technical guy. This itself defeats the main purpose of Cucumber that the stakeholder can read and understand Cucumber language. Also how many people other than a QA guy, cares about test steps. Unless there is some critical customer issue, nobody wants to go through the test steps in detail, even a developer have to be reminded multiple times to review a testcase.
Now for argument sake, lets say our customer is a technical person. How likely is it that he knows BDD Steps? Is he going to learn cucumber-BDD for reviewing your tests? Not yet, until he is forced upon :).
5. DuplicateStepDefinitionException seems to occur often: We have been seeing this often times. This occurs when we have duplicate methods in different class files and its likely to occur as different testcases have similar steps. Best solution to avoid this is to checkout all tests and run it locally before committing yours. The problem is people doesn't follow the solution, out of laziness or are in a hurry to automate more :).
Finally, saying all this drawbacks does-not mean that we are stopping with Cucumber. We are loving every bit of it and with each and every issue we are happy we learnt something new today. After all we are engineers and we keep learning!!!
No comments:
Post a Comment