Another year is about to end and we will move on to the next year with a lot of learnings. One of those learnings is around our endeavor on End-to-end Frontend Testing. I will try to keep this blog as close as to what we learned during the entire journey and provide useful hyperlinks because Cypress’s own document is pretty darn good.
Quarter 1 – Piloting with Selenium
At Jio Haptik, we currently have three major internal tools to monitor and build bots. They are namely, Agent Chat, Analytics, and Bot builder.
With our increasing frequency of pull requests every day and the ever-growing complexity of these web applications, we chose Selenium to automate integration tests and get rid of man-hours for manual testing. One of the strong reasoning behind choosing Selenium was to have the code consistency across tools, platforms and extend its capability to test mobile applications as well.
Three weeks down the line, we quickly realized that keeping pace with the development cycle is extremely difficult because of these inherent reasons
- The learning curve for manual testers is steep
- Setting up a test environment with selenium is not straight forward and often requires OS-specific quirks
- Focus diversion: We observed that testers are spending more time figuring out the capabilities/syntax of the language instead of focusing on writing the range of test cases.
- Writing tests in Selenium is more like building with lego blocks; for every new feature, you’ll have to rely on a third-party add-on. For example, we used Cucumber for screenshot capturing, AutoIt for system popups. This creates a huge dependency and is prone to stale libraries.
This lead us to ask the obvious question…
Why does everyone still use Selenium despite the inherent problems?
After an extensive interview with a couple of our QAs, putting down some of the major pointers, as to why Selenium is a goto solution for End-to-end Testing:
- Community support: Being one of the oldest and matured solutions around, Selenium has grown a wonderful and large community. So, getting a solution to any of your queries is just a Google search away
- Language support: Selenium is language-agnostic; its core API is ported in many languages namely, Java, C#, Ruby, Python, etc.
- Multi-browser support: This is a valid reason as many companies will save on maintenance overhead.
- Reports: For Sass/Service based enterprise, client test reports are vital. Selenium ecosystem provides multiple such add-ons for report generation, test case aggregation on tags, which helps in monitoring and final delivery.
Quarter 2 and 3 – Cypress! The new guy
At the end of the 1st quarter, we needed a solution that would enable us to increase the development and testing cycle instead of being a hindrance. We decided to try out Cypress, and I would like to highlight a few pointers which were immediately beneficial.
- Setting up and writing our initial test case were done in a matter of minutes.
- Increased collaboration: All our platform repositories in-housed the testing code, which enabled front-end developers and QAs to simultaneously build features and write tests. This new workflow also enabled us to set up contracts between the frontend team and QAs, to define better DOM selectors, which previously was an afterthought.
- Use of Mocha for test case structuring, Chai for the assertion along with Cypress own API, enforced consistency of code and readability.
- Time travel – Debugger: The power of scrubbing through different states of the application gives a broader insight into why a test might fail. Furthermore, the experience of writing tests is enhanced due to the integration of Cypress test runner with already powerful chrome dev tools.
- Retries and browser event wait:The core API of cypress to find DOM elements has an implicit timeout of 4 seconds. This really takes a lot of overhead code to provide explicit wait to find DOM elements. This is just one of many such browser events, that Cypress keeps watching, to describe:
a. The DOM load event.
b. Bootstrapping of the framework being used.
c. Wait for the XHR/Asynchronous request.
d. Wait for Animation to be completed. - Intercept network:Cypress allows you to intercept network calls and mock requests/responses. It really helps to write robust test cases instead of relying on dynamic responses from the server end.
- Native access: Because Cypress operates within the browser, it has native access to its APIs. Whether it is the window, the document, a DOM element, your application instance, a function, a timer, a service worker, or anything else – you have access to it in your Cypress tests.
- Out of box recording for every test-run helps to find application/test loopholes faster:
Other than these, which we immediately got benefited from, please do read the other great features which might relate to your domain-specific problem.
Cypress.io – Our learnings
During our usage of Cypress for more than two quarters, we have faced a few different challenges and learned a lot. Let’s have a look at those:
- Using data attributes to find DOM elements makes test cases resilient for a longer period of time and immune to development changes.
- Using assertion alternatively in chained commands to enable retries.
cy.get('ul[data-testid=todo-list] li', { timeout : 10000 }) /*command*/.should('have.length', 1) /*assertion*/.find('label') /*command*/.should('contain', 'todo A') /*assertion*/
- Wait for a route instead of explicit waiting. Cypress allows you to track API calls via the route method. To verify the success or failure of an asynchronous call, waiting for the network call provides accurate waiting time.
- Check for events before the click. Cypress often tries to click on a DOM element even before an event listener has been attached to it. Adding an explicit wait makes the test flaky. The ideal approach would be to check if there are any listeners attached to a DOM element. This makes the test robust. Here is a great blog about it.
Reference and Roadmap
So far we’ve been very happy with what Cypress has to offer out-of-the-box and pretty much satisfies our needs. With its fast growth, they keep reducing the existing problems. Do keep track of their current roadmap, which might help you to take a long-term decision. Furthermore, adding some important articles that are worth pinning
- Best practices of Cypress
- Cypress Architecture
- Trade-offs
- Click retries
Conclusion
As I stated at the start of this article, my experience of automation testing is not a good one where a lot of money, time and pain are spent keeping thousands of hard to maintain tests afloat for not great payout. Automation testing has only ever guaranteed a long CI build in my experience.
We, as developers, need to be better at automation testing. We need to write fewer tests that do more and are useful. We’ve left some of the most difficult code to write to some of the least experienced developers. We’ve made manual testing seem outdated when, for my money, this is still where the real bugs are found.
We need to be sensible about what automation testing can achieve. Cypress is great because it makes things synchronous, this eliminates a whole world of pain, and for this, I am firmly on board. This, however, is not the green light to write thousands of cypress tests. The bulk of our tests are manual with a layer of integration tests before we get to a few happy path automation tests.Haptik is hiring. Do visit our careers page.
