E2E Development challenges
End to end (E2E) testing plays a significant role in the quality of any product, and it is undoubtedly more than desirable for it to be part of the process of developing software solutions. If we look at the test pyramid, we will see the relationship of these tests as part of the bigger picture. As you can see, they are not additional or optional tests. So why do we rarely see them in projects?
Although in this pyramid diagram, these E2E tests look like a small part of the testing process, they are in fact, a big part of the testing and quality strategy.
Taking the iceberg analogy, in terms of impact, they hold considerable value. Yet because they are not as visible as other testing regimes to the development cycle and teams, they are often given lower priority and are left for better times when the conditions allow for it. Which as many of us know, is never.
The conditions that cause this missed opportunity can vary from lack of funding, limited experience or availability of time to invest in it.
However, in recent years, E2E testing has shown a resurgence, where the popularity and value of the approach are having a breath of new life breathed back into it. The increased speed of new software releases and features is increasingly crucial to whether a product will succeed or be overtaken by the competition. Significant innovation and integration with other systems also play an impactful role here. Manual testing of all substantial changes in short periods can no longer guarantee quality and the desired end-user experience. In the next image below, you can see a simple application and all the relevant and essential environments.
So to embrace this essential testing approach, its necessary to add a new environment. Not surprisingly, this is called the E2E test environment, where tests can be executed continuously without false-positives or false-negatives. So when tests, no matter how many times run, almost always give the same results. The more complex the system is, and the more dependent it is on other components, the more likely it is that one of the tests will fail due to problems with consistent behaviour of components. For example, communication with the database may be slow or non-existent; or the identity provider is not available in a given moment, etc.
The need for speedy software development has led to the development of various tools and libraries that will allow large parts of the software dependencies to be minimized. So now we can separate the development environment in the previous diagram, allowing each member of the team to develop, maintain and use E2E tests in isolation (in their local environment).
This means that we need to set all the configurations for the system in the source code. i.e. part of the software. So for each change made, team members are presented with the same changes to their environment automatically. (e.g. upgrading database version: if it’s not configuration in the code, developers would have to upgrade the database version manually)
So this solves one of the biggest obstacles to development, E2E testing and maintenance. The following diagram shows us what that environment would look like.
Similar to local setup configurations, E2E tests are part of the code base, and maintained in parallel with system changes. So the tests can be created/changed/performed independently of other developers, allowing for rapid development of tests local to a given developers setup. If tests are run for extended periods (a few hours for example), there is the risk of these local systems (such as the developers PC) affecting the performance of the system under test. In this case, a more stable local environment is needed, one that is insulated from local interference. e.g. CPU resources being used by other applications. There are also techniques on grouping the tests, where some are executed on a local environment for the current changes. All other tests that are not directly related to the change being performed on another test environment.
So it’s crucial to choose the right framework for developing the E2E tests based on the software requirements and what the frameworks offer.
E2E testing frameworks
There are several environments for the execution of the E2E automated tests. By simulating the user flow from start to finish, the completion of this testing will not only validate the system under test but will also ensure that all other systems work and behave as expected.
It should also be noted that with E2E tests, we don’t need to check all possible scenarios. This is because much of the test coverage will already have been done with the unit tests. The idea here is we want to check that those units all work together as they should as an integrated user flow.
Nightwatch is one of the most popular test automation frameworks that include highly transparent and readable code. Uses Selenium WebDriver API and allows the user to perform end to end testing, by simplifying the process of writing automated tests and setting up CI in the development cycle.
Nightwatch has several features that make it a highly popular testing tool. It has a built-in test runner that has many useful options such as sequential and parallel run of tests. It also allows the user to set implicit waits on the tests, or to retry a test execution for failed tests. Users can group test suites and add tags. Another powerful feature is, it has an inbuilt command-line test runner suite with Grunt support for executing the automated tests.
Compared to other frameworks, cloud support is one of its most significant benefits. It means tests can be run on a specific browser version on SauceLabs and BrowserStack, avoiding users needing to have multiple installations of any particular browser on their machines.
There is also the support of Page Object Model for improved organization of elements and sections (similar to Selenium), with automatic management of Selenium or WebDriver services.
As mentioned previously, Nightwatch supports CI, allowing tests to be created and integrated with systems such as TeamCity, Jenkins and Hudson.
The main advantages of using Nightwatch.js are the following:
- Quick setup
- Supports multiple browsers
- Real browser is not required for test execution, supports headless browser execution, and supports cross-browser on the desired browser version
- Supports good test organization
- Supports 3rd party integration with Cucumber
- Compatible with CI Integration
The main disadvantages of using Nightwatch.js are the following:
- Weak documentation – you may not find all the information you need, compared to other frameworks on the market that have excellent resources.
- Slow performance – test executions may have long periods with lots of waits
- Slightly less available support compared to other frameworks
Cypress is a relatively new product that has gained industry trust in a relatively short amount of time, with many satisfied users.
The major plus of using this framework is that it is an all-in-one, with assertion library, mocking and stubbing, and no connection or usage of Selenium. With all of this in mind, users won’t need to install multiple tools to set up the environment where they will write and run the tests. A good illustration of this is provided in the image below. You will observe on the left-hand side numerous tools need installing to run E2E tests. However, on the right side, there is Cypress all on its own, doing all of that by itself.
Cypress is also more developer-centric and focuses on making TDD part of the development process for its users. Since it’s not Selenium-based, like most of the other testing frameworks, it has a different architecture to Selenium. Selenium-based testing frameworks use WebDriver, which runs remotely outside of the browser. Whereas for Cypress, it runs inside the browser, helping to provide more consistent test results. This also means test execution is a lot faster than any other tool.
Cypress also takes a screenshot of every step of the test, which makes the debugging process more straightforward. There is also comprehensive documentation to make the development of the tests easier for developers.
Cypress also supports visual regression, but their approach is more manual compared to other tools. Additionally, they support Percy.io, a cloud-based visual testing tool that neatly integrates into Cypress, allowing for visual regression testing.
The main advantages of using Cypress are the following:
- Easy to set up and write E2E tests
- Test runner makes debugging straightforward
- It has excellent documentation and support
- Fast test execution
- Takes a screenshot of each executed step (no matter if we get a pass or fail of a test)
- Compatible with CI Integration
The main disadvantages of using Cypress are:
- Some functions are still not built-in and require workarounds and libraries (e.g. file upload, SSO Login (like login with AWS Cognito) and many others)
- You need to pay to unlock the full version (free version supports up to 500 tests)
Puppeteer is a NodeJS library that runs headless by default but can be run in a browser as well (note: only in Chrome or Chromium’s latest versions). Every single run creates its browser user profile which is cleaned on every next run.
It has some of the same functionalities as Cypress, such as clicking and filling out fields etc. but also can handle popups and SSO logins, which Cypress cannot.
Puppeteer can be used in combination with other frameworks; for example, it can fill in the functionality gaps that Cypress has. For instance, SSO login, since the first E2E test will be the login step that will be reused in many more tests
The main advantages of using Puppeteer are the following:
- Runs a real browser when executing the tests, which means the quality of the test is much higher.
- It also supports headless browser execution.
- Works well for visual testing
- Supports testing in offline mode
- Can take screenshots of webpages
The main disadvantages of using Puppeteer are:
- Not suitable for cross-browser testing since it supports Chromium only
It’s one of the earliest testing frameworks for automated E2E testing and is dedicated to testing Angular JS and Angular software. It allows for cross-browser testing since this library wraps Selenium, but also has additional locators for selecting elements like repeater, model, binding, etc.
It also supports the Page Object model, so the tests and locators can be organized in a desired manner.
Installation is easy, and Protractor has good documentation of the API.
The big plus for the framework is automatic waiting. Protractor is capable of executing the next step of the test at the exact moment the web application finishes pending tasks.
The main advantages of using Protractor are the following:
- Allows use of angular specific commands (identifying the elements for angular.js)
- Runs on multiple browsers in parallel
- A real browser is required for test execution, supporting headless browser execution and cross-browser (via selenium utility)
- Compatible with CI Integration
The main disadvantages of using Protractor are the following:
- Hard to debug
- Returning a value from a protractor promise is hard
- You cannot simulate real user
All testing frameworks have their pros and cons. Selecting the right one depends on several factors.
It is essential the framework that you use has excellent support, and the engineers that are in charge of developing the E2E tests are familiar with it. Don’t use a tool just because it’s popular and widely used. If the team lacks the knowledge to use it fully, they won’t be able to take full advantage of it.
Depending on the software requirements, it is essential to factor in if the tests need to be executed on multiple browsers and browser versions. Or for example, your application only supports Chrome, thus making various browser executions irrelevant.
Maybe your project has specific CI infrastructure so you will need to take into consideration how the E2E framework you are using is compatible with it.
To summarise, choose the right framework, based on criteria that suits your capabilities and needs.