The Problem

JavaScript is now a popular choice of language for developing web applications. To test the applications developed in JavaScript, there seem too many end to end testing frameworks and tools which help in Unit testing, UI testing. However, to use one of them, one has to go through and analyse each tool and framework to know which suits their application best. We, through this document make an attempt to weigh three popular (E2E) testing tools in JavaScript.

The Approach

We have selected three popular automation tools which can automate web applications in different browsers and in different operating systems. These are based on selenium webdriver API.

  1. Protractor (v4.0.10)
  2. WebDriverIO (v4.4.0)
  3. Nightwatch.js (v0.9.8)

We have analysed these automation tools in different categories to identify the advantages and disadvantages of each so that the reader of this document can choose a tool that is best suitable for their needs.

We have compared the three automation tools against different categories which we think are essential for automating a web/mobile application. Below comparison table shows which tool is good at which category.

The Comparison

Category Protractor WebDriverIO Night Watch
Architecture Protractor is a wrapper around WebdriverJS (selenium-webdriver node module). WebdriverIO is custom implementation of W3C webdriver JSON wire protocol. Night Watch is custom implementation of W3C webdriver JSON wire protocol.
Supported browsers
  • Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs
Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs Edge, Chrome, Firefox, Safari, Opera, HtmlUnit, PhantomJs
Angular specific locator strategies Supported Partially supported by adding ‘webdriverjs-angular’ node module. Only ‘WaitForAngular’ has been implemented. Not supported. But still we should be able to automate angular js application as selenium does.
Inbuilt test runner Available Available Available
Supported testing frameworks Jasmine, Mocha, Cucumber Jasmine, Mocha, Cucumber Mocha, inbuilt framework
Reporting Junit Xml Reporter,

Protractor Html screenshot reporter, Jasmine spec reporter

Allure reporting, Jasmine Spec reporter, Junit xml reporter, Cucumber reporting Junit xml reporter,

Mocha reporting

Parallel Execution Supported Supported Supported
  1. Cloud Execution Sauce Labs
  2. BrowserStack
  3. Testing bot
Supported Supported Supported
Mobile Support (Appium) Partial support (Supports mobile browsers but does not support native apps) Android, IOS Android, IOS
Synchronous execution Supported Supported Supported
Page Object Pattern Supported Supported Supported
Continuous Integration Supported Supported Supported
Retrying flaky tests Possible Possible Possible
TypeScript support Inbuilt support Possible when using Codecept.js Not yet supported
Additional Features Angular application support CSS regression testing using ‘webdrivercss’ npm package.

The Summary

Protractor

Advantages

  1. Protractor is the only automation tool which has inbuilt support for AngularJS element identification strategies. Better to go with protractor if the application is developed using AngularJS.
  2. It has support for different unit test frameworks like (Jasmine, Mocha, Cucumber etc..)
  3. Good reporting mechanism (html, xml etc..)
  4. Compatible with Continuous integration
  5. TypeScript support is available which will help Angular 2 developers to write unit tests very easily.

Disadvantages

  1. It does not have support to automate mobile Apps.
  2. It is implemented as a wrapper to the WebdriverJs. So there is one more layer added in between Selenium server and the Protractor. If there is an issue with WebdriverJs the Protractor team should wait for the WebDriverJs team to fix that issue.

WebDriverIO

Advantages

  1. WebDriverIO is a custom implementation of W3C webdriver API. This gives it an edge to have full control over implementation rather than depending on WebDriverJS implementation.
  2. It is fully extensible and is written to be as flexible and framework agnostic as possible. It can be applied in any context and serves not only the purpose of testing.
  3. It has a command line interface ‘wdio’ which makes test configuration as easy and simple as possible so that a non-programmer can configure the setup.
  4. It has support for most BDD and TDD test frameworks.
  5. It has good support and enthusiastic developer community and end users which gives it an edge over NightwatchJS.
  6. It can be used with ‘webdrivercss’ to compare css stylings of an element in the webpage.

Disadvantages

  1. Since it is a custom implementation, it is also a disadvantage as it deviates from generic syntax which may confuse selenium developers coming from other languages.
  2. It can be used for automating AngularJS apps but it is not as customized as Protractor.

NightwatchJS

Advantages

  1. Similar to WebDriverIO, it is also a custom implementation of W3C webdriver API.
  2. As all other JS libraries, it is extensible easily – one can write custom commands, assertions extending it.
  3. It has its own testing framework and assertions mechanism which solves the programmer’s headache of choosing a testing framework.

Disadvantages

  1. It does not have much of choices for unit test frameworks as it has an own testing framework and also support Mocha.
  2. Slightly lesser support than WebDriverIO and Protractor.
Authors
  1. Vijay Kumar Daram
  2. TulasiRam A