Saturday, December 6, 2014

Software Testing

A solid testing strategy depends on a combination of clear business requirements, documentation, resources, and analysis. It helps to leverage tools in order to maximize the amount of coverage and speed up the process, but it's important not to compromise the integrity that comes from taking a methodical, hands-on approach. Today, we're going to highlight some of the specific components involved in the testing of software. Because of the sheer breadth of this topic, this will be an ongoing series of articles.

Test Planning

A test plan is a document that describes the strategy and scope of testing a system such as a process or a product. It is the basis for which all systematic forms of testing should be carried out. This document should detail test items, risks, features to be tested, features not to be tested, pass/fail criteria, suspension and resumption criteria, test deliverables, and other objectives. It helps, tremendously, to create this document alongside the software specifications (user stories, wireframes, flowcharts, pixel perfect mocks) and product roadmap.

Unit Testing

This level of testing focuses on the validity of isolated pieces, or units, of code. Because these tests are written at a very low level, unlike forms of black box testing, they should be designed prior to development and written by the developers, themselves. This also has the added benefit of encouraging engineers to really think through the software requirements prior to coding the implementation. Here's an example of a unit test written in JavaScript, using the Jasmine test suite.

# src/utils/ArrayUtils.js
var ArrayUtils = {
    swapByIndex: function(arr, a, b) {
        var tmp = arr[b];
        arr[b] = arr[a];
        arr[a] = tmp; 

First, we declare our ArrayUtils object with a swapByIndex method that swaps two elements of an array.

# spec/utils/ArrayUtils.spec.js
  describe("utils/ArrayUtil", function() {
      var arr;
      # setup
      beforeEach(function() {
        arr = [1,2,3,4];

      # teardown
      afterEach(function() {
        arr =  null;

      # unit test
      describe('#swapByIndex', function() {
          it('should swap two elements by index', function() {
              ArrayUtils.swapByIndex(arr, 0, 3);

Next we write our spec with the appropriate setup and teardown functions, which run before and after every test. Whether this spec is run in the browser or in a headless browser like Phantom.js, Jasmine will generate a pass fail report. This can then be piped into a Continuous Integration system or simply skimmed over. If you'd like an example using Python, check out this article on creating a Python project skeleton.

Integration Testing

The purpose of integration testing is to verify that the overall system, including functionality, performance, and stability are operating reliably according to the established requirements in the test plan. Integration testing involves multiple components, including server-side software, client-side applications, databases, load balancers, deployment, and other pieces. A Continuous Integration system leverages software to systematically run tests either on a schedule or in response to changes in the codebase. Lastly, this testing needs to happen early and often in order to speed up diagnostics and increase development.

Integration testing is also referred to as end-to-end testing. Conceptually, they are the same, but in the context of this article, we define end-to-end testing as operating on a smaller scale.

End-to-end Testing

This is a form of black box testing that ensures software is functioning from a much higher level than unit testing, but not as high as integration testing. Where unit testing is concerned with testing functions, one at a time, end-to-end testing will test a feature as a whole, regardless of how many components are involved. A good example of a feature could be user authentication, which could involve multiple steps. Take the following example:

  • User opens the application
  • The application checks to see if the user is already logged in
  • If not, the user is redirected to the login page
  • The user enters their credentials into a form and clicks the submit button
  • Given the proper credentials, an authenticated session is created and the user is redirected to the main view
  • Given the improper credentials, the user is notified and prompted to re-enter their username and password

An end-to-end test aims to automate the execution of these steps in order to deliver a pass/fail report that determines if the application is functioning properly. Let's take a look at an example of an end-to-end test written with Protractor for a JavaScript application.

describe('Authentication', function() {
  var username = element('username'));
  var password = element('password));
  var error = element(by.model('error'));
  var submit = element(by.xpath('//form[1]/input[@type="submit"]'));
  var loginURL = '';

  it('should redirect to the login view if the user is not logged in', function() {
    loginURL = browser.getCurrentUrl();


  it('should notify the user when credentials are missing', function() {
    // empty the form fields

    // fill in the password field with 'test' 

    // assert that the username is empty
    expect(error.getText()).toMatch('empty username');

    // fill in the username field with 'test'

    // empty the password field

    // assert that the password field is empty
    expect(error.getText()).toMatch('missing password');

  it('should accept a valid username and password', function() {
    // empty the fields

    // fill in the credentials

    // assert that the user was logged in

  it('should redirect the user to the login page on log out', function() {
    var logoutButton = $('#logout');;

As you can see above, in contrast to a unit test, our end-to-end test automates the process of logging in across multiple views of the application. It is not concerned with the testing the smallest unit of code, but the feature as a whole.

Acceptance Testing

Acceptance testing is another variant of testing in which it must be determine if the requirements have been met. This type of testing is closer to the end user or the client that will be receiving the product. Acceptance tests and criteria are usually developed by the customers themselves and are formatted in more generic terms that relate more to the user stories than other, more technical, specifications.

That concludes this article. If you're interested in other forms of testing, check back for more articles in this series.

No comments:

Post a Comment