Sunday, March 1, 2015

How to Benchmark JavaScript Applications with BenchmarkJS

Making a Case for Benchmarking

Of all the process-oriented tasks a programmer weighs against writing actual code, I'd say benchmarking is often one of the most sidelined. Other processes that fall into this category, but take precedence include source control, documentation, bug tracking, task tracking, uml class diagrams, unit tests, and integration tests. If I could provide a reason as to why, based on my experience with various companies, it would be a general lack of understanding of the benefits of benchmarking.

Benchmarking is really about leveraging data to make decisions about how the code or the product should evolve, and to really capitalize, it helps to have a process in place. This process is going to depend on the proper implementation of a benchmarking tool, benchmark writing standards, reporting, analysis, and the agility and resources to refactor towards more performant code. Because of these dependencies, most companies will wait to implement something like benchmarking and that's not such a bad thing. Process improvements are expensive, especially when there's a large amount of infrastructure in place or the impact of change is high. If your company can justify the use of benchmarking solutions, however, the benefits are far-reaching. The argument will very likely be influenced by one or more of the following:

  • Performance is a key selling point of the product
  • You develop game engines or games
  • You develop applications or services that handle a large amount of requests
  • Performance has to be weighed against the competitive landscape
  • Sheer size of the code base warrants the need to understand how each component impacts performance
  • Performance is suffering for unknown reasons
  • You need to decide between two different solutions prior to a long term commitment

Prerequisites: It helps to have a basic understanding of memory management in the browser. This guide was written for Mac OS X users.

Setup

We'll be covering client-side performance. The two main tools you'll want to be familiar with are jsPerf and benchmarkjs. BenchmarkJS was written by Mathias Bynens and John-David Dalton. Start by navigating to http://benchmarkjs.com/, and downloading the latest development source. You'll have the option to run your benchmarks directly in the browser or through node or Ringo.Running the benchmarks through node or Ringo has the added advantage of easily automating the process cross-platform. Keep in mind that if you plan to run these benchmarks within a CI step, they take much more time to run than unit tests and you're better off running them outside of the integration process. Benchmark reports provide supplementary information and should be run periodically through a different process.

Ideally, you'll want to create a "benchmarks" or "bench" folder at the same level of your tests folder. We'll use the following directory structure:

|~project/
| |+bench/
| |~lib/
| | `-benchmark.js
| |+src/
| `+test/

You can skip directly to the source files here.

Or build the project from scratch by navigating to your project's lib folder through the terminal and typing the following commands:

mkdir project
cd project
mkdir bench lib src test
curl -O lib https://raw.githubusercontent.com/bestiejs/benchmark.js/v1.0.0/benchmark.js
touch index.html

You should now have the following folder structure:

project/
|~project/
| |+bench/
| |~lib/
| | `-benchmark.js
| |+src/
| |+test/
| `-index.html

Now that we have the library downloaded, we can begin to set up the functions we want to benchmark. There are three main scenarios I often find myself writing benchmarks for. I'm either comparing my implementation against native implementation, my implementation against alternative implementations, or comparing two different versions of my own implementation. In the following example, we're going to compare three types of for loops. Now type the following in your terminal:

touch bench/loops.js

Let's add some code to loops.js.

var suite = new Benchmark.Suite;
arr = [4, 8, 15, 16, 23, 42];

// add tests
suite.add('normal', function() {
    var total = 0;
    for (var i = 0; i < arr.length; i++) {
       total+= arr[i]; 
    }
})
.add('cached length', function() {
    var total = 0;

    for (var i = 0, len = arr.length; i < len; i++) {
       total+= arr[i]; 
    }
})
.add('cached length reverse', function() {
    var total = 0;

    for (var i = arr.length - 1; i >= 0; i--) {
       total+= arr[i]; 
    }
})
// add listeners
.on('cycle', function(event) {
  console.log(string(event.target));
})
.on('complete', function() {
  console.log('fastest is ' + this.filter('fastest').pluck('name'));
})
// run async
.run({ 'async': true });

Now set up your index page and navigate to it.

<!DOCTYPE html>
<html>
    <head>
        <script src="lib/benchmark.js"></script>
        <script src="bench/loops.js"></script>
    </head>
    <body>
        <p>Please open your console</p>
    </body>
</html>

You should see the following output in your console.

normal x 114,476,386 ops/sec ±0.95% (91 runs sampled) loops.js:27
cached length x 115,634,617 ops/sec ±1.15% (90 runs sampled) loops.js:27
cached length reverse x 952,590,815 ops/sec ±1.13% (94 runs sampled) loops.js:27
fastest is cached length reverse 

As you can see, iterating over an array in reverse is significantly faster than the other two approaches. We don't see a significant increase when caching the length because we ran these tests in Chrome. Newer browsers will cache the length for you, but older browsers still lag behind.

So that's the gist of it. We didn't go over how to run these benchmarks in node, or how to pipe the results into an analysis tool, but you should have a basic understanding of benchmarking on the client side. Even with all of this properly set up, micro benchmarks are often not the best way to gauge performance. You'll still want to extract a test from a real world scenario to take all factors into consideration.

Also, don't forget to checkout jsPerf and this for loop benchmark.

2 comments:

  1. You'll find that reverse is not that much faster or even slower. The test was written incorrectly.

    .add('cached length reverse', function() {
    var total = 0;

    for (var i = arr.length - 1; i >= 0; i--) {
    total+= arr[i];
    }
    })

    ReplyDelete