November 2, 2018

Property based testing using jsverify

I wanted to write about property based testing for a long time. Since I usually write about things I have built, in order to write a post I had to built something. So I started building a runners calculator, using react-native. The calculator helps with estimating the pace (min/km) a runner needs to keep in order to finish by given duration.

What is property based testing you might ask?

It simply is an approach to testing where you verify your code's properties. According to QuickCheck article on wikipedia

... the programmer writes assertions about logical properties that a function should fulfil.

A good example of this are properties of multiplication operation.

  • Every integer multiplied by 0 gives 0.
  • Every integer multiplied by 1 gives same number.

These are the properties of a multiplication function that can be verified. They are quite obvious to anyone who had math in primary school. Your function is probably more complex and testing each of the cases may be cumbersome. Here property based testing comes into play. Given the definition of a property you are able to test your code against large number of cases that might not be obvious to you in the first place.

Why use it?

It helps with proving your solution. Tools which support property based testing can really save you time on writing test cases, as they generate a lot of them. It is worth watching Hillel Wayne in talk Beyond Unit Tests: Taking Your Testing to the Next Level. This talk inspired me to give this a go again. So you may try yourself if you are not convinced.

How to design for property based testing?

To be honest, applying this technique in my work was difficult at first. Problems I'm solving are most of the time not suited for this kind of testing. I thought this is rather obscure tool for specific cases. Then after watching a talk by Gary Bernhardt from SCNA 2012 titled Boundaries I had a "it's not you it's me" moment. I realised that software I'm writing may not be designed in a way that is testable in such manner.

The talk is worth watching, however, the spark that lit the fire was one specific thought of his: "creating functional cores". When you have a core of your software (class or function) that is functional, it is more likely that you can use property based testing. I also think that using primitive types may help as well. The tools I have found were limited to such types and constructing more elaborate types could be costly. I'm going to explore contract testing which may have better support for more complex types and objects, as well as static type checking.

What I have used in my small project?

Let's have a look at some code. I'm using react-native and javascript so I had settled on jsverify. Since I have created a small functional core I could test it pretty easily with regular tests and property tests as well.

test('given one input expect string of 8 chars', () => {
    checkForall(nat, (a) => {
        return toTime(a).length === 8;
    });
});

This test function will verify that my toTime function will create a string out of an int, like this toTime(65) => "00:01:05". I'm testing that no matter the input, my function will create an 8 characters long string.

const timeGenerator = bless({
    generator: nat.generator.map(n => `${n.toString().padStart(2, '0')}`)
});
test('toTime and toSeconds should be commutative', () => {
    checkForall(timeGenerator, timeGenerator, timeGenerator, (t, x, y) => {
        const time = [t, x, y].join(':');
        return toTime(toSeconds(time)) === time;
    });
});

Another test verifies that functions operating on time are commutative. Given the time like "01:49:56" converting it to seconds and then back to time will give the same result. In my opinion, this is a good place to test the property with such tools as jsverify. You will get hundreds of test cases for free based on the definition of your property. You will also receive a failing test case.

● Console

console.info node_modules/jsverify/lib/jsverify.js:334
  OK, passed 100 tests
console.error node_modules/jsverify/lib/jsverify.js:325
  Failed after 2 tests and 0 shrinks. rngState: 036f5d837c9e042b1a; Counterexample: "00"; "30"; "12";  [ '00', '30', '12' ]

I wrote a dozen or more unit tests verifying my solution, however, my input arguments were too simple and obvious. Next, I tried jsverify and it found a bug on first run, simply because I have not anticipated such an input that had been created by this library.

This to me is the best incentive to use property based testing even more :)

Tags: testing jsverify property-based functional javascript