Time Travelling and Fixing Bugs with Property-Based Testing (2019)

Summary

Property-based testing (PBT) is a powerful testing technique that helps us find edge cases and bugs in our software. A challenge in applying PBT in practice is coming up with useful properties. This tutorial is based on a simple but realistic system under test (SUT), aiming to show some ways you can test and find bugs in such logic using PBT. It covers refactoring, dealing with non-determinism, testing generators themselves, number of examples to run, and coupling between tests and implementation. The code is written in Haskell and the testing framework used is Hedgehog. This tutorial was originally written as a book chapter, and later extracted as a standalone piece. Since I’m not expecting to finish the PBT book any time soon, I decided to publish the chapter here. System Under Test: User Signup Validation The business logic we’ll test is the validation of a website’s user signup form. The website requires users to sign up before using the service. When signing up, a user must pick a valid username. Users must be between 18 and 150 years old. Stated formally, the validation rules are: 0≤length(name)≤5018≤age≤150(1) \begin{aligned} 0 \leq \text{length}(\text{name}) \leq 50 \\ 18 \leq \text{age} \leq 150 \end{aligned} \qquad(1) The signup and its validation is already implemented by previous programmers. There have been user reports of strange behaviour, and we’re going to locate and fix the bugs using property tests. Poking around the codebase, we find the data type representing the form: data SignupForm = SignupForm { formName :: Text , formAge :: Int } deriving (Eq, Show) And the existing validation logic, defined as validateSignup. We won’t dig into to the implementation yet, only its type signature: validateSignup :: SignupForm -> Validation (NonEmpty SignupError) Signup It’s a pure function, taking SignupForm data as an argument, and returning a Validation value. In case the form data is valid, it returns a Signup data structure. This data type resembles Signu...