Programmable Property-Based Testing

A team of researchers including Alperen Keles, Justine Frank, Ceren Mert, Harrison Goldstein, and Leonidas Lampropoulos has published a paper proposing a fundamental shift in how property-based testing (PBT) frameworks are designed. Their work, 'Programmable Property-Based Testing,' introduces a new language based on 'deferred binding abstract syntax' that treats properties as reified data structures rather than executable code tightly coupled to specific testing runners.

The technical innovation centers on a mixed embedding approach that decouples property definitions from the mechanisms that test them. Current PBT frameworks like QuickCheck use shallowly embedded domain-specific languages where properties are written directly in the host language, limiting customization to what framework authors anticipated. The new approach allows properties to be manipulated as first-class data, enabling users to create specialized testing runners without rewriting entire frameworks.

Implemented in both Rocq (leveraging dependent types) and Racket (using dynamic types), the system demonstrates practical flexibility through rapid prototyping of various property runners. This addresses a key limitation in modern software testing where developers often face a choice between limited configuration options in existing frameworks or the significant overhead of building custom testing infrastructure from scratch.

The implications are significant for software engineering teams working on complex systems requiring specialized testing approaches. By making PBT more programmable, the research enables domain-specific testing improvements that were previously impractical, potentially reducing bugs in critical software while lowering the barrier to creating tailored testing solutions for specific application domains.