Propilot uses GPT 5.5 to auto-generate property-based tests for AI compilers
Cuts test redundancy by 49% and eliminates invalid tests with algebraic skeletons.
Propilot addresses a critical gap in AI compiler testing: while fuzzing generates well-formed inputs, it cannot detect semantic drifts caused by graph transformations. The framework, powered by GPT 5.5, encodes tensor algebra into reusable property skeletons—each coupling operator constraints, shape/value rules, and oracle templates. These skeletons are then instantiated into executable property-based tests (PBTs) for a target compiler, including paired tensor computation graphs, concrete inputs, and expected semantic relations.
Propilot validates each PBT candidate before execution to avoid degenerate or uninformative tests, using feedback and coverage signals to guide generation. In experiments on TVM with 212 operators and 20 skeletons, Propilot produced 4,579 PBTs, cutting redundancy by 49% compared to direct LLM-based generation. The approach successfully uncovered real semantic errors and numerical issues, proving that structured algebraic knowledge dramatically improves both test quality and bug-finding ability in DL compilers.
- Propilot uses GPT 5.5 to generate 4,579 property-based tests across 212 TVM operators using 20 algebraic skeletons.
- Achieves 49% less redundancy than direct LLM-based PBT generation, with automatic validation of test validity.
- Catches semantic errors and numerical discrepancies in AI compiler optimizations—problems fuzzing alone cannot detect.
Why It Matters
Automates finding semantic bugs in AI compilers, improving reliability of optimized tensor computations in production.