The Impact of Documentation on Test Engagement in Pull Requests in OSS

A new paper from Teal Amore, Nathan Berman, and Siyuan Jiang, titled 'The Impact of Documentation on Test Engagement in Pull Requests in OSS,' investigates whether documentation on testing can proactively encourage contributors to include tests in their pull requests. Traditionally, interventions like code coverage metrics or reviewer feedback are reactive, applied after a PR is opened. The study introduces the Test Engagement Ratio (TER) to measure testing frequency. Analyzing data from 160 open-source software repositories, the researchers found a weak but statistically significant positive correlation (ρ=0.36, p<0.001) between documentation comprehensiveness and TER. This relationship strengthens to a moderate correlation (ρ=0.44) in repositories with higher pull request activity, indicating that documentation's impact is more pronounced in busier projects.

Specific documentation categories, such as 'How to Run Tests' and 'How to Write Tests,' showed the strongest correlation with testing engagement, suggesting that clear, actionable guides are most effective. Additionally, TER was moderately correlated (ρ=0.52, p<0.001) with Test Code Ratio, providing preliminary evidence of its validity as a metric. The findings suggest that documentation on testing may be associated with increased testing engagement, offering a proactive strategy for maintainers to improve code quality. Future work will explore causality, documentation quality at a granular level, and cross-repository exposure effects, aiming to deepen understanding of how documentation influences contributor behavior in open-source software.