PETRA dataset boosts petroleum engineering search by 44% on benchmarks
A new pipeline turns noisy web data into a curated search corpus for oil and gas.
PETRA addresses the supervision gap in petroleum-engineering search, where general retrievers struggle due to scarce domain-specific relevance labels. The pipeline converts noisy public web text into a curated corpus using high-recall energy-domain curation, an energy-domain classifier (98.4% accuracy), chunk-grounded query generation, LLM-written hard negatives, and retrieval-mined candidate lists. The resulting dataset contains 1.36M curated chunks (~2B tokens), 859k embedding training rows from 224k anchors, and roughly 400k teacher-scored reranker candidate rows.
Key results show that first-stage retrieval nDCG improves from 0.703 to 0.763 via score fusion, while reranker adaptation yields a 44% relative improvement on the Earth Science benchmark and 23% on a six-task reasoning-intensive panel. An important negative finding: high train-holdout accuracy on synthetic labels does not predict retrieval gains. Retrieval-mined data only helps after being repackaged as teacher-scored candidate lists sampled from the inference-time candidate distribution. This work demonstrates a systematic approach to domain adaptation for specialized fields, with implications for industrial search in energy and beyond.
- PETRA contains 1.36M curated chunks and ~859k embedding training rows from ~224k anchors.
- The classifier achieves 98.4% test accuracy for energy-domain classification.
- Reranker adaptation improves Earth Science benchmarks by 44% and reasoning tasks by 23%.
Why It Matters
Enables accurate search in a high-value domain where relevance labels are scarce, improving knowledge retrieval for oil and gas professionals.