New optimal trigger rules cut LLM invocation costs in streaming systems

Streaming inference pipelines often pair lightweight models with expensive LLMs for semantic understanding, but deciding when to invoke the LLM has lacked formal treatment. Wang's work casts this as a risk-based sequential stopping problem: a trigger policy fires when a risk functional over observation history exceeds a threshold. The paper proves six key results: a minimum inter-event time bound to prevent chattering, optimality of threshold policies via smooth pasting, approximate SPRT guarantees under estimated parameters, regret bounds of O(sqrt(T log T)) for stationary streams and O(sqrt((C_T+1)T log T)) under changepoints, O(1/sqrt(T)) convergence for online gradient descent, and a calibration-to-miss-rate transfer inequality. Classical families like event-triggered, SPRT, CUSUM, and Bayesian triggers emerge as special cases.

Empirically, the framework was tested on the CMAPSS turbofan degradation dataset with real LLM calls. It was compared against six baselines including RouteLLM-style routers and contextual bandits. Results confirm sublinear regret and high diagnostic quality: 92.9% of 1600 LLM diagnoses reached a grounding score >=0.75 under the paper's rubric. Critically, anomaly-score-driven risk functions dominated alternatives by roughly an order of magnitude on the Pareto AUC. The work also analyzes cost sensitivity and LLM failure modes, providing practical guidance for deploying cost-efficient LLM-based streaming analytics in production environments.