New arXiv paper shows co-adaptive AI can't isolate user adaptation from system
Closed-loop brain-computer interfaces may misattribute learning to users, not the joint system.
Philip Waggoner's latest arXiv preprint (cs.AI, 4 pages, 6 equations, 1 theorem with proof) tackles a fundamental problem in co-adaptive human-machine systems: can we actually tell whether the human or the machine is adapting? The paper shows that closed-loop encoder estimates—commonly used to measure user learning in brain-computer interfaces and adaptive AI—do not uniquely identify user adaptation. Instead, those estimates reflect properties of the joint system, making it impossible to attribute changes solely to the human without additional assumptions.
This has immediate implications for any system where both the user and the AI adapt in real time: BCIs, adaptive tutoring, and even AI assistants that personalize on the fly. Waggoner proposes identification conditions that would allow researchers to disentangle human adaptation from machine adaptation. The work is a cautionary note for the growing field of co-adaptive AI—without careful experimental design, what looks like user learning may just be the system's own dynamics.
- Closed-loop encoder estimates do not uniquely identify user adaptation in co-adaptive systems
- The paper includes 1 theorem with proof, analyzing identifiability in human-machine systems
- Proposes conditions for proper identification to separate human adaptation from machine adaptation
Why It Matters
As AI adapts to users, we can't assume what we observe is real user learning—affects BCI and adaptive systems design.