UCB-NOM algorithm cracks non-observable Markovian bandits with near-log regret

This paper tackles a tough online learning problem: Markovian bandits where the internal states of each arm are completely hidden from the algorithm, and decision epochs may be constrained. The authors focus on 'pure' regret—comparing against the best arm that never switches. They define a new subclass called self-degrading Markovian bandits, proving that pure policies are asymptotically optimal within it. Without prior knowledge, any algorithm that switches arms infrequently must incur super-logarithmic regret, scaling as ω(log T). Yet the authors propose UCB-NOM, an optimistic algorithm inspired by UCB, that achieves nearly logarithmic regret despite this theoretical barrier.

When given a prior bound on the bias functions of each arm, UCB-NOM attains exact O(log T) regret and a worst-case O(√T log T) bound—both independent of the number of Markov chain states. This independence is notable, as typical bandit analyses degrade with state space size. The results suggest that state non-observability is a mild inconvenience in self-degrading settings, shifting the complexity burden from state tracking to intelligent arm-switching and optimistic exploration. For practitioners, this offers a principled way to handle hidden-state decision problems with strong theoretical guarantees.