AI That Spots Ships From Space Fails on Unfamiliar Waters

A team of researchers tested six ready-made AI models on the job of naming ships from satellite radar pictures. This kind of radar, called SAR, works day, night, and through clouds, which is why it's popular for watching the ocean. The models were trained on two different collections of ship images, then graded on three tasks: identifying ships they'd already studied, identifying ships from a completely different image collection, and flagging ship types they'd never seen before.

The first task went fine. On familiar images, the best model was right about 73% of the time on one dataset and 53% on the other. The second task fell apart. One model got 64.5% of its guesses correct — which sounds decent — but only 33.3% on a fairer score that counts how well it does across every ship type. That's basically the same as flipping a coin, and it typically means the AI is just guessing the most common ship over and over. Researchers call this a cross-dataset failure: the AI memorized one dataset's quirks instead of learning what ships actually look like.

The third task, spotting unknown ship types, was also shaky. The AI's built-in confidence signals were often "close to random," meaning the model was unsure even when it shouldn't have been, and confident when it was wrong. Interestingly, simpler uncertainty measures beat the fancier ones researchers usually reach for, though which one wins depends on the dataset and the ship class being hidden.

So what? Ship-spotting AI increasingly feeds into coast guard patrols, illegal-fishing crackdowns, sanctions enforcement, and shipping insurance. If a model that looks accurate in a lab quietly degrades in a new bay or a new season, the result is missed vessels or false alarms sent to real crews. The bigger lesson applies far beyond ships: an AI's published accuracy is only as good as the data it was tested on.