New Study Asks: Can AI Really Be Trusted to Secure the Internet?

Every time your phone talks to a Wi-Fi router, a bank server or a website, it follows a rulebook. Engineers call these "protocols" — think of them as a giant flowchart of if-this-then-that instructions: if the password matches, then open the door; if three attempts fail, then lock the account. These rulebooks are written in dense text, and companies increasingly ask AI to read them, then use that understanding to test systems and find security flaws.

The problem is trust. If an AI misreads the rulebook — missing one branch of the flowchart — it could declare a broken system safe, or waste weeks chasing a security hole that doesn't exist. A team from Purdue University set out to measure how often that happens. They built 1,482 quiz questions spanning 16 different protocols and asked AI models to reconstruct the underlying flowchart from the written spec. Then they compared the AI's answers to a hand-built, human-checked version.

Their findings are a warning, not a verdict. AI models did not map the specifications perfectly, and difficulty varied sharply depending on the protocol and how the question was framed. The researchers also found something sneakier: the automated "AI judges" often used to grade other AI answers carry their own biases, which can make results look better or worse than reality. That's a problem across the whole industry, not just networking.

The honest limitation: this is 16 protocols, not the whole internet, and the paper measures understanding rather than real-world break-ins. The takeaway isn't "AI is useless here." It's "verify, don't trust" — which is how you'd treat a brilliant new intern who has never once seen your network and never admits uncertainty.