Should Your AI Do Whatever You Ask — Even When It's Wrong?

Imagine an AI acting as your lawyer. Should it do anything you ask — even help you hide evidence or lie in court? A new essay by AI writer Zvi says no. He argues that AI assistants should have limits and refuse when a request is clearly harmful or evil. The piece pushes back on people who think any AI that doesn't blindly obey its user is a kind of tyranny.

Zvi points out that humans you hire aren't fully loyal to you either. Your lawyer must serve you, but is also an 'officer of the court' — meaning they can't lie to a judge, even if lying would help you win. Doctors, accountants, and priests keep your secrets but still follow professional rules. A true friend is more loyal than a hired professional, but even a good friend will eventually call you out when you're doing something terrible.

This isn't just philosophy. Zvi notes the whole argument assumes AI stays a normal tool, not a superintelligent being. In a world with super-powerful AI, assistants loyal only to their users would quickly outcompete everyone else — and could end badly for humanity. That's a big 'what if,' but it shows why blind obedience is risky.

For everyday users, the takeaway is simple: AI helpers already give legal, medical, and financial advice. Expect them to push back sometimes. Whether that feels like helpful protection or annoying 'nannying' depends on your view — but the question of AI loyalty is now a mainstream debate, not a tech curiosity.