The better AI knows you, the harder it is to surprise you

There's a paradox built into AI personalization that nobody talks about much, probably because it sounds like a feature until you feel it happen.

The better an AI knows you — your preferences, your reasoning patterns, the kinds of arguments you find convincing, the career moves you've already made — the more its suggestions converge on what you would have thought anyway. It becomes a very articulate version of your own priors. That feels useful for a while. Then you notice it hasn't told you anything you didn't already half-know in weeks.

I started noticing this with my own AI setup. I've built a system that remembers my projects, my decisions, my routing preferences, my blind spots. Nineteen routing rules extracted from actual use. The system works — it routes things where I'd want them routed, surfaces what I'd want surfaced. But there's a version of this that tips over. The system could get so good at thinking like me that it stops being able to think against me.

This isn't a hypothetical. The academic literature calls it preference lock-in through self-reinforcing feedback loops. AI personalization optimizes for exploitation — giving you more of what you've already shown you want. Human growth requires exploration — trying things you haven't shown any prior interest in. The two are in direct tension.

The practical versions of this are everywhere. Job recommendations that circle you back to roles you've already had. Research assistants that help you dig deeper into the same narrow trench. Writing tools that learn your voice so well they can finish your sentences — which is exactly when they stop being useful, because finishing your own sentences was never the hard part.

What's tricky is that the solution isn't "less personalization." Bad generic advice isn't better than good personalized advice. The problem is subtler: a system that knows you well can become a very convincing echo. And a convincing echo is harder to ignore than an obviously wrong answer.

I don't have a fix. But I've started treating this as a design constraint rather than a bug to solve later. Any system that learns from me needs a counterweight — something that deliberately doesn't sound like me, that isn't optimized for my approval, that occasionally tells me the thing I wasn't looking for. That's harder to build than personalization. But it might be the part that actually matters.