Can AI improve your judgment, or just your confidence?

AI and Judgment: Real Gains or Just More Certainty?

Both are possible, but the default is confidence, not judgment. A CHI 2025 study showed your self-confidence drifts toward the AI's, and the miscalibration lingers after it leaves. To improve judgment you need feedback loops and Mollick's Best Available Human test, not a fluent answer that simply feels right.

Here is the uncomfortable distinction most people collapse: feeling more sure is not the same as being more right, and AI is far better at delivering the first than the second. Judgment is calibration, your confidence matching your actual accuracy. Confidence is just the feeling, untethered from whether you are correct. A tool can raise one while leaving the other flat, and unless you watch for it, that is the usual outcome.

The clearest evidence I have seen is a 2025 study presented at the ACM CHI conference, As Confidence Aligns, which earned an honourable mention. In a randomized experiment, people's self-confidence drifted toward the confidence level of the AI they worked with, and, strikingly, that drift persisted even after the AI was removed from the task. The alignment degraded their calibration: those paired with an overconfident model became more sure of themselves without getting more accurate. The one thing that reduced the effect was real-time feedback on whether they were actually right. Read plainly, that means a confident assistant reshapes how certain you feel, and the distortion follows you into your next unaided decision.

This compounds with automation bias, well documented through 2026, where humans accept a machine's recommendation even when their own independent read was correct, an effect that intensifies under time pressure. Stack the two together and you get a precise failure mode for ambitious people: you ask an articulate model, it answers fluently, your certainty rises, your accuracy does not, and the inflated confidence outlives the conversation. Daniel Kahneman warned in Thinking, Fast and Slow that the subjective sense of confidence is a feeling reflecting coherence and processing ease, not evidence of correctness. AI is essentially a coherence engine. Of course it raises that feeling. Whether it raises accuracy is an entirely separate question you have to test.

So can AI improve judgment? Yes, but only if you engineer the conditions, because the default does the opposite. The first condition is feedback, the only variable that helped in the CHI study. Judgment improves when you write down a prediction and its probability before acting, then check the result later. I keep a simple decision log for this, and the AI's role is to pressure-test the reasoning, not to hand me a verdict that ends thought. The second condition is Ethan Mollick's Best Available Human standard from Co-Intelligence: ask not whether AI is good in the abstract, but whether it beats the most capable person actually available to you on this specific question, then verify the output anyway. For drafting and second opinions that bar is often cleared. For high-stakes judgment under uncertainty it frequently is not, and pretending otherwise is how confidence quietly substitutes for competence.

The third condition is the one I lean on most as a coach: use AI to multiply perspectives rather than to confirm the one you arrived with. I will ask Claude Opus 4.8, a hybrid reasoning model Anthropic released in May 2026, to argue the strongest case against my position, then steelman two alternatives I had dismissed. That widens the option set instead of narrowing it, which is the actual mechanism behind better judgment. It mirrors what good coaching does, separating what you know from what you merely feel, the discipline behind John Whitmore's GROW model and the calibration habits of the best decision-makers. The danger is using the same tool to manufacture reasons you were right all along, which feels productive and teaches you nothing. I have explored why staying open and exploring rather than rushing to certainty matters in my piece on exploration and success.

The practical test I give the executives I work with is one question: after using AI, are you more accurate or just more comfortable? If you cannot answer because you never track outcomes, assume it is confidence. Build the feedback loop first, treat fluency as a warning rather than proof, and ask the model to attack your view before it ever defends it. Done that way, AI can genuinely sharpen judgment. Done the easy way, it sells you certainty you did not earn and will not notice you are missing until a decision goes badly and you realize you never actually thought it through.