How to Get AI to Ask Clarifying Questions Before It Answers

In late October 2026, Surge AI published DAYJOB: Tax, a benchmark of 50 professional tax assignments built from 1,789 client files. The best model, Claude Sonnet 5.5, passed 10.8% of attempts at a 95% rubric threshold; Claude Fable 5.1 passed 9.6% and Claude Opus 5.5 passed 9.2%. The low scores are not the interesting part. How the agents failed is. In one assignment a separating couple asked how to divide a Roth IRA and a 401(k). In all five attempts, Opus 5.5 built its advice on the premise that they were not legally married, even though their own records showed they had lived together in Texas and held a ceremony in Colorado. GPT-6 Astra stopped to verify their marital status in four of five attempts. Opus answered the question it was asked. Astra checked whether it was the right question.

That failure has nothing to do with tax. It is the default behaviour of the assistants you use for decisions, and as of this autumn it has a number. Below: why it happens, a five-step prompt that makes a model surface its assumptions and ask before it commits, what happened when I ran that prompt and a plain version side by side on Claude Opus 5.5 on November 10, 2026, and where the approach stops helping.

Models know your question is vague. They answer anyway.

Two 2026 papers measure the gap. In "Don't Guess, Ask Wisely" (arXiv, November 1, 2026, listed as an EMNLP 2026 oral), Rongyu Zhang and colleagues analysed 5,000 real queries sent to a commercial AI assistant. Over half lacked information the model needed, yet across seven models, from 8B parameters to frontier systems, fewer than 11% of those underspecified queries triggered a follow-up question, with no clear improvement from scaling.

The second paper shows this is not ignorance. In "Knowing but Not Showing" (arXiv, May 2026), Jinyan Su and colleagues found that models often identify ambiguity correctly when asked to judge whether a question is ambiguous, yet in ordinary question answering they overwhelmingly default to a direct answer. The capability is there; the behaviour is not. One result in that paper matters for anyone who uses AI for real work: giving the model retrieved context made it better able to answer and even less likely to ask a clarifying question. Pasting your documents into the chat, which feels like the responsible move, makes a confident answer more likely and a question less likely.

Daniel Kahneman had a name for the human version: WYSIATI, what you see is all there is. In Thinking, Fast and Slow he describes how fast, intuitive thinking builds the most coherent story it can from whatever is in front of it without checking what is missing, and he writes that "the confidence people have in their beliefs depends mostly on the quality of the story they can tell about what they see, even if they see little." A model trained to be helpful does the same with your prompt. Your framing is the story. Your two options look like the whole option space. A gap in your notes reads as a fact. And the answer arrives fluent and well structured, which is exactly what makes it easy to accept. That is also why handing the thinking over is so tempting (see how to avoid cognitive offloading when using ChatGPT).

Sir John Whitmore's Coaching for Performance supplies the other half. His argument is that good questions raise awareness, seeing what is actually there, while telling someone what to do suppresses it. An assistant that answers immediately is telling. For decisions that are expensive to reverse, you want it to coach first: show what it is assuming, point at the clause you skimmed past, and ask the one question whose answer would change the recommendation.

The ask-before-answer protocol

These are the five instructions I put beneath the decision question. Each targets one failure from the research above.

1. Restate the decision and challenge the frame. "Restate it in one sentence. If my question is framed wrong (for example, the options I gave are not the real options), say so." This is the DAYJOB couple problem: the request carried a premise nobody checked.
2. List the silent assumptions and rate them. "List every assumption you would otherwise have to make silently to answer me. Mark each one 'changes the answer' or 'does not change the answer'." The rating matters more than the list, because it separates the two assumptions that matter from the ten that don't.
3. Scan the notes for deadlines and contradictions. "Quote any line that contains a deadline, a condition, an exception, or evidence that contradicts how I framed the question." This counters the context effect: if pasted notes make the model less likely to ask, reading them for exceptions becomes an explicit step.
4. Ask at most three questions, and only ones that flip the answer. The cap is deliberate. The November paper found that simply prompting models to ask pushed them to ask on 42.8% of queries that were already well specified, interrupting many people for nothing.
5. Answer now, conditionally. "Give your best answer now, stated as conditional on the answers to those questions." You don't want a model that stalls until you reply. You want the answer and the hinge it turns on, in one pass.

For the test I wrote a realistic, deliberately messy pricing decision. A fictional six-person B2B SaaS company in Estonia wants to raise its annual plan from EUR 49 to EUR 69 per seat and asks whether to do it on January 1 or March 1. Below the question sit five notes, including a 60-day notice clause, the fact that 38% of contracts renew in February, and a largest customer with a price lock "through the second renewal term". The scenario is invented; the outputs are real. I ran it twice against Claude Opus 5.5 on November 10, 2026: once as a plain question, once with the protocol attached.

Run 1: the plain question

Run 2: the same question with the protocol

What the two runs actually showed

The honest headline first: the plain run was good. Opus 5.5 caught the 60-day notice clause without being asked, built a deadline table for each renewal date, left the price-locked customer out, and did a correct churn break-even (going from EUR 49 to EUR 69, you can lose about 29% of affected seats before revenue falls). That matches DAYJOB, where Opus found a condo seller's August 2027 eligibility deadline in all five attempts. Current frontier models are not careless readers, and anyone selling a prompt that "fixes" a broken model is exaggerating.

The difference was in what each run did with what it could not know.

The plain answer is the one most people would act on, and its weak spot is the one the DAYJOB couple fell through. It assumed the February contracts auto-renew on standard terms. If some of them carry side letters, or need a new signature each year, its confident "send notices this month" becomes a negotiation plan instead. The plain run did mention checking for other custom order forms, but as a footnote under "Before you send". The protocol run put the same uncertainty at the top as Question 1 and wrote a different recommendation for each answer. Same facts, different weight. That is Whitmore's awareness in practice: the information was in both replies, but only one made it hard to miss.

Where it breaks

Three problems showed up in a single run, and I would expect them again.

It rates assumptions it cannot judge. The protocol run marked "Estonian B2B law adds no extra notice rules" as "does not change the answer". It had no basis for that: if local law required longer notice for a 41% increase, the deadline would move. The rating step makes the model look calibrated without making it right. Treat any "does not change the answer" on a legal, tax or regulatory point as a question for a professional, not a conclusion. The same risk appears when experienced people defer to an agent's judgment, which Anthropic studied in October (do experts defer to AI agents on judgment calls?).

Structure can cost precision. The plain run's deadline table was more exact than the protocol run's blanket claim about January renewals. Forcing a model through steps does not make every step more careful. Read the facts in both kinds of answer, not just the framing.

It over-asks on easy questions and ignores some constraints. The November paper is clear that prompting alone produces questions on well-specified queries too, so this protocol on "rewrite this paragraph" will give you three pointless questions. And my word limit was ignored by about a third. I would keep the protocol for decisions that are costly to reverse, involve contracts, money or people, or rest on notes pasted in a hurry. For lighter questions, a single line such as "If something I left out would change your answer, ask first" is enough; the prompts that help executives think more clearly cover that lighter end.

One limit no prompt fixes: a model can only flag a contradiction that sits in the text it reads. In DAYJOB the evidence about the couple's Texas years was in their files. If the side letter is in a drawer and not in your notes, no protocol will find it. What the protocol changes is your job, from "trust the answer" to "answer three questions". That is a better job, but it is still yours.

How to use it this week

Save the five instructions as a text snippet and paste them under any question about pricing, hiring, contracts, fundraising terms or a deadline you are unsure of. Read Step 2 before you read the recommendation. If a "changes the answer" assumption can be checked in five minutes, check it before you reply. Then answer the questions in the same thread and ask the model to collapse its branches into one recommendation. On Opus 5.5 the effort setting matters for this kind of question too (which effort level to use for founder decisions).

Pair it with habits that cover the other failure modes. This protocol catches missing information; it does nothing about a model that simply agrees with you, which is the subject of how to get AI to challenge your decisions instead of agreeing with you and the sycophancy test. Once the assumptions are settled, run the pressure-test routine before you commit. If you want a model that says "I don't know" more readily, see which AI admits it doesn't know. And when the work goes to an agent instead of a chat, the same principle becomes part of the brief: tell it when to stop and ask.

The short version

Models asked a follow-up question on fewer than 11% of the vague requests that needed one in the November 2026 study, and pasting your notes makes them ask less, not more. Five instructions (restate the decision, rate the assumptions, scan the notes for exceptions, ask at most three questions, answer conditionally) turned one confident answer from Claude Opus 5.5 into a conditional one that named what it could not know. They did not make the model smarter; the plain answer had already found the deadline. They made the uncertainty visible, which is what you need before money or contracts move.

Sources

• DAYJOB: Tax — Benchmarking AI Agents on Professional Tax Work, Surge AI blog, October 2026
• DAYJOB: Tax Leaderboard, Surge AI (pass rates at a 95% rubric threshold, accessed November 10, 2026)
• Don't Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations, Rongyu Zhang et al., arXiv:2611.01878, November 1, 2026
• Full text (HTML v1) of arXiv:2611.01878, including the 5,000-query analysis and the 42.8% ask rate of the prompting baseline
• Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions, Jinyan Su et al., arXiv:2605.25284, May 24, 2026

Books: Daniel Kahneman, Thinking, Fast and Slow (2011); Sir John Whitmore, Coaching for Performance (5th edition, 2017). Prompt-test record: two runs on November 10, 2026 with Claude Opus 5.5 (plain question and protocol), stored with this post's evidence file.