Everyday Data Science
Latest
Agentic workflows now power a third of surveyed enterprise automationAfrica's AI startup ecosystem posts record funding yearNew benchmark results reshape the coding-agent leaderboardNigeria launches national AI strategy with major investment planRwanda's sovereign AI cloud enters public betaThe future of AI agents: from tools to teammates
Research DigestResearch Brief

Thinking Harder Bought Confidence, Not Better Shopping: Ten Frontier Models, Three Benchmarks

Two University of Chicago researchers gave ten frontier models a shopping list and three effort settings. Accuracy barely moved. Confidence rose for nine of ten models, and the models that failed together failed on the same wrong product.

Ibrahim Denis FofanahIbrahim Denis FofanahData Scientist & AI Researcher13 min read·Agents and Evaluation · Research Brief

The question

Inference-time scaling is one of the few reliable levers in modern LLM work. Give a model more tokens to think and it does better on maths and coding. Shopping agents are a different kind of task. The hard part is reading a listing carefully, not deriving a multi-step answer. A model that misses a secondary constraint at low effort, a size, a scent, a price ceiling, does not obviously recover it by thinking longer.

Adi Krishnamoorthy and X.Y. Han at the University of Chicago tested that directly. Their paper, "Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents" (arXiv:2611.22033, submitted 17 November 2026), benchmarks ten frontier models on three product-choice benchmarks and asks two questions: does more thinking buy better choices, and does the confidence the model reports tell you how likely it is to be wrong?

What they ran

The design is deliberately stripped down. Each task hands the model a shopping query and a fixed, shuffled list of ten candidate products, exactly one of which is correct. Search and navigation are removed, so every error is a judgment error and every model sees identical evidence at every effort level.

The three datasets cover different ground. WebShop contributes 496 tasks built from Amazon listings with images, where a choice is correct only if it earns the maximum reward of 1 on attributes, options, and price. Amazon SQID contributes 500 tasks where correctness is a human Exact label from the ESCI annotation scheme, with short typed queries like "running sunglasses for men" and "makita inflator 18v". Wayfair WANDS contributes 330 tasks, text only, no images, prices, or options, with products described by a median of 54 catalog attributes. The models are GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna; Claude Fable 5.1 and Claude Opus 5.5; Gemini 4 Argon and Gemini 3.1 Pro; DeepSeek V4.1 Flash; Qwen 3.8 Max; and Kimi K3. Each was queried through OpenRouter at low, medium, and high reasoning effort, except DeepSeek and Kimi, which offer only low and high. In total the study collected 37,128 responses, 99.92 percent of which contained a parseable choice and confidence rating.

Confidence was elicited as an integer from 0 to 10 in the same JSON object as the answer, then rescaled to 0 to 100 so it can sit next to the success rate. Calibration is measured with expected calibration error, Brier score, and AUROC. Every effort comparison is a paired within-task test with a 95 percent confidence interval, and the headline significance claims survive a Holm correction for 26 comparisons per metric and dataset.

Finding 1: Accuracy barely moved when effort went from low to high

On WebShop, success rates across all 28 model-effort conditions sat between 74.0 and 88.1 percent. The low to high change was small everywhere. Only three of the thirty model-dataset pairs showed a nominally significant accuracy gain on WebShop: Qwen 3.8 Max at +5.5 points, Gemini 3.1 Pro at +3.2, and Gemini 4 Argon at +2.4. On WANDS, only GPT-6 Luna improved significantly, +5.2 points. On SQID, no model improved significantly at all.

Then the correction arrives. After Holm correction for multiple comparisons, only Qwen on WebShop survives among the accuracy gains. The authors note in their discussion that 8 of 11 nominally significant accuracy effects disappear under correction. The two cases where accuracy did respond, Qwen on WebShop and GPT-6 Luna on WANDS, delivered the whole gain at the low to medium step and plateaued after that. That pattern suggests the low setting was genuinely insufficient for those models on those tasks, not that more thinking keeps helping.

Finding 2: Confidence rose for nine of ten models on WebShop

The same low to high change that left accuracy flat moved confidence almost everywhere. On WebShop, confidence rose significantly for nine of ten models. On SQID it rose for five of ten, and on WANDS for six of ten. Counting every effort contrast, confidence changed significantly in 20 of 26 comparisons on WebShop, 11 on SQID, and 14 on WANDS, with almost all of those changes upward.

The clearest case is Gemini 3.1 Pro. Its confidence rose significantly on all three datasets while its accuracy rose only on WebShop, and its WebShop AUROC actually fell while the share of answers rated 10 out of 10 rose heavily. The one exception runs the other way: Kimi K3 is the only model whose confidence fell significantly, down 0.6 points on WebShop and 0.8 on WANDS. The increases are modest on the 0 to 10 scale, usually one to three points when rescaled, but they are systematic, and they start from a level that is already too high for eight of the ten models.

Finding 3: Calibration belongs to the model, not the setting

This is the finding that matters most for anyone choosing a model to ship. The two Claude models were near calibrated on every dataset. Claude Fable 5.1 had an expected calibration error of 3.4 points on WebShop, 13.2 on SQID, and 5.0 on WANDS. Claude Opus 5.5 scored 6.7, 12.4, and 5.3. The other eight models ranged from 8.9 to 18.2 on WebShop, 27.6 to 34.3 on SQID, and 11.8 to 19.9 on WANDS.

In overconfidence terms, the Claude models were slightly underconfident on WebShop, at minus 2.8 and minus 6.7 points, and roughly calibrated on WANDS at plus 0.7 and minus 1.5. The other eight were overconfident by 7 to 16 points on WebShop, 10 to 20 on WANDS, and 27 to 33 on SQID, where accuracy falls to around 50 percent while stated confidence falls much less. The ranking of models by calibration barely changes across datasets whose accuracy levels differ by nearly forty points: the Spearman correlation of the ECE rankings runs 0.79 to 0.92. Effort did not change any of this. Reading across the low, medium, and high columns, ECE moves by less than two points for most model-dataset pairs.

AUROC tells a subtler story. It ignores the level of confidence and asks only whether correct answers carry higher confidence than wrong ones. Here the gap narrows: Claude leads on WebShop and WANDS at 0.69 and 0.80, but most models cluster between 0.56 and 0.69 on WebShop and 0.67 and 0.76 on SQID. The overconfident models do carry information in their confidence. They report it on a scale shifted upward by 10 to 30 points. A model can be informative and still unusable as a gate, because the number it prints is not the number you should believe.

Finding 4: "High effort" meant wildly different amounts of thinking

The effort control is a provider-defined abstraction, and the token counts show it. On WebShop, Qwen 3.8 Max went from 394 reasoning tokens per task at low effort to 2,033 at high, and from 488 to 3,926 on SQID. DeepSeek V4.1 Flash went from 445 to 634. Gemini 3.1 Pro went from 305 to 772, and Gemini 4 Argon from 0 to 774. GPT-6 Astra and GPT-6.1 Sol stayed below 100 reasoning tokens per task even at high effort.

Two providers' "high" is a rounding error next to another's. That matters for evaluation: a benchmark that reports results at "high effort" without reporting the tokens actually produced is comparing settings that differ by a factor of twenty. It also matters for cost. Qwen's fivefold token increase bought the only Holm-surviving accuracy gain in the study, 5.5 points on WebShop. The GPT models' near-zero token increase bought nothing, but also cost nothing. If you are paying per token, the effort knob is a price control as much as a quality control, and this paper gives you the numbers to price it.

Finding 5: When models failed, they failed together, on the same wrong product

The most unsettling result is not about effort at all. If the ten models made mistakes independently, the count of models wrong on a given task would follow a binomial distribution, clustered around the average. Instead the distribution is bimodal. On WebShop at low effort, 319 of 493 tasks were solved by all ten models, where independence predicts about 52, and 47 were failed by all ten, where independence predicts 0.0001.

The variance of the wrong-model count is 6 to 8 times what independence would produce, and the implied pairwise error correlation is 0.70 to 0.71 on WebShop, 0.79 to 0.81 on SQID, and 0.77 to 0.81 on WANDS. None of these quantities moves with effort. The models are as aligned in their mistakes at high effort as at low.

They do not merely fail on the same tasks. They fail in the same way. On tasks where at least two models erred at low effort, the share of wrong choices landing on the single most popular wrong product is over 80 percent on all datasets, against 32 to 36 percent if wrong models picked uniformly among the nine distractors. On WebShop the shared wrong product is typically a plausible near miss that satisfies the product type and a headline attribute but fails a secondary constraint such as size, scent, or price. On SQID it is most often a Substitute whose title is nearly identical to the Exact product's.

What this means for data and AI practitioners

First, treat calibration as a selection criterion, not a tuning target. The paper's own conclusion is that choosing a calibrated model matters more than turning up its effort. If your agent hands decisions back to a user when it is unsure, or buys only above a confidence threshold, the model with the lowest ECE is worth more than the model with the highest effort setting. Claude's near-calibration here is a property of the model. You cannot buy it with tokens.

Second, price the effort knob before you turn it. On this evidence, moving from low to high buys confidence and tokens, not correctness, for most models. The exception pattern matters: where gains existed, they arrived at the low to medium step and plateaued. Test medium before you pay for high, and measure the tokens each setting actually produces, because "high" spans under 100 to over 2,000 tokens per task across providers.

Third, do not use agreement as your safety check. Two agents agreeing on a purchase is weak evidence when their errors correlate at 0.7 to 0.8. If you need a real check, it has to come from a different kind of evidence: a rule-based constraint check on price, size, and attributes, a retrieval step that re-reads the listing, or a human in the loop for high-value orders. The paper's near-miss analysis points the way. The shared wrong product fails a secondary constraint. That is exactly the kind of thing a deterministic checker can catch and a second LLM will miss for the same reason the first one did.

Fourth, report calibration and effort-conditioned results alongside accuracy in your own evaluations. The authors ask for exactly this, plus paired designs with multiple-comparison control and token counts per setting. Their own numbers justify the ask: without the Holm correction, the accuracy story looks like three models improving. With it, the story is one model, one dataset, and a plateau at medium.

Against

The study isolates the decision and deliberately excludes search, pagination, and multi-step navigation, where effort might matter more. A shopping agent that has to find the product first may benefit from thinking in ways this design cannot see. Confidence was elicited once, as an integer in the same JSON as the answer, and other elicitation protocols might yield different levels, though the stability of the rankings across three datasets suggests the model differences are robust. Each condition was run once with provider-default sampling, so run-to-run variance is absorbed into the task-level noise rather than estimated separately.

In favour

The design is the study's strength. Fixed ten-product choice sets, identical evidence at every effort level, paired within-task tests, and Holm correction for 26 comparisons per metric and dataset make the null result hard to dismiss as noise or a weak baseline. The three datasets differ in retailer, domain, modality, and ground truth, yet the pattern repeats: flat accuracy, rising confidence, stable calibration rankings, and shared errors. The token counts are reported per model and setting, which is more than most effort studies offer.

What would make me wrong

  • A replication with search and navigation included, where high effort improves end-to-end success by a margin that survives correction, would show the null result is an artifact of the isolated-decision design.
  • A different confidence elicitation, such as token probabilities or a separate verbalized estimate after the answer, that moves the ECE rankings or closes the Claude gap would show the calibration result is a measurement artifact.
  • A model family trained explicitly for calibration that reaches Claude-level ECE with higher accuracy would show the trade-off is not structural.
  • A repeat run with fresh sampling that flips the Qwen WebShop gain or finds several Holm-surviving gains would show the accuracy null is underpowered rather than real.

Sources

Would you pay for high reasoning effort in a shopping agent after seeing these numbers, or would you spend that budget on a calibrated model and a deterministic constraint check instead?

About the writer

Ibrahim Denis Fofanah
Ibrahim Denis Fofanah

Data Scientist & AI Researcher

3 followers

Data scientist and AI researcher at Pace University. I coined Artificial Frictional Unemployment, and built the first machine learning model for crop yield prediction in Sierra Leone. Author of Understanding Agentic AI. I write about agentic systems and applied ML, with a bias toward what actually works, and who gets left out when it doesn't.

Share

Found this useful? Passing it on to someone who builds is the best way to help the publication grow.

Built something worth sharing? Write it up for us →