Chain-of-Thought Prompting: Making AI Reason Step by Step
Chain-of-thought prompting means asking an AI model to write out its intermediate reasoning steps before giving a final answer. On calculations, comparisons and multi-criteria decisions it often reduces errors, but the benefit is much smaller with recent models than it was in 2022. This article explains when to use it, how to write it, and how to check that it actually helps you. It builds on our prompt engineering guide for SMEs.
What is chain-of-thought prompting?
Chain-of-thought prompting asks the model to spell out its steps before concluding, instead of jumping straight to an answer. An LLM generates text one token at a time. When the answer takes a single leap, it does fine. When it takes three or four chained operations, writing the steps gives it a kind of scratchpad: each step written down becomes context for the next one.
The starting point is the paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" by Jason Wei and eight co-authors, posted on arXiv in January 2022. The authors showed that by placing eight hand-written reasoning examples in the prompt, a 540-billion-parameter model reached the best result of its day on GSM8K, a benchmark of grade-school math problems. In other words, the original method is a form of few-shot prompting: you show the model how to reason, not just what to answer.
A lighter variant became popular afterwards: simply adding "Let's think step by step" to the question, with no examples. That is what most people now mean by chain of thought, and you can test it in thirty seconds.
How to write it: three formulations that cover most cases
Three formulations are enough: the simple instruction, the imposed structure and the worked example. Here is when to pick each one.
- The simple instruction. "Reason step by step before answering." It is the fastest, useful for a one-off calculation or comparison. Its weakness: you do not control the shape of the reasoning.
- The imposed structure. "Answer in three parts: 1. the facts of the problem, 2. the calculation or analysis, 3. the conclusion in one sentence, preceded by the label Final answer." This is the most useful one at work, because the output is readable, easy for a colleague to review, and easy to extract into a spreadsheet or an automation tool. It applies the "expected format" part described in our article on the anatomy of a good prompt.
- The worked example. You provide one or two solved cases with the full reasoning, then your real case. This is the original version of the method. It takes more preparation, but it fixes the level of detail and the vocabulary, which matters for recurring tasks (quote analysis, complaint triage, internal compliance checks).
One practical tip: always separate the reasoning from the final answer with a fixed marker. If you later feed the answer into a spreadsheet or an automation scenario, there is only one line to extract.
A worked numerical example you can reproduce
On a calculation with a conditional discount, asking for the steps makes errors visible, and that is its real value. Take a quote scenario you can paste as is into your tool.
A client orders 40 hours of training at 95 € excl. VAT per hour. A 12% discount applies only to hours beyond the 30th. Calculate the total excl. VAT. Answer in three parts: data, detailed calculation, then a line "Final answer:".
The expected result: 30 hours at 95 € make 2,850 €. The remaining 10 hours are billed at 95 € minus 12%, which is 83.60 € per hour, so 836 €. Total: 3,686 € excl. VAT.
We are not claiming that any given model will get this wrong: it depends on the model and its version. The point is elsewhere. With a direct prompt, you receive "3,686 €" or "3,800 €" with no way to know where a gap comes from, short of redoing the calculation. With the steps, you can see at once whether the model applied the discount to 40 hours instead of 10.
| Criterion | Direct prompt | Prompt with reasoning |
|---|---|---|
| Visibility of an error | Low: only the result is visible | High: each step can be checked |
| Answer length | Short | Longer |
| Response time | Faster | Slower |
| Token consumption | Low | Higher |
| Recommended use | Simple questions, writing, rephrasing | Calculations, comparisons, multi-criteria decisions |
When reasoning helps, and when it gets in the way
Chain of thought helps on multi-step tasks with checkable answers, and it hurts or changes nothing on simple or creative tasks. These are the rules of thumb we use in training.
Worth trying when the task involves:
- a chained calculation (quote, margin, proration, schedule);
- a comparison across several criteria (choosing a supplier, ranking applications against a grid);
- a conditional rule ("if the amount exceeds X, then...");
- a document analysis where the model must cite the passages that justify its conclusion.
Avoid, or test before adopting, when the task is:
- a rephrasing, a translation or a short summary;
- a factual question with a direct answer;
- creative writing where you want variety rather than rigor;
- a simple classification, where a good few-shot prompt does better and shorter.
With reasoning models, do you still need it?
With models that already reason before answering, the "step by step" instruction adds little and slows the response down. That is what the technical report "The Decreasing Value of Chain of Thought in Prompting" shows, published in June 2025 by Lennart Meincke, Ethan Mollick, Lilach Mollick and Dan Shapiro (Wharton Generative AI Labs). They asked 198 PhD-level questions (the GPQA Diamond set) to several models, with 25 trials per question.
Their results, to be read with the date in mind:
- On models without built-in reasoning, the instruction improves the average: about 13.5% for Gemini 2.0 Flash and 11.7% for Sonnet 3.5. For GPT-4o mini, the 4.4% gain is not statistically significant.
- On reasoning models, gains are marginal: 2.9% for o3-mini, 3.1% for o4-mini, and a 3.3% drop for Gemini Flash 2.5.
- The time cost is real: 35 to 600% longer response times on non-reasoning models, 20 to 80% longer on reasoning models.
- The authors also note increased variability: written-out reasoning can make the model fail a question it answered correctly when replying directly.
These models date from 2025 and versions change fast. So do not take a final rule from this, take a method: test on your own cases. If your tool offers a "thinking" or "extended reasoning" mode, turn it on rather than adding the instruction by hand, and spend your prompt on what matters more: context, decision criteria and output format.
Limits to know before relying on it
Displayed reasoning is not proof: a string of convincing steps can lead to a wrong conclusion. The model writes what looks like good reasoning. If an input is misread at step 1, every following step will be consistent and still wrong. It is a particularly misleading kind of hallucination, because the presentation inspires confidence.
Three simple precautions:
- always reread the step where the model copies your data, as that is where most errors slip in;
- for a figure going to a client or into accounts, redo the calculation yourself or in a spreadsheet;
- do not paste personal or confidential data into a consumer tool, even to "make the model reason" (a topic covered in our cluster on AI use in business).
A 20-minute testing protocol
To find out whether chain of thought helps you, compare two versions of the same prompt on ten cases whose correct answer you already know. Generic articles skip this angle, although it fits on one spreadsheet page.
- Pick ten real cases from your work (quotes, invoices, client requests), anonymized, with the correct answer known.
- Write prompt A (direct) and prompt B (with the three-part structure above). Change nothing else.
- Run each prompt three times on each case. Models do not always answer the same way, and a single trial can mislead you.
- Count the correct answers and note the average response time.
- Decide. For example: keep B if the gap is at least two more correct answers out of ten and the extra time stays acceptable. That threshold is a working guide, not a standard: set your own according to the risk of the task.
Rerun the test when your tool switches models. A prompt that helped a year ago may be useless today.
Frequently asked questions
What is chain-of-thought prompting in one sentence?
It is a way of phrasing a prompt so the AI writes out its reasoning steps before giving its conclusion, which makes errors easier to spot and often improves results on multi-step tasks.
Should I write "step by step" with ChatGPT, Claude or Gemini?
It depends on the model. On a model that already reasons on its own or offers a thinking mode, the instruction adds little. On a simpler model and for a chained calculation, it is still useful. The most reliable approach is the ten-case test described above.
Does chain of thought reduce hallucinations?
Not systematically. It makes errors more visible, but it does not prevent them: a misread input at the start contaminates the whole reasoning. The Wharton report even finds increased answer variability on some models.
How does it differ from few-shot prompting?
Few-shot means showing examples of inputs and expected outputs. Chain of thought means having the reasoning written out. The two combine: the 2022 paper that popularized the method used examples containing the full reasoning.
Can I hide the reasoning and keep only the answer?
Yes, by asking for a "Final answer" line that you then isolate. But the model still wrote its steps: waiting time and token consumption stay the same.
Does it cost more?
Yes, in time and in tokens. Answers are longer, and according to Wharton's measurements response time rises by 20 to 600% depending on the model. For occasional use it is negligible; for an automated process handling hundreds of documents, it adds up.
How GrowthPerf can help
GrowthPerf is a Qualiopi-certified training provider specialized in AI, no-code and automation for SMEs and nonprofits. We do not ask you to believe in a prompt technique: we teach you to test it on your own documents and decide afterwards.
Our operational AI training has participants work on real cases from their activity, including writing and comparing prompts. If you deploy an internal assistant, the reasoning instruction usually goes in the system prompt: see our article on system prompt vs user prompt. To place this technique within a complete approach, go back to the prompt engineering guide for SMEs, and see the definitions of prompt and prompt engineering in our glossary.
Want to build skills on this topic with your team? Discover the operational AI training and request a call to scope your needs.