## Elaborating on Chain-of-Thought Evolution in Reasoning Models #ai/reasoning #paper/environmentalaestheticsofai #paper/generatingphilosophy ### From CoT as Prompting to Search-like Reasoning via RL The story is a progression. First, _chain-of-thought_ (CoT) reframes reasoning as next-token prediction: the model keeps “continuing” text, but we nudge it to continue with steps. Then, _reasoning models_ train—via [[reinforcement learning]] (RL)—to _internally_ generate longer, higher-quality traces. Those traces often display search-like behaviours (exploring alternatives, revising steps) and, with verifiable rewards, they become increasingly effective. This is why models such as OpenAI’s o-series and DeepSeek-R1 outperform standard LLMs on hard benchmarks: they don’t just _prompt_ for CoT, they _learn_ to produce and use it. **Three anchors** - **Text continuation as the substrate.** CoT leverages the model’s native “continue the sequence” machinery. In few-shot form, it yields big jumps on grade-school maths (e.g., PaLM-540B on GSM8K: ~18% → ~57–58% with CoT; ~74% with self-consistency decoding). Zero-shot CoT (“Let’s [[think step]] by step”) exists too, but the famous 57–58% figure comes from _few-shot_ CoT, not the zero-shot trick. - **Search-like long CoTs via RL.** Reasoning models “bake in” long hidden CoTs and, under RL pressure, exhibit behaviours _analogous_ to algorithmic search (branching, backtracking, self-critique). The search analogy is descriptive, not a claim that the model runs an explicit tree search. ([openai.com](https://openai.com/index/learning-to-reason-with-llms/ "Learning to reason with LLMs | OpenAI")) - **Verifiable rewards drive improvement.** With rewards keyed to _correct final answers_ (and light formatting constraints), RL preferentially selects policies that generate more reliable traces. DeepSeek-R1-Zero, trained with GRPO and verifiable rewards, rises on AIME-2024 from 15.6% to 71.0% pass@1 (86.7% with 64-sample consensus), while learning to “think longer” over training. OpenAI’s o-series similarly scales with more train-time RL and more test-time “thinking”. Below I expand each point and connect them. --- ### 1) CoT “works” because reasoning is cast as continuation Modern chat LLMs are typically _decoder-only_ transformers: they generate one token at a time conditioned on prior context. CoT simply asks them to _continue with steps_, leveraging the [[statistical structure]] of stepwise prose seen in pre-training. In few-shot form, CoT led PaLM-540B to ~57–58% on GSM8K, up from ~18% with standard prompting; adding _self-consistency_ (sample multiple traces, take the majority answer) raised accuracy to ~74% on GSM8K. Zero-shot CoT (“Let’s [[think step]] by step”) provides further gains but is distinct from the few-shot result. > [!nice!] > CoT simply asks them to _continue with steps_, leveraging the [[statistical structure]] of stepwise prose seen in pre-training. Two clarifications help precision: - **Architecture scope.** While many frontier “chat” models (e.g., Llama 3) are decoder-only, the broader LLM family also includes encoder–decoder models such as T5. So it’s better to say “most current chat LLMs are decoder-only” than “LLMs are decoder-only”. ([ai.meta.com](https://ai.meta.com/blog/meta-llama-3-1/?utm_source=chatgpt.com "Introducing Llama 3.1: Our most capable models to date")) - **Prompting vs training.** CoT in Wei et al. (2022) is _few-shot prompting_; the dramatic GSM8K jump belongs there. The later zero-shot effect was shown by Kojima et al. (2022). Self-consistency is a _decoding_ strategy (Wang et al., 2022) layered on top, which pushed PaLM-540B to ~74% on GSM8K. --- ### 2) Reasoning models: long hidden CoTs and search-like behaviour (via RL) OpenAI’s o-series explicitly trains models to _think before answering_: they generate a hidden chain of thought, then summarise. OpenAI reports that o1’s performance improves with more RL (“train-time compute”) and with more time spent thinking at inference (“test-time compute”). In the o3-mini release, OpenAI exposes `reasoning_effort` (low/medium/high); at _high_ effort, o3-mini reaches **83.6%** on AIME-2024. ([openai.com](https://openai.com/index/learning-to-reason-with-llms/ "Learning to reason with LLMs | OpenAI")) DeepSeek-R1-Zero demonstrates the same pattern in an open setting. Using _pure_ RL (no SFT) with GRPO, it learns to produce long traces and displays _emergent_ behaviours—reflection, alternative exploration—_without_ hard-coding those behaviours. [[The authors]] explicitly describe a “self-evolution” process and show improved accuracy as training progresses. They also set very high maximum generation lengths (up to 32,768 tokens) and document that the model naturally increases “thinking time” (longer traces) during RL. This is where [[the analogy]] to search comes from: the text trace branches, revisits, and prunes in ways reminiscent of tree search or backtracking, even though the underlying mechanism is just language-model continuation steered by RL. _Why call it an analogy?_ Because no paper claims these models are literally running MCTS during inference; rather, the _observed_ behaviours look search-like when you read the traces. That framing keeps us honest about mechanisms while still describing what we see. --- ### 3) Rewards and “natural” evolution of reasoning traces The modern recipe is _RL with verifiable rewards (RLVR)_: use programmatic checks to score outcomes (string-match answers, unit tests, etc.), sometimes with a light _format_ constraint that enforces where the hidden reasoning should live. In DeepSeek-R1-Zero, the reward is two-part: (i) accuracy (verifiable correctness) and (ii) a small format component that requires the reasoning to appear between `<think>…</think>` tags. No neural reward model is used, specifically to avoid reward hacking. Under this signal, the policy shifts toward longer, self-correcting traces and higher accuracy—e.g., AIME-2024 pass@1 climbs from **15.6% to 71.0%**, and **86.7%** with 64-sample majority vote. Beyond DeepSeek, RLVR has been formalised in the open Tülu-3 line of work and follow-on analyses: train a policy to generate a CoT and an answer, pay only for verifiably _correct_ outcomes (plus any minimal structure), and skip subjective preference models. This framing helps explain why _longer, more structured_ traces get selected [[over time]]: they increase the chance of hitting a verifiably correct end state. ([arXiv](https://arxiv.org/html/2411.15124v3?utm_source=chatgpt.com "Tülu 3: Pushing Frontiers in Open Language Model Post- ...")) OpenAI reports the same compute-sensitivity on its side: o1 improves with more RL and with more time thinking; in its public system card, o1 achieves **74.4%** pass@1 and **83.3%** cons@64 on AIME-2024 (and o3-mini-high later posts **83.6%** at high effort). These are consistent with the “more thinking helps (up to a point)” pattern that RLVR tends to elicit. ([openai.com](https://openai.com/index/learning-to-reason-with-llms/ "Learning to reason with LLMs | OpenAI")) --- ### Pulling it together - CoT is not a separate “reasoning module”; it is the familiar continuation task, _steered_ toward steps. Few-shot CoT and self-consistency establish the baseline improvements. - Reasoning models _train_ this behaviour: they generate hidden, longer CoTs and learn search-like moves under RL—reflection, revising steps, exploring alternatives. The mechanism is token continuation; the _behaviour_ is search-like. ([openai.com](https://openai.com/index/learning-to-reason-with-llms/ "Learning to reason with LLMs | OpenAI")) - Verifiable rewards and (light) formatting constraints supply a clean signal, which naturally selects for longer, better-structured traces. That’s why accuracy scales with “effort” at inference and with more RL at training. For a clear narrative overview of how these pieces fit—standard CoT, RLVR, and long-trace behaviours—Cameron R. Wolfe’s “[[Demystifying Reasoning Models]]” is a helpful map. ([cameronrwolfe.[[substack]].com](https://cameronrwolfe.substack.com/p/demystifying-reasoning-models?utm_source=chatgpt.com "[[Demystifying Reasoning Models]] - by Cameron R. Wolfe, Ph.D.")) --- ## References (inline) - **Few-shot CoT (PaLM-540B)** and GSM8K 18% → 57–58%; **self-consistency** to ~74%: Wei et al. 2022; Wang et al. 2022. - **Zero-shot CoT** (“Let’s [[think step]] by step”): Kojima et al. 2022. ([arXiv](https://arxiv.org/abs/2205.11916?utm_source=chatgpt.com "Large Language Models are Zero-Shot Reasoners")) - **Decoder-only chat LLMs vs T5**: Llama-3 (decoder-only); T5 (encoder–decoder). ([ai.meta.com](https://ai.meta.com/blog/meta-llama-3-1/?utm_source=chatgpt.com "Introducing Llama 3.1: Our most capable models to date")) - **OpenAI o1: hidden CoT, scales with train/test compute**; **AIME-2024** (74.4 pass@1; 83.3 cons@64): OpenAI blog + system card. ([openai.com](https://openai.com/index/learning-to-reason-with-llms/ "Learning to reason with LLMs | OpenAI")) - **OpenAI o3-mini: `reasoning_effort` and AIME-2024 83.6% at “high”**: o3-mini release page. ([openai.com](https://openai.com/index/openai-o3-mini/ "OpenAI o3-mini | OpenAI")) - **DeepSeek-R1-Zero: pure RL (GRPO), verifiable rewards, `<think>` format reward, long traces, AIME-2024 15.6% → 71.0% (86.7% majority vote), increasing “thinking time”**: DeepSeek-R1 paper. - **RLVR framing in open literature**: Tülu-3 (AI2) and follow-ups. ([arXiv](https://arxiv.org/html/2411.15124v3?utm_source=chatgpt.com "Tülu 3: Pushing Frontiers in Open Language Model Post- ...")) --- _Note on claims about “tree search/backtracking”: these are intentional analogies to the _observed_ text-trace behaviours (branching, revisiting, pruning). No cited source claims that the models are running a formal MCTS algorithm at inference._ If you want, I can drop this straight into your doc with compact author–year inline citations (and a short bibliography) so it’s publication-ready.