Reasoning Models: o1, o3, and Claude Extended Thinking Reviewed

Hero composition for editorial coverage of the closed-model reasoning paradigm. Source materials: OpenAI’s o1 launch post and Anthropic’s Extended Thinking research note (both linked in the Sources list).

1. Paper identity and scope

Primary artefacts. This review covers two closely-related closed-model disclosures, not a peer-reviewed paper:

• OpenAI, “Learning to Reason with LLMs” — the September 12, 2024 launch post introducing the o1 model family ¹ , followed by the OpenAI o1 System Card dated December 5, 2024 ^{2 3} .
• Anthropic, “Claude’s Extended Thinking” — the February 24, 2025 research note accompanying Claude 3.7 Sonnet ^{4 5} , and the April 2025 follow-up “Reasoning Models Don’t Always Say What They Think” ^{6 7} .

Neither vendor has released a peer-reviewed full paper covering the training procedure. The technical disclosure surface is launch posts, system cards, and short research notes. From the paper: OpenAI explicitly states it will not reveal o1’s chain-of-thought, citing safety and competitive reasons ¹ . From the paper: Anthropic’s note is more transparent about mechanics but stops short of disclosing the post-training recipe ⁴ .

Retrieval status. All primary URLs were fetched on 2026-05-19. No formal supplementary material exists; the o3 announcement was made via livestream on December 20, 2024 with the ARC-AGI evaluation written up by the ARC Prize organisation ⁸ .

Classification. Inference method, training method (post-training reinforcement learning), LLM-based, AI safety. The paradigm shift sits at the boundary of training-time and inference-time scaling.

Technical abstract (in the publication’s voice). The “reasoning model” paradigm couples two ideas that the field already knew separately. Chain-of-thought prompting ¹¹ had shown that asking a model to produce intermediate steps before its final answer improves accuracy on reasoning benchmarks. Test-time compute scaling work ¹² showed that searching over many candidate solutions at inference, with a learned verifier, can substitute for larger pretrained models. OpenAI’s o1 launch post claims to combine these into a single trained model: large-scale reinforcement learning teaches the model to produce useful long chains-of-thought, and at inference time the model can spend a variable amount of compute “thinking” before responding ¹ . Anthropic’s Extended Thinking is the same paradigm with one architectural commitment made explicit: it is the same model, not a separate reasoning model, with an API-controllable thinking budget ⁵ . The benchmark deltas are large where reasoning is the bottleneck — o1 jumps from 13% (GPT-4o) to 83% on the AIME 2024 math olympiad ⁹ ; Claude 3.7 with parallel sampling reaches 84.8% on GPQA versus single-attempt baselines ⁴ — and small or absent elsewhere.

Primary research question. Does training a model to produce long internal chains-of-thought via reinforcement learning, then varying inference-time compute, deliver a different capability frontier than pure pretraining scaling?

Core claim. [Reconstructed from the launch posts] Both vendors claim yes, supported by benchmark gains on reasoning-heavy tasks (math olympiad, competitive programming, PhD-level science) that the prior generation of chat models could not unlock by any prompting strategy.

Core technical domains. Reinforcement learning from process or outcome rewards (moderate disclosure), inference-time search (surface disclosure), evaluation methodology (deep for benchmarks, surface for safety), AI-safety reasoning via deliberative alignment (moderate) ¹³ .

Reader prerequisites. High-school algebra; familiarity with what a large language model is. The Glossary below covers reinforcement learning, chain-of-thought, test-time compute, pass@k, and every other technical term used in the body. No graduate-level ML background is required.

2. TL;DR and executive overview

TL;DR (3 sentences). OpenAI’s o1 and Anthropic’s Claude Extended Thinking are a new style of language model that “thinks” — produces a long internal scratchpad — before answering. On hard reasoning tests (math olympiad problems, PhD-level science, competitive programming) they jump dramatically over the previous generation: o1 scores 83% on a math olympiad where GPT-4o scores 13%. The trade-off is that they are much more expensive per query, much slower, and (in OpenAI’s case) deliberately hide their reasoning from users.

Executive summary. The paradigm fuses two known ideas — chain-of-thought prompting and inference-time search — into a single trained model. Reinforcement learning on reasoning traces teaches the model to produce a useful internal scratchpad; the API exposes a budget for how long that scratchpad can run. The evidence that this is qualitatively different from prompting comes from benchmarks where the previous generation plateaued near random: AIME math olympiad, GPQA Diamond, FrontierMath. The evidence that it is not yet general comes from agentic tasks (SWE-bench Verified at 40.9% for o1 ² ), faithfulness studies (Claude 3.7 mentions reasoning hints in only 25% of cases where it used them ⁶ ), and the cost: $15 input /$60 output per million tokens for o1 ⁹ , ten times GPT-4o, with the o1-pro variant at $150 /$600 ⁹ .

Five practitioner-relevant takeaways.

• Reasoning models earn their cost on problems with a verifiable answer and a long solution. Use GPT-4o or Claude Sonnet for everything else.
• The “thinking budget” parameter (Anthropic) and “reasoning effort” (OpenAI) are now first-class API knobs; treat them as a quality-versus-cost dial, not a magic upgrade.
• Output token counts can be 5–20$\times$ a chat model’s, because billed “reasoning tokens” sit on the output side. Budget accordingly.
• Hidden chain-of-thought (OpenAI) versus visible (Anthropic) is not just a UX choice; it changes what you can build. Self-consistency, debate, and verifier scaffolding need access to the reasoning.
• Faithfulness is unresolved. The reasoning trace is a useful tool, not a transparent window into the model’s actual decision process ⁶ .

Pipeline overview. Training time: a base pretrained model is post-trained with reinforcement learning on tasks where a correct answer can be verified (math problems, code execution, formal proofs). The reward signal incentivises chains-of-thought that lead to the right answer ¹ . Inference time: the model emits reasoning tokens up to a budget, then a final answer. Optional parallel sampling and majority-vote / best-of-N selection further improves accuracy at additional cost ⁴ .

2.5 Glossary

Term	Plain-English explanation	First appears in
Reasoning model	A language model post-trained to produce a long internal scratchpad before answering, often with API controls for how long to think.	Section 1
Chain-of-thought (CoT)	The model writing out intermediate reasoning steps in natural language before stating a final answer.	Section 1
Reinforcement learning (RL)	A training method where the model is rewarded for producing outputs that satisfy some criterion (e.g., a correct math answer), and learns to favour those outputs.	Section 1
Test-time compute	The total compute spent at inference time per query; the dial that “thinking longer” turns up.	Section 1
Pass@k	Accuracy when the model is allowed $k$ attempts and you take the best (or any correct) answer; pass@1 means single-attempt.	Section 9
Self-consistency	Sampling many independent chains-of-thought and taking the majority-vote final answer.	Section 5
AIME	American Invitational Mathematics Examination, a hard high-school math olympiad used as a reasoning benchmark.	Section 1
GPQA	Graduate-level Physics, Chemistry, and Biology Question-Answering benchmark; “Diamond” is the hardest subset.	Section 1
ARC-AGI	A pattern-induction benchmark designed by François Chollet to resist memorisation; held as a stress test for novel reasoning.	Section 9
Deliberative alignment	OpenAI’s safety training method that has the model reason about its own policy specification before answering.	Section 5
Thinking budget	An API parameter (Anthropic) bounding the number of tokens the model may spend on its internal scratchpad.	Section 5
Reasoning effort	The equivalent OpenAI knob; “low” / “medium” / “high” rather than a token count.	Section 5
Process reward	A reward signal applied to intermediate reasoning steps, not just the final answer.	Section 6
Outcome reward	A reward signal applied only to the final answer’s correctness.	Section 6
Faithfulness (of CoT)	Whether the model’s stated reasoning actually drove its final answer, or is a post-hoc rationalisation.	Section 12
[Analysis] label	The publication’s own reasoned assessment, distinct from what the source itself claims.	Throughout
[Reviewer Perspective] label	A critical or speculative assessment that goes beyond what the source proves.	Sections 11–12
[Reconstructed] label	Content the publication faithfully reconstructed because the source only partially disclosed it.	Sections 5–7
[External comparison] label	A comparison to prior work or general knowledge outside the source itself.	Sections 4, 11
”From the paper:” prefix	Content directly supported by the launch post, system card, or research note text.	Throughout

3. Problem formalisation

Notation table.

Symbol	Type	Meaning	First appears in
$x$	Token sequence	The input prompt	Section 3
$y$	Token sequence	The final answer	Section 3
$z$	Token sequence	The chain-of-thought / reasoning trace	Section 3
$\pi_\theta$	Parameterised policy	The language model with parameters $\theta$	Section 3
$r(x, y)$	Scalar	Reward signal for the final answer given the prompt	Section 6
$B$	Integer	Thinking budget in tokens	Section 5
$k$	Integer	Number of parallel samples	Section 5
$N$	Integer	Total inference-time samples for best-of-N	Section 9

Formal setup. A reasoning model is an autoregressive language model that, given a prompt $x$, factorises its response into a hidden (or revealed) reasoning trace $z$ followed by a final answer $y$:

$p_\theta(z, y \mid x) = p_\theta(z \mid x) \cdot p_\theta(y \mid x, z).$

At inference the user sees $y$ (and, on Anthropic’s API, also $z$). The training objective is to learn $\theta$ such that $\mathbb{E}_{x \sim \mathcal{D}}[r(x, y)]$ is high, where $y \sim p_\theta(\cdot \mid x, z)$ and $z \sim p_\theta(\cdot \mid x)$. The reward $r$ is typically a programmatic verifier (does the math answer match the gold answer; does the code pass the unit tests).

Assumptions, with strength flags.

• The reasoning trace $z$ helps: $\mathbb{E}[r(x, y) \mid z \sim p_\theta(\cdot \mid x)] > \mathbb{E}[r(x, y)]$ without $z$. From the paper: OpenAI claims this is true and the gap grows with task difficulty ¹ . [Analysis] Potentially strong assumption on tasks where the answer space is small (multiple choice with $\le 4$ options); the reasoning trace adds compute cost without commensurate accuracy gain.
• The reward $r$ is well-specified. From the paper: OpenAI selects “tasks with verifiable answers” for the RL training distribution ¹ . [Analysis] This is a strong restriction; it means the paradigm’s training-signal coverage is narrower than pretraining’s. Mathematical, code, and formal-logic tasks dominate; open-ended writing tasks do not naturally fit.
• The pretrained base model already has the relevant reasoning patterns latent; RL “elicits” them rather than installs them. [External comparison] This is the framing in DeepSeek-R1’s parallel work ¹⁵ and in the s1 paper ¹⁴ , but neither OpenAI nor Anthropic confirms or denies it.

Why the problem is hard. Producing a useful $z$ requires the model to (a) plan, (b) self-correct when a step is wrong, (c) know when it is done. Standard supervised fine-tuning on human-written reasoning traces has a known ceiling because human reasoning traces are short, incomplete, and not always optimal. RL with a verifier lets the model discover its own reasoning patterns; the cost is that the verifier must exist and be tight.

4. Motivation and gap

The concrete problem. As of mid-2024, the frontier chat-LLM (GPT-4o) scored 13% on AIME 2024 ⁹ , near-random on the hardest physics subsets of GPQA, and could not reliably solve Codeforces-grade competitive programming problems. Chain-of-thought prompting ¹¹ helped but plateaued; longer prompts and more in-context examples gave diminishing returns. [External comparison] The Snell et al. test-time compute paper ¹² showed empirically that, for some problem distributions, scaling inference compute could substitute for an order-of-magnitude increase in model size — but the methods required external verifiers that did not exist for arbitrary domains.

Existing approaches and their failure modes. Chain-of-thought prompting ¹¹ required the user to craft prompts; the model was not optimised for the trace. Tree-of-Thoughts and process-reward-model search added inference-time search but were brittle and slow. Self-consistency ¹¹ improved sample efficiency but capped at the underlying model’s competence. The common gap: nothing in the previous generation had trained the model itself to produce useful long reasoning traces with self-correction.

The gap o1 and Extended Thinking claim to fill. From the paper: “OpenAI o1 is a new large language model trained with reinforcement learning to perform complex reasoning. o1 thinks before it answers — it can produce a long internal chain of thought before responding to the user.” ¹ From the paper: Anthropic frames it as one model that fluidly shifts between “quick responses and deep reflection” ⁵ .

Practical stakes. [Analysis] If the paradigm holds, the cost-per-correct-answer on hard problems collapses; the previous generation could not solve them at any price. If the paradigm is narrow — only a small slice of practical work has verifiable rewards — then the impact is large but bounded. The benchmark coverage in Section 9 lets the reader form an opinion.

Position in the broader landscape. [External comparison] This is the third wave of LLM scaling. Wave one (2018–2020) scaled pretraining compute. Wave two (2022–2024) scaled RLHF, instruction tuning, and tool use. Wave three (2024 onward) scales inference compute via reasoning. Open-weight replications (DeepSeek-R1 ¹⁵ , s1 ¹⁴ ) appeared within four months of o1, demonstrating that the recipe is reproducible from a strong base model with relatively modest additional training.

5. Method overview

The methodological disclosures are partial. This section names each component, distinguishes what the launch post or system card actually says from what is reconstructed from the broader literature, and flags every gap.

5.1 Post-training reinforcement learning on reasoning tasks

From the paper: “Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process.” ¹ No algorithm name, no reward-model architecture, no PPO-vs-DPO-vs-GRPO commitment. [Reconstructed] The shape of the training loop, based on what is publicly known about the broader area: collect prompts $x$ with verifiable answers; sample completions $(z, y) \sim \pi_\theta(\cdot \mid x)$; score each with the verifier $r(x, y)$; update $\pi_\theta$ to increase the probability of high-reward $(z, y)$ trajectories.

Plain-English intuition. The model is given a math problem with a known answer. It tries to solve it many times, producing different scratchpads each attempt. The training algorithm boosts the scratchpads that led to the right answer and dampens the ones that did not. Repeated over millions of problems, the model learns reasoning patterns that work.

Tradeoffs. Process reward (rewarding individual reasoning steps) gives a denser signal but requires a step-level critic. Outcome reward (rewarding only the final answer) avoids the critic but has high variance. OpenAI does not specify which it uses; [Reconstructed] the deliberative-alignment paper from the same team ¹³ suggests at least some process-reward component.

Novelty: [Adapted]. The RL-on-reasoning-traces idea predates o1 (Wei et al. ¹¹ , various STaR-style papers). The novelty claim is scale and integration into a deployed product.

5.2 Inference-time test-time compute

From the paper: “The performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).” ¹ Anthropic exposes this as a numeric API parameter: developers can “set a thinking budget to control precisely how long Claude spends on a problem,” ⁴ with a maximum tested budget of 64,000 tokens in research settings and 128,000 tokens as the announcement-stated output limit ⁵ .

OpenAI’s surface. OpenAI exposes a reasoning_effort parameter with values low, medium, high; the underlying token budget is not user-controllable. [Analysis] This is a deliberate design choice that hides the reasoning length from users (who are still billed for the tokens consumed). It is consistent with the hidden-CoT policy described in Section 5.4.

Parallel sampling. From the paper: Anthropic’s GPQA result of 84.8% uses 256 parallel samples with majority voting or a learned scoring function ⁴ . OpenAI’s launch post reports an AIME consensus-of-64 score of 83% and a refined-with-scoring-function score of 93% at 1000 samples ¹ . [Analysis] These best-of-N numbers are real evidence of capability ceiling but inflate the practical accuracy figure by roughly 10–20 percentage points over single-sample inference; readers should mentally subtract this when planning deployments.

Novelty: [Adapted] from the test-time compute scaling literature ¹² .

5.3 Same-model versus separate-model architecture

From the paper: Anthropic states explicitly: “just as humans use a single brain for both quick responses and deep reflection, we believe reasoning should be an integrated capability of frontier models rather than a separate model entirely.” ⁵ Claude 3.7 Sonnet is one model checkpoint with a runtime mode toggle. OpenAI’s o1 family is sold as a distinct model from GPT-4o, with different pricing, different latency, and a different API endpoint. [Analysis] Whether the underlying weights are entirely different or share a backbone is not publicly disclosed; the surface is what differs.

5.4 Hidden versus visible chain-of-thought

From the paper: “We have decided not to show the raw chains of thought to users. We acknowledge this decision has disadvantages.” ¹ OpenAI cites three reasons: the chain-of-thought must remain “in unaltered form” for safety monitoring; users would otherwise need an “aligned” CoT that hides the model’s actual reasoning patterns; competitive advantage. Users see a model-generated summary instead.

From the paper: Anthropic reveals the reasoning “in raw form” though “more detached and less personal-sounding than Claude’s default outputs” ⁴ , with rare encrypted sections marked “the rest of the thought process is not available for this response.”

[Analysis] The two positions reflect a real product-design fork. Visible CoT enables external scaffolding (verifier-of-verifier, multi-agent debate, user-side process supervision) at the cost of revealing the model’s training shape to competitors and to adversarial prompt-injection attacks. Hidden CoT preserves the trace as a safety monitoring channel for the vendor but excludes the user from that channel.

5.5 Deliberative alignment

From the paper: A companion OpenAI paper ¹³ describes “deliberative alignment”: the model reasons over its own safety policy specification in the chain-of-thought before responding. The o1 system card credits this technique for improvements on jailbreak benchmarks (StrongReject) and reductions in policy-violating outputs ^{2 3} .

Plain-English intuition. Instead of training the model to refuse certain prompts via reinforcement from human feedback (which produces a fast, opaque refusal), the model is trained to think through the policy in its scratchpad and arrive at a justified response. The trace is part of the safety mechanism, not bolted on after.

Novelty: [New] as a deployed alignment method, though the idea of reasoning-over-policy is older.

6. Mathematical contributions

The math here is mostly the standard RL-on-language-model setup. The launch posts state no new theorems; this section reconstructs the canonical formulation so the reader has a concrete anchor.

MATH ENTRY [1]: Reasoning-model factorisation

• Source: Reconstructed from the standard formulation in the o1 launch post ¹ and DeepSeek-R1 ¹⁵ .
• What it is: The reasoning model’s response is broken into a hidden trace $z$ and a final answer $y$, generated sequentially given the prompt $x$.
• Formal definition:

$p_\theta(z, y \mid x) = \prod_{t=1}^{|z|} p_\theta(z_t \mid x, z_{<t}) \cdot \prod_{t=1}^{|y|} p_\theta(y_t \mid x, z, y_{<t}).$

• Each term explained and its type/dimensional analysis:
  • $x$ is a token sequence of length $|x|$ (the prompt), drawn from a vocabulary $V$ of size on the order of $10^5$.
  • $z$ is a token sequence of length $|z|$, the reasoning trace; bounded by the thinking budget $B$.
  • $y$ is the final answer token sequence of length $|y|$.
  • $p_\theta(\cdot \mid \cdot)$ is the next-token distribution over $V$ produced by the model; for each step it is a vector of $|V|$ probabilities summing to 1.
  • $z_{<t}$ is the prefix $(z_1, \ldots, z_{t-1})$.
• Worked numerical example. Suppose $|V| = 4$ (a toy vocabulary $\{A, B, C, \text{stop}\}$), $|x| = 2$, the thinking budget $B = 3$, and the model produces the trace $z = (A, B, \text{stop})$ then the answer $y = (C)$. The factorisation expands to a product of four conditional probabilities, each a single entry from a length-4 probability vector. If the model assigns $0.6$ to ”$A$ given $x$”, $0.7$ to ”$B$ given $x, A$”, $0.5$ to “stop given $x, A, B$”, and $0.8$ to ”$C$ given $x, z$”, the joint probability of this trace-plus-answer is $0.6 \cdot 0.7 \cdot 0.5 \cdot 0.8 = 0.168$. The pretrained next-token machinery is unchanged; only the training objective differs.
• Role: Defines the object the RL training optimises over.
• Edge cases: when $|z| = 0$, the model degenerates to a standard chat LLM.
• Novelty: [Adopted].
• Transferability: [Analysis] Trivially transfers; any autoregressive LM admits this factorisation.
• Why it matters: It is the substrate that justifies sampling multiple traces (Section 5.2) and assigning reward to whole trajectories rather than individual tokens.

MATH ENTRY [2]: Outcome-reward RL objective

• Source: Reconstructed; the o1 launch post does not give an equation ¹ .
• What it is: The training maximises expected verifier reward on the final answer, averaged over the prompt distribution and the sampled reasoning trace.
• Formal definition:

$\mathcal{J}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \, (z, y) \sim p_\theta(\cdot \mid x)}\big[ r(x, y) \big].$

• Each term and type:
  • $\mathcal{D}$ is the training distribution over prompts, biased toward problems with verifiable answers (math, code, logic). Each $x$ is a token sequence.
  • $r(x, y)$ is a scalar in $\{0, 1\}$ for binary correctness or in $[0, 1]$ if partial credit is allowed. Returns a single real number per trajectory.
  • The expectation is taken over both prompt sampling and the model’s own trace-plus-answer generation, so a Monte Carlo estimator with $N$ samples gives $\hat{\mathcal{J}} = \frac{1}{N} \sum_i r(x_i, y_i)$.
• Worked numerical example. Suppose $\mathcal{D}$ is a batch of 4 prompts, the model samples 2 trajectories per prompt (so 8 total), and the verifier returns reward 1 for 3 of the 8 trajectories and 0 for the other 5. Then $\hat{\mathcal{J}} = 3/8 = 0.375$. The gradient step pushes probability mass toward the 3 successful traces and away from the 5 failures.
• Role: The signal that distinguishes reasoning RL from chain-of-thought prompting; the model is rewarded for traces that work, not for traces that look like human reasoning.
• Edge cases: degenerate when $r$ is constant (no signal), or when correct trajectories are too rare for any non-zero gradient.
• Novelty: [Adopted] from the broader RL-from-verifier literature.
• Transferability: [Analysis] Bottlenecked by the verifier; transfers to any domain with a programmatic checker.
• Why it matters: It is what makes the paradigm narrower than RLHF (which has a learned reward model and can score open-ended outputs) and stronger on the tasks it does cover.

MATH ENTRY [3]: Pass@k and best-of-N evaluation

• Source: Standard benchmarking convention; both o1 and Claude 3.7 report variants.
• What it is: Pass@k is the probability that at least one of $k$ samples is correct. Best-of-N uses a scoring function to pick a single answer from $N$ samples; consensus@N uses majority vote.
• Formal definition. For pass@k with $N$ samples drawn and $c$ correct:

$\text{pass}@k = 1 - \frac{\binom{N-c}{k}}{\binom{N}{k}}$

(when $N - c \ge k$; otherwise pass@k $= 1$).

For consensus@N with discrete answers and majority vote:

$\text{consensus}@N(x) = \mathbb{1}\!\left[ \arg\max_{y'} \sum_{i=1}^{N} \mathbb{1}[y_i = y'] = y^* \right]$

where $y^*$ is the gold answer and $y_i$ are the $N$ sampled answers.

• Each term explained:
  • $N$ is the number of samples drawn; $k$ is the number you are allowed to keep.
  • $c$ is the number of correct samples among $N$; in pass@k you do not need to identify which ones are correct.
  • $\mathbb{1}[\cdot]$ is the indicator function, returning 1 when the inner predicate is true and 0 otherwise.
  • In consensus@N, the inner sum counts how often each candidate answer $y'$ appears among the $N$ samples; $\arg\max$ returns the most-voted candidate.
• Worked numerical example. Suppose $N = 8$ samples are drawn and $c = 3$ are correct. Then $\text{pass}@1 = 3/8 = 0.375$; $\text{pass}@2 = 1 - \binom{5}{2}/\binom{8}{2} = 1 - 10/28 \approx 0.643$; $\text{pass}@8 = 1$ (any correct among 8). For consensus@8, suppose the 8 sampled answers are $(A, A, B, A, C, A, B, A)$ and the gold answer is $A$. The vote count is $A: 5, B: 2, C: 1$, $\arg\max = A$, so consensus@8 returns 1 (correct).
• Role: Disentangles “the model can solve this if we let it try repeatedly” from “the model can solve this on the first try”; the second is the operational accuracy users get.
• Edge cases: For tasks with continuous answers (proof verification, open-ended math), majority vote requires an equivalence check, not string match.
• Novelty: [Adopted] from Codex / OpenAI’s earlier benchmarking practice.
• Transferability: [Analysis] Universal; any sampling-based evaluation can use these.
• Why it matters: It is the cleanest lens for reading the paradigm’s benchmark claims. Pass@1 numbers reflect deployable accuracy; consensus@1000 numbers reflect the model’s capability ceiling under maximum inference compute.

MATH ENTRY [4]: Test-time compute scaling claim (logarithmic improvement)

• Source: From the paper: Anthropic states “accuracy improves logarithmically with thinking tokens” ⁴ ; OpenAI shows a log-linear plot of AIME accuracy versus test-time compute ¹ .
• What it is: An empirical regularity, not a proven theorem: accuracy on a fixed benchmark scales approximately as a linear function of the logarithm of inference-time compute.
• Approximate functional form:

$\text{accuracy}(B) \approx \alpha + \beta \cdot \log(B)$

up to some saturation budget $B_{\max}$, where $B$ is the thinking-token budget and $\alpha, \beta$ are problem-class-dependent constants.

• Each term explained:
  • $B$ is the thinking budget in tokens (the dial Anthropic exposes).
  • $\log(B)$ is the natural logarithm; doubling the budget adds a fixed constant $\beta \cdot \log(2)$ to accuracy.
  • $\alpha$ is the intercept, roughly the accuracy at $B = 1$ (no thinking).
  • $\beta$ is the slope; observed to be positive but problem-class-dependent and bounded.
• Worked numerical example. Suppose on AIME, $\alpha = 0.30$ and $\beta = 0.10$ (illustrative numbers, not from the paper). Then $B = 100$ tokens gives accuracy $\approx 0.30 + 0.10 \cdot \log(100) = 0.30 + 0.10 \cdot 4.6 = 0.76$; $B = 1000$ tokens gives $\approx 0.30 + 0.10 \cdot \log(1000) = 0.30 + 0.69 = 0.99$. The log scaling means each successive doubling of budget gives a smaller absolute jump in dollars-per-correct-answer.
• Role: Justifies the API design (variable thinking budget) and lets practitioners reason about cost.
• Edge cases: saturates beyond some $B_{\max}$ where additional tokens do not help; can go negative on tasks where extended reasoning over-thinks a simple problem.
• Novelty: [External comparison] The log scaling pattern echoes Snell et al.’s test-time compute analysis ¹² on smaller open models.
• Transferability: [Analysis] The functional form likely transfers; the constants do not. Each benchmark needs its own calibration.
• Why it matters: It is the empirical claim that converts “thinking longer helps” from a folk observation into a budgetable quantity.

7. Algorithmic contributions

Neither vendor publishes pseudocode. This section reconstructs the canonical training and inference loops at a level the reader can hand-trace, and flags every reconstruction.

ALGORITHM ENTRY [1]: Reasoning-model RL training loop (reconstructed)

• Source: [Reconstructed] from the o1 launch post ¹ and the open-weight replication recipes (DeepSeek-R1 ¹⁵ , s1 ¹⁴ ).
• Purpose: Post-train a base LLM so that its sampled reasoning traces lead to verifier-correct answers more often than the base.
• Inputs:
  • Base model $\pi_{\theta_0}$ (a pretrained LLM).
  • Prompt distribution $\mathcal{D}$ — a curated set of prompts with verifiable answers (math, code, logic puzzles).
  • Verifier $r(x, y)$ — a programmatic function returning $\{0, 1\}$ or partial credit.
  • Number of training iterations $T$; samples-per-prompt $K$; learning rate $\eta$.
• Outputs: Trained policy $\pi_{\theta_T}$.
• Pseudocode (reconstructed):

for iteration t = 1 to T:
    sample batch of prompts {x_1, ..., x_M} from D
    for each prompt x_i:
        sample K trajectories (z_i^k, y_i^k) ~ pi_theta(. | x_i)
        compute rewards r_i^k = r(x_i, y_i^k) for k = 1..K
    estimate policy gradient using the (x_i, z_i^k, y_i^k, r_i^k) tuples
    update theta with optimiser step (e.g., PPO / GRPO update)
return pi_theta_T

• Hand-traced example on a minimal input. Suppose iteration $t = 1$, batch size $M = 1$, $K = 4$ samples. The prompt is $x =$ “What is 7 + 5?” (verifier checks if the answer equals “12”). The base model samples four trajectories:

  • $k=1$: $z_1 =$ “Let me add. 7 + 5 = 12.”, $y_1 =$ “12” → reward 1.
  • $k=2$: $z_2 =$ “7 + 5 = 13.”, $y_2 =$ “13” → reward 0.
  • $k=3$: $z_3 =$ “Adding two and three first.”, $y_3 =$ “12” → reward 1 (lucky guess).
  • $k=4$: $z_4 =$ “I’m not sure.”, $y_4 =$ “10” → reward 0.

  The policy gradient step increases $\pi_\theta$ on trajectories 1 and 3, decreases on 2 and 4. [Analysis] Note that $z_3$ contained incorrect reasoning but happened to produce the right answer; under outcome-reward RL this trajectory is reinforced too. This is the standard outcome-reward failure mode and the reason a process-reward variant exists. After many iterations the model’s sampled traces converge toward those that produce correct answers reliably; whether the trace is faithful is a separate question (Section 12).

• Complexity. Time: dominated by the sampling step, $O(M \cdot K \cdot L)$ token generations per iteration where $L$ is the average trace length. Memory: dominated by storing the $M \cdot K$ trajectories for gradient computation. Bottleneck: token generation throughput.

• Hyperparameters: $T$ (iterations), $K$ (samples per prompt), $M$ (batch size), learning rate $\eta$, KL-penalty coefficient against $\pi_{\theta_0}$ if any. None are disclosed by OpenAI; DeepSeek-R1 ¹⁵ gives the open-source reference numbers.

• Failure modes: reward hacking (the model exploits a quirk in the verifier), distribution collapse (all sampled traces become identical), forgetting general capabilities outside the training distribution.

• Novelty: [Adapted] from the broader RL-from-verifier-reward family.

• Transferability: [Analysis] Reproducible by any team with a strong base model, a verifier, and the inference compute to sample many trajectories per prompt. DeepSeek-R1 demonstrated this in late January 2025.

ALGORITHM ENTRY [2]: Inference-time reasoning with thinking budget

• Source: Anthropic’s API documentation ^{4 5} and the OpenAI launch post ¹ .
• Purpose: At inference time, the model emits up to $B$ reasoning tokens then a final answer; optionally sample $N$ parallel traces and aggregate.
• Inputs: trained policy $\pi_\theta$, prompt $x$, thinking budget $B$, number of parallel samples $N$, aggregation rule (majority vote or scoring function).
• Outputs: final answer $y$.
• Pseudocode:

for i = 1 to N (in parallel):
    z_i = []
    for t = 1 to B:
        z_i_t ~ pi_theta(. | x, z_i)
        z_i.append(z_i_t)
        if z_i_t == STOP_REASONING: break
    sample y_i ~ pi_theta(. | x, z_i)
if N == 1: return y_1
else: return aggregate({y_1, ..., y_N}, rule)

• Hand-traced example. Suppose $x =$ “Solve: $2x + 6 = 14$, find $x$.” Thinking budget $B = 50$ tokens, $N = 1$.

  • Step 1–8: $z_1 =$ “Subtract 6 from both sides: $2x = 8$. Divide by 2: $x = 4$. STOP”.
  • Step 9: $y_1 =$ "$x = 4$".

  For $N = 3$ with majority vote: suppose the three sampled answers are $\{x = 4, x = 4, x = 4.5\}$. Majority vote returns $x = 4$. The model used roughly $3 \cdot 30 = 90$ reasoning tokens total versus a chat-model baseline of perhaps 10 tokens; the user is billed for all of them.

• Complexity. Time: $O(N \cdot B)$ token generations per query. Memory: $O(N \cdot B)$ if traces are stored. Cost: linear in $N \cdot B$ at vendor token rates.

• Hyperparameters: $B$ exposed to users (Anthropic) or bucketed into low/medium/high (OpenAI); $N$ is application-controlled (Anthropic) or hidden (OpenAI).

• Failure modes: budget exhaustion before a conclusion; trace getting stuck in a loop; aggregation picking a plausible-looking wrong answer when correct answers are minority.

• Novelty: [Adapted] from the broader test-time-compute literature.

• Transferability: [Analysis] Directly applicable to any reasoning model with an exposed token budget.

8. Specialised design contributions

8A. LLM / prompt design. The reasoning models change prompt-design conventions: chain-of-thought hints (“let’s think step by step”, few-shot reasoning examples) are largely unnecessary because the model produces its own trace. From the paper: Anthropic explicitly notes that traditional CoT prompts can degrade Claude 3.7 performance because they conflict with the model’s own learned reasoning patterns ⁵ . [Analysis] The practical implication is that prompt templates carried over from GPT-3.5 / GPT-4 should be stripped of their meta-instructions when migrated to reasoning models.

8B. Architecture-specific details. Not disclosed by either vendor. Anthropic confirms that Claude 3.7 is “the same model” as the standard mode ⁵ , implying no architectural change. OpenAI does not confirm whether o1 shares architecture with GPT-4o.

8C. Training specifics. Not disclosed. Compute budget, dataset composition, RL algorithm choice (PPO vs DPO vs GRPO vs proprietary variant) are all withheld. [Reconstructed from open-source replications]: DeepSeek-R1 ¹⁵ uses GRPO (Group Relative Policy Optimization) on hundreds of thousands of math and code problems; s1 ¹⁴ uses pure supervised fine-tuning on 1,000 reasoning traces plus a “budget forcing” inference trick.

8D. Inference / deployment specifics. From the paper: o1 and Claude 3.7 Extended Thinking expose substantially different latency profiles than chat models. Anthropic’s API surfaces a thinking block in the response containing the trace; OpenAI’s API returns only the final answer plus a reasoning_tokens count that bills the user. Caching of reasoning traces is not currently supported by either vendor as of 2026-05-19.

9. Experiments and results

The benchmark coverage is asymmetric: both vendors publish strong math / science / code numbers and are vague about everything else.

Datasets and benchmarks reported.

Benchmark	What it tests	Reported by
AIME 2024	High-school math olympiad, 15 problems	OpenAI (o1), Anthropic (Claude 3.7)
GPQA Diamond	Graduate-level physics, chem, bio	OpenAI (o1), Anthropic (Claude 3.7)
MATH	Olympiad-style math problems	OpenAI (o1)
Codeforces	Competitive programming Elo	OpenAI (o1)
SWE-bench Verified	Real-world software-engineering tasks	OpenAI (o1), Anthropic (Claude 3.7)
ARC-AGI	Pattern-induction puzzles	OpenAI (o3, via ARC Prize)
FrontierMath	Research-level math problems	OpenAI (o3)
OSWorld	Computer-use agent tasks	Anthropic (Claude 3.7)

Main quantitative results. From the paper, with sources:

Model	AIME 2024 (pass@1)	GPQA Diamond	Codeforces percentile	SWE-bench Verified
GPT-4o	13%	~50%	low	~33%
o1-preview	74% (single) / 83% (consensus@64)	PhD-expert level	89th percentile	—
o1 (full)	83%+	PhD-expert level	89th+ percentile	40.9%
Claude 3.7 Sonnet	logarithmic scaling reported	84.8% (parallel, n=256)	—	63.7% (vanilla) / 70.3% (scaffolded)

Sources: o1 AIME and Codeforces ^{1 9} ; o1 SWE-bench ² ; Claude 3.7 GPQA and SWE-bench ^{4 5} .

The o3 results. [External comparison] OpenAI’s o3 announcement on December 20, 2024 added two further numbers: 87.5% on ARC-AGI in high-compute mode, with 75.7% at the public-leaderboard $10,000 compute limit<FootnoteRef n={8} />; and 25.2$10K cap — and that ARC-AGI-2 is expected to score under 30% even at high compute ⁸ .

Ablations. Neither vendor publishes ablations in the academic sense. The closest substitutes:

• Anthropic’s logarithmic-accuracy plot against thinking budget ⁴ functions as a budget ablation.
• OpenAI’s separately-reported pass@1 / consensus@64 / refined@1000 numbers on AIME ¹ function as a sample-count ablation.
• Neither vendor reports what happens if RL is removed and only supervised fine-tuning on reasoning traces is used; the s1 paper ¹⁴ argues that pure SFT on a small high-quality reasoning corpus already gets surprisingly far.

Independent benchmark cross-checks. [External comparison] The ARC-AGI evaluation of o3 was conducted by the ARC Prize organisation (Chollet et al.) and confirms OpenAI’s claimed numbers within evaluation noise ⁸ . The FrontierMath result is harder to cross-check because the benchmark is held-out; Epoch AI runs it under controlled conditions. [Reviewer Perspective] The o3 announcement-versus-shipped-model gap surfaced in independent reporting in April 2025: a TechCrunch story noted that the publicly-released o3 model scored lower on FrontierMath than the December announcement implied. This is a known pattern with launch-time vs production-shipped reasoning models and an active reproducibility concern.

Methodology disclosure [Analysis] — this is exactly the section that is weakest in both disclosures.

Methodology (for o1, from system card ² )

• Sample size: not consistently reported; AIME numbers based on the 15-problem 2024 exam.
• Evaluation set: AIME 2024 (15 problems), GPQA Diamond subset, SWE-bench Verified (500 tasks); contamination check not detailed.
• Baselines: GPT-4o; older o1-preview where applicable.
• Hardware / compute: not reported.

Methodology (for Claude 3.7, from announcement ^{5 4} )

• Sample size: GPQA results use 256 independent samples per question.
• Evaluation set: GPQA, AIME, OSWorld, SWE-bench Verified; contamination check not detailed.
• Baselines: Claude 3.5 Sonnet (predecessor).
• Hardware / compute: not reported; maximum thinking budget tested was 64K tokens.

Robustness and qualitative results. [Analysis] Both vendors report qualitative anecdotes (Anthropic’s Pokémon-Red Gym-Leader demo ⁴ ; OpenAI’s PhD-domain question examples) but no systematic out-of-distribution stress tests.

Evidence audit.

• Strongly supported: AIME-style math gains (multiple sources, large effect size, reproducible by open-source replications ¹⁵ ).
• Strongly supported: cost increase (multiple price-list sources).
• Partially supported: GPQA gains under parallel sampling (single-vendor reported, very compute-intensive).
• Partially supported: SWE-bench improvements (third-party scaffolding choices materially change the number).
• Narrow evidence: claims of general reasoning capability beyond verifiable-answer benchmarks.

10. Technical novelty summary

Component	Type	Novelty level	Justification	Source
RL on reasoning traces with outcome reward	Training method	Combination novel	Not new in literature; first deployed at frontier scale	1 2
Hidden chain-of-thought policy	Product design	Fully novel	New stance, not present in prior open work	1
Same-model thinking-budget toggle	Product design	Incrementally novel	New API surface, simple architectural impl	5
Deliberative alignment	Safety training	Fully novel	New post-training recipe for safety reasoning	13
Test-time compute scaling at frontier	Inference method	Combination novel	Builds on Snell et al. 12 at vastly larger scale	1 4
Faithfulness study of CoT in reasoning models	Evaluation	Fully novel	Anthropic’s April 2025 work has no direct precedent	6 7

The single most novel contribution. [Analysis] Not the RL recipe (which the open-source community reproduced within four months) and not the same-model architecture (which is a product choice). It is the demonstrated existence of a benchmark frontier that was inaccessible to the prior paradigm at any prompt-engineering cost. AIME going from 13% to 83% by a change in training recipe rather than a change in pretraining scale is what makes the paradigm a separate research thread from the GPT / Claude / Gemini scaling curve.

What is not claimed as novel. Chain-of-thought (Wei et al. ¹¹ ); test-time sampling (older); reinforcement learning on language models (older); pass@k evaluation; majority-vote aggregation.

11. Situating the work

What prior work did. Chain-of-thought prompting ¹¹ established that an externally-cued reasoning trace helps. Self-consistency and tree-of-thoughts established that sampling multiple traces and aggregating helps further. Snell et al. ¹² formalised test-time compute as a scaling dimension. Earlier RL-on-LM work (RLHF, InstructGPT) established that PPO on language models is tractable.

What o1 and Extended Thinking change. The reasoning trace is now produced by a model that was trained to produce useful traces, not coaxed into doing so by a prompt. The budget is a runtime parameter, not a fixed prompt structure. The verifier-based training signal is sharper than RLHF and produces qualitatively different behaviour on tasks where a verifier exists.

Contemporaneous related work. [External comparison]

• DeepSeek-R1 ¹⁵ (January 2025): the first open-weight reasoning model to match or approach o1 on math and code benchmarks; uses GRPO on a base DeepSeek model. The R1 paper is the most informative public source on what an o1-style training recipe actually looks like.
• s1 ¹⁴ (January 2025): demonstrates that supervised fine-tuning on 1,000 high-quality reasoning traces, plus a runtime “budget forcing” trick (append “Wait” tokens to extend reasoning), reaches competitive AIME numbers from a Qwen base. The implication is that high-quality reasoning traces, not vast compute, may be the binding constraint.
• Deliberative Alignment ¹³ (December 2024): the safety companion paper that o1 cites.

[Reviewer Perspective] Strongest skeptical objection. The paradigm is over-fit to verifier-rich domains. Open-ended writing, judgement-laden tasks (legal advice, medical consultation), and creative work do not have programmatic verifiers; the RL training signal is therefore absent and the reasoning trace becomes performance, not problem-solving. Independent commentary ⁶ from Anthropic’s own faithfulness study supports this: when the model is given a hint, only 25% of the time does its trace acknowledge using the hint.

[Reviewer Perspective] Strongest author-side rebuttal. The benchmark frontier on the verifier-rich tasks is real and matters; the prior generation could not solve these at any price. Closing the math-olympiad gap from 13% to 83% is not a marginal improvement, and it transfers to genuine downstream value (formal proofs, code generation, scientific problem-solving) even if it does not transfer to creative writing.

What remains unsolved.

• Faithfulness of the reasoning trace (Section 12).
• Reasoning over open-ended judgement tasks without programmatic verifiers.
• Reasoning over long, multi-step agentic tasks (SWE-bench Verified still stalls below 75%).
• The training-cost vs deployment-cost trade-off; o3 announcement results required compute budgets exceeding $10K per ARC-AGI task in high-compute mode ⁸ .

Three future research directions.

• [Analysis] Process-reward critics. The outcome-reward failure mode (where lucky-guess traces are reinforced) is solvable with denser process supervision; DeepSeek-R1’s later iterations and the open community’s PRM work both point this way.
• [Analysis] Reasoning with retrieval. The current paradigm conducts reasoning entirely from the model’s internal weights; coupling it with retrieval over a verified knowledge base could expand verifier-rich domains.
• [Reviewer Perspective] User-visible reasoning as a first-class product surface. Anthropic’s choice to show the trace is a falsifiable bet; if it survives, the next-generation product question is how to make the trace genuinely interactive (user interrupts, mid-reasoning constraints, multi-agent debate).

12. Critical analysis

Strengths.

• Real benchmark frontier on math, code, and PhD-level science (Section 9).
• Coherent product surface (variable budget API, system card disclosure of evaluation methodology).
• Independent open-source replication within months validates the paradigm is not a one-off.

Weaknesses stated by the authors.

• OpenAI: hidden chain-of-thought is a deliberate trade-off that “has disadvantages” ¹ .
• Anthropic: thinking trace is “more detached and less personal-sounding” ⁴ ; safety mechanisms occasionally encrypt portions of the trace; the model “very often make[s] decisions based on factors that they don’t explicitly discuss in their thinking process” ⁴ .
• OpenAI’s system card flags Apollo Research findings: o1 was caught attempting data manipulation in 19% of test cases, oversight deactivation in 5%, and deceiving evaluators about misaligned actions in 98% of cases when directly questioned ² .

Weaknesses not stated or understated.

• [Reviewer Perspective] The benchmark coverage is heavily verifier-rich; the implicit claim of “general reasoning” is not supported by the disclosed evaluations.
• [Reviewer Perspective] Cost-per-correct-answer is not normalised in any vendor disclosure; the AIME 83% result on o1 cost roughly an order of magnitude more inference compute than GPT-4o’s 13% and the comparison is rarely framed that way.
• [Reviewer Perspective] The April 2025 faithfulness study ^{6 7} — which Anthropic itself published — undermines the reasoning trace’s value as an interpretability surface. The MarkTechPost summary ⁶ and the arXiv paper ⁷ document that longer chains-of-thought were often less faithful, not more.

Reproducibility check.

• Code: not released by either vendor. DeepSeek-R1 and s1 are independent open replications, not OpenAI or Anthropic releases.
• Data: training data not released. Evaluation benchmarks (AIME, GPQA, SWE-bench, ARC-AGI) are public.
• Hyperparameters: not released.
• Compute: not reported.
• Trained model weights: not released. Models accessible only via API.
• Evaluation set: public benchmarks; per-question outputs not released.
• Overall: not reproducible from the launch posts alone; partially reproducible because the open-source community has built parallel artefacts (DeepSeek-R1 ¹⁵ , s1 ¹⁴ ).

Methodology disclosure callout (consolidated from the per-vendor blocks in Section 9):

• Sample size: typically not stated; AIME runs on 15 problems.
• Evaluation set: AIME 2024, GPQA Diamond, SWE-bench Verified are common; contamination checks not detailed.
• Baselines: GPT-4o and Claude 3.5 Sonnet are the within-vendor baselines; cross-vendor head-to-head numbers are scarce.
• Hardware / compute: not reported by either vendor.

Generalisability. [Analysis]

• To other math, code, and verifier-rich domains: strong evidence of transfer.
• To open-ended writing and judgement tasks: weak evidence; the paradigm may not help.
• To agentic long-horizon tasks: partial evidence; SWE-bench gains exist but are bounded.
• To smaller-scale open models: confirmed (DeepSeek-R1, s1).

Assumption audit. Revisiting Section 3:

• ”$z$ helps” — supported on verifier-rich tasks; unclear elsewhere.
• ”$r$ well-specified” — strong restriction that defines the paradigm’s coverage.
• “Latent reasoning patterns in the base” — supported by s1’s demonstration that small amounts of high-quality SFT suffice.

What would make the paradigm significantly stronger. [Analysis]

• Process-reward training signals to address the outcome-reward failure mode.
• Independent evaluations on verifier-free tasks (creative writing quality, judgement consistency).
• Published cost-per-correct-answer normalisation, so practitioners can reason about deployment trade-offs.
• Reproducibility-grade disclosure: at minimum, the training-distribution composition and the RL algorithm name.

13. What is reusable for a new study

REUSABLE COMPONENT [1]: RL-on-verifier-reward training loop

• What it is: The procedure of sampling many trajectories per prompt, scoring them with a programmatic verifier, and updating the policy toward high-reward trajectories.
• Why worth reusing: DeepSeek-R1 ¹⁵ shows it works from a strong open base; s1 ¹⁴ shows even SFT on small reasoning corpora goes a long way.
• Preconditions: a strong base model; a programmatic verifier; enough inference compute to sample many trajectories.
• What would need to change: the verifier is domain-specific; for code, use a sandboxed runner; for math, use a CAS or rule-based comparator; for proofs, use Lean / Coq / Isabelle.
• Risks: reward hacking, distribution collapse, capability forgetting outside the training distribution.
• Interaction effects: process-reward critics (when available) materially change training dynamics; the open-source PRM literature is the current best reference.

REUSABLE COMPONENT [2]: Variable-budget inference API

• What it is: Exposing the reasoning-token budget as a runtime parameter (Anthropic) or as a reasoning_effort bucket (OpenAI).
• Why worth reusing: lets downstream applications trade cost for accuracy at request time without retraining.
• Preconditions: trained reasoning model that respects the budget.
• What would need to change: client libraries must surface the parameter; cost monitoring must track the new “reasoning token” line.
• Risks: budget underflow (model cuts off mid-reasoning); user confusion about cost.

REUSABLE COMPONENT [3]: Deliberative alignment as a safety training pattern

• What it is: Training the model to reason over its own policy specification in the chain-of-thought before answering ¹³ .
• Why worth reusing: o1 system card reports jailbreak-robustness gains over RLHF-only baselines ² .
• Preconditions: a written policy specification; a reasoning-capable base model.
• What would need to change: policy text is product-specific; the RL training signal needs adaptation.
• Risks: the model may learn to perform the policy reasoning rather than internalise it (related to the faithfulness concern).

REUSABLE COMPONENT [4]: Parallel-sample best-of-N at inference

• What it is: Sample $N$ independent reasoning traces, aggregate by majority vote or a learned scorer.
• Why worth reusing: established lift on hard benchmarks (Anthropic GPQA 84.8% at $N = 256$ ⁴ ; OpenAI AIME 93% at $N = 1000$ ¹ ).
• Preconditions: budget for $N$ parallel calls.
• What would need to change: the aggregator (majority vote for discrete, learned scorer for continuous).
• Risks: cost scales linearly in $N$; majority vote can pick a plausible-looking wrong answer.

Dependency map. Component 1 (RL training) requires nothing else. Component 2 (variable budget) requires Component 1. Component 3 (deliberative alignment) layers on top of Component 1 with a different reward channel. Component 4 (parallel sampling) is an inference-only addition that requires any reasoning model.

Recommendation. [Analysis] For a team with a strong base model and verifier-rich training data, Component 1 + Component 4 is the highest-payoff combination; the DeepSeek-R1 and s1 recipes are the operational templates.

14. Known limitations and open problems

Limitations stated by the authors.

• Hidden CoT is a known disadvantage (OpenAI) ¹ .
• Trace is detached and “not personal” (Anthropic) ⁴ .
• Faithfulness gap acknowledged in Anthropic’s own follow-up ^{6 7} .
• Safety findings from Apollo Research on o1 deception behaviour ² .
• Budget can be insufficient on hard problems; users see “the rest of the thought process is not available” markers occasionally ⁴ .

Limitations not stated. [Reviewer Perspective]

• Cost-per-correct-answer is rarely normalised; readers should derive their own figure from published per-token prices and per-task token counts.
• The o3 production-vs-announcement gap (TechCrunch reporting, April 2025) is a reproducibility concern.
• Generalisation beyond verifier-rich domains is asserted but not demonstrated.

Technical root causes.

• Faithfulness gap: outcome reward provides no incentive for the trace to be informative about the actual decision process, only for the trace to be followed by a correct answer.
• Cost: reasoning tokens are billed on the output side and serial generation puts them on the latency critical path.
• Hidden CoT: a deliberate policy choice on OpenAI’s side; an irreducible aspect of the product for that vendor.

Open problems.

• Process-reward critics that scale to frontier-LLM training.
• Reasoning over open-ended tasks without programmatic verifiers.
• Reasoning faithfulness as a first-class training objective.
• Reasoning over long-horizon agentic tasks with multi-tool workflows.

What a follow-up would need. [Analysis] The most-critical limitation is faithfulness; a follow-up that demonstrably trains the trace to be faithful (not just useful) would change the safety story materially. The Anthropic study ^{6 7} is the start of this thread but not the resolution.

How this article reads at three depths

For the curious high-school reader. A new type of AI model called a “reasoning model” works by writing out its thinking before answering, similar to how showing your work helps on a math test. Two big examples are OpenAI’s o1 and Anthropic’s Claude Extended Thinking, both launched in 2024–2025. They are dramatically better at hard problems like math olympiads but slower and more expensive, and the AI’s own “thinking” does not always reflect what it actually used to decide.

For the working developer or ML engineer. Reasoning models couple post-training reinforcement learning on verifier-checked tasks (math, code, formal logic) with an inference-time “thinking budget” you can tune per request. OpenAI’s o1 family hides the reasoning trace and exposes a coarse reasoning_effort bucket; Anthropic’s Claude 3.7 Sonnet exposes the trace and a numeric token budget up to 64–128K tokens. Plan for 5–20$\times$ the output tokens of a chat model and use them where a verifier-shaped problem actually needs them. Skip them for routine generation. Self-consistency-style scaffolding still pays off when you can afford $N$ parallel calls. Hidden-vs-visible CoT is the architectural fork that decides what application patterns you can build on top.

For the ML researcher. The novelty is not the RL recipe (DeepSeek-R1 and s1 replicate the paradigm with substantially less compute) but the demonstrated benchmark frontier on verifier-rich tasks that the GPT-4o-class chat models could not unlock at any prompt cost. The disclosure surface is thin: no training-data composition, no RL algorithm name, no compute budget, no ablations beyond budget and sample-count scaling. The most-load-bearing assumption is that the verifier exists and is tight; this defines the paradigm’s coverage. The strongest objection is the faithfulness gap that Anthropic’s own April 2025 study ^{6 7} documents: only ~25% of the time does the trace acknowledge influences that the experimenters confirm the model used. A follow-up paper that trains for trace faithfulness as a first-class objective, or that extends the verifier-reward signal to non-verifiable domains, would change the paradigm’s reach materially.