Limit of RLVR

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

We systematically study Reinforcement Learning with Verifiable Rewards (RLVR) across math, coding, and vision benchmarks and uncover that RL fine-tuning enhances sampling efficiency without expanding the reasoning capacity already present in base models.

  • Base models surpass RL at large pass@k. RL-trained variants shine at low sampling budgets, yet the original base models consistently overtake them as k increases.
  • RL narrows exploration. rewarded trajectories are amplified, but the broader solution space shrinks, revealing that RLVR optimizes within, rather than beyond, the base distribution.
  • Current RL algorithms plateau. PPO, GRPO, and Reinforce++ behave similarly and remain far from the optimal sampling efficiency achieved by broader strategies such as distillation.
Paper arXiv Code

Video: pass@k curves of base models and their zero-RL-trained counterparts across multiple mathematical benchmarks. When k is small, RL-trained models outperform their base versions. However, as k increases to the tens or hundreds, base models consistently catch up with RL-trained models across all benchmarks and LLM families without exception. Eventually, base models surpass RL-trained models.

NeurIPS 2025 Best Paper Runner-Up Award

ICML 2025 Workshop AI4Math Best Paper Award

Yang Yue1*† Zhiqi Chen1* Rui Lu1 Andrew Zhao1 Zhaokai Wang2 Yang Yue1 Shiji Song1 and Gao Huang1‡
1 Tsinghua University, LeapLab    2 Shanghai Jiao Tong University
* Equal Contribution    † Project Lead    ‡ Corresponding Author
Correspond to: {le-y22,zq-chen23}@mails.tsinghua.edu.cn , gaohuang@tsinghua.edu.cn

Yang Yue is currently focused on developing new paradigms for incentivizing LLM/MLLM reasoning, generalized world models, and exploring the generalization of VLA. He is seeking active collaboration opportunities with companies that offer the freedom to explore these frontier and fundamental questions, alongside abundant resources and a strong technical atmosphere. Additionally, he is seeking a Ph.D. visit. Please feel free to reach out if there is potential for collaboration.

Read This First

Before You Read

Thank you for visiting our website and reading our paper! If you are already familiar with the paper or have questions about its conclusions, you may start with this section. If not, we recommend reading the main content first, then returning here.

Why Study the Capability Boundary Under Multiple Sampling?

This paper does not focus on user experience or practical utility, but rather on the limits of model capability. A key question we ask is: since RL is often viewed as a core pathway toward stronger models, can RL training alone one day enable models to solve open problems, such as mathematical proofs or biological discoveries, that the base model and human cannot? Other paradigms, such as AlphaEvolve, have already demonstrated the ability to achieve breakthroughs on open problems previously unsolved by humans.

Our Core Conclusion

Current RLVR methods (specifically those using binary 0/1 rewards) do not appear to teach models fundamentally new reasoning abilities. Instead, they primarily amplify reasoning capabilities that already exist within the base model. The observed performance gains (e.g., in avg@n) mostly arise from improved sampling efficiency. A key piece of evidence is that RL models rarely solve problems that the base model cannot solve at all.

The Intuition Behind This

Effective exploration in the vast language space is extremely difficult. If the base model cannot sample correct solutions, then under a 0/1 reward scheme, RL receives no useful gradient signal—thus, it cannot learn better strategies. In such settings, the RL-trained model naturally struggles to surpass the base model.

This Is Not a Rejection of the RL Paradigm

The above intuition applies only to the 0/1 reward setting, especially GRPO-style methods that omit a value network. In a broader RL context, there are still many promising directions to break the limit:

  1. Dense / process rewards provide partial learning signals even when the full solution isn’t reached;
  2. Credit assignment via value networks, reward intermediate reasoning steps;
  3. Data (env) scale and curriculum learning use larger, dynamic data so the model can progress from simpler subproblems to harder ones;
  4. Stronger exploration mechanisms move beyond token-level sampling;
  5. Agent frameworks extend capability boundaries through tool use and external information retrieval.

While these directions are theoretically promising, none have yet worked reliably in practice. We hope the community will continue advancing these ideas.

If you still have questions—such as about the “monkey-typing” analogy (infinity sampling), lucky guessing, whether RL is useful at all, or whether the training scale was sufficient, please refer to the Q&A section below for detailed explanations.

Introducing Our Work

Recent breakthroughs in reasoning-focused large language models (LLMs) like OpenAI-o1, DeepSeek-R1, and Kimi-1.5 have largely relied on Reinforcement Learning with Verifiable Rewards (RLVR), which replaces human annotations with automated rewards (e.g., verified math solutions or passing code tests) to scale self-improvement. While RLVR enhances reasoning behaviors such as self-reflection and iterative refinement, we challenge a core assumption:

Does RLVR actually expand LLMs' reasoning capabilities, or does it merely optimize existing ones?

By evaluating models via pass@k, where success requires just one correct solution among k attempts, we uncover that RL-trained models excel at low k (e.g., pass@1) but are consistently outperformed by base models at high k (e.g., pass@256). This demonstrates that RLVR narrows the model's exploration, favoring known high-reward paths instead of discovering new reasoning strategies. Crucially, all correct solutions from RL-trained models already exist in the base model's distribution, proving RLVR enhances sampling efficiency, not reasoning capacity, while inadvertently shrinking the solution space.

Video: The effect of RLVR on LLM's reasoning ability. Search trees are generated by repeated sampling from the base and RLVR-trained models for a given problem. Grey indicates paths that are unlikely to be sampled by the model, while black indicates paths that are likely to be sampled. Green indicates correct paths, which has positive rewards. Our key finding is that all reasoning paths in the RLVR model are already present in the base model. For certain problems like Problem A, RLVR training biases the distribution toward rewarded paths, improving sampling efficiency. However, this comes at the cost of reduced scope of reasoning capacity: For other problems like Problem B, the base model contains the correct path, whereas that of the RLVR model does not.

Conclusion

  1. RL-trained models perform worse than base models in pass@k at large k values.
    While RL-trained models outperform base models at low sampling sizes (small k), base models consistently surpass them at larger k across all benchmarks, even achieving higher pass@k scores. Manual inspection reveals that base models can solve problems thought to require RL training by generating diverse reasoning paths, with at least one correct solution per problem. This indicates that RL training does not enhance—and may even limit—the full reasoning potential of LLMs compared to aggressive sampling in the base model.
  2. RL boosts sampling efficiency but reduces the reasoning capacity boundary.
    The analysis reveals that RLVR-trained models generate reasoning paths already within the base model's output distribution, meaning RLVR biases the model toward higher-rewarded solutions rather than creating entirely new reasoning abilities. However, this focus on rewarded paths reduces the model's exploration capacity, limiting its coverage of solvable problems at larger sampling sizes. These findings suggest that RLVR does not fundamentally transcend the base model's reasoning capabilities but instead optimizes existing pathways at the cost of broader problem-solving diversity.
  3. RLVR algorithms perform similarly and remain far from optimal.
    The study compares various RL algorithms (PPO, GRPO, Reinforce++) and finds their performance differences minor, as measured by the sampling efficiency gap (∆SE), which assesses how close they get to optimal sampling efficiency. Despite slight variations in ∆SE among algorithms, the gap remains large across all methods. This indicates that current RL approaches, focused on improving sampling efficiency, still fall far short of optimal performance.
  4. RLVR and distillation are fundamentally different.
    While RL improves sampling efficiency, distillation can genuinely introduce new knowledge into the model. As a result, distilled models often exhibit an expanded scope of reasoning capability beyond that of the base model by learning from distilled models, in contrast to RLVR-trained models whose capacity remains bounded by the base.

Q&A

01 Q

You're using pass@k instead of majority vote—doesn't that make the results invalid?

A
[1/3] We use pass@k… View full answer Hide answer

[1/3] We use pass@k not to measure practical utility, but to explore the reasoning capacity boundary of LLMs, as outlined in our paper.
[2/3] If a model can solve a difficult problem at least once in k samples, we consider that problem within its potential reasoning scope. If RL training truly expands reasoning, we would expect the RL model to solve more such problems than the base.
[3/3] However, we observe the opposite: RLVR models often solve fewer problems at large k, suggesting a narrowing of this boundary. This implies RLVR is optimizing within the base model's capabilities, not extending them.

02 Q

Isn't pass@k meaningless since you could eventually guess the right answer through randomly sampling k times?

A
[1/3] It’s true pass@1024… View full answer Hide answer

[1/3] It's true that pass@1024 can be noisy for datasets like AIME whose answers are within limited integer space. But we also evaluated coding benchmarks, where guessing is nearly impossible to pass unit test case, and yet similar patterns hold, with base models performing better at large k.
[2/3] For AIME and GSM8K, we manually inspected the CoT outputs and found that, in most cases, the base model produced at least one correct reasoning path—not just lucky guesses.
[3/3] On datasets such as MATH500, where answers involve complex forms (roots, fractions, symbolic math) and are hard to guess, we again observed similar trends.
Together, these results highlight the often underappreciated reasoning potential of base models. We're considering including a random sampling baseline in future work.

03 Q

Even random sampling can eventually generate the correct answer with a large enough k. So doesn't that make your result—that RL improves over base model's pass@k—meaningless?

A
Not quite. … View full answer Hide answer

Not quite. “More is different.” While it's true that in theory, even random typing has a non-zero chance of producing a correct answer—about 1/V^L, where V is the vocabulary size (~30k) and L is the output length (>200). However, in practice, that search space is astronomically large.
The key point is that the magnitude of that probability matters. If the base model has a prior that gives the correct answer a 1 in 10⁴ or 10⁵ chance, then RL might find it with millions of samples. But if that probability is 1 in 10¹⁰ or smaller, RL is extremely unlikely to escape local optima and reach meaningful reward.
In our paper, we show that for most problems, this probability is not negligible—we observe correct outputs with k = 128 or 1024, which is feasible with today's resources. So rather than being meaningless, pass@k reveals that base models already possess the necessary reasoning paths.

04 Q

Isn't it common sense that RL should turn pass@k into pass@1?

A
It’s not surprising… View full answer Hide answer

It's not surprising that RLVR turns pass@k into pass@1—that's what RL is designed for.
But what's more interesting is that RLVR doesn't do much beyond that in our experiments. It doesn't seem to introduce new reasoning abilities: if the base model can't solve a problem, the RL-trained model still can't either. This clearly highlights the upper bound of RL in reasoning.
And that's not something obvious. In traditional RL such as Atari or Go, RL is known to explore and discover new strategies, continuously self-improving without an inherent bound. But in the case of LLMs, RLVR seems constrained by the base model's existing capabilities.
Actually, this phenomenon that RL-trained models perform worse than base models in pass@k surprises lots of researchers.

05 Q

Is the training model too small in your experiment?

A
In camera-ready… View full answer Hide answer

In camera-ready, we include preliminary scaling experiments using Magistral, a near-frontier pure RLVR model, and the conclusion remains consistent. But it's still an open problem that if we scale RLVR compute to 10–1000× of Magistral, would it actually produce new knowledge beyond pretraining?

06 Q

Is your paper saying RL is useless?

A
No. RL is still… View full answer Hide answer

No. RL remains practically useful because it improves sample efficiency. However, if we want LLMs to solve truly harder problems beyond pretraining, we may need a new training paradigm that can go beyond the base model's ceiling.

07 Q

DeepSeek-Math reported similar results. How is your work different?

A
Yes, DeepSeek-Math… View full answer Hide answer

Yes, DS-Math did observe similar trends, but their study was limited to a single instruction-tuned model and two math benchmarks.
In contrast, Our work systematically investigates this across true base models in a zero-RL setting, covering multiple LLM families and a wider range of benchmarks.
We also dive further by providing deeper analyses—including perplexity trends, different RL algorithms, and evaluation against distilled models—offering a more comprehensive view of RLVR's capabilities and limitations.
We believe the fact that the reasoning scope of RLVR models is bounded by the base model is a notable phenomenon that deserves deeper attention.

08 Q

Do your conclusions contradict those in ProRL and AceReason-Nemotron?

A
FYI: The follow-up work… View full answer Hide answer

FYI: Following paper invisible leash also tested Nvidia's proRL v1, v2 and AceReason-Nemotron. The same observations that base surpasses RL on both math and coding tasks.

Experiments

We conducted experiments across three representative domains to evaluate the effect of RLVR on the reasoning ability boundaries of base and RLVR models.

Math

In the math experiments, we evaluate multiple LLM families (Qwen-2.5 and LLaMA-3.1) and their RL-trained variants on benchmarks like GSM8K, MATH500, and AIME24. We analyze pass@k curves to compare base and RL-trained models, observing that RL improves low-k performance but reduces problem coverage at high k. We manually inspect CoT validity to ensure correct answers stem from valid reasoning, not lucky guesses. Additionally, we examine Oat-Zero-trained models and filter guessable problems to focus on challenging cases. The results show base models maintain broader reasoning coverage despite RL's initial accuracy gains.

Coding

In the coding experiments, we evaluate the RLVR-trained model CodeR1-Zero-Qwen2.5-7B, derived from Qwen2.5-7B-Instruct-1M, on benchmarks like LiveCodeBench, HumanEval+, and MBPP+. We assess performance using pass@k metrics, measuring correctness based on predefined test cases. The results show RLVR improves single-sample pass@1 scores but reduces coverage at higher sampling counts (k = 128). The original model exhibits continued potential for improvement with larger k, while RLVR's performance plateaus. This indicates RLVR enhances deterministic accuracy but limits exploration diversity.

Visual Reasoning

In the experiments on visual reasoning, we evaluate Qwen-2.5-VL-7B on filtered visual reasoning benchmarks (MathVista and MathVision), removing multiple-choice questions to focus on robust problem-solving. The improvements from RLVR in visual reasoning align with those seen in math and coding benchmarks, indicating that the original model already covers a broad range of solvable problems, even in multimodal tasks. The consistency across domains suggests that RLVR enhances reasoning capabilities without fundamentally altering the model's problem-solving approach.

Case Study

We present ONE of the sampled correct CoTs from the base model, manually selected from 2048 samplings for the hardest questions in AIME24. The responses from the base model tend to be long CoTs and exhibit reflective behavior, highlighting the strong reasoning ability inherent in the base model.

Explore the full case study

Example

AceReason-Nemotron reported that after conducting 64 samples, their RL-trained model successfully solved four problems (No. 3, 14, 29, and 30) from the AIME24 dataset that DeepSeek-R1-Distill-Qwen-7B (base model in their RL training) failed to solve. While their RL-trained model is well-trained and impressively powerful, our findings suggest that the base model itself still exhibits considerable potential. With increased sampling, we observed that these four problems can indeed be solved by the base model: Problem No. 3 was solved after 5120 samples, and Problems No. 14, 29, and 30 were solved within 1024 samples. For each case, we provide representative examples of correct Chains of Thought (CoTs) and corresponding final answers:

Condensed: The process of consecutive reflective contents starting with "Wait," which are long and similar in the CoT, is summarized and abridged for better readability.

BibTeX

@article{yue2025limit-of-rlvr,
  title={Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?},
  author={Yue, Yang and Chen, Zhiqi and Lu, Rui and Zhao, Andrew and Wang, Zhaokai and Yue, Yang and Song, Shiji and Huang, Gao},
  journal={arXiv preprint arXiv:2504.13837},
  year={2025}
}
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? Poster