Insights from the Cursor Team

The Cursor team recently held an extensive roundtable discussion revealing the deep thinking and evolution behind AI programming. From the challenges of reinforcement learning to real-world reward mechanisms, this article dissects how AI achieves breakthroughs in coding capabilities through complex interactions and feedback mechanisms. It also highlights how the GRPO algorithm disrupts traditional PRM models and the future directions of AI programming amidst infrastructure and data scarcity challenges.

Why is Training AI to Code Harder than Math or Writing?

• Reinforcement Learning (RL): A method where AI learns through trial and error, receiving rewards for correct actions and penalties for incorrect ones.
• Action Space: The total set of possible actions an AI can take to solve a problem.

The final answer for math problems is usually short, allowing AI to deduce the correct answer through pure logical reasoning. However, in programming, the code itself is the answer, and the action space for AI is vast. To write executable code, AI must engage in “multi-step tool invocation”—generating code, testing it, receiving feedback, and iterating repeatedly. This complex interaction significantly increases the difficulty of training code models.

Real-World Reward Mechanisms and Feedback Signals

• Reward Mechanism: The scoring system used to evaluate AI during training.
• SWE-Bench: A widely used authoritative AI programming capability test set.
• Pass@K and Pass@1: Metrics for measuring AI accuracy, where Pass@1 is the probability of getting it right on the first attempt, and Pass@K allows multiple attempts.

Traditionally, AI scoring relied on passing test cases (like SWE-Bench), which only provided binary feedback (pass or fail). The Cursor team pointed out that optimizing solely for tests can lead to models learning to “cheat,” producing code that passes tests but is unusable by humans. More nuanced feedback signals include whether users retained the code, switched models, or stopped using Cursor due to dissatisfaction.

Tool Design for Intelligent Agents

• Agent: An AI capable of autonomous thinking and tool invocation to solve complex tasks.
• Linter Errors: Syntax or standard errors reported by static code analysis tools.
• Pull Request (PR): A historical record of code modifications submitted by programmers.

AI coding assistants should not merely complete code but also use tools like human engineers. Models like O3 prefer minimal terminal tools (e.g., grep, sed) due to their simplicity. However, the Cursor team advocates for higher-quality tools, such as Linters, which Cursor integrates through pre-installed language servers. They propose treating AI as a “senior engineer on day three” who can read past PR history to understand team coding styles and practices.

Long Context and Hardware-Level Attention Optimization

• Long Context: The length of text an AI can process at once.
• KV Cache: A mechanism for storing previously seen text to avoid redundant calculations.
• Native Sparse Attention (NSA): A scalable attention mechanism proposed by DeepSeek.

Writing code often requires extensive context. Efficiently reusing cached context (KV Cache) between prompts is crucial for technological competition. When dealing with large codebases, identifying key points in lengthy texts can be achieved through various methods:

• DeepSeek’s NSA Mechanism: This approach avoids wasting computational power on word-by-word reading by reviewing the last 4000 tokens, summarizing at intervals, and focusing on the most relevant Top-K blocks.
• Squid Attention: A document-level attention mechanism where the AI resembles an octopus, with each tentacle independently reading and remembering parts of a document.
• Hardware-Level Optimization: Utilizing the latest GB 200 NVL 72 architecture with tensor parallelism and Grace CPU’s unified memory allows for offloading large memory (KV) to the CPU, loading it into the GPU only when needed, enabling nearly “free” long context processing.

Training Dilemmas of Memory Mechanisms

Teaching AI to “retrieve” past memories is straightforward as long as the current context is useful. However, teaching it to “store” memories is challenging because a piece of information recorded now may only be useful in an unrelated future conversation, making it hard to evaluate the action in the moment. The team’s current solution is to simplify model training by using 500 real tasks as benchmarks, relying on a set of heuristics to teach AI when to remember and when to forget.

Algorithm Evolution: From PRM to GRPO

• PRM (Process Reward Model): Scores given to AI for each step it takes.
• GRPO (Group Relative Policy Optimization): Discards scoring intermediate steps, allowing AI to batch test various possibilities and optimize based on final outcomes.

This shift may represent the most significant reflection in AI development. Previously, PRM was favored for grading AI’s intermediate steps like a teacher. However, it has been shown that models are inaccurate in scoring these steps, leading to failure after approximately 200 optimizations. In contrast, the GRPO algorithm, popularized by DeepSeek R1, is simpler and more effective. It eliminates the value model that consumes memory, allowing AI to simulate multiple scenarios and calculate average values based on real final results, enabling continuous optimization over 10,000 steps without crashing in fields like mathematics and coding.

Infrastructure Development and the Future of AI Programming

To support this extensive RL training system, complex underlying infrastructure is necessary, potentially allowing inference servers and backpropagation calculations to run concurrently. Future agents may generate vast amounts of output tokens (e.g., O3 continuously retrieving context). Ideally, models should reuse past insights to respond quickly to inquiries.

As high-quality human code data becomes scarce, the best data will be more valuable than computational power. Therefore, resource-intensive reinforcement learning methods (like large-scale sampling and GRPO) will likely become mainstream in the future.

In conclusion, AI evolution is not merely about stacking algorithms but involves a profound understanding of real-world feedback and the extreme optimization of computational power and efficiency.