Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI

JITAO SANG∗, Beijing Jiaotong University, China JINLIN XIAO, Beijing Jiaotong University, China JIARUN HAN, Beijing Jiaotong University, China JILIN CHEN, Beijing Jiaotong University, China XIAOYI CHEN, Beijing Jiaotong University, China SHUYU WEI, Beijing Jiaotong University, China YONGJIE SUN, Beijing Jiaotong University, China YUHANG WANG, Beijing Jiaotong University, China

The rapid evolution of agentic AI marks a new phase in artificial intelligence, where Large Language Models (LLMs) no longer merely respond but act, reason, and adapt. This survey traces the paradigm shift in building agentic AI: from Pipeline-based systems, where planning, tool use, and memory are orchestrated by external logic, to the emerging Model-native paradigm, where these capabilities are internalized within the model’s parameters.

We first position Reinforcement Learning (RL) as the algorithmic engine enabling this paradigm shift. By reframing learning from imitating static data to outcome-driven exploration, RL underpins a unified solution of ?????? + ???? +???????? across language, vision and embodied domains. Building on this, the survey systematically reviews how each capability—Planning, Tool use, and Memory—has evolved from externally scripted modules to end-to-end learned behaviors. Furthermore, it examines how this paradigm shift has reshaped major agent applications, specifically the Deep Research agent emphasizing long-horizon reasoning and the GUI agent emphasizing embodied interaction.

We conclude by discussing the continued internalization of agentic capabilities like Multi-agent collaboration and Reflection, alongside the evolving roles of the system and model layers in future agentic AI. Together, these developments outline a coherent trajectory toward model-native agentic AI as an integrated learning and interaction framework, marking the transition from constructing systems that apply intelligence to developing models that grow intelligence through experience.

§: https://github.com/ADaM-BJTU/model-native-agentic-ai

Additional Key Words and Phrases: agentic AI, AI Agent, model-native, pipeline, reinforcement learning, large language models

Contents

Abstract 1

2

1 Introduction 3

1.1 Paradigms . . 3
1.2 Applications 5
1.3 Algorithms . 6
1.4 Survey Structure . 8

2 Algorithm: RL for LLM 8

2.1 Necessity: The Shortage of Procedural Data . 8
2.2 Feasibility: Classical RL vs. RL for LLM 1 0
2.3 A Data Synthesis Perspective . 12

3 Core Capability: Planning 13

3.1 Overview 13
3.2 Pipeline-based Paradigm 1 4
3.3 Model-native Paradigm 15
3.4 Summary and Discussion 19

4 Core Capability: Tool Use 21

4.1 Overview 21
4.2 Pipeline-based Paradigm 2 2
4.3 Model-native Paradigm 23
4.4 Summary and Discussion 26

5 Core Capability: Memory 28

5.1 Overview 28
5.2 Short-term Memory: Long Context 29
5.3 Short-term Memory: Context Management 32
5.4 Long-term Memory 3 6
5.5 Summary and Discussion 38

6 Applications 39

6.1 Deep Research Agent 39
6.2 GUI Agent 47

7 Future Direction and Discussion 55

7.1 Emerging Model-native Agentic Capabilities . 56
7.2 System vs. Model: Evolving Roles in Agentic AI 6 0

8 Conclusions 61

References 61

“I hear and I forget. I see and I remember. I do and I understand.”

— Confucius

1 Introduction

In recent years, the field of Artificial Intelligence (AI) has been dominated by the rapid progress of generative AI, which excels at producing human-like text, images, and other modalities [13, 164, 210]. Yet, the outputs of generative AI remain largely reactive: it produces content as prompted but does not pursue goals, sustain long-horizon reasoning, or interact with environments. To move beyond passive generation toward autonomous action, researchers have increasingly focused on agentic AI, which emphasizes self-directed behavior, complex reasoning abilities, and environment interaction. The rise of agentic AI is widely regarded as the next stage in the evolution of AI systems.

Across both academia and industry, three core capabilities are consistently highlighted as central to agentic AI:

• Planning: to decompose high-level goals into coherent, multi-step strategies. • Tool use: to invoke and coordinate external resources such as APIs, databases, or other models. • Memory: to retain, retrieve, and manage information over extended horizons.

The development of agentic AI is deeply intertwined with the evolution of how its core capabilities are implemented, a process that has been undergoing a profound paradigm shift.

1.1 Paradigms

Pipeline-based Paradigm. Early attempts for constructing agents can be characterized as a “pipeline” paradigm, where the agent’s core capabilities were largely facilitated by external structures in a workflow-style architecture:

• Planning in early systems often relied on external symbolic planners such as PDDL (Planning Domain Definition Language), e.g., LLM+P [126] generated plan by domain-independent engines rather than by the model itself. Later, this evolved to eliciting reasoning chains directly from the model through prompts like Chain-of-Thought (CoT) [244] and Tree-of-Thought (ToT) [276], where the model articulates its intermediate thinking process step-by-step. • Tool use initially appeared as single-turn functional calls [165], where the model generated a structured API request parsed and executed by the system. This later advanced to multi-turn frameworks like ReAct [277], which prompts the model to interleave reasoning traces with action calls following an external Thought-Action-Observation loop. • Memory has long been addressed with external modules. Short-term memory was typically maintained through conversation summary [18], where long interaction histories were summarized and reinserted to fit within the context window. Long-term memory usually relied on retrieval-based methods, most prominently through Retrieval-Augmented Generation (RAG) [97], which stored past interactions in a vector database and retrieved them on demand.

In this paradigm, the essential capabilities of agency were not intrinsic to the model, but rather engineered externally and imposed through handcrafted pipelines. We introduce the following formal definition:

Definition 1 (Pipeline-based Agentic AI). Pipeline-based agentic AI conceptualizes the agent as a system where a Large Language Model (LLM) serves as a functional component orchestrated by external, handcrafted logic. Formally, the agent’s policy ??agent is a composite function, where its final action ?? is constrained and manipulated by an external pipeline (either system workflow or model prompt) ??pipeline, acting upon the LLM’s internal policy ???? : ?? = ??pipeline (???? ).

While the pipeline-based paradigm offers modularity and a degree of interpretability, its limitations were apparent: such systems heavily rely on carefully engineered pipelines, making them both rigid and brittle. Since the execution logic follows a pre-scripted procedure, these systems struggle to adapt when faced with unforeseen circumstances or dynamically changing environments. Pipeline-based paradigm treats the LLM as a passive, reactive tool rather than a proactive, autonomous decision-maker.

Model-native Paradigm. In response to the limitations of pipeline-based paradigm, agentic AI is undergoing a paradigm shift centered on the principle of “model-native”. This represents a move away from building complex external agentic systems, toward training a powerful agentic model that effectively becomes the system itself. In the old pipeline-based paradigm, an agent was conceptualized as a composite system linked together through prompts or workflow scripts. By contrast, the emerging model-native paradigm envisions the agent as a single, unified model that, through end-to-end training, has learned to autonomously perform high-level functions. Planning, tool use, and memory management are no longer external scripts or templates but progressively internalized within the model itself

• Planning was first internalized in OpenAI’s reasoning model o1 [166] through large-scale reinforcement learning, demonstrating the feasibility that LLM could learn to “think” and plan autonomously. This line of work was advanced by DeepSeek’s R1 [38], where reinforcement learning with outcome-based rewards was sufficient to train reasoning and planning behaviors, significantly reducing the need for costly, step-by-step supervision.

• Tool use has likewise moved inside the model. OpenAI’s o3 model [167] exemplified the integration of tool use into reasoning process, learning when and how to invoke a diverse set of tools as part of its internal policy. Moonshot’s K2 [88] further scaled this direction by synthesizing large-scale tool-use trajectories combined with a multi-stage RL process, strengthening agentic tool use and multi-step decision making.

• Memory has also begun to shift from external modules to model-native mechanisms. For short-term memory, Qwen-2.5-1M [268] leveraged synthetic long-sequence data to extend the native context windows, addressing the challenge of “remembering” long-horizon information. Beyond remembering, recent works attempt to effectively “using” the information, e.g., MemAct [311] reframed context management as a tool the agent learns to call, proactively deciding when and how to store or retrieve information based on dynamic state and environmental feedback. For long-term memory, representative works like MemoryLLM [238] pioneered parameterizing memory directly, where a set of latent memory tokens is continuously updated as part of the model’s forward pass, resulting in automatically updated internal knowledge.

It is easy to see that, in contrast to the pipeline-based paradigm, this emerging model-native paradigm regards LLMs as autonomous decision-makers which learn to generate plans, invoke tools, and manage memory as intrinsic behaviors. This shift leads to the following definition:

Definition 2 (Model-native Agentic AI). Model-native agentic AI refers to the paradigm where the core capabilities of agency are progressively internalized within the LLM’s parameters. Formally, the agent’s policy is monolithic and synonymous with the LLM’s internal policy: ??agent ≡ ???? . The model directly learns to map a state ?? to a final action ?? by optimizing its policy ???? (??|??) to maximize a task-oriented objective.

1.2 Applications

The paradigm shift in how core agentic capabilities are acquired has also transformed the way agent applications are developed. Currently, agent applications have evolved along two major lines: (1) the Deep Research agent, which acts as the “brain” and excels at complex reasoning and analysis, and (2) the GUI agent, which acts as the “eyes and hands” and simulates human interaction with graphical environments. The Deep Research agent is designed for knowledge-intensive tasks, such as writing literature review, conducting market analysis, where long-horizon reasoning, information retrieval, and critical synthesis are essential. By contrast, the GUI agent is well suited to operation-intensive tasks such as automated software testing, workflow automation, where the agent is required to click, type, and manipulate graphical elements with high precision.

Deep Research Agent. An early form of the Deep Research agent was AI search, exemplified by systems like Perplexity [4], which constructed an agent pipeline involving steps such as query expansion, information retrieval, and answer generation. Google was the first to introduce a “Deep Research” agent [58], upgrading the single-turn AI search to a multi-turn, iterative research process. However, early versions of Google’s Deep Research agent still relied on a carefully engineered pipeline to manage the multi-turn search and final report generation.

The paradigm shifted with OpenAI’s introduction of the first model-native Deep Research agent [163], which was fine-tuned based on its agentic foundation model o3. This approach, where the model’s internal policy learns to strategize the entire research process, significantly enhanced long-horizon consistency and the depth of information discovery, establishing Deep Research as an advanced assistant capable of tackling rigorous professional tasks. More recently, Tongyi Lab’s WebAgent series have advanced the open-source implementation of model-native Deep Research agents. This includes WebSailor [102], which addressed the critical challenge of synthesizing highquality trajectory data for uncertain tasks, and culminated in the Tongyi DeepResearch model [92], which is capable of executing complex, multi-step research tasks in dynamic web environments.

Compared with pipeline-based systems, model-native Deep Research agents demonstrate longhorizon consistency, deeper exploration, and greater adaptability to diverse information environments. Nevertheless, two key research challenges remain. First, operating on the open web exposes the agent to pervasive information noise, and outcome-driven RL may amplify hallucinations by rewarding spurious correlations rather than factual grounding. Second, defining rewards for open-ended research tasks is inherently difficult: unlike tasks with verifiable answers, research outputs are judged by subjective qualities such as insightfulness and critical analysis. Developing reward models that capture these nuanced criteria remains a key frontier.

GUI Agent. Early GUI agents often adopted a pipeline-based approach, built a workflow that orchestrated powerful, closed-source LLMs. For instance, AppAgent [300] builds a pipeline where the LLM is orchestrated through XML view-hierarchy information, using structured metadata to perceive UI elements and generate tap/swipe actions. Mobile-Agent [228, 229] developed a multimodal agent pipeline, invoking specialized perception tools such as object detection and OCR to ground UI elements directly from screenshots. These systems typically paired a general-purpose model with specialized tools for perception and action, relying on an external workflow to guide the agent’s behavior.

Fig. 1. From Pipeline to Model-native: RL-driven evolution of agentic capabilities and applications.

More recently, the trend has shifted towards developing model-native solutions, which internalize perception, planning, grounding, and action execution into a unified policy. UI-TARS [185] represents an early step in model-native GUI agent design. Instead of wrapping a model with orchestrated pipelines, UI-TARS was trained end-to-end to predict low-level actions from visual and UI context under supervised trajectory datasets. GUI-Owl [279] and OpenCUA [237] advanced this paradigm further by fully internalizing GUI planning and execution via reinforcement learning. By optimizing outcome-based rewards over long horizons, they gained robustness, adaptability, and the ability to decide when and how to act beyond imitation.

Compared to pipeline-based systems, model-native paradigm enables GUI agents to tackle more complex and flexible tasks without brittle external scripts. Nevertheless, model-native GUI agents face unique challenges. First, unlike high-level text-based tasks, their inputs and outputs are inherently fine-grained and low-level, requiring the agent to reason over pixel-level visual cues, widget hierarchies, and precise action sequences such as taps, swipes, and text entries. Small perception or grounding errors can easily cascade into task failure. Second, GUI environments evolve as dynamic interface states, e.g., the same webpage may differ across time due to layout changes, pop-ups, or backend updates. This non-stationarity makes parallel exploration and reinforcement learning particularly difficult, since trajectories collected once may not generalize to later executions of the same task.

1.3 Algorithms

A central driver behind the paradigm shift from pipeline-based toward model-native agentic AI is the application of large-scale Reinforcement Learning (RL) in LLM training. Following the release of DeepSeek-R1 technical report [38], significant progress has been made in end-to-end RL for LLMs, demonstrating that core agentic capabilities can be acquired through exploration without requiring costly, step-by-step supervision. This highlights the transformative potential of RL for enabling LLMs to evolve their own behavioral policies and adapt to novel environments. Fig. 1 illustrates the paradigm shift from pipeline-based to model-native agentic AI, contextualized by the evolution of RL algorithms.

From SFT to RL. Before RL became the focus, Supervised Fine-Tuning (SFT) was the primary method for enhancing LLM capabilities. The premise of SFT is to train the model to imitate a dataset of ground-truth trajectories. However, unlike perception-level tasks such as object classification where humans can easily and reliably provide labels, agentic tasks usually operate at the cognitive and executive level. Real-world agent tasks, such as writing a research report, involve multi-step reasoning, iterative information retrieval before drafting the final report. Constructing complete trajectories for such tasks is prohibitively expensive and, in many cases, practically infeasible for human annotators.

Reinforcement learning elegantly bypasses the need for explicit procedural supervision by fundamentally reframing the learning problem. It shifts the objective from imitating a static dataset of how to act to exploring within a dynamic environment to learn what actions lead to success. This process can be formalized as a Markov Decision Process (MDP): at each step, the model observes a task context (state) and produces an action—a sequence of text or a specific decision. The environment then provides a feedback signal in the form of a reward, reflecting the quality or utility of the model’s action with respect to the overall task. The model’s objective is to learn a policy ???? that maximizes the expected cumulative reward over the task horizon. By learning from the relative value of different trajectories, the model can gradually refine its policy through experience without step-by-step instruction. This transformation is critical because it allows the model to discover novel and potentially more optimal strategies that may not exist in human-curated data, turning it from a passive imitator into an active explorer.

RL for LLMs. Early RL methods for LLMs largely treated the model as a static sequence generator, with the objective of aligning the output to human preference data. A representative example is Proximal Policy Optimization (PPO) [191] and Direct Preference Optimization (DPO) [189], which were widely applied in the Reinforcement Learning from Human Feedback (RLHF) [169] framework. In this setup, a separate reward model is first trained to convert human preferences into numerical rewards, which are then used to fine-tune the LLM via PPO/DPO. Such methods proved highly effective at optimizing single-turn behaviors, like enabling models to better follow instructions and generate content aligned with human values.

Yet, these RL methods are insufficient to train agentic models. Agents operate in multi-turn interactions and dynamic environments, often entailing long-term dependencies and sparse rewards. PPO/DPO and RLHF, which rely heavily on dense and step-level supervision, cannot efficiently optimize policies for long-horizon tasks.

To address these issues, a new set of outcome-driven RL algorithms has emerged to address the practical challenges of training stability and efficiency in long-horizon tasks. To counter the inefficiency of PPO’s large critic network, Group Relative Policy Optimization (GRPO) [193] was proposed. It introduces a lightweight evaluation paradigm that computes advantages based on the relative rewards within a group of sampled responses, circumventing the need for an absolute value critic and improving training stability. Further refining this, Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) [285] improves performance in multi-turn interactions by decoupling the clipping mechanism for positive and negative advantages and employing dynamic sampling strategies, making it particularly effective for training long-horizon agents.

Following these advancements, other RL innovations [109, 223, 271] have continued to emerge, further enabling large-scale, high-efficiency training on LLMs. This progression has culminated in what is now considered a unified training solution: ?????? + ???? + ????????, where a base model is enhanced via a RL learning algorithm within a well-defined task environment. Together, these algorithmic advances collectively form the central driver behind the model-native paradigm of agentic AI.

1.4 Survey Structure

The remainder of this survey is organized as follows. Section 2 begins by discussing the necessity of RL for training based on the unique characteristics of agent tasks, and compares classical RL with RL for LLMs to establish its feasibility. Section 3, 4, and 5 will systematically review the paradigm shift from pipeline-based to model-native approaches for each of the three core agentic capabilities: planning, tool use, and memory. Section 6, concurrent with the evolution of core capabilities, will examine the evolution of agent applications, specifically outlining the progression of Deep Research agents and GUI agents. Finally, Section 7 will discuss developing model-native trends for other agentic capabilities, such as multi-agent collaboration and reflection, and explore the evolving distinct roles of the system layer and the model layer in building agentic AI.

2 Algorithm: RL for LLM

2.1 Necessity: The Shortage of Procedural Data

Consider an LLM parameterized by ??, whose base policy can be expressed as a conditional distribution ???? (??|??), where ?? denotes the input instruction (query), and ?? denotes the final answer. In this formulation, the model directly maps an instruction to an answer without modeling intermediate procedural steps. In the following, by analyzing how pipeline-based methods elicit model to generate procedural behaviors, we discuss the necessity of using RL to internalize the agentic capabilities within the LLM.

Pipeline as an External Scaffold. Using the planning capability as an example, we can examine how the CoT prompting-based pipeline guides the base policy ???? to perform multi-step reasoning.

We first formalize that:

A LLM is trained to generate an answer given a query , such that ?? .• A CoT prompt consists of ?? few-shot exemplars, denoted as ?? = {(?? (??), ?? (??)1:???? , ?? (??) )}????=1, where each exemplar consists of an instruction ?? (??) , a reasoning trajectory ?? (??)1:???? , and a final answer ?? (?? ) . • The full prompt fed to the model is a concatenation ??prompt = [??; ??].

When the model’s policy ???? processes the concatenated prompt ??prompt, the probability of generating the subsequent reasoning chain ?? = ??1:?? and answer ?? is the product of the conditional probabilities for each token:

The presence of the exemplar ?? in the conditioning context creates a strong pattern, significantly increasing the probability of generating a sequence that begins with a reasoning chain ?? that resembles ?? (??)1:???? . In other words, CoT prompting externally injects procedural structure into the?? an internalized behavior, but rather an elicited one.

The Out-of-Distribution Gap. This reliance on in-context pattern matching is precisely why the pipeline-based paradigm is brittle. The model has not learned why the reasoning steps are logical or effective, but only learned that they are textually plausible in the given context. The effectiveness of such handcrafted pipelines is thus limited because large-scale natural corpora, used for pre-training, rarely contain the kind of structured, procedural data that would directly support learning the conditional probability.

Specifically, the conditional distribution invoked by CoT prompting, ???? (??, ??| [??; ??]), often deviates from the distribution that the LLM was exposed to during pretraining. Pretraining primarily optimizes ???? (??|??) over naturalistic instruction-answer pairs. Since the model rarely encountered trajectories of the form (??, ??1:?? , ??), the structured prompt context ??prompt = [??; ??] is often Out-ofDistribution (OOD) with respect to the data model was trained.

The model’s ability to follow CoT patterns is therefore fragile: it may succeed when the test query is similar to exemplar queries, but fails to generalize in OOD cases. Without an internalized planning policy, the model may fail to follow the reasoning structure, and produce incoherent reasoning steps and ungrounded answers.

The Necessity of RL. As discussed above, pipelines such as CoT prompting do not update the model’s parameters ??. Instead, they manipulate the input sequence to make the desired output pattern more probable. To bridge the OOD gap and create an internalized capability, the model’s parameters must be explicitly optimized.

Still taking planning as the example, we can reframe the planning problem from a probabilistic perspective. Instead of directly mapping a query to an answer, a model with native planning ability will first reason and then answer. This can be viewed as marginalizing the final answer probability over the space of all possible reasoning trajectories ??:

where ?? = (??1:?? ) denotes the latent reasoning trajectory. To internalize this planning process, the model’s policy must learn to model both terms: the policy over reasoning trajectories, ?? (??|??), and the conditional generation of the answer given the reasoning trajectory, ?? (??|??, ??).

One approach to achieve this is through SFT during post-training. This requires datasets of (??, ??, ??) triples obtained via either human annotation or data synthesis. However, as previously discussed, human annotation of optimal reasoning paths is prohibitively expensive and sometimes even impossible for complex tasks. While, synthetic data generation is ultimately limited by the capabilities of the base model.

RL addresses this by allowing the policy ???? to explore complete reasoning trajectories ?? in an environment, and to update its parameters based on outcome-driven rewards R (??) without requiring full procedural supervision. Formally, the optimization objective can be expressed as:

where ?? = (??1:?? , ??) denotes a reasoning trajectory consisting of intermediate steps ??1:?? and a final answer ??.

Furthermore, RL offers two fundamental advantages over SFT for internalizing the agentic capabilities: (1) RL enables a shift from static data feeding to dynamic sample generation. In SFT, the model is a passive recipient of a fixed dataset. In contrast, RL enables adaptive and exploratory learning: the model continuously updates its behavior through interaction, generating new trajectory samples ?? ∼ ???? as its policy ???? evolves. (2) RL provides a more flexible feedback mechanism, moving from absolute ground-truth fitting to relative value learning. Unlike SFT, which minimizes loss against a single, pre-defined “correct” trajectory, RL optimizes for relative outcomes by rewarding trajectories that lead to better task performance even without explicit ground truth.

In this sense, RL transforms the model from a passive imitator into an active explorer, providing the fundamental mechanism for the model-native internalization of agentic capabilities.

2.2 Feasibility: Classical RL vs. RL for LLM

Large-scale pretraining endows LLMs with extensive world knowledge and structured priors, which fundamentally reshape how RL can be applied. These pretrained priors not only provide a knowledgeable starting point that improves exploration efficiency, but also supply a universal interface for representing environments, actions, and rewards across diverse tasks. Consequently, RL for LLMs is no longer confined to narrow, domain-specific settings, but becomes a general mechanism for internalizing agentic capabilities.

Exploration Efficiency: from Random Search to Prior-guided Exploration. In classical RL, the agent begins with a random policy and learns solely through trial and error. The policy ???? (??|??) is initialized randomly and gradually refined by maximizing expected cumulative rewards across trajectories sampled from an initially unstructured distribution. This process is computationally expensive and sample-inefficient, as the agent must repeatedly explore a large amount of low-value or irrelevant states before converging to an optimal policy.

By contrast, an LLM pretrained on massive corpora already encodes extensive factual and procedural knowledge, which acts as a strong prior over both the state and action spaces. As a result, the model’s policy is no longer initialized randomly but anchored by a structured prior ??prior(??|??, K), where K represents world knowledge embedded in the pretrained weights. The RL objective thus becomes:

which can be viewed as fine-tuning the knowledge-conditioned policy to better align with taskspecific reward signals R (??). This prior-guided exploration significantly improves sample efficiency, enables meaningful trajectory discovery in early training, and prevents the policy from degenerating into random or repetitive behaviors. In essence, RL for LLMs does not learn from scratch, but rather performs exploratory refinement of a pretrained world model through feedback-driven optimization.

Generalization across Tasks: Universal Environment, Action and Reward Interfaces. Classical RL operates within narrowly defined environments, where both the state and action spaces are explicitly specified and the reward function is handcrafted for a single objective. Each policy ???? (??|??) is trained to optimize performance within this constrained setup, which is tightly coupled to its environment. As a result, classical RL agents, including systems like AlphaGo [198] and robotic manipulation controllers, are highly specialized and rarely generalize beyond their predefined domains.

By contrast, RL for LLMs operates within an open-ended, language-mediated environment where every element of the RL tuple is represented through text or symbolic tokens. The state ???? corresponds to the evolving textual or multimodal context, capturing task descriptions, retrieved evidence, and interaction history. The action ???? is expressed as generated text, tool invocations, or GUI operations, serving as a universal and compositional control interface that allows one policy to act across diverse domains. Crucially, the reward R (??) can also be defined in flexible and semantic terms: success in reasoning, factual correctness, user preference alignment, or even programmatic verification (e.g., passing unit tests or symbolic proofs).

This language-based representation collapses the boundaries between distinct RL tasks, creating a unified state-action-reward interface for different forms of reasoning and interaction. Through this abstraction, LLM serves as both the policy learner and an implicit world model, transforming

	MethodologicalSingularity	Shift in Research Focus
	MethodologicalSingularity	ApplicationExpansion	CapabilityEnhancement	TheoreticalInnovation
Physics	NewtonianMechanics + Calculus	Fluid dynamics,Celestial mechanics	Many-Body,Turbulence problem	ElectromagneticTheory, Quantum
AI	LLM + RL	Scientific Discovery,Social Systems	Continual Learning,Safety&Alignment

Table 1. Physics vs. AI: methodological singularities and subcequent shifts in research focus.

RL from a domain-specific algorithm into a general optimizer, bridging reasoning, decision-making, tool use, and memory management within one integrated agentic framework.

Unified Solution: LLM + RL + Task. Recently, it is widely discussed that the development of AI has entered “the second half” [274]. Rather than designing solutions for specific problems, research now starts from a unified methodology, LLM with RL, and seeks appropriate tasks through which to evaluate and further enhance model capabilities. The proposal of challenging benchmarks has thus become critical for advancing LLMs and agents. This includes GAIA [153] for assessing general-purpose agent capabilities, SWE-Bench [81] for evaluating project-level programming, AndroidWorld [190] for real-world GUI operations, BrowseComp [243] for challenging deep research tasks, MCP-Universe [140] for evaluating multi-agent collaboration and planning, and most recently, FutureX [298] for examining the predictive abilities of LLM agents.

The unified paradigm of model-native agentic model training can be formulated as:

where Mbase provides general world knowledge and reasoning priors, Alearn (e.g., RL or preference optimization) adapts and refines these capabilities through interaction and optimization, and Etask defines the environment, tool set, and reward signals that contextualize learning. Recently, a growing body of work demonstrates how research across NLP, vision, robotics, and scientific domains is increasingly converging on this formula.

This is strikingly analogous to the unification of Physics after Newton. Before Newton, physical subfields were fragmented: Celestial mechanics was governed by Kepler’s laws describing planetary motion, Terrestrial mechanics was pioneered by Galileo’s studies of falling bodies, and fields like Optics and Fluid dynamics had their own disparate empirical and geometric principles. Newton’s “Three Laws of Motion”, “Law of Universal Gravitation”, together with the powerful mathematical tool of Calculus, provided a universal framework that unified the experimental research of the heavens and the earth under a single set of principles, achieving an unprecedented unification for Physics.

This precisely resembles the role that “LLM + RL” plays today for AI. LLM provides a unified model of world knowledge and foundational reasoning, much like Newton’s laws. Meanwhile, RL offers a dynamic, goal-oriented optimization framework, analogous to the problem-solving engine of Calculus. Following the development of Physics after Newton, as summarized in Table 1, we discuss the shift in AI’s research focus after the “LLM + RL” methodological singularity. The new focus may range from the expansion of applications into practical and complex domains like healthcare, scientific discovery and social system simulation, to the enhancement of advanced agentic capabilities such as continual learning, safety and alignment, and ultimately toward possibly new theories on general intelligence.

Fig. 2. RL-driven data synthesis as the engine of compute-intelligence conversion.

2.3 A Data Synthesis Perspective

Compute–Intelligence Conversion in AI Development. To further understand the role of RL in training agentic models, it is useful to review the evolution of AI development. With computational power growing at a rate of 3-4?? per year, the total increase in compute over the last two decades has been on the order of trillion-fold, making compute a non-negligible driving force. An intriguing view holds that, an underlying thread of AI development over this period has been the effort to convert the ever-growing compute into intelligence gains with maximum efficiency [31]. As illustrated in the upper half of Fig. 2, this thread has unfolded in two major stages:

• Model architecture advancement. During the first ten years (approx. 2010-2020), progress was mainly driven by architectural innovations that enabled the efficient consumption of massive compute and data. The evolution from Support Vector Machines (SVMs) [32] to Deep Neural Networks (DNNs) [90] and then to Transformer [220] was creating models capable of effectively utilizing and encoding data into an ever-increasing number of parameters. • Data-centric scaling. Over the last five years, the focus has shifted to the data side. SelfSupervised Learning (SSL), particularly next-token prediction [188], first unlocked the ability to use the entire internet as a pretraining corpus. The current frontier is the use of RL in posttraining to convert compute into high-quality synthetic data, which can be further divided in two sub-stages: (1) RL for LLMs (Internal Reasoning): The synthesized data is internal without environment interaction, i.e., the procedural data such as reasoning trajectories that do not exist in the pre-training corpus. (2) RL for LLM Agents (Environmental Interaction): This involves interactive RL learning through tool invocation and environmental feedback, which generates interaction data that captures the consequences of the agent’s actions.

We can see that the ever-growing computational capacity has been systematically converted into more powerful model intelligence by scaling both model parameters and data volume. Focusing on the post-training stage, both forms of RL-driven data synthesis rely on strong pretrained base models but also create new procedural and interaction data that can, in turn, enhance the next round of pretraining. This establishes a positive feedback loop linking pretraining, post-training, and inference, in which RL serves as the critical mechanism continually transforming compute into intelligence. From this perspective, the involved three scaling laws in pretraining, post-training and inference should no longer be viewed as independent, one-way relationships, but as interconnected components of a cyclic system driving the self-improving evolution of agentic intelligence.

Extrapolative and Interventional Data Synthesis. As previously discussed, RL mitigates the shortage of procedural data and alleviates OOD issues by synthesizing new data beyond the natural corpora used in pretraining. Understanding the characteristics of these synthesized data provides an alternative perspective on why RL is critical for acquiring agentic capabilities. As illustrated in the lower half of Fig. 2, RL generates two types of synthetic data, extrapolative and interventional data through internal reasoning and environmental interaction, respectively.

Extrapolative data arise from internal cognitive tasks performed by the LLM itself. The model is incentivized to produce procedural data that are not present in original pretraining corpus, with a reward signal then used to select and amplify high-quality samples. For example, in mathematical reasoning, internet-scale corpora may contain queries ?? and answers ??, along with the requisite knowledge (e.g., axioms, theorems), but they rarely contain complete, step-by-step solution trajectories (??, ??1:?? , ??). RL encourages the model to explore within its existing knowledge space via ???? (??, ??1:?? |??), combining known concepts to generate previously unseen reasoning paths. When a path leads to a correct answer, it is positively reinforced. This process is essentially a form of structured extrapolation from the model’s pre-trained knowledge.

Interventional data is generated when an agent is trained to interact with an external environment to complete tasks. Natural data are typically observational (for instance, (screenshot, click_position) pairs from human GUI operation logs), which merely reveal behavioral correlations: what humans tend to do in certain contexts. In contrast, RL allows the agent to actively perform interventions ???? (???? ) that change the environment from state ???? to ????+1, receiving a reward ???? . Through learning from interventional data ???? (????+1, ???? |???? (???? ), ???? ) rather than passive observations ?? (????+1 |???? , ???? ), the agent acquires a causal mapping from action to outcome, thereby learning to predict the consequences of its action. In essence, synthesizing interventional data through RL-driven environment interaction embodies what Rich Sutton describes as “experience”: the agent acquires experience by “doing things,” and through that experience, it builds its understanding of the world [199].

3 Core Capability: Planning

3.1 Overview

Planning is formulated as an automated reasoning process that, given an initial state, seeks a feasible sequence or strategy of actions to achieve one or more specified goals, leading the system to a desired state. Traditional symbolic planning methods offer strong interpretability but rely on manually constructed, explicit environment models, leading to high domain-specific customization costs and limited applicability. The advent of LLMs has fundamentally transformed this paradigm. Leveraging implicit world knowledge acquired from vast datasets, LLMs can directly interpret natural language instructions and utilize their inherent commonsense and reasoning capabilities for task decomposition and dynamic adjustment.

Within this context, early explorations primarily adopted a “pipeline” mode. For instance, approaches such as LLM+P [126] and LLM+PDDL [62] pioneered a hybrid pathway by using the LLM as a front-end for automatic generation of formal planning descriptions (e.g., PDDL [55]), which are then solved by an external planner. Such neuro-symbolic methods integrate the strengths of both paradigms to some extent. Another category of pipeline-based methods, such as CoT [244] and ToT [276], elicits the model’s step-by-step reasoning capability during decoding through prompt engineering to accomplish planning. Although these approaches differ in implementation, they share a common characteristic: the LLM remains within an external collaborative framework, dependent on external planner or prompting strategies to complete the planning process.

Fig. 3. Overview of agentic planning methods.

However, pipeline-based paradigm fails to fully exploit the LLM’s intrinsic potential. As research progresses, the focus of the planning paradigm is gradually shifting from reliance on manual design toward harnessing the model’s internal reasoning capabilities. Planning, as a goal-directed decision-making process, fundamentally relies on the model’s reasoning abilities in comprehension, decomposition, logic, and causality. Inspired by this, a clear trend has emerged: the internalization of planning capacities into the model. Through techniques such as supervised fine-tuning and reinforcement learning, complex planning and reasoning abilities are directly embedded into the model parameters, enabling more flexible and robust autonomous planning in open-world environments and ultimately eliminating dependence on external frameworks. An overview of agentic planning methods is illustrated in Fig. 3.

3.2 Pipeline-based Paradigm

3.2.1 Symbolic Planning. Traditional symbolic planning is characterized by its dependence on explicit, manually constructed action models and logical deduction. Classical planners, exemplified by STRIPS [48] and PDDL [55], operate on symbolic state representations and perform state-space search through logical inference based on pre-defined action preconditions and effects, thereby producing a plan to achieve a goal. These methods exhibit robustness and are readily verifiable in closed domains where prior knowledge is sufficient and amenable to precise modeling. However, the translation of flexible natural language tasks into formalized symbolic domains and action schemas necessitates domain expertise, leading to high modeling and maintenance costs and a pronounced sensitivity to modeling errors.

The emergence of LLMs has partially alleviated the burdens associated with semantic extraction and environment modeling. This has prompted the development of integrated pathways such as LLM+PDDL [62], which combines LLMs with external planners. In this framework, the LLM automatically induces objects, predicates, and action templates from unstructured text or example trajectories, generating a consistent PDDL domain file for solution by a symbolic planner. Although these hybrid methods demonstrate strong performance and interpretability in closed domains, their capacity for cross-domain generalization and their operational robustness remain constrained in open, dynamic, or noisy environments.

3.2.2 Prompt-based Planning. LLM-based planning methods regard planning as a sequence generation task for the model, leveraging the world knowledge and reasoning capabilities acquired during pretraining. This approach significantly surpasses traditional planners relying on rigid symbolic logic in terms of generalization and flexibility. Among them, prompt-based planning, an early and widely explored category, requires no additional training. It operates exclusively by designing appropriate input prompts to guide the model in generating planning paths. Based on the topology of path generation, these methods can be roughly categorized into two types: linear and non-linear.

Linear-structured Planning Methods. These methods decompose tasks into a sequence of reasoning steps executed in order. A representative approach is Chain-of-Thought (CoT) prompting, which guides the model to produce intermediate reasoning steps. This is achieved by providing few-shot exemplars or zero-shot cues such as “Let’s think step by step,” enabling the model to progressively arrive at the final answer. This approach breaks down complex problems into logically coherent sub-steps, improving the transparency and interpretability of reasoning and helping reduce errors caused by the model omitting steps. However, the linear structure is inherently a local, sequential reasoning mode, lacking a global perspective on the overall task, making it difficult to handle complex planning scenarios that require multi-path exploration or backtracking.

Nonlinear-structured Planning Methods. These methods generate multiple possible “thought nodes” during reasoning, forming a tree or graph-based search space, and incorporate external or internal evaluation mechanisms for path selection and optimization. For example, Tree-of-Thought (ToT) [276] structures reasoning as a tree, expanding multiple candidate reasoning paths at each step and selecting the optimal one via self-evaluation or external scoring. LLM+MCTS [267] integrates Monte Carlo Tree Search with LLMs, leveraging simulation and backtracking to balance exploration and exploitation. RAP [68] formalizes reasoning as a planning problem in state space, guided by policy and value functions to generate reasoning paths. Non-linear methods significantly enhance the model’s global planning capability by introducing search mechanisms during reasoning. However, their performance heavily depends on the quality of the external evaluator, and the generation and evaluation of numerous candidate paths lead to a substantial increase in computational cost.

In summary, prompt-based planning methods, characterized by training-free, ease of implementation, and high flexibility, have provided a crucial foundation for the exploration of LLM-based planning. Nevertheless, their limitations are evident: they are highly sensitive to prompt design, exhibit unstable performance in complex tasks, and incur high token consumption costs due to the generation of extensive intermediate content. Consequently, the current research trend is shifting toward internalizing planning capacities into model parameters, a shift that allows models to perform reasoning without relying on explicit search or external evaluators, thereby maintaining strong planning capabilities while significantly improving reasoning efficiency and stability.

3.3 Model-native Paradigm

Based on the source of learning signals, model-native methods can be classified into two categories: (1) supervised learning, which acquires planning capabilities by imitating existing reasoning trajectories; and (2) reinforcement learning, which optimizes decision-making policies through self-exploration and learning from rewards.

3.3.1 Supervised Learning. Through supervised learning, models acquire planning capabilities by learning from high-quality reasoning process data. This paradigm is fundamentally constrained by its reliance on offline datasets, with final model performance being a direct function of data quality. A critical challenge is the scarcity of such procedural data containing sequential reasoning logic in natural web text, coupled with the high cost and lack of scalability of manual annotation. To address this data bottleneck, contemporary research has pursued two primary directions for constructing training sets: data synthesis and data distillation.

Data Synthesis. Synthetic data denotes reasoning-enriched, long Chain-of-Thought (longCoT) data generated automatically via programmed rules, LLM-based self-generation, or search mechanisms, eliminating labor-intensive manual annotation. The core challenge lies in systematically generating diverse, challenging, and verifiable reasoning traces, replacing costly human labeling with automated quality control. Currently, the synthesis of high-quality Chain-of-Thought data primarily relies on two approaches: multi-path reasoning process sampling and tree-search methods, both requireing integration with effective quality filtering strategies.

• Multi-path reasoning-trajectory sampling. Multiple complete reasoning paths are sampled from the model and then filtered and corrected under pre-defined rules to obtain highquality data. For example, LIMO [280] demonstrates that selecting high-quality reasoning trajectories is pivotal for boosting model capability. s1 [158] further shows that a small, carefully curated dataset can fine-tune models more effectively than a large, unfiltered one. BOLT [174] uses in-context learning to elicit detailed reasoning and filters low-quality responses via outcome-based rewards, thereby automating quality control. In vertical domains, HuatuoGPT-o1 [20] targets verifiable medical problems, sampling correct paths and refining erroneous ones to construct reliable reasoning chains.

• Tree search. These typically combine Monte Carlo estimation or process reward models to enable fine-grained control over step-level quality [105, 106]. For instance, Marco-o1 [321] employs Monte Carlo Tree Search (MCTS), using model confidence to guide exploration and synthesize high-quality reasoning data. WebSynthesis [52] introduces a world model in Web-UI agent scenarios, combined with MCTS to generate web interaction trajectories at scale, supported by a two-stage curriculum learning strategy to enhance policy capability. ReST-MCTS* [301] integrates a process reward model with MCTS to collect higher-quality trajectories by assessing the value of each reasoning step. Regarding multimodal reasoning, AStar [248] defines an executable visual action space and leverages MCTS to obtain highquality visual reasoning data.

To sum up, multi-path sampling focuses on the holistic evaluation and selection of complete reasoning paths, whereas tree search methods enable finer, step-level quality control through structured search and process rewards. Together, these two data synthesis approaches facilitate an effective balance between the quality and scale of synthetic data.

Data Distillation. Data distillation refers to the process of extracting high-quality reasoning chains and final answers from a strong, System-2 teacher model with advanced reasoning capabilities. After verifying answer correctness and applying simple filtering, this data is used to train a student model. The goal is to transfer the teacher’s reasoning style and step-by-step strategies to a more lightweight student model, thereby internalizing stable planning capabilities. The performance of this approach is highly dependent on the reasoning ability of the teacher model. In DeepSeek-R1 [38], for instance, the reasoning capability of the base model was first improved through direct RL, and then followed by data distillation. Only simple verification and filtering were required to obtain high-quality reasoning process data. After the release of DeepSeek-R1, thanks to its powerful reasoning performance, many studies have attempted to use DeepSeek-R1 as the teacher model to distill high-quality reasoning data [63, 77, 94, 146, 148]. A common characteristic of these works is the achievement of scalable and automated data production, significantly reducing the cost of manual annotation.

3.3.2 Reinforcement Learning. In the training of reasoning models, supervised learning remains effective but is constrained by its heavy reliance on large volumes of high-quality reasoning data. Moreover, the learned patterns often exhibit limited generalization [30, 255], making them less adaptable to complex and dynamic scenarios. As foundation models continue to improve, they increasingly possess strong inherent reasoning capabilities, which means the key challenge lies in how to effectively elicit and activate these abilities. This has made RL-based optimization a viable and promising path. Early academic explorations primarily focused on process reward, aiming to shape the model’s reasoning trajectory through fine-grained supervision. However, the release of DeepSeek-R1 [38] has profoundly revealed the practical challenges associated with process reward, while successfully demonstrating the potential of using mere outcome reward. Accordingly, our discussion of reinforcement learning will also center around these two types of rewards.

Process Reward. Process reward refers to the mechanism of evaluating the correctness of each intermediate step in a model’s multi-step reasoning process and providing corresponding reward signals. The core idea is to ensure the reliability of the entire reasoning path through dense reward guidance. By providing fine-grained supervision over intermediate steps, Process Reward Model (PRM) is expected to not only identify and correct errors more effectively but also enhance the model’s performance on unseen problems. However, its effectiveness hinges on one critical prerequisite: the availability of large-scale, high-quality step-level supervision data. However, providing accurate labels for every step in a reasoning process is an extremely challenging task. This reintroduces the bottleneck of supervised learning, i.e., the need for extensive, granular annotation, which represents the central challenge in PRM development. Based on the source and method of supervision signals, we categorize process reward into explicit process reward and implicit process reward.

• Explicit process reward. Explicit process reward relies on high-quality human annotations or reliable automatic labeling methods to obtain precise step-level reward signals. For instance, ProcessBench [322] constructs specialized step-level benchmarks through expert human annotation, providing reliable metrics for model performance.

To address scalability limitations of manual labeling, many studies employ Monte Carlo (MC) estimation or LLM-as-Judge [99] methods, sampling and comparing multiple reasoning paths to assess step quality. For example, Math-Shepherd [233] assigns correctness scores to each mathimatical inference step via MC estimation, eliminating dependency on human annotation. OmegaPRM [136] enhances this approach by combining MC estimation with Monte Carlo Tree Search (MCTS), using binary search to locate the first erroneous step in reasoning chains, thereby improving data collection efficiency. Qwen-2.5-Math-PRM [318] notes that both MC estimation and LLM-as-Judge introduce noise in automatic labeling, while human annotation remains cost-prohibitive, thus proposing a hybrid approach to reduce noise and improve accuracy.

In executable domains like code generation, execution results can serve as process supervision signals: PRLCoder [281] determines line correctness through compilation and test execution, enabling precise line-by-line labeling. ORPS [286] integrates execution feedback directly into the reasoning loop, with each step evaluated based on program execution results. The advantage of execution feedback lies in its absolute objectivity (pass/fail test criteria), making this method particularly suitable for tasks with clear verification standards like code generation and theorem proving.

• Implicit process reward. Instead of relying on explicit process-level annotations, implicit process reward methods derive stepwise supervision directly from the model’s internal dynamics. The key idea is to approximate process rewards by using the language model’s own likelihood changes as a proxy signal, i.e., measuring how each intermediate reasoning step increases or decreases the final outcome probability. This allows for the efficient, on-thefly generation of dense rewards without requiring any process-level labels.

The foundational work on Implicit PRM [287] first demonstrated this principle. It showed that an Outcome Reward Model (ORM), trained only on final outcome labels, could effectively generate process-level rewards during inference by calculating log-likelihood ratios. Building on this, PRIME [33] integrated this mechanism into a full online reinforcement learning loop. Instead of just using the implicit rewards for inference-time evaluation, PRIME utilizes these implicitly generated, per-token dense rewards to directly compute advantage estimates for policy updates during training. This approach avoids the need to train a separate, explicit PRM, significantly reducing development costs and complexity, and has shown notable improvements in performance on mathematical and code reasoning tasks.

Overall, PRMs offer a viable way for enhancing the controllability and generalization of reasoning models by providing fine-grained, step-level supervision, thereby effectively mitigating the challenge of reward sparsity in multi-step reasoning. However, as highlighted by DeepSeek-R1 [38], several critical challenges remain. First, the correctness of intermediate steps is often difficult to define and evaluate unambiguously, as step-quality criteria tend to be subjective and task-dependent. Second, obtaining high-quality process supervision data is challenging: manual annotation is prohibitively expensive, while automated labeling methods are generally prone to noise. Furthermore, PRMs are susceptible to reward hacking, where models may learn to exploit loopholes in the reward function rather than genuinely improving reasoning quality. These challenges collectively constrain the further development and application of PRMs, underscoring the need for future breakthroughs in evaluation frameworks, data construction, and training mechanisms.

Outcome Reward. Outcome reward, also referred to as outcome supervision, provides reward signals based solely on the correctness of the final answer. It offers a sparse yet unambiguous reward signal, grounded in the core hypothesis that a powerful foundation model can, through feedback on end results, autonomously explore and internalize effective reasoning policy. Exemplified by the RLVF approach adopted in DeepSeek R1 [38], outcome reward completely bypasses the complex evaluation of intermediate steps and only requires verifying the correctness of the final answer [73, 216]. In tasks with easily verifiable answers, such as multiple-choice questions and mathematical problems, RL is typically applied using verifiable outcome rewards to enhance the model’s reasoning capability [137, 217, 313]. Similarly, in the domain of code generation, methods like o1-Coder [312] and RLEF [53] validate answer correctness based on program execution feedback.

The primary advantages of outcome reward are its low cost and high reliability. The annotation cost is minimal because only the final answer needs verification, making the approach highly scalable. Furthermore, the reward signal is accurate and verifiable, as the correctness of a final answer usually meets clear, objective standards, thereby avoiding the subjectivity and inconsistency issues that challenge process reward. As a result, outcome reward has become one of the mainstream approaches for strengthening reasoning abilities in automatically verifiable tasks such as mathematics and code generation.

Beyond learned outcome models, a parallel line of work designs explicit rule-based rewards that score outputs or intermediate reasoning steps according to deterministic rules. These rules may enforce output formats, verify structural consistency, or constrain reasoning-chain length to mitigate redundant computation and balance accuracy with efficiency. Unlike learned reward models, this approach uses explicit rules to constrain and shape reasoning behaviors, offering a simple and scalable method for supervision. Widely-used rules include: (1) format reward [38], which enforces a specific output structure to ensure the stable extraction and verification of reasoning processes and final answers, and (2) length-based reward [1, 135, 195, 283], which imposes constraints on the length of reasoning chains to control for redundant computation (“overthinking”) or to achieve a controllable trade-off between accuracy and computational cost. Similar to outcome rewards, rule-based rewards are characterized by their simplicity in design and ease of scalability. Their key function, however, is to provide auxiliary structural supervision that guides reasoning beyond the final answer’s correctness. Therefore, they are often combined with outcome rewards to create a more holistic objective function that jointly guides the optimization of the model’s reasoning behavior.

3.4 Summary and Discussion

This section outlines the shift in LLM planning from the external framework-dependent “pipeline” paradigm toward the autonomous capability-oriented “model-native” paradigm. The core distinction lies in where planning capability resides: the former positions the LLM as a front-end or collaborator, whose planning efficacy depends on integration with external symbolic planners or complex prompt engineering—yet remains constrained by the high costs of knowledge formalization and prompt design; the latter aims to directly encode planning and reasoning abilities into the model’s parameters, transforming the model into an independent, end-to-end planning agent. The reviewed representative studies are summarized in Table 2.

Within the evolution of the model-native paradigm, we observe two major shifts. First, there is a transition from SFT to RL. While SFT serves as the foundation for ability internalization, its effectiveness is limited by the scarcity and high cost of high-quality annotated data. As base models become more capable, RL emerges as a more effective post-training paradigm, alleviating the dependency on costly process-level supervision. Second, within RL itself, there is a shift from process reward to outcome reward. Although process reward offers dense, step-by-step guidance, it faces serious challenges such as ambiguous definitions of intermediate step correctness, high labeling costs, and reward hacking. Exemplified by DeepSeek-R1 [38], research has established that sparse yet objective outcome rewards are sufficient to guide capable models toward effective reasoning, making this approach a mainstream choice for verifiable domains like mathematics and code. It is often combined with rule-based rewards (e.g., for formatting or length constraints) to achieve more stable optimization.

It is worth noting that this evolution from “pipeline” to “model-native” is not confined to linguistic planning. A similar trajectory is observed in multi-modal reasoning: early multi-modal tasks often relied on external visual tools or complex prompting chains to connect vision and language modules, whereas current research aims to enable models to inherently process and reason over multi-modal information through end-to-end training, achieving genuine "what you see is what you think" reasoning [120].

Looking forward, research on model-native planning capabilities will evolve along the following directions:

• From explicit to implicit reasoning: Current explicit reasoning processes relying on Chainof-Thought enhance interpretability but incur significant computational and storage overhead. A key future direction is achieving an implicit CoT, where the planning process occurs within the model’s hidden activations rather than generating lengthy textual steps [245]. This could substantially improve computational efficiency and better approximate the intuitive, rapid decision-making patterns of human experts. • From supervised to self-supervised/unsupervised internalization: To reduce reliance on external reward signals, future research will explore self-supervised or unsupervised learning mechanisms. For example, by designing intrinsic reward signals such as exploration rewards based on information-theoretic principles like information gain or entropy [306], models could be motivated to autonomously discover novel and complex reasoning strategies, enabling continuous learning without explicit external objectives.

Table 2. Overview of agentic planning methods.

Method Task Modal Affiliation Access Date
Pipeline-based Paradigm
SymbolicPlanning	STRIPS [48]PDDL [55]	General	Text	Academia	No		1971
	STRIPS [48]PDDL [55]	General	Text	Academia	No		1998
	LLM+P [126]LLM+PDDL [62]	GeneralGeneral	Text	Academia	Yes		23.04
	LLM+P [126]LLM+PDDL [62]	GeneralGeneral	Text	Academia	Yes		23.05
Prompt-basedPlanning	CoT[244]ToT[276]	General	Text	Academia	No		22.01
	CoT[244]ToT[276]	General	Text	Academia	Yes		23.05
	RAP [68]LLM+MCTS [267]	General	Text	Academia	Yes		23.05
	RAP [68]LLM+MCTS [267]	General	Text	Academia	Yes		24.09
Model-native Paradigm
Supervised	ReST-MCTS*[301]Marco-01 [321]	General	Text	Academia	Yes		24.06
	ReST-MCTS*[301]Marco-01 [321]	General	Text	Industry	Yes		24.11
	HuatuoGPT-01 [20]	Others	Text	Academia	Yes		24.12
	Bespoke-Stratos [94]	General	Text	Industry	Yes		25.01
	s1[158]	Math	Text	Academia	Yes		25.01
	R1-Distill-SFT [146]	General	Text	Industry	Yes		25.01
	LIMO [280]	Math	Text	Academia	Yes		25.02
Learning	BOLT[174]	General	Text	Industry	No		25.02
	AStar [248]	General	Multi-modal	Academia	No		25.02
	FastMCTS [105]	General	Text	Academia	No		25.02
	OpenThoughts [63]OpenR1-Math-220k [77]SYNTHETIC-1[148]WebSynthesis [52]	General	Text	Academia	Yes		25.02
		Math	Text	Industry		Yes	25.02
		General	Text	Industry		Yes	25.02
		Others	Multi-modal	Academia	Yes		25.07
ReinforcementLearning	Math-Shepherd [233]	Math	Text	Academia	Yes		23.12
	ReFT[217]	Math	Text	Industry	Yes		24.01
	OmegaPRM [136]	Math	Text	Academia	No		24.06
	OpenAI o1[166]	General	Multi-modal	Industry	No		24.09
	RLEF[53]	Code	Text	Industry	No		24.10
	01-coder [312]	Code	Text	Academia	Yes		24.11
	Implicit PRM [287]	Math	Text	Academia	No		24.12
	ORPS [286]OpenRFT[313]	Code	Text	Academia		Yes	24.12
	ORPS [286]OpenRFT[313]	Others	Text	Academia		Yes	24.12
	DeepSeek R1 [38]	General	Text	Industry		Yes	25.01
	Qwen-2.5-Math-PRM [318]Kimi k1.5 [214]	Math	Text	Industry		Yes	25.01
	Qwen-2.5-Math-PRM [318]Kimi k1.5 [214]	General	Multi-modal	Industry		No	25.01
	O1-Pruner[135]	General	Text	Academia		Yes	25.01
	PRIME [33]	General	Text	Academia		No	25.02
	DeepScaleR [137]	Math	Text	Academia	Yes		25.02
	PRLCoder [281]	Code	Text	Academia	No		25.02
	L1[1]	General	Text	Academia	Yes		25.03
	DAST [195]	General	Text	Industry		Yes	25.03
	QwQ[216]	General	Text	Industry		Yes	25.03
	Skywork or1[73]	General	Text	Industry		Yes	25.05
	Demystify-long-cot [283]	General	Text	Academia	Yes		25.05
	LLM-as-Judge [99]	Others	Text	Academia	No		25.09

• From single-task to general and transferable planning: Current models are often trained for planning within specific domains (e.g., mathematics or code). A significant future challenge lies in achieving cross-domain generalization and transfer of planning abilities [123, 204]. This requires models not only to learn problem-solving schemas for specific tasks but also to abstract general meta-cognitive planning abilities, such as task decomposition, subgoal management, and dynamic strategy adjustment, enabling improved generalization or rapid adaptation to new tasks.

4 Core Capability: Tool Use

4.1 Overview

While the previous section addressed the agent’s high-level task planning capability—the strategic reasoning process of decomposing a complex goal into a sequence of logical sub-goals, this section focuses on a distinct, more tactical layer of planning that is inherent to tool use. Tool use refers to the decision-making process in which an intelligent agent invokes external tools to extend its capability and accomplish complex tasks. The challenge of tool use is not solved by high-level planning alone: each sub-goal must be executed. Therefore, for an agent to act upon the world, it must also be able to plan the concrete sequence of actions required to achieve these sub-goals.

To systematically analyze this capability, we deconstruct tool use into two layers: planning and execution. It is critical to distinguish the planning discussed here from the task planning in the previous section. Here, planning refers to the action-level orchestration of tool invocations: determining the timing and sequence of tool calls and refining this action-plan based on feedback. The execution layer, in contrast, is responsible for generating the specific, syntactically correct invocation commands and interacting with the environment.

Early explorations into tool use were notably marked by the introduction of single-turn functional calls [165], which largely focused on the execution layer. The model’s task was confined to generating a single, structured API request in response to a direct user query. However, for a truly agentic system, tool use involves more sophisticated decision-making process with multi-turn interaction and dynamic adaption, i.e., the model must master both the strategic planning of when to use tools and the subsequent execution of those calls. This section will examine the paradigm shift in how both the planning and execution layers of tool use are implemented, from external pipeline-based systems to internalized, model-native policies.

Early research methods primarily followed the pipeline-based paradigm, characterized by the externalization of tool-use decision logic. Specifically, one category of approaches relies on hardcoded system workflows, embedding the model into predefined execution nodes. Another category, carefully designing prompts, grants the model a degree of autonomy by embedding the decision logic for tool use, covering invocation timing, tool selection, and parameter specification, within structured prompts to guide dynamic planning and execution during reasoning.

Recently, the research on agentic tool use has shifted toward the model-native paradigm, which internalizes tool-use decision-making capabilities within the model’s parameters. This approach emphasizes the model’s ability to autonomously perform tool selection and invocation during reasoning. Along the two layers of planning and execution, existing studies can be divided into two categories: (1) Modular training decouples planning and execution, typically optimizing only the planner while delegating execution to external modules; (2) End-to-end training, in contrast, emphasizes the joint optimization of planning and execution, imposing supervision signals on the model’s multi-step planning and execution to cultivate a unified policy handling the complete process from decision-making to invocation. An overview of agentic tool use methods is illustrated in Fig. 4.

Fig. 4. Overview of agentic tool use methods.

4.2 Pipeline-based Paradigm

4.2.1 System-based Workflow. System-based workflow methods follow a predefined sequence. The model is embedded at specific nodes within this flow, acting as a subtask executor. Its responsibility is to generate specified outputs based on given inputs, without the authority to autonomously plan the overall task path or decide on tool usage. In the early implementations, the dialogue system BlenderBot 2.0 [89, 262] externalized long-term memory and web search modules as controllable components, enabling the dialogue engine to explicitly trigger queries and generate responses based on the retrieved results. HuggingGPT [194] employs a pipeline of task planning, model selection, execution, and response generation, utilizing ChatGPT as a coordinator to select and orchestrate multi-modal expert models. Code as Policies [117] treats LLM-generated executable code as a policy interface, enabling robots to implement coded skills and controllers.

In summary, for system-based workflows, the advantages include high predictability due to the fixed process, stable execution less prone to errors, and ease of debugging and replication. However, this hard-coded design also leads to insufficient flexibility, difficulty in handling exceptions outside the predefined flow or complex tasks, poor generalization capability, and potential need for extensive system restructuring to add new tools or functions, limiting the rapid iteration and expansion of agent capabilities.

4.2.2 Prompt-based Methods. Prompt-based methods delegate the decision-making authority for tool use to LLM itself, where the overall control flow is dynamically generated by the model’s reasoning chain. Based on the relationship between planning and execution, we categorize existing prompt-based methods into: (1) Plan-and-Execute Separation: exemplified by Plan-andExecute, where a planner generates a high-level plan, and an executor carries it out step-by-step. (2) Interleaved Planning and Execution: exemplified by ReAct [277], where “Thought” and “Action”

are alternately generated within a single reasoning sequence, and subsequent decisions are updated based on immediate “Observation”, forming a closed-loop control adaptive to environmental feedback.

Plan-and-Execute Separation. The model first formulates a complete action plan, subsequently followed by step-by-step execution by the system or the model itself. This mode offers clear logic but struggles to adapt to unexpected deviations during execution. For instance, the PAL [51] method requires the model to write Python subroutines, which are then executed by an interpreter at runtime to obtain results. PoT [24] further expresses the entire reasoning chain as program code, effectively decoupling the reasoning process from numerical computation. For multimodal and knowledge-intensive tasks, Chameleon [131] explicitly separates planning and execution: an LLM acts as the planner, composing tool sequences (e.g., retrieval, Python functions, vision models, heuristic modules) as needed, which are then executed uniformly to produce the answer.

Interleaving Planning and Execution. The model performs immediate reasoning at each step to decide the next action, observes the result after execution, and continues reasoning accordingly, forming a dynamic decision-making process capable of adapting to environmental feedback. The ReAct framework [277] structures each interaction round into “Thought-Act-Observation”, forming a multi-turn reasoning-action loop. This facilitates simultaneous evidence gathering and plan adjustment during reasoning, significantly enhancing the model’s adaptability in dynamic environments. It has demonstrated significant advantages in task success rate, reasoning transparency, and result reproducibility for knowledge-intensive QA and interactive environments. For example, Self-Ask [180] explicitly deconstructs complex questions into “selfask-retrieve-synthesize” substages, proving particularly adept at handling compositional reasoning tasks requiring multi-step information integration.

In summary, prompt-based methods represent a paradigm shift in tool-augmented language models, transitioning system control from rigid architectures to dynamic LLM-driven reasoning. While plan-and-execute separation provides structural clarity, interleaved planning and execution offers superior adaptability through real-time feedback integration. The choice between these approaches depends on task requirements: the former suits well-defined problems with predictable execution paths, whereas the latter excels in dynamic environments requiring iterative refinement.

4.3 Model-native Paradigm

4.3.1 Modular Training. Modular training decouples planning from execution: a compact, trainable planner selects actions and tool calls, while a frozen executor formats API invocations and interacts with the environment. This separation concentrates learning on decision making and mitigates credit-assignment issues introduced by execution noise. For example, Agent-asTool [310] argues that end-to-end optimization dilutes the reward signal and complicates credit assignment. It therefore factorizes the agent into a trainable planner for high-level reasoning and a separate toolcaller for execution. RL is applied exclusively to the planner, with environmental feedback masked during training to isolate the learning signal to the decision-making policy. AI-SearchPlanner [150] assigns retrieval and call sequencing to a small trainable planner while delegating QA to a parameter-frozen large model (e.g., GPT-4, DeepSeek-R1), thereby avoiding simultaneous optimization of heterogeneous capabilities in a single loop. RLTR [116] reaches a similar conclusion: since end-to-end training struggles to focus updates on planning, it optimizes only the planner, converting a multi-objective problem into a single-objective one.

Overall, the modular training paradigm treats planning as the core of agent decision-making. In end-to-end optimization, structured parameter generation and interaction with the environment introduce additional noise, thereby weakening the effective propagation of learning signals. By contrast, the execution layer has a single, verifiable objective that can typically be realized robustly via templates, rules, or frozen models. On this basis, removing the execution layer from the training loop and modularizing it while concentrating on optimizing the planner not only conserves the training budget for the primary task, avoiding expenditure on format learning and noise handling, but also substantially improves sample efficiency and training stability.

4.3.2 End-to-end Training. End-to-end training refers to coupling planning and execution under a single objective, such that one model jointly learns the abilities for multi-step planning and action execution. While modular training offers greater control by isolating individual components, the shift toward a fully end-to-end paradigm introduces two principal bottlenecks that the unified policy must now learn to handle directly: (1) cross-step credit assignment, which centers on the granularity of attribution—whether it is assigned at the trajectory level or the turn level; and (2) uncertainty in environmental feedback, which is largely determined by the type of environment— static or simulated environments typically being more controllable and less noisy, whereas dynamic or real-world environments exhibit greater stochasticity and noisier feedback. Accordingly, the discussion of end-to-end training methods is organized along two axes: the granularity of credit assignment and the type of the environment.

Credit-assignment Granularity. Credit assignment refers to attributing a task’s final return or verifiable outcome to each tool-related decision (and its preceding reasoning) along the action sequence, thereby producing learnable signals for policy optimization. The design of the creditassignment scheme directly determines both the informativeness of these signals and the stability of training. Along this dimension, existing approaches can be organized into two categories.

• Trajectory-level credit assignment. Rewards are computed solely from the final, verifiable outcome and applied uniformly across the entire trajectory, implicitly treating all intermediate steps as equal contribution. Representative works include Search-R1 [82], R1-Searcher [201], and ReSearch [22], which optimize multi-turn retrieval with simple outcome-based rewards and use retrieval-token masking to limit overfit. AutoCoA [314] internalizes a Chain-ofAction agent: while its SFT stage contains step-level action triggers, its RL phase adopts trajectory-level credit assignment. ARTIST [200] unifies agentic reasoning, tool integration, and RL in a single framework where the model autonomously decides when, how, and which tools to invoke; optimization relies on outcome-based RL without step-level supervision.

In multimodal settings, trajectory-level supervision is likewise common: VTool-R1 [250] alternates textual reasoning with intermediate visual operations, integrates a Python imageediting tool into the RL loop, and trains with accuracy-linked outcome rewards; DeepEyes [325] explores “thinking with images” via a unified multimodal agent and emphasizes end-to-end, outcome-driven RL without cold-start SFT. Overall, trajectory-level methods are simple, scalable, and practical for large, real-world tasks. However, in long-horizon, multi-tool, multi-turn scenarios, reward sparsity and diffuse credit can weaken policy-improvement signals.

• Step-level credit assignment. The goal is to identify which intermediate steps or turns substantially contribute to the final outcome. In multi-turn agent RL, finer-grained credit enables targeted improvement of specific actions rather than averaging rewards across all steps. For example, RAGEN [241] shows that robust reasoning fails to emerge in multiturn agent RL without fine-grained rewards, underscoring the need for granular attribution. Several lines of work pursue finer credit in multi-turn RL: Multi-Turn-RL [296] estimates advantages directly at the turn level; SPA-RL [224] decomposes the terminal reward into per-step progress; GiGPO [47] combines trajectory- and step-level signals without adding a critic or extra sampling, significantly improving multi-turn training; StepSearch [242] frames retrieval and tool use through policy updates guided by information gain at each step, to mitigate policy drift from sparse global signals. Collectively, these methods shift reward from the terminal outcome to individual step, aligning gradients with effective actions and stabilizing training.

Environment Type. Regarding tool use, feedback arising from agent-environment interactions is incorporated into the context and conditions subsequent reasoning. Because this feedback is not generated autonomously by the model and is affected by environmental noise and temporal non-stationarity, it often amplifies uncertainty in subsequent outputs. The magnitude of this uncertainty is largely determined by environmental characteristics: it tends to be lower in static or well-controlled simulated settings and substantially higher in dynamic or real-world environments.

• Static/simulated environments. These environments are fixed and typically preprocessed, yielding stable feedback and low noise. As a result, training is more stable than in real settings. For instance, ZeroSearch [205] simulates a search engine with an LLM and applies curriculum RL to progressively degraded synthetic documents, substantially reducing external API costs and stochastic noise. MaskSearch [252] develops “deep research” capabilities under controllednoise conditions by using an offline retriever. ReSearch [22] jointly optimizes reasoning and retrieval via RL and is commonly trained on offline or semi-online, controlled corpora. AutoCoA [314] internalizes a world model to reduce reliance on real-world interactions. In aggregate, this class of methods makes a pragmatic trade-off: sacrificing a degree of real-world adaptivity in exchange for the benefits of lower interaction costs, more stable training, and greater reproducibility.

• Dynamic/real-world environments. The environments in this category evolve over time or change during interaction; feedback is unstable and can be delayed, which may destabilize training and thus necessitates additional stabilization techniques. Researchers have developed several lines of work to address these challenges, primarily focusing on either mitigating environmental noise or designing more robust training methodologies.

A primary strategy involves directly handling the noise and complexity of feedback from live environments. On code-oriented tasks, rStar2-Agent [192] explicitly tackles tool- and environment-induced noise in the coding loop, showing that mitigating task-irrelevant disturbances improves efficiency and reliability. Similarly, to manage noise during online training with live retrieval, Search-R1 [82] couples autonomous multi-turn querying with retrievaltoken masking and outcome rewards to stabilize learning. Other approaches use intermediary modules to preprocess the environment’s feedback. For instance, DeepResearcher [324] introduces a dedicated web-browsing agent that supplies the main agent with a preprocessed, filtered, and high-quality information stream. WebDancer [249] adopts a similar principle by compressing verbose and noisy web page content into a concise “evidence + summary” format to reduce context length and interference.

A second line of work focuses on the training framework itself to manage the inherent instability of online interaction. SkyRL [14], for example, provides a training framework for real-world tasks that uses high-throughput asynchronous parallelism to execute environments more efficiently, thereby reducing latency effects and demonstrating the feasibility of online RL for long-horizon, multi-tool tasks. Taking a hybrid approach, SimpleDeepSearcher [208] synthesizes offline datasets from the real web. This strategy aims to mitigate the instability of purely online training while achieving a closer match to real-world conditions than purely offline or simulated setups.

Overall, these methods offer stronger domain fit and closer alignment with end-use applications, but they do so at the cost of significantly higher training and engineering complexity.

Beyond credit-assignment granularity and environment type, the choice of base models also critically influences the internalization of tool-use capabilities. Compared with non-reasoning base models, Large Reasoning Models (LRMs) possess stronger long-horizon reasoning and planning abilities, enabling more effective acquisition of tool use under both end-to-end and modular optimization.

While a majority of studies integrate tools on non-reasoning base models [34, 82, 246], several recent end-to-end training studies (AutoCoA [314], WebDancer [249], SkyRL [14], etc.) have begun to explore training on more capable LRMs. For instance, ReTool [46] trains tool integration within the same tool-augmented RL framework using, respectively, a non-reasoning base (Qwen2.5- 32B-Instruct [187]) and a reasoning base (DeepSeek-R1-Distill-Qwen-32B [38]). The results show that, under matched training conditions, the reasoning base performs better, corroborating that base-model reasoning capacity enhances the learning of tool-integrated reasoning.

In summary, achieving robust capability internalization in end-to-end training requires a coordinated strategy that addresses three key factors: (1) the chosen granularity of credit assignment, (2) the challenges to training stability posed by the environment type, and (3) the selection of a foundation model with reasoning strength and tool-use alignment appropriate for the task.

4.4 Summary and Discussion

This section has reviewed the evolution of agentic tool use, elaborating its progression from the externally orchestrated pipeline-based paradigm to the internalized model-native paradigm. This shift represents a move away from predefined processes toward greater decision autonomy and adaptability in open environments. In the model-native approach, tool use is decomposed into two interdependent layers, planning and execution, presenting a multi-objective optimization problem where the model must produce logical action sequences while also reliably executing API calls and interpreting environmental feedback. The reviewed representative studies are summarized in Table 3.

The ongoing model-native tool use research faces two challenges. The first is credit assignment: how to reliably attribute a final outcome to the specific decision steps in a long action sequence. The second is environmental noise: how to maintain training stability when faced with uncertainties such as tool timeouts, inconsistent returns, and dynamic content.

To address these issues, research is likely to progress along several directions. As a pragmatic solution, a trend has emerged toward back to modular training. By decoupling the planner from the executor, the learning of the core decision-making policy can be isolated from the noise generated by the execution layer, which has been shown to improve both sample efficiency and training stability. This can help distinguish between model decision errors and environmental interference.

Within end-to-end training, the refinement is expected from coarse, trajectory-level rewards toward more granular, step-level or turn-level credit assignment. Since tool use is often a multi-turn interactive process, relying solely on a final outcome signal dilutes the contribution of key intermediate decisions. Methods that decompose rewards or estimate advantages at a finer granularity help align the learning signal more directly with effective actions, thereby stabilizing training. Simultaneously, training environments are gradually shifting from low-cost, controllable, low-noise static simulators to dynamic real-world settings to narrow the simulation-to-reality gap. The latter provides more complex and diverse scenarios, facilitating more robust and generalizable policies.

Based on this analysis, future research on model-native tool use will likely advance along the following directions:

Table 3. Overview of agentic tool use methods, including task, model, tool (CI = code interpreter; EM = external model), environment (Environ.) and other attributes.

Method

Task Model Tool Environ. Affiliation Access Date

Pipeline-based Paradigm

System-basedWorkflow

WebGPT[160]SayCan [3]

Retrieval

LLM

API

Dynamic

Industry

No

21.12

Others

LLM

Others

Dynamic

Industry

Yes

22.04

WebShop[275]Code as Policies [117]HuggingGPT[194]AutoGen [251]SWE-agent [270]

Retrieval

LLM

UI

Static

Academia

Yes

22.07

Others

LLM

CI

Dynamic

Industry

Yes

22.09

General

LLM

EM

Dynamic

Academia

Yes

23.03

General

LLM

API

Dynamic

Industry

Yes

23.08

Code

LLM

CI

Dynamic

Academia

Yes

24.05

Prompt-basedMethods

Self-Ask [178]ReAct [277]PAL [51]PoT[24]Reflexion [197]ViperGPT[211]CRITIC [61]

RetrievalHybrid

LLM

API

Dynamic

Academia

Yes

22.10

LLM

API

Static

Academia

Yes

22.10

Code

LLM

CI

Static

Academia

Yes

22.11

MathGeneral

LLM

CI

Static

Academia

Yes

22.11

LLM

Others

Static

Academia

Yes

23.03

Hybrid

LMM

EM

Static

Academia

Yes

23.03

Hybrid

LLM

CI

Dynamic

Industry

Yes

23.05

Model-native Paradigm

ModularTraining

Agent-as-Tool [310]AI-SearchPlanner [150]RLTR [116]

Retrieval

LLMLRM

API

Dynamic

Academia

No

25.07

Retrieval

API

Dynamic

Industry

No

25.08

General

LRM

API

Dynamic

Academia

No

25.08

End-to-endTraining

R1-Searcher [201]ReSearch [22]

Retrieval

LLM

API

Static

Academia

Yes

25.03

Retrieval

LLM

API

Static

Academia

No

25.03

ToRL [114]

Math

LLM

CI

Dynamic

Academia

Yes

25.03

Search-R1 [82]

Retrieval

LLM

API

Static

Academia

Yes

25.03

AutoCoA [314]

Retrieval

LRM

API

Static

Academia

Yes

25.03

ReTool [46]

Math

LRM

CI

Dynamic

Industry

Yes

25.04

ToolRL [181]

General

LLM

API

Dynamic

Academia

Yes

25.04

OTC [225]

Retrieval

LLM

API

Static

Academia

No

25.04

DeepResearcher [324]

Retrieval

LLM

API

Dynamic

Academia

Yes

25.04

SWiRL [8]

Retrieval

LLM

API

Static

Academia

Yes

25.04

ARTIST [200]

General

LLM

API

Static

Industry

No

25.04

WebThinker [112]

Retrieval

LRM

API

Dynamic

Academia

Yes

25.04

RAGEN[241]

General

LLM

API

Dynamic

Academia

Yes

25.04

WebDancer [249]

Retrieval

LRM

API

Dynamic

Industry

Yes

25.05

Tool-N1[305]

General

LLM

API

Static

Industry

Yes

25.05

Satori-SWE[295]

Code

LLM

CI

Dynamic

Academia

Yes

25.05

MaskSearch [252]SkyRL [14]

Retrieval

LLM

API

Static

Academia

Yes

25.05

Code

LRM

CI

Dynamic

Academia

Yes

25.05

ZeroSea

Retrieval

LLM

API

Static

Industry

Yes

25.05

Math

LLM

CI

Dynamic

Academia

Yes

25.05

GIGPO [47]

Retrieval

LLM

API

Static

Academia

Yes

25.05

VTool-R1 [250]

Others

LMM

API

Static

Academia

Yes

25.05

DeepEyes [325]

Others

LMM

API

Static

Industry

Yes

25.05

Multi-Turn-RL [296]

Others

LLM

UI

Dynamic

Industry

Yes

25.05

StepSearch [242]

Retrieval

LLM

API

Static

Industry

Yes

25.05

Spa-RL [224]O²-Searcher [149]

General

LLM

Others

Staic

Academia

Yes

25.05

Retrieval

LLM

API

Static

Academia

Yes

25.05

continued on next page

Table 3 Overview of agentic tool use methods (continued).

Method		Task	Model	Tool	Environ.	Affiliation	Access	Date
	MMSearch-R1 [247]	Retrieval	LMM	API	Dynamic	Academia	Yes	25.06
	Agent-RLVR [34]	Code	LLM	CI	Dynamic	Industry	No	25.06
	AutoTIR [246]	Retrieval	LLM	API	Static	Academia	Yes	25.07
	Agent Lightning [139]	Hybrid	LLM	API	Staic	Industry	Yes	25.08
	FunRL [67]	General	LLM	API	Dynamic	Academia	Yes	25.08
	rStar2-Agent [192]	Math	LRM	CI	Dynamic	Industry	Yes	25.08
	ASearcher [50]	Retrieval	LRM	API	Dynamic	Academia	Yes	25.08

• Hybrid architectures. While pure end-to-end training holds the highest theoretical performance ceiling, it often suffers from the training instabilities discussed. Conversely, fully modular designs, though easier to debug and optimize, may sacrifice synergy between components. A key research direction will be the development of hybrid architectures that strategically combine modular and end-to-end training to balance these trade-offs.

• Robust training in open environments. The transition from simulated to real, open environments requires models to handle significant challenges related to latency, noise, and uncertainty. This necessitates the construction of more stable and sample-efficient training frameworks capable of learning reliable policies from inconsistent real-world feedback.

• From tool user to tool creator. Current research primarily focuses on using a predefined set of tools. A significant leap will be to enable agents to dynamically create, compile, and validate new tools based on task demands. This represents a fundamental shift from leveraging existing resources to creatively generating novel solutions.

5 Core Capability: Memory

5.1 Overview

In LLM-based agent systems, memory is evolving from a single external module into a set of capabilities integral to the entire task lifecycle. Its functions include preserving historical and world states, selectively organizing factual evidence within a limited context, and injecting this evidence to support planning, tool use, and reasoning. This section conceptualizes memory as a form of action-oriented evidence governance: its role is not only to store information effectively but also to ensure that information is utilized proficiently to guide actions. Accordingly, this deconstructs the memory process into four core functions: storing, the decision of what information to write; managing, the organization and compression of that information; retrieval, the extraction of relevant evidence when needed; and utilization, the effective employment of that evidence in the agent’s reasoning and actions.

Two primary paradigms have emerged to implement these memory functions. The first, the pipeline-based paradigm, relies on external workflows to orchestrate information. This approach uses techniques such as indexing, summarization, routing, and rule-based strategies to manage data before and after the model’s reasoning process. While this method offers advantages in controllability, interpretability, and ease of implementation, it is often limited by a performance ceiling caused by error accumulation across modules and inflexibility in long-tail scenarios. The second, the model-native paradigm, aims to internalize memory capabilities directly within the model. This is achieved through methods like internal model editing or by integrating memoryrelated workflows into the model’s training process. By learning a unified policy for storing, managing, and retrieving information, the model-native paradigm holds a higher potential for performance but requires more substantial data, computational resources, and training stability.

Fig. 5. Overview of long context methods for short-term memory. Method categories include: SW (Sliding Window), CS (Compression & Summarization), RAG (Retrieval-Augmented Generation), Method categories include: PE (Position Encoding extrapolation), LS (Long-sequence Synthesis), and AO (Attention Optimization).

In practice, these two paradigms are not mutually exclusive and often coexist within real-world systems.

From a developmental perspective, many advancements that enabled modern agent memory did not initially target it as an explicit goal. Instead, they emerged from parallel research in system optimization and model architecture, such as KV-cache management [127, 257, 316], efficient attention mechanisms [10, 36, 294], and position encoding extrapolation [179, 203]. These foundational technologies effectively expanded the model’s capacity for short-term information retention. Building on this, engineering practices established the RAG paradigm as the default baseline, using external pipelines to manage knowledge [86]. The recent paradigm shift is defined by the internalization of these processes, as models learn a unified policy for deciding what to store, how to manage it, and when to retrieve and utilize it through end-to-end training.

To systematically review this evolution, this section divides memory into two primary layers: short-term memory and long-term memory. Our discussion of short-term memory follows its own developmental trajectory. We begin with techniques for long context, which address the fundamental challenge of processing large volumes of information within a single session [10, 36, 294]. We then examine context management, a more advanced capability focused on the selective organization and utilization of information to mitigate attention dilution and improve reasoning quality [7]. In contrast, the section on long-term memory reviews methods for persistently storing knowledge across sessions, either in external repository or within model parameters [17, 240, 263, 326]. Notably, since long-term memory content is typically not rewritable online, the paradigm shift in this area focuses more on internalizing retrieval and usage strategies rather than the entire storage process.

5.2 Short-term Memory: Long Context

This subsection examines how an agent, within a single inference session, can reliably process a large set of task-relevant evidence. The effectiveness of long-context processing is not determined solely by the absolute size of the context window but by the model’s ability to use the information contained within it. We analyze this capability across three ascending levels of difficulty:

• Retrieval. This is the foundational ability to accurately identify and pinpoint task-relevant information within a long and potentially noisy input.
• Basic Reasoning. This level involves performing basic, single-step operations on one or more localized pieces of evidence, such as extraction, summarization, or straightforward logical comparisons.
• Complex Reasoning. The most advanced level, this requires the model to integrate multiple, often distributed, pieces of evidence to construct a complete causal or logical chain for solving multi-step problems.

Consequently, effectively handling long contexts requires coupling the window’s physical capacity with training strategies that promote its stable use. When these mechanisms fail, systems may either overlook crucial facts or suffer from attention dilution. This can lead to an illusion of perception without actual use, a phenomenon described as the “Lost in the Middle” problem [122]. The overview of long context methods for short-term memory is illustrated in Fig. 5. More details are summarized in Table 4.

5.2.1 Pipeline-based Paradigm. Pipeline-based methods manage long contexts by externally processing information before it is fed to the model. These modular strategies can be grouped into several method categories:

Sliding Window. The sliding window approach is a simple and broadly applicable baseline for managing context. Under a fixed computational budget, it concatenates recent interaction segments with critical history, such as initial instructions, based on temporal or topical continuity. This method is easy to implement, computationally light, and provides predictable latency, making it suitable for tasks with strong topical continuity like dialogue or code completion. However, its limitations become apparent when relevant evidence is distant or requires cross-segment alignment, as proximity-based truncation can permanently discard key facts. In practice, this baseline is often extended with more sophisticated mechanisms, such as streaming strategies and dedicated memory controllers, which have been developed to create more dynamic and efficient systems [171, 257].

Compression and Summarization. When the context window is constrained but the history is extensive, hierarchical and selective summarization can increase information density while preserving core semantics. Traditional summarization techniques are often irreversible and produce highly abstract text, saving space at the cost of detail and traceability. To overcome this, contemporary systems increasingly construct structured evidence cards with citation anchors that point back to exact source locations. This approach effectively turns summaries into a session state cache that can be decompressed on demand, a technique explored in systems such as Selective Context [115] and LLMLingua [79]. During inference, the model can operate on compact summaries while retaining the ability to rapidly follow anchors to verify details, thereby maintaining interpretability and a coherent reasoning chain.

Retrieval-Augmented Generation. When the dialogue context far exceeds the model’s budget and the necessary knowledge resides in external stores, RAG dynamically constructs the working context for each turn. Compared with traditional RAG pipelines, which typically treat retrieval as a stateless, per-query operation, conversational variants are session-aware: they employ query expansion to capture dialogue intent and multi-path retrieval with reranking to optimize the evidence set across turns. This approach excels at knowledge-intensive, needle-in-ahaystack problems like log analysis and technical question answering. Various frameworks have demonstrated this approach, from those employing MapReduce paradigms [69] to more advanced systems that utilize graph-based structures [43] or memory-augmented techniques [182, 213]. A key challenge remains that excessive injection of retrieved information can lead to attention dilution. Therefore, practical deployments often treat long windows and session RAG as complementary, using the window to cover retrieval misses and falling back to precise injection when the window becomes too noisy.

Table 4. Overview of long context methods for short-term memory.

	Method	Long-Context Adaptivity		CapabilityLevel	Affiliation	Acces	Date
Pipeline-based Paradigm
SlidingWindow	StreamingLLM [257]MemGPT[171]	Streaming to~millions; 8K windowStreaming,infinite		Basic Rea.Basic Rea.	AcademiaAcademia	YesYes	23.0923.10
Compression&Summarization	SelectiveContext [115]LLMLingua [79]	Compression (≥2×)Compression (3~20×)		RetrievalRetrieval	AcademiaIndustry	YesYes	23.1023.10
RAG	MapReduce [69]GraphRAG [43]MemoRAG [182]ILM-TR [213]	Variable (chunked)Variable (graphretrieval)Variable (memoryRAG)Variable (iterativeretrieval)		RetrievalBasic Rea.Basic Rea.Complex Rea.	IndustryIndustryAcademia/IndustryAcademia	YesYesYesNo	22.1024.0424.0924.10
Model-native Paradigm
PositionEncodingExtrapolation	RoPE [203]	Unknown		Retrieval	Academia	Yes	21.04
	ALiBi [179]YaRN [177]LieRE [168]UltraLLaDA [71]	Unknown128KUnknown(multimodal)128K		Retrieval	Academia	Yes	21.08
				Retrieval	AcademiaAcademiaAcademia	Yes	23.09
				RetrievalComplex Rea.	AcademiaAcademiaAcademia	24.0625.10
Long-sequenceSynthesis	Qwen2.5-1M [215]Gemini 2.5 [37]	1M1M		Complex Rea.	IndustryIndustry	YesNo	25.0125.03
Long-sequenceSynthesis	Qwen2.5-1M [215]Gemini 2.5 [37]	1M1M		Complex Rea.	IndustryIndustry	YesNo	25.0125.03
AttentionOptimization	Longformer/LED [10]BigBird [294]Performer [29]FlashAttention [36]LightningAttention [307]LightningAttention-2 [308]SKVQ[40]MoBA[156]	16K8K		Basic Rea.	Academia	Yes	20.04
		16K8K		Basic Rea.	Industry	Yes	20.05
		4K+N/AN/A		Retrieval	Academia	Yes	20.06
				Retrieval	Academia/Industry	Yes	22.05
					Academia/Industry	Yes	22.05
				Retrieval	Academia	Yes	23.07
		T	Theoretically Infinite		Retrieval	Academia	Yes	24.01
		1MScalable		1M	Retrieval	Academia	Yes	24.05
		1MScalable		Complex Rea.	Industry	Yes	25.02

5.2.2 Model-native Paradigm. Model-native solutions enhance the model’s core architecture to directly handle longer sequences. These methods primarily fall into the following three categories:

Position Encoding Extrapolation. The first challenge for long-context short-term memory is that models pretrained on shorter sequences struggle to generalize to longer ones. Architectural modifications to positional encodings address this. For instance, Rotary Position Embedding (RoPE) [203] uses rotations to encode relative positions, which inherently improves length extrapolation. Other methods, such as Attention with Linear Biases (ALiBi) [179], inject distance-aware penalties into attention scores to aid generalization at inference time. Subsequent refinements like YaRN [177] and LieRE [168] further stabilize behavior in the extrapolation regime. However, research has shown that these architectural changes are most effective when combined with continued training on long-sequence data. For example, UltraLLaDA [71] couples position encoding extrapolation with targeted training to extend context stability to 128K tokens for retrieval and 32K tokens for complex reasoning, demonstrating that training is critical for activating a model’s latent long-context abilities.

Long-sequence Synthesis. Expanding the context window’s capacity does not automatically teach a model how to use it effectively. Therefore, task-driven training curricula are central to developing this capability. These curricula often include tasks like “needle-in-a-haystack” extraction and cross-document reasoning, which explicitly incentivize the model to attend to distant information. Reports on large models such as Qwen2.5-1M [215] and Gemini2.5 [37] confirm that curricula aligned with real-world applications markedly improve performance on ultra-long context tasks. Training typically follows a schedule of increasing difficulty, starting with shorter sequences and gradually extending the length, number of distractors, and reasoning complexity.

This curriculum-centric approach functions as a bridge from pipeline-based engineering to model-native capability. With sufficient training pressure, models can internalize functions once delegated to external modules like retrievers and re-rankers. The result is a shift away from reliance on external workflows and toward robust, end-to-end long-context reasoning.

Attention Optimization. The quadratic computational cost of the standard self-attention mechanism makes processing very long sequences prohibitively expensive. This has spurred the development of more efficient attention algorithms. Algorithmic methods reduce this cost through sparse connectivity patterns, as seen in models like Longformer [10] and BigBird [294], or through linear and kernelized approximations such as the Performer [29]. In parallel, systems-level optimizations focus on improving hardware utilization. A notable example is FlashAttention [36], which preserves exact attention while reducing high-bandwidth memory access through kernel fusion. This line of work has been extended by subsequent improvements like Lightning Attention [307, 308]. Additional cache-aware and shape-adaptive variants further refine attention allocation to better match task structure [40, 156]. Together, these advances are shifting long-context capabilities from an exotic feature to a practical default, providing a solid substrate for higher-level algorithms and applications.

5.3 Short-term Memory: Context Management

While long-context capabilities expand the volume of information a model can process, context management addresses the quality and relevance of that information. The central challenge is attention dilution: even when the total input length is within a model’s theoretical limits, irrelevant or redundant data can severely impair reasoning performance. In the “Needle in a Haystack” test, for instance, model performance often drops significantly when key information is placed in the middle of a long, distracting context [93, 122].

Effective context management aims to produce a sparse yet high-value stream of evidence that maximizes relevance while minimizing attention dilution. The evolution of these techniques can be examined along two core paths: Context Re-structuring (CR), which focuses on optimizing the presentation and ordering of information within the context window, and Retrieval and Routing (RR), which concerns the filtering and recall of information from external sources or internal history.

While the complete memory cycle includes an initial storing phase, the core challenge of context management lies in dynamic, in-session processes. Therefore, we categorize the methods in this section based on their primary focus within three main stages: Management (M), Retrieval (R), and Utilization (U). To clarify application scenarios, we also distinguish between single-agent and multi-agent paradigms. In single-agent systems, context management prioritizes evidence selection for a single model, whereas in multi-agent systems, it must also handle inter-agent messaging, role-conditioned retrieval, and shared memory.

Fig. 6. Overview of context management methods for short-term memory.

We further divide the progression toward full internalization into two distinct stages: the hybrid paradigm and the fully model-native paradigm. It reflects the difference between learning to operate components within a system versus the model becoming the system itself. The hybrid paradigm represents a transitional stage where pipeline-based architectures are retained, but individual rule-based modules are replaced with learnable components (modular optimization), or the model is trained to execute a predefined external workflow (workflow internalization). In contrast, the fully model-native paradigm eliminates this external scaffolding entirely, internalizing the entire memory management process into a single, unified policy learned end-to-end. The overview of context management methods for short-term memory is illustrated in Fig. 6. More details are summarized in Table 5.

5.3.1 Pipeline-based Paradigm. The pipeline-based paradigm for context management is often referred to as context engineering [125], where information is manipulated externally through rule-based or modular workflows before being passed to the model.

Context Re-structuring. In this technical path, context re-structuring involves the static prearrangement of input. Techniques include chunking and reordering content based on relevance, such as placing critical information at the beginning or end of a prompt to circumvent the “lost-in-themiddle” effect. Structured prompt templates, which assign fixed positions for instructions, examples, and retrieved evidence, are also common, as seen in systems like MemoChat and MemInsight [6, 45]. For specialized domains, hierarchical organization may be used to manage complex evidence and resolve conflicts, as demonstrated by HiAgent and AWM [76, 240].

Retrieval and Routing. When evidence is drawn from heterogeneous sources, retrieval and routing becomes critical. Retrieval systems typically combine sparse and dense methods to ensure both keyword and semantic coverage [86]. A rule-based routing module then selects the most appropriate information based on user intent or knowledge domain. To increase information density, diversity constraints may be applied to reduce redundancy. This approach is exemplified by systems like COPPER, Memary, and COLA, which offer auditable selection policies [84, 219, 256]. Production systems like ZEP and Mem0 further integrate joint sparse-dense retrieval for fast and semantically faithful recall [28, 299]. While controllable and auditable, the rigidity of these rule-based policies limits their adaptability, motivating a shift toward hybrid, learnable approaches.

5.3.2 Hybrid Paradigm. The hybrid paradigm enhances adaptability by replacing fixed rules with learned policies while retaining a modular structure. This approach represents a transitional stage, bridging rule-based pipelines and fully internalized models.

Context Re-structuring. In the hybrid paradigm, context re-structuring evolves from simple information arrangement to dynamic knowledge construction [240]. A representative technique is the use of learned memory slots, where a lightweight policy network learns to distill and write key information into a structured, finite set of slots rather than passively receiving raw text [231, 263, 293]. Another approach uses a fine-tuned model to create associations between disparate pieces of knowledge, effectively synthesizing new, structured evidence. Systems such as EMU, Optimus-1, and Nemori illustrate this synthesis-oriented restructuring [159, 218, 284], which has been shown to significantly improve reasoning performance [28, 263].

Retrieval and Routing. Retrieval and routing in the hybrid paradigm aim to replace fixed rules with learned policies to enhance adaptability. The evolution proceeds mainly along two paths. The first path is modular optimization, which retains the classic pipeline structure but replaces individual rule-based components (e.g., a retriever or re-ranker) with trainable neural modules. This improves performance on specific tasks while maintaining the overall architecture’s interpretability, as exemplified by RAP, SeCom, M+, TWM, RCR-Router, and Learn-to-Memorize [17, 44, 173, 186, 235, 315].

The second path is workflow internalization, towards a deeper integration. The LLM is trained to execute the steps of a predefined workflow, learning to decide when to retrieve, rewrite, or evaluate information. Self-RAG, for instance, introduced reflection tokens that allow the model to trigger retrieval and self-critique its generated content based on the results [7]. Related approaches like TiM, HippoRAG, MemAgent, and Memory-R1 similarly entwine selection and synthesis within the model’s decision-making process, improving adaptability while preserving auditability through explicit step definitions [222, 265, 266, 278].

5.3.3 Model-native Paradigm. The fully model-native paradigm eliminates external scaffolding, internalizing the entire memory management process into a single, unified policy learned end-to-end.

Context Re-structuring. In this paradigm, context re-structuring is deeply integrated with the model’s internal state management. This is often achieved by modifying the attention mechanism itself, for example, through a learnable KV-cache policy that proactively decides which historical key-value pairs to retain, discard, or compress. Systems like Scissorhands and H2O exemplify this dynamic cache management [127, 316]. Another direction couples model parameterization with external memory, allowing memory operations to be tied to model weights while remaining grounded in an auditable store, as seen in MEM1 and H-MEM [206, 327]. In all cases, restructuring moves from an external pre-processing step to an intrinsic, policy-driven behavior during inference.

Retrieval and Routing. The highest level of internalization merges the entire memory process into a unified policy space, where the model learns not only how to reason but also how and when to access and manage its memory. Very recently, MemAct [311] proposes to reframe context management as a learnable, intrinsic capability. In MemAct, the agent actively curates its own working memory by executing explicit editing operations as part of its unified decision-making process, trained via a novel DCPO learning algorithm to balance memory curation with long-term task objectives. This approach contrasts with methods like CARE, which focus on co-adapting retrieval and reasoning without giving the agent explicit control over its memory state [142]. In the fully native paradigm, the retriever and generator become deeply coupled, often sharing parameters and enabling end-to-end optimization. External knowledge repositories remain valuable as a factual foundation for compliance, but the model itself assumes full autonomy over their use, marking the final transition from an assembly of external modules to a system with internally coherent capabilities.

Table 5. Overview of context management methods for short-term memory

Method Category	Method	Tech. Path	Main Stage			Agent Archi.	Affiliation	Access	Date
Method Category	Method	Tech. Path	M	R	U	Agent Archi.	Affiliation	Access	Date
Pipeline-based Paradigm
Prompt Structuring	MemoChat [45]	CR	√			Single	Academia	No	23.08
	MemInsight [6]	CR	√			Single	Industry	No	25.03
	HiAgent[76]	CR	√			Multi	Academia	Yes	24.08
	AWM[240]	CR	√			Single	Academia	Yes	24.10
Rule-based Selection & Retreival	COPPER [219] Memary [84]	RR RR	√	>>>>		Multi Single	Academia Academia	No Yes	24.09 24.10
						Multi	Academia
	COLA [256]	RR						No	25.03
	ZEP[299]	RR				Single	Industry	Yes	25.01
	Mem0 [28]	RR				Single	Industry	Yes	25.04
Hybrid Paradigm
Slot-based Memory	G-MEM [231] A-Mem [263] Intrinsic Memory Agents [293]	CR CR CR	√ √ √			Single Single Multi	Academia Academia Academia	Yes Yes No	24.12 25.02 25.08
Knowledge Synthesis	EMU[159] Optimus-1[284] Nemori [218] RAP[173] SeCom[186]	CR CR CR RR	√		专	Single Single Single Single	Academia Academia Academia Industry Industry	Yes Yes No Yes No	24.03 25.06 25.08 24.02 25.02
Optimization	M+[17] TWM [235] RCR-Router [44]	CR CR RR	<	√		Single Single Multi	Academia Academia Academia	No Yes No	25.02 25.02 25.08
	Learn-to- Memorize [315]	CR	√			Single	Academia/ Industry	Yes	25.08
	Self-RAG [7]	RR		√		Single	Academia	Yes	23.10
	TiM [222] HippoRAG [278]	CR RR		√	√	Single	Industry	No	23.11
	Internalization MemAgent [265]					Single	Academia	Yes	24.05

		Memory-R1[266]	CR			√	Single	Industry	Yes	25.07
			CR			√	Single	Academia	No	25.08

		Scissorhands [127]	Model-native Paradigm	√					No	23.05
CR Single
√
State Management Hybrid		H2O [316] MEM1[327]	CR CR			√	Single Single	Academia Academia	Yes No	23.06 25.06
Parameterization	H-MEM [206]	CR	√			Single	Academia	Yes	25.07
				√
Unified Policy Learning	CARE [142] MemAct [311]	RR RR			√	Single Single	Academia Academia	No No	25.10 25.10

5.4 Long-term Memory

Long-term memory refers to persistent knowledge and experience maintained across sessions or tasks. Its contents can be held in two primary types of carriers: external repositories, such as documents, vector stores, or graphs, and the model’s internal parameters, including adapter modules or the weights of the model itself [39, 41, 170, 175, 196, 234, 254, 323].

Key properties of long-term memory include low write frequency and strong traceability, which support compliance and versioning. The content is typically not editable online during a task; instead, any online optimization focuses on retrieval and utilization strategies. This paradigm provides a factual basis for an agent’s planning and tool use but can face challenges with crosssource consistency and information timeliness. Storing knowledge directly in model parameters can yield lower latency and deeper integration, but it often comes at the cost of reduced interpretability and an increased risk of interference between stored facts [319]. An overview of methods for long-term memory is summarized in Table 6.

5.4.1 External Repository as Memory Carrier.

Retrieval-Augmented Generation. When using external repositories, the emphasis shifts from how to store information to how to retrieve it precisely and use it reliably [12, 78]. A common practice is to build hybrid indexes offline by combining inverted indices, dense vectors, and metadata. During inference, multi-path retrieval and re-ranking are used to select and compress candidate evidence. This process is often accompanied by deduplication and consistency checks to ensure the injected evidence is both relevant and traceable [170, 323]. While this approach is robust for tasks requiring deep knowledge from large corpora, injecting excessive or poorly selected information can dilute the model’s attention. Consequently, recent methods have moved from single-path semantic retrieval toward more complex pipelines that incorporate learned re-ranking and delegate decisions about when and what to retrieve to learned policies [41].

Structure and Compressed Summarization. Because long-term knowledge sources are often lengthy and heterogeneous, simply expanding the context window is not always effective. Layered summarization and structuring are therefore used to manage information density. Documents can be segmented into structured evidence cards with explicit citations and timestamps for verifiability. This structured layer reduces noise and stabilizes the candidate pool for retrieval systems [170, 175]. The focus has shifted from generating human-readable text summaries to creating machine-usable evidence objects that integrate tightly with retrieval and routing strategies [41, 78].

5.4.2 Model Parameters as Memory Carrier.

Global Parameter Internalization. For knowledge that is stable and frequently reused, writing it directly into the model’s parameters can reduce inference latency and dependence on external stores. In practice, this is often achieved through continual pretraining or instruction distillation, where domain-specific knowledge, API schemas, or reasoning patterns are integrated into the training data. Replay and regularization techniques are used to mitigate catastrophic forgetting. This allows the internalized knowledge to serve as a fallback when external retrieval fails or returns incomplete information [12, 78, 319].

Table 6. Overview of long-term memory methods

Carrier	Method	Content Structure	Memory Update	Retrieval Strategy	Affiliation	Access	Date
ExternalRepository	RETRO [12]	Text+Vector	Static	Chunk Retrieval+Cross-attention	Industry	No	21.12
	Atlas [78]	Text	Static	Embedding KNN	Industry	No	22.08
	GenerativeAgents [175]	StructuredText	Rule-basedSummarization	Scoring	Academia	No	23.04
	Synapse [323]	TrajectoryExemplars	Offline curation	Embedding KNN	Academic	No	23.06
	MemGPT[170]	Text +Vector	Rule-basedSummarization	Embedding KNN+ Scoring	Academia	Yes	23.10
	GraphRAG [41]	KnowledgeGraph	Rule-basedSummarization	Graph Retrieval	Industry	Yes	24.04
	LMLM [319]	Text	Static	Instruction-Trigger	Academia	No	25.05
Model Parameters	K-Adapter [234]	Adapters /LoRA Bank	OfflineTraining	Rule-based+Router	Academia	Yes	21.08
	MemorizingTransformers [254]	kNN Cache+KV	Online KVwrites	Embedding KNN	Academia	Yes	22.03
	S-LoRA [196]	Adapters /LoRA Bank	OfflineTraining	Rule-based +Router	Academia	No	23.11
	MixLoRA [98]	Adapters/LoRA Bank	JointFine-Tuning	Router (top k)	Academia	Yes	24.04
	ELDER [101]	Adapters /LoRA Bank	AutomaticOfflineTraining	Rule-based +Router	Academia	No	24.08
	MoM[39]	Linear StateLayers	AutomaticOffline Traning	Router	Academia	Yes	25.02

Targeted Parameter Intervention. When a small number of critical facts need to be corrected or updated, model editing offers a more targeted approach. Representative methods apply lowrank adjustments to specific layers or intermediate representations, allowing for the injection or replacement of target knowledge with minimal disruption to the model’s overall capabilities [151, 152, 154]. This method is effective for sparse, important factual corrections but is not suitable for large-scale knowledge migration. In practice, this has evolved from single edits to batch pipelines that can be triggered by upstream changes in an authoritative source, enabling near real-time corrections [154].

Lightweight Parameter Injection. To balance the high cost of global internalization with the limited capacity of model editing, lightweight modules can be used. This strategy involves freezing the base model and training or attaching small, specialized components that carry specific knowledge. LoRA-style adapters and adapter banks can encode user preferences or task-specific skills in compact modules, while linear state layers can provide additional internal memory that is routed on a per-input basis [39, 98, 101, 196, 234]. This approach is computationally efficient and allows for per-task or per-domain customization.

5.5 Summary and Discussion

The evolution of agent memory also follows the paradigm shift from externally orchestrated modules toward an intrinsic, unified capability. This trend reframes memory from a simple information storage mechanism into a system for action-oriented evidence governance, responsible for preserving state, retrieving information, and injecting it into the agent’s reasoning process. This shift from pipeline-based to model-native solutions is particularly evident in short-term memory.

Short-term memory was initially developed to compensate for models’ limited context processing abilities. Early pipeline-based approaches, such as MemoChat [45], used structured templates and decomposable modules to filter and arrange evidence before it reached the model. While interpretable and easy to implement, these systems were limited by cross-module error accumulation and inflexibility, casting the model as a passive participant. As native context windows have expanded, the focus has shifted from mere capacity to effective context management. The challenge is no longer how much information can be processed, but how well the model can filter noise, locate critical evidence, and utilize it for complex reasoning. This has driven the evolution from coarse-grained pipelines to more intelligent, learned policies.

This transition is not an abrupt replacement but a gradual process of internalizing validated pipeline functions into the model’s native abilities. A key bottleneck in this path lies in longsequence data synthesis and curriculum design. Future breakthroughs will likely involve training models to endogenously learn operations like retrieval, compression, and verification as explicit objectives [12, 78, 319]. As models learn attention mechanisms akin to a retriever and resource allocation strategies similar to a router [44], external modules like vector stores will recede from online decision-making. Their role will shift to serving as backend infrastructure for compliance, persistent storage, and offline content preparation, supporting the model’s native governance capabilities.

Long-term memory began primarily as external repositories, with engineering practices emphasizing traceability, control, and governance. Retrieval relied on cascaded re-ranking [86], but this design faced two key limitations: the content was not rewritable online, and there was often a mismatch between retrieved relevance and actual utility in the reasoning chain [7]. A key recent trend is personalization, where long-term memory evolves from a unified knowledge warehouse into a dynamic, user-specific asset composed of traceable evidence cards [170, 175]. A second trend is the deepening coupling with short-term memory. Although most long-term stores are not yet fully model-native due to their offline nature, methods are emerging that blur this boundary. This includes co-training the retriever and generator to improve evidence utilization [78, 309], using online updates to write back critical facts, and employing lightweight adapters to inject knowledge into model parameters [234]. Systems like Mem0 [28] sit at this intersection, combining the governability of external memory with the fast read-write capabilities needed for session-level adaptation, pointing toward an eventual unification of long-term and short-term memory systems. In summary, several key trends are shaping the future of memory-enhanced agents:

• Benchmark evolution: evaluation will move beyond raw capacity toward stringent reasoning benchmarks that stress-test long-range dependency, interference robustness, and logical closure.
• Capability internalization and unification: the internalization of capabilities will continue, with the progressive unification of short-term and long-term memory into model-native skills and the default co-training of retrieval and generation components.
• Personalization with stronger governance: long-term memory will become highly personalized and dynamic, increasing need for stronger governance, privacy controls, and user-facing correction to ensure consistency and reliability.

6 Applications

6.1 Deep Research Agent

6.1.1 Overview. Recent progress in search technologies has extended model capabilities beyond their internal parameterized knowledge, which is often outdated and unreliable. The RAG paradigm [97] enables direct access to external knowledge bases, providing accurate and up-to-date information. Building upon the RAG framework, Deep Research agents employ adaptive search strategies that control search iterations and step depth to retrieve relevant evidence and generate definitive answers or generative synthesis. Depending on whether the search mechanism is embedded within model parameters, the paradigms of Deep Research agents can be categorized into two complementary yet closely related forms: the pipeline-based and the model-native approaches.

The pipeline-based paradigm provides controllable and interpretable workflows, supporting debugging and updates at any stage, which makes it well-suited for industrial deployment. However, it depends on predefined, prompt-driven procedures that demand substantial manual design. The performance of such systems is constrained by the model’s instruction-following capability, longhorizon reasoning stability, and sensitivity to prompt formulation, making them less reliable for complex reasoning tasks.

In contrast, the core idea of model-native methods is to internalize search strategies into model parameters through reinforcement learning, enabling end-to-end autonomous research capability. This paradigm enhances long-horizon reasoning consistency and allows the model to dynamically adapt to diverse search contexts. From the perspective of reasoning consistency, internalizing strategies at the parametric level leverages stored knowledge to preserve goal and evidence coherence across multiple turns [202], thereby improving reasoning stability and reducing phenomena such as goal drift, lost-in-conversation, and information forgetting in extended prompt chains [93, 122, 239]. With respect to adaptability, the model dynamically adjusts retrieval depth and evidence integration according to task complexity, without relying on predefined workflows, thereby reducing unnecessary tool calls, redundant information retrieval, and overall token costs [82]. By shifting from external orchestration to internalized capability, this paradigm ensures stable reasoning across diverse environments and task structures, enhancing the generalization of Deep Research agents to open-domain, multi-hop, and unstructured evidence tasks.

The pipeline-based and model-native approaches differ fundamentally in implementation but converge in their evolution toward more autonomous, efficient, and adaptive research processes, exhibiting consistent trends across key dimensions such as information acquisition, task objectives, base model design, and agent architecture. The remainder of this paper is organized as follows: Section 6.1.2 introduces the pipeline-based paradigm; Section 6.1.3 presents the model-native paradigm; and Section 6.1.4 compares their evolutionary patterns and discusses future directions of Deep Research agents.

6.1.2 Pipeline-based Paradigm . Pipeline-based methods enable retrieval-augmented knowledge integration and multi-turn reasoning through carefully designed prompt templates and orchestrated tool chains. Two main implementation paths exist: (1) single-turn retrieval and synthesis, exemplified by RAG [97], which optimizes the “retrieve–integrate–generate” pipeline; and (2) multiturn reasoning–action cycles, exemplified by ReAct [277], which integrates retrieval into iterative reasoning, allowing models to decide when and what to retrieve through a “think–retrieve–rethink” process.

Single-turn Retrieval and Synthesis. Single-turn retrieval and synthesis methods perform one-shot retrieval and generation, introducing external knowledge into LLM reasoning to address the limitations of static, parameterized memory. These methods typically follow a two-stage process:

Fig. 7. Overview of deep research agent methods.

query enhancement and evidence processing. During the first phase, the model or its external components construct or rewrite task-specific queries to retrieve candidate evidence from search engines, vector databases, or knowledge bases. Subsequently, the retrieved content is filtered, compressed, and structurally organized to preserve essential and relevant context, which is then injected into the model’s context for answer synthesis or report generation [85]. The two stages are elaborated in detail below.

• Query enhancement. Query enhancement improves retrieval effectiveness by rewriting, expanding, or constructing task-specific queries [236], thereby increasing retrieval quality and answerability. Query2doc [232] prompts an LLM in a few-shot manner to generate pseudo-documents, which are concatenated with the original query for search, effectively expanding query scope. Ma et al. [143] introduce a query rewriting framework for RAG, where an LLM or a lightweight model first rewrites the query, retrieves results via the Bing API, and then passes the retrieved content to a reader model for final generation. FreshLLMs [221] focus on time-sensitive knowledge by combining multiple evidence sources with structured metadata, and guide attention according to temporal order. Jin et al. [83] present BIDER, which improves retrieval coverage and matching accuracy through bidirectional expansion of both documents and queries from the index side.

• Evidence processing. To ensure that retrieved information is relevant, concise, and structurally coherent before being injected into the model context, evidence processing refines and organizes raw retrieval outputs. Xu et al. [261] propose RECOMP, which introduces a compression layer prior to injection, combining extractive and abstractive compressors that can selectively omit uninformative content. Jiang et al. [80] develop LongLLMLingua, applying global reranking and span-level extraction to mitigate context explosion. Vu et al. [221] normalize search results into structured evidence lists for more consistent integration. Li et al. [111] design CorpusLM, which treats a unified corpus as fixed external memory, retrieving fragments online and directly reading from a clean repository during inference. Li et al. [113] present RetroLLM, converting web pages into Markdown to build writable memory, enabling continuous accumulation and reuse of retrieved evidence. Edge et al. [42] proposed GraphRAG that replace scattered documents with knowledge graphs and community summaries, supporting path-level interpretability and global evidence aggregation.

Multi-turn Reasoning-Action . To overcome the limitations of single turn retrieval and synthesis, including insufficient evidence, hallucinations, and error propagation, multi turn reasoning and action methods introduce iterative retrieval within a closed reasoning loop. Yao et al. [277] propose ReAct, a representative framework that interleaves reasoning and acting, forming a feedback process where reasoning guides actions and observations refine subsequent reasoning. This paradigm enhances reasoning reliability through two key mechanisms: the orchestration of reasoning and retrieval, which enables iterative evidence acquisition, and context management with evidence refinement, which maintains focus and reduces noise accumulation. Building on this foundation, recent studies have advanced these two mechanisms through various designs and optimization strategies, as detailed below.

• Orchestration of reasoning and retrieval. This line of work focuses on coordinating reasoning and retrieval to iteratively accumulate relevant evidence. He et al. [72], Gur et al. [65], and Alzubi et al. [5] develop agents that solve tasks step by step, repeatedly invoking search tools and feeding textual observations or web snapshots back into the reasoning process. Li et al. [110] propose Search o1, which employs a large reasoning model without handcrafted templates and introduces a Reason in Documents mechanism that orchestrates pause, retrieve, and resume operations for retrieval augmented answering.

• Context management and evidence refinement. Managing large amounts of retrieved information is essential to preserve reasoning coherence and reduce redundancy. He et al. [72] prune historical context, retaining only the three most recent observations and the complete reasoning trace to maintain recency and relevance. Gur et al. [65] train HTML T5 to handle long HTML documents using hybrid local and global attention with long span denoising pretraining, extending input length and context window during fine tuning. Li et al. [110] design an in document reasoning and refinement module that compresses retrieved content and reinjects only the most relevant knowledge. Alzubi et al. [5] adopt chunking and reranking, segmenting web pages into paragraph level chunks and keeping only those above a relevance threshold. Wu et al. [253] mitigate information overload through periodic context summarization, which removes redundant content while preserving key evidence, enabling continuous exploration.

6.1.3 Model-native Paradigm . The model-native paradigm redefines how Deep Research agents perform reasoning and tool use. Instead of relying on externally orchestrated workflows such as step-by-step prompting in ReAct, model-native agents internalize planning and retrieval within their parameters. This integration allows end-to-end, proactive, and dynamically adaptive information seeking, where the model itself acts as both planner and executor. Depending on the interaction environment during training, model-native agents can be developed through offline or online training.

Offline Training . Offline training conducts external information retrieval within closed, locally hosted databases that operate without internet access. This setup provides a stable and controllable training environment, allowing the model to learn retrieval and reasoning behaviors under low noise and fixed data conditions. Representative works such as Search-R1 [82], R1- Searcher [201], ReSearch [22], R-Search [320], and R1-Searcher++ [202] adopt Wikipedia as their static retrieval corpus, enabling supervised or reinforcement learning over consistent textual contexts. Similarly, M2IO-R1 [259] employs a pre-constructed dataset (M2IO-Inserter) to simulate information insertion and retrieval tasks in a controlled manner. Such designs ensure reproducibility and avoid issues commonly encountered in online environments, including network instability, API rate limitations, and unpredictable web responses.

However, the static and bounded nature of local datasets constrains the scope of knowledge and limits exposure to real-time information. As a result, models trained exclusively offline may struggle with information obsolescence and reduced generalization in open environments. To overcome these limitations, ZeroSearch [205] introduces a simulated search engine that uses an LLM-generated document space to emulate online retrieval. This approach maintains controllable training dynamics while approximating the variability of real web search, achieving a balance between data freshness, quality, and stability.

Online Training . Online training situates model learning within real-world, connected environments, allowing agents to retrieve and reason over dynamic and continuously updated information sources. This paradigm provides direct access to the open web, enabling models to acquire real-time knowledge and effectively extend their informational coverage beyond static corpora. DeepResearcher [324] demonstrates large-scale reinforcement learning in live web environments, where agents interact directly with search engines and webpages during training. Following this direction, WebThinker [112], WebWatcher [54], SFR-DeepResearch [161], MMSearch-R1 [247], and DeepDive [132] perform end-to-end reinforcement learning under realistic web settings, where model actions directly determine query formulation, browsing behaviors, and document reasoning.

Although online training captures the complexity of open-world information retrieval, it also introduces significant engineering and optimization challenges. Factors such as network fluctuations, page latency, API quotas, anti-crawling mechanisms, and advertisement noise can disrupt the learning signal and escalate computational costs. Building robust and scalable online training pipelines therefore requires careful system design, adaptive control strategies, and efficient reward shaping to ensure stability and performance in dynamic environments.

6.1.4 Summary and Discussion. The preceding subsections have separately introduced the implementation mechanisms and representative works of the pipeline-based and model-native paradigms. In this subsection, we synthesize these two trajectories from a broader evolutionary perspective. Specifically, we analyze four converging dimensions that characterize the ongoing transformation of the Deep Research paradigm: (1) information acquisition methods diversifying across API-based and browser-based approaches; (2) task objectives expanding from deterministic question answering to open-ended generative synthesis; (3) base models evolving from LLMs to LRMs and further to LMMs (Large Multimodal Models; and (4) agent architectures shifting from single-agent coordination to multi-agent collaborative systems. An overview of deep research agents is summarized in Table 7.

Information Acquisition . Information acquisition denotes the use of external tools to obtain information from the environment during the reasoning and answering process of Deep Research Agents. According to the mode of interaction, existing approaches can be broadly categorized into API-based and browser-based methods.

• API-based methods. These methods acquire external information through APIs that interface with either static, local databases or live, third-party search engines.

A significant line of work focuses on interacting with live search engines to access realtime, open-domain information. For example, systems like Search-o1 [110], ODS [5], and ReSum [253] utilize APIs from commercial search engines like Bing or Google, often in combination with web page parsing tools like Jina Reader. This approach has also been extended to multimodal search: MMSearch-R1 [247] and WebWatcher [54] employ SerpAPI for both image and text retrieval. To handle more specialized information needs, WebResearcher [184] integrates multiple APIs, including Google Search and Google Scholar, and uses a dedicated summarization model to process the results.

Table 7. An overview of deep research agent methods, with their task objective (Def. for definitive answering and Gen. for generative synthesis), base model, information acquisition method (Info.), agent architecture (Arch.) and among other attributes.

	Method	Objective	Model Info.		Arch.	Affiliation	Access Date
Pipeline-based Paradigm
Single-turnRetrieval&Synthesis	Query2doc [232]Query Rewriting [143]	Def.	LLM	API
					Single
					Single	Industry	Yes		23.03
		Def.	LLM	API	Single	Academia	Yes		23.05
	RECOMP [261]LongLLMLingua [80]FreshLLMs [221]	Def.	LLM	API	Single	Academia	Yes		23.10
		Def.	LLM	API	Single	Industry	Yes		23.10
		Def.	LLM	API	Single	Academia	Yes		23.10
	BIDER [83]CorpusLM[111]	Def.	LLM	API	Single	Academia	No		24.02
	BIDER [83]CorpusLM[111]	Def.	LLM	API	Single	Academia	No		24.02
	GraphRAG 42]RetroLLM[113]	Gen.Def.	LLM	API	Single	Industry	Yes		24.04
	GraphRAG 42]RetroLLM[113]	Gen.Def.	LLM	API	Single	Academia	Yes		24.12
Multi-turnReasoning-Action	Real-World WebAgent [65]WebVoyager [72]Search-o1[110]	Def.	LLM	Browser	Multi	Academia	Yes		23.07
		Def.Def.	LMM	Browser	Single	Academia	Yes		24.01
		Def.Def.	LRM	API	Single	Academia	Yes	Yes		25.01
	ODS [5]ReSum [253]	Def.Def.+Gen.	LRMLRM	API	Single	Academia	YesYes			25.03
	ODS [5]ReSum [253]	Def.Def.+Gen.	LRMLRM	API	Multi	Industry	YesYes		25.09
Model-Native-Based Paradigm
OfflineTraining	R1-Searcher [201]	Def.	LLM	API	Single	Academia	Yes		25.03
	Search-R1 [82]	Def.	LLM	API	Single	Academia	Yes		25.03
	ReSearch [22]R1-Searcher++ [202]R-Search [320]M2IO-R1[259]	Def.Def.Def.Def.	LLM	API	Single	Academia	No		25.03
			LLM	API	Multi	Academia	Yes		25.05
			LLM	API	Single	Academia	Yes		25.06
			LMM	API	Single	Academia	Yes		25.08
OnlineTraining	DeepResearcher [324]WebThinker [112]ZeroSearch [205]MMSearch-R1[247]WebWatcher [54]	Def.Def.Def.	LLMLRM+LLM	API+Browser	Multi	Academia	YesYes		25.0425.04
			LLMLRM+LLM	API+Browser	Multi	Academia	YesYes		25.0425.04
			LLM	API	Multi	Academia	Yes		25.05
		Def.	LMM	API	Single	Industry	Yes		25.06
		Def.	LMM	API	Single	Industry	Yes		25.09
	SFR-DeepResearch [161]	Gen.	LRM	API	Single	Industry	No		25.09
	DeepDive [132]WebResearcher [184]	Def.	LRM	API+Browser	Single	Academia	YesYes		25.0925.09
	DeepDive [132]WebResearcher [184]	Gen.	LRM	API+Browser	Multi	Industry	YesYes		25.0925.09

Another line of research prioritizes reproducibility and training stability by using static databases, such as a Wikipedia dump, as the external knowledge source. This controlled environment allows for more rigorous evaluation of an agent’s reasoning and retrieval strategies without the noise and non-stationarity of the open web. Representative works in this category include Search-R1 [82], ReSearch [22], R-Search [320], and the R1-Searcher series [201, 202]. M2IO-R1 [259] further adapts this approach for multimodal scenarios by retrieving from a pre-constructed database of text-image pairs. To bridge the gap between static and live environments, ZeroSearch [205] introduces a novel training strategy where an LLM is used to simulate a search engine, generating both relevant and noisy documents. This allows the agent to be trained with curriculum-based difficulty, improving robustness while avoiding the high cost and instability of live API calls.

• Browser-based methods. Complementary to API-based approaches, browser-based methods enable more human-like web interaction through automated browser control, supporting dynamic page rendering, element interaction, and the handling of complex web scenarios. Related methods can be distinguished by their level of abstraction.

Some works focus on the low-level mechanics of direct browser interaction. For example, WebVoyager [72] renders HTML into visual web pages and interacts with elements using a combination of textual information and screenshots. This allows it to handle dynamic elements like floating advertisements, thereby improving robustness in real-world web environments.

Other approaches operate at a higher level of abstraction, focusing on information and workflow management. DeepResearcher [324] invokes a dedicated browsing agent to sequentially read and extract relevant information, managing a short-term memory to condense the information before returning it to the main agent. Similarly, WebThinker [112] uses an auxiliary LLM to summarize crawled pages based on the agent’s click intent, and in its report generation mode, stores explored pages in a document memory for later retrieval. Finally, some frameworks represent a hybrid approach; DeepDive [132], for instance, abstracts browser interactions into a set of discrete function calls like (??????????ℎ, ??????????, ????????), combining the direct control of browser-based methods with the structured nature of API-based invocation.

Task Objectives . According to the goal of information research, Deep Research agents can be broadly divided into two categories: definitive answering and generative synthesis.

• Definitive answering. This category focuses on concise, verifiable factual questions that emphasize rapid information localization and multi-hop verification.Representative works such as WebVoyager [72], Real-World WebAgent [65], Search-o1 [110], ODS [5], and SearchR1 [82] effectively employ single-hop or multi-hop search strategies to locate precise answers to user queries.

• Generative synthesis. This category addresses complex, open-ended tasks that require integrating multiple sources of evidence and constructing coherent arguments around thematic questions. Such works often employ multimodal or structured representations—such as tables and images—to enhance information density and presentation quality. WebThinker [112] generates research-style reports incorporating Markdown-formatted tables for structured summarization. M2IO-R1 [259] produces interleaved text–image outputs, combining retrieved multimodal content with contextual reasoning. SFR-DeepResearch [161] supports long-form report generation, synthesizing retrieved knowledge into extended, well-organized documents.

Base Models . Across both pipeline-based and model-native paradigms, Deep Research agents have evolved in base model design, from LLMs to LRMs and further to LMMs. Pipeline-based methods rely on LLMs or LRMs guided by explicit prompting frameworks (e.g., ReAct), while modelnative approaches use reinforcement learning to internalize reasoning and retrieval. With the emergence of reasoning-optimized LRMs such as OpenAI-o1 [166] and DeepSeek-R1 [38], models increasingly integrate tool use within their architecture. To handle multimodal environments, LMMs extend these capabilities to images, webpages, and videos, enabling richer evidence acquisition.

• LLM. Search-R1 [82], R1-Searcher [201], ReSearch [22], R-Search [320] utilize a single LLM that, after reinforcement learning for deep research work, becomes an LRM that internalizes tool usage mastery, responsible for completing the entire process of reasoning, generating search queries, understanding retrieval results, and generating answers. • LRM. Search-o1 [110] enables the model to autonomously decide when to retrieve and refine documents into compact “knowledge steps” that support current reasoning. WebThinker [112] performs end-to-end reasoning, invoking tools such as search, navigation, and report generation within its reasoning chain. SFR-DeepResearch [161] and DeepDive [132] adopt multiple reasoning-optimized LRMs as their core base models. • LMM. Multimodal extensions further enhance Deep Research capabilities. WebVoyager [72] defaults to gpt-4-vision-preview model. MMSearch-R1 [247] uses three types of LMMs. Qwen2.5-VL-7B-Instruct serves as the policy model, learning to execute on-demand search through RL training. Qwen3-32B serves as the webpage summarization model for content summarization in text search pipelines. GPT-4o serves as the judge model for evaluating model response accuracy. WebWatcher [54] uses two types of LMMs. Qwen2.5-VL-7B and Qwen2.5-VL-32B serve as policy models trained through SFT and RL. GPT-4o serves as an auxiliary model for data generation, trajectory annotation, and evaluation. M2IO-R1 [259] uses Qwen2.5-VL series for image insertion tasks, with GPT-4o for text answer generation and auxiliary data construction.

Agent Architectures . Whether in pipeline-based or model-native paradigms, Deep Research Agents have developed along two architectural lines: single-agent and multi-agent. Single-agent systems integrate planning, retrieval, reasoning, and generation within one model, featuring simplicity, efficiency, and low cost. In contrast, multi-agent architectures assign specialized roles such as planning, retrieval, summarization, and writing to multiple collaborating models, improving scalability, robustness, and adaptability for complex, long-horizon, or multimodal tasks.

• Single-agent architecture. Single-agent architectures unify planning, retrieval, comprehension, and generation within one model, achieving end-to-end execution with minimal computational overhead and a shorter training pipeline. Search-R1 [82], R1-Searcher [201], and ReSearch [22] follow this pattern, using a single LLM trained through reinforcement learning to perform reasoning, generate search queries, interpret retrieved content, and produce final answers. Such an approach is well suited to factual or task-specific applications. • Multi-agent architecture. Compared with single agents that are limited by context capacity and role overload in complex reasoning, multi-agent architectures distribute specialized roles such as planning, retrieval, summarization, and writing to enhance scalability, robustness, and overall performance. Real-World WebAgent [65] separates planning and summarization using HTML-T5, while Flan-U-PaLM generates executable programs, improving coordination and efficiency. DeepResearcher [324] employs a Web Browsing Agent for segmented retrieval and summarization, reducing context load on the main agent while preserving essential information for reasoning. ReSum [253] integrates a reasoning policy model with a summarization tool that compresses historical context when limits are reached, enabling continuous exploration. WebThinker [112] assigns its main LRM to reasoning and planning, and an auxiliary LLM to search intent generation, content summarization, and report composition, improving logical coherence. R1-Searcher++ [202] adopts dual-model collaboration during reinforcement learning, where a rewrite model transforms retrieved documents into internal reasoning paths, allowing the policy model to internalize external knowledge and reduce retrieval frequency without compromising accuracy.

Reward Design . Beyond the common trends shared by both pipeline-based and modelnative paradigms, reward design in reinforcement learning is a mechanism unique to the latter. It determines whether models can internalize retrieval and reasoning strategies—when to search, how to filter evidence, and how to generate reliable results. Through fine-grained reward shaping, model-native approaches transform externally guided behaviors into self-regulated reasoning capabilities. Existing studies mainly explore three types of reward signals: outcome, format, and process rewards.

• Outcome rewards. Outcome rewards assess only the final result, encouraging correct outputs without considering intermediate reasoning quality. DeepSeek-R1 [38] demonstrates that outcome rewards alone can effectively guide reasoning. Similarly, Search-R1 [82], ZeroSearch [205], and SFR-DeepResearch [161] adopt simple outcome-based supervision.

• Format rewards. Format rewards ensure structural correctness within reasoning chains, aligning thinking steps, tool calls, and outputs. R1-Searcher [201], ReSearch [22], DeepResearcher [324], WebWatcher [54], and M2IO-R1 [259] constrain models to wrap reasoning, retrieval, and final answers within predefined tags. DeepDive [132] further applies a strict binary reward that requires both correct format and correct answer simultaneously.

• Process rewards. Process rewards provide stepwise feedback, guiding models to balance reasoning depth, efficiency, and traceability. R1-Searcher++ [202] combines format, outcome, and group rewards, encouraging accurate answers with minimal retrievals. R-Search [320] extends this approach with multi-stage reward mechanisms to jointly optimize reasoning quality and resource efficiency.

Future Directions . Building on the evolutionary framework of Deep Research agents, we identify key challenges and outline future research trajectories. These directions collectively aim to advance agents from instruction-following tools to intelligent research partners that understand problems, explore knowledge boundaries, and co-evolve with humans.

• Information acquisition. Agents should dynamically choose acquisition strategies—using APIs for structured, time-sensitive queries and browsers for interactive, multimodal scenarios. Emerging AI-native browsers (e.g., Browserbase, Browser Use, Perplexity’s Comet) are narrowing this gap by providing stable DOM access, asynchronous execution, and automated anti-scraping, bringing browser-based methods closer to API-level reliability.

• Task objectives. Future research may progress in three directions: (1) Interactive research with user feedback loops to refine intent and deepen exploration; (2) Continuous “Living Research” for persistent topic tracking and periodic incremental reporting; (3) Predictive analysis that anticipates future trends and potential breakthroughs from historical data.

• Agent architecture. Agents should assess task complexity, evaluate interdependencies, and autonomously decide between serial or parallel collaboration. This can be achieved by modeling workflows as mutable graphs and optimizing them through evolutionary or adaptive algorithms.

• Reward design. Incorporating curiosity-driven and self-reward mechanisms [176, 289] can encourage exploration and self-assessment. Intrinsic rewards promote uncertainty reduction, while self-evaluation reduces reliance on external supervision.

Fig. 8. Overview of GUI agent methods.

6.2 GUI Agent

6.2.1 Overview. Graphical User Interface (GUI) agents refer to intelligent entities capable of autonomously perceiving, planning, and executing tasks in GUI environments. Their development also reflects a clear evolution from an external, pipeline-based paradigm to an internalized, modelnative paradigm.

Early GUI automation relied on a pipeline-based paradigm, initially driven by system-based workflows like record-and-replay [9, 64] or rule-based scripts [2, 108]. While stable for repetitive tasks, the hard-coded system workflow lacked the necessary understanding of interface semantics and user intent. The advent of LLMs introduced prompt-based methods, replacing rigid scripts with natural language instructions and enabling agents like AppAgent [300] and Mobile-Agent [229] to exhibit a preliminary degree of autonomy. However, these agents still operated within manually orchestrated frameworks, invoking LLMs as external tools that were not specifically optimized for the nuances of GUI interaction, thus failing to fully leverage their inherent potential.

To overcome these limitations, current research is progressively moving towards the model-native paradigm, which internalizes the core capabilities of a GUI agent within the model’s parameters. This transition has progressed along a path of increasing integration, beginning with modular training. This approach solved critical bottlenecks by individually training components, such as the perception-focused models in UGround [59] and Aria-UI [273], or by fusing modules as seen in CogAgent [75]. This evolutionary path is ultimately leading toward end-to-end training, where one single model learns the direct mapping from perceptual inputs to executable actions. This fully integrated approach, demonstrated by recent methods like UI-TARS [185] and GUI-owl [279] significantly enhances an agent’s autonomy and generalization, allowing it to move beyond its dependency on external frameworks. An overview of GUI Agent methods is illustrated in Fig. 8.

6.2.2 Pipeline-based Paradigm. In their early development, GUI Agents primarily relied on manually orchestrated pipelines that decomposed tasks into fixed modules linked by explicit logic. Within this framework, models were assigned only limited functions at specific steps of the process. Based on the autonomy of the agent’s decision-making, the pipeline paradigm can be further divided into two sub-stages: system-based and prompt-based. The former, corresponding to the traditional Robotic Process Automation (RPA) paradigm, relies on record-and-playback or rule-based scripts to replay a fixed sequence of operations. The latter utilizes prompts to drive LLMs to autonomously handle functions such as task understanding, planning, and tool calling to complete a GUI task.

System-based Workflows. In the early development of GUI Agents, models did not yet have the ability to perceive interface states or make independent decisions. Consequently, automation relied on system-level orchestration to reproduce human operations in fixed environments, aiming for greater efficiency and repeatability. According to the control method, this stage can be divided into two types: record-and-replay and rule-based scripting automation.

• Record-and-replay. This class of methods achieves task automation by recording a user’s action sequence (e.g. clicks and inputs) and replaying it verbatim within an identical environment. For instance, CoScripter[96] first proposed a method for users to record actions directly within a web browser, which could then be replayed as scripts to enable the automation and sharing of web-based workflows. Subsequently, RERAN[57] extended this concept to mobile platforms, achieving high-precision touch replay by capturing low-level Linux input events.

To improve robustness, subsequent research introduced more intelligent matching and waiting mechanisms. Ringer[9], for example, enhanced replay success rates across different website versions by recording multi-dimensional attributes of UI elements and calculating their similarity during the replay phase. It could also automatically insert wait conditions to handle asynchronous scenarios. In the mobile domain, SARA[64] introduced adaptive replay and context-capturing mechanisms, enabling recorded scripts to be executed across heterogeneous devices.

Although these improvements enhanced the environmental adaptability of the scripts, this approach still depended on fixed environments and specific examples. Consequently, it suffered from poor generalization and was unable to support open-ended tasks.

• Rule-based scripting. To overcome the brittleness of record-and-replay methods and to support more complex and stable automation, the research community shifted towards rulebased scripting. This approach involves controlling the operational logic of an agent through explicit rules or conditional statements. For example, Chickenfoot[11] pioneered the use of natural language-like JavaScript to manipulate web elements, which enabled users to control tasks through semantic commands such as “ click(login)”. Subsequently, Sikuli[282] advanced this approach by using screenshot matching instead of coordinate-based positioning, making scripts more robust against layout changes.

As this technology progressed towards industrial applications, the research focus shifted to automating and enhancing the intelligence of rule-based scripts. SmartRPA[2], for instance, proposed automatically synthesizing rule-based scripts from user interface logs to reduce manual effort. Meanwhile, Sugilite[108] explored a collaborative human-computer programming paradigm that combined voice and demonstration, allowing end-users to generate conditional scripts from natural language instructions.

While rule-based scripting offered greater flexibility and a degree of logical control compared to record-and-replay, it was still critically constrained by their lack of perceptual and semantic understanding capabilities.

In summary, the system-based workflow is defined by its reliance on deterministic control through either explicit rules or direct action replay. This approach offered high stability and interpretability but was ultimately limited by its lack of semantic understanding and its inability to generalize to novel tasks.

Prompt-based Methods. With the advent of LLMs, the research focus for GUI Agents shifted from scripted execution to language-driven decision generation. This prompt-based stage leverages the in-context learning capabilities of a pre-trained LLM, using prompts to guide the model in understanding task instructions, parsing interface states, and generating action plans without requiring additional training. This transition enabled agents to adapt to multi-platform GUI environments using a unified language interface, thereby achieving a preliminary form of generality. As work in this area progressed to enhance the coherence and reliability of planning, four primary developmental paths emerged:

• Single-step reactive agents. Early prompt-based GUI agents often adopted a single-step reactive approach, where the agent generates and executes a single action immediately following each observation of the interface. The ReAct framework [277] pioneered this by having an LLM generate a “Thought” and an “Action” in each round, enabling zero-shot completion of tasks in text-based environments like ALFWorld. Building on this, AutoGUI[317] constructed a multimodal context that included action history to inform its singlestep decisions. This concept was further adapted to mobile platforms by MobileAgent [229], which concatenates screenshots, OCR information, and user instructions into a multimodal prompt. This process allows the agent to produce short-term reasoning and an action at each step, achieving low-latency reactive operations.

However, a significant limitation of purely reactive methods is their lack of a global plan, which can lead to inefficient looping behaviors or failure to address the overall task objective.

• Hierarchical planning agents. To address the limitations of purely reactive methods, researchers introduced the hierarchical planning paradigm. In this paradigm, the agent first generates a high-level, global plan and then executes it, with the ability to dynamically modify sub-plans. For instance, WebAgent [66] employs a three-stage process where an LLM first decomposes a task, a program synthesizer converts the plan into a script, and the script is then executed. In the mobile domain, the MobileAgent-v2 [228] series extended this concept into a multi-agent, layered architecture where a high-level planner summarizes task progress to guide a separate low-level decision-making agent.

Such hierarchical structures compensate for the lack of global control found in reactive strategies. However, their reliance on pre-defined plans can limit their ability to generalize across platforms or to correct errors during execution.

• Reflective agents. This approach introduces a self-evaluation and correction loop, enabling the agent to identify and rectify its own errors. The Reflexion [197] framework allows an agent to analyze task failures and store these reflections in memory to avoid repeating mistakes, which has been shown to improve success rates. Similarly, the RCI [87] framework demonstrated increased accuracy in GUI form-filling tasks by incorporating a self-correction mechanism. In mobile scenarios, Mobile-Agent-V [230] achieves self-correction through a fusion of visual backtracking and linguistic reflection, reviewing past demonstration videos to evaluate its actions and generate improvement strategies.

• Exploratory agents. In contrast to the post-execution error correction of reflective agents, exploratory agents proactively generate and evaluate multiple candidate paths via simulation or search before selecting the optimal one for execution. The Tree-of-Thoughts (ToT) [276] framework, for example, extends the LLM’s reasoning process into a tree structure, enabling parallel reasoning and self-evaluation to prune less promising paths at critical decision points. AppAgent [300] improves stability by leveraging a knowledge base built during an initial, task-agnostic exploration phase. Furthering this idea, MobileGPT [95] combined exploratory decision-making with a hierarchical memory to enable long-term knowledge accumulation, demonstrating improved generalization and stability in zero-shot, cross-application scenarios.

The prompt-based methods demonstrated the potential of using LLMs for GUI agent tasks, freeing them from the constraints of manual scripting and enabling zero-shot completion of complex tasks via natural language.

However, this paradigm also revealed significant limitations: (1) perceptual constraints, due to a reliance on textual descriptions of the GUI; (2) insufficient state tracking, because of limited context windows; and (3) high inference costs from multi-round reflection and exploration. These collectively motivated the research community to advance toward the model-native paradigm, with the aim of improving representational stability and generalization through direct model training.

6.2.3 Model-native Paradigm.

To overcome the limitations of the pipeline-based approaches, research has shifted toward the model-native paradigm, which aims to internalize core GUI agent capabilities into the model’s parameters through data-driven training. These capabilities can be trained either individually or jointly, leading to the development along two primary paths distinguished by their degree of integration: modular training and end-to-end training.

Modular Training. Modular training serves as a critical transitional stage between pipelinebased systems and fully end-to-end learning. In this approach, the complex GUI interaction process is decomposed into several learnable functional components such as perception, planning, and execution, which are trained either independently or in a partially coupled manner. This methodology enhances interpretability and data efficiency by allowing individual capabilities to mature under targeted supervision before their eventual integration. To provide a clearer overview of development in this stage, we categorize the related work into three main directions:

• Foundational modules: perception and execution. In the initial phase of modular training, research focused on establishing the agent’s fundamental interaction loop, which consists of perception, planning, and execution. In this loop, the perception module analyzes the structure and semantic elements of the graphical interface; the planning module decomposes high-level user goals into sub-tasks and generates executable steps; and the execution module maps these abstract plans into concrete actions.

Early work on the perception module focused on parsing the screen into structured elements. For instance, OmniParser[133] converts screen content into readable bounding boxes and semantic labels. Subsequent research began to focus more specifically on grounding. UGround[59] trained a “universal GUI grounding” model on a large-scale dataset consisting of 1.3 million screenshots and tens of millions of UI elements. Further advancing this work, AriaUI[273] achieved purely visual grounding across diverse instruction formats and in dynamic contexts, setting new state-of-the-art results on multiple online agent benchmarks. More recently, research has begun to introduce reinforcement learning paradigms into perception training. For example, SE-GUI[291] utilized self-evolving fine-tuning with dense, point-level rewards through GRPO to stably improve localization robustness. Overall, this line of research has collectively established a stable correlation: improvements in the performance of the grounding-centric perception module are consistently linked to higher overall task success rates.

Research in planning module follows two main approaches: imitative and generative planning. Imitative planning learns from expert demonstrations. For instance, CoAT[303] makes the “chain of action” and “chain of thought” explicit for supervised fine-tuning, enhancing interpretability and robustness. Similarly, WebAgent[66] constructs a modular pipeline to normalize and orchestrate natural language instructions. In contrast, generative planning enables agents to create plans more autonomously. WMA-Agent[15], for example, allows the planner to “imagine before deciding” by learning the environment’s dynamics, showing performance gains on benchmarks like WebArena. Another approach, WebSynthesis[52], uses a world model with MCTS to synthesize high-quality training trajectories, thereby augmenting its policy learning process.

Execution modules can be divided into two categories based on their action space: atomiclevel and widget-level executors. Atomic-level executors decode primitive, coordinate-based actions like click and type, focusing on precision and reliability. Research in this area includes UI-R1[134], which enhanced atomic action accuracy using rule-based rewards; and OSKairos[27], which enhances the robustness of its atomic-level actions by jointly optimizing for confidence estimation. Widget-level executors, in contrast, perform semantic selection on UI elements to align with human intent. For instance, WEPO[121] used preference learning to reduce selection errors caused by element ambiguity. In summary, the two executor types offer a clear trade-off. Atomic-level executors provide high precision and generality but are vulnerable to perceptual noise and lack semantic context. Conversely, widget-level executors align well with human intent and offer better generalization, but they depend on high-quality perception and struggle with fine-grained operations.

• Advanced modules: reflection and exploration. As foundational perception and execution capabilities matured, the research focus shifted to higher-level agentic capabilities to self-assess, self-correct, and proactively explore. The reflection module helps the model identify and correct erroneous decisions through continuous evaluation and feedback on the execution process. In parallel, the exploration module is responsible for autonomously generating high-quality interaction data to support continuous learning and generalization. The introduction of these two module types marks the evolution of GUI Agents from passive executors into proactive entities capable of learning and adaptation.

Training for the reflection module follows two main paradigms. Discriminative Reflection uses supervised signals to judge the correctness of local actions, enabling real-time selfchecking and rollbacks. In contrast, Evaluative Reflection assesses entire sequences through reward modeling or preference learning to guide policy optimization.

In discriminative reflection, the focus is on step-by-step verification. For instance, STEVE[129] reflects on each action’s validity by using a large multimodal model to verify the screen transition (previous screen, action, next screen). V-Droid[35], on the other hand, reflects by using a verifier to evaluate a set of candidate actions based on learned task progress preferences. In evaluative reflection, the goal is to assess the overall quality of a sequence. GUI-PRA[260] employs a Process Reward Model (PRM) to reflect on entire action sequences, using this evaluation to shape rewards during training and re-rank actions during inference. Similarly, UI-Genie[258] uses a learned reward model to score actions. This reflective scoring serves a dual purpose: re-ranking actions during inference and providing a reward signal for fine-tuning, thereby creating a self-improvement loop.

The exploration module aims to autonomously produce high-quality interaction data for GUI agents, creating a “data flywheel” that improves sample efficiency and robustness. Early research focused on the method of exploration, evolving from broad synthesis to more targeted and efficient strategies. Initially, the focus was on scalable trajectory synthesis. For instance, Explorer[172] explored the web using heuristic and self-learning strategies to synthetically generate large-scale, diverse interaction data. Subsequently, the research shifted to backward task synthesis, where agents explore more freely. OS-Genesis[207], for example, explores a real environment without a predefined task and then derives high-quality, labeled trajectories by backtracking from its interaction logs, using a reward model for quality control. To maximize efficiency, GUI-Xplore[209] demonstrated that an agent can perform just one or a few explorations to learn strategies that are generalizable across sub-tasks, highlighting the power of targeted, few-shot exploration.

• Module fusion. To mitigate the information mismatches and error propagation that can arise from training modules in isolation, researchers have attempted to incorporate multiple sub-modules into a unified training objective. The related studies can be categorized by which modules are fused.

One line of work focuses on perception-planning fusion. For example, CoCo-Agent [144] uses joint training of separate modules, AGUVIS [264] connects them via a structured “Inner Monologue” to improve information flow, and CogAgent [75] unifies both capabilities within a single model through multi-task pre-training.

A second line, planning-execution fusion, aims to bridge the gap between high-level intent and low-level action. MagicGUI [212] achieves this by training a single, unified policy network with reinforcement learning. In contrast, PILOT-RL [130] fuses these stages by embedding a world model into the decision loop, guiding the policy search with imagined outcomes. These fusion strategies represent a critical step toward reducing inter-module error and are a clear progression towards fully end-to-end training paradigms.

The future progression of this research is toward more efficient joint training and cross-task generalization, while preserving interpretability. However, the inherent need to reduce error propagation continues to drive the field from module fusion toward fully end-to-end training paradigms.

End-to-end Training. While modular training advanced individual components, its limitations such as error accumulation and weak generalization became increasingly apparent. To address these challenges, the research focus has shifted toward a higher level of integration with end-to-end training. In this paradigm, a single, unified model learns the complete mapping directly from perceptual inputs (like screen images) to executable actions, eliminating the need for manually designed intermediate modules. The work in this area can be categorized by the training environment into two primary approaches: offline and online training.

• Offline training. In the early exploration of offline end-to-end training, researchers primarily relied on data-driven imitation learning and multimodal pre-training. The objective was for a model to learn the complete mapping from interface perception to action execution within a single, unified structure.

The initial breakthrough in this area came from GUIDE[19], which was the first work to systematically construct an execution-oriented GUI behavioral dataset. It unified interface screenshots, task descriptions, the previous action, and the spatial location of the next action into supervised training samples. This enabled a multimodal model to directly learn the vision-to-action transformation in an offline setting, laying the data foundation for subsequent end-to-end agents. Building on this idea, GUICourse[23] further proposed a systematic training curriculum that included sub-datasets for interface understanding (GUIEnv), action knowledge (GUIAct), and task interaction (GUIChat). Through a process of staged supervised fine-tuning, this curriculum enabled a general-purpose vision-language model to progressively acquire integrated capabilities for perception, planning, and execution. As a result, the model could perform task-level operations without the need for an explicitly designed workflow. More recently, GUI-R1[138] built upon this foundation by introducing a rule-based

Table 8. Overview of GUI Agents.

Method		Platform	Modal	Action	Affliation	Access	Date
Pipeline-based Paradigm
System-based Workflow	Ringer[9]	Web	Text	Widget	Academia	Yes	16.11
	Sugilite [108]	Mobile	Hybrid	Widget	Academia	Yes	17.05
	SARA [64]	Mobile	Text	Widget	Academia	Yes	19.07
	SmartRPA [2]	PC	Text	Widget	Academia	Yes	22.11
	Reflexion [197]	General	Text	Widget	Academia	Yes	23.10
Prompt-based Methods	RCI [87]	Web	Text	Widget	Academia	Yes	23.11
	AppAgent [300]	Mobile	Hybrid	Widget	Industry	Yes	24.04
	Mobile-Agent [229]	Mobile	Hybrid	Widget	Academia	Yes	24.04
	Mobile-Agent-V2 [228]	Mobile	Hybrid	Atomic	Academia	Yes	24.06
	MobileGPT[95]	Mobile	Text	Atomic	Academia	Yes	24.11
	Mobile-Agent-V [230]	Mobile	Hybrid	Widget	Academia	No	25.06
			Model-native Paradigm

Modular Training	CogAgent [75]	General General	Visual	Atomic	Academia	Yes	23.12
	UGround [59]		Visual	Atomic	Academia	Yes	24.10
	CoAT [303]	Mobile	Hybrid	Widget	Academia	Yes	24.11
	WEPO [121]	Web	Text	Widget	Academia	Yes	24.12
	STEVE [129]	PC	Hybrid	Widget	Academia	Yes	25.03
	Explorer [197]	Web	Hybrid	Widget	Academia	Yes	25.05
	OS-Genesis [207]	General	Hybrid	Widget	Academia	Yes
							25.06
	Aria-UI [273]	General	Visual	Atomic	Academia	Yes	25.07
	V-Droid [35]	Mobile	Text	Widget	Industry	Yes	25.09
End-to-end Training	SeeClick [26]	General Mobile	Visual	Atomic	Academia	Yes	24.02
	MobileFlow [162]		Visual	Atomic	Industry	No	24.07
	UI-TARS [185]	General	Visual	Atomic	Industry	Yes	25.01
	GUICourse [23]	General	Visual	Atomic	Academia	Yes	25.05
	ZeroGUI[269]	General	Visual	Atomic	Academia	Yes	25.05
	ARPO [128]	PC	Visual	Atomic	Academia	Yes	25.05
	UItron [297]	General	Visual	Atomic	Industry	Yes	25.08
	GUI-Owl[279]	General	Visual	Atomic	Industry	Yes	25.08
	DART[104]	PC	Visual	Atomic	Academia	Yes	25.09
	GUI-R1[138]	General	Visual	Atomic	Academia	Yes	25.10
OpenCUA [237]	PC	Visual	Atomic		Academia	Yes	25.10

Reinforcement Fine-Tuning (RFT) mechanism. It performs policy-level optimization on a pre-trained model using a small amount of high-quality offline data. Through a unified action space and a relative reward design, it further improves the stability and generalization of end-to-end execution.

Overall, this body of work signifies the evolution of offline GUI agents from data-driven imitation learning to policy-level adaptive optimization. In this new paradigm, the model no longer relies on explicit modules or a pipeline structure. Instead, it directly learns the mapping from perception, through planning, to execution within a unified architecture, laying a solid foundation for future online reinforcement learning and hybrid training paradigms.

• Online training. Online end-to-end training aims to overcome the coverage limitations of offline data. It enables a GUI agent to autonomously optimize its entire decision-making chain, from visual perception to action execution, through continuous interaction with a real or simulated environment.

The earliest work in this area, UI-TARS[185], was the first to implement end-to-end online reinforcement learning in a native application environment. It demonstrated the online learnability of a “pixels-to-actions” policy by having a model take screenshots as direct input and produce action commands as output, using real interactions and environmental feedback to optimize its policy function. Subsequently, ARPO[128] built upon this foundation by proposing a mechanism that decouples experience replay from policy updates. By converting collected GUI trajectories into reusable samples and introducing a high-quality task filtering strategy, it achieved efficient and stable convergence for end-to-end reinforcement learning. This allowed the model to continuously improve its perception, planning, and execution capabilities through sustained online interaction. ZeroGUI[269] further advanced this line of research by breaking the dependence on human supervision. It achieved a “zero human cost” online end-to-end training pipeline through automatic task generation, automatic reward signal evaluation, and an adaptive sampling mechanism. This enabled the model to self-explore and correct its policy within a closed loop. In parallel, DART[104] significantly improved the efficiency and stability of online training in multi-task, long-sequence environments by introducing a decoupled multi-turn reinforcement learning framework that separates environment exploration from policy updates. Building on these advances, UI-TARS-2[226] extended the training to multi-turn interactions and task transfer. It constructed a complete multi-task online optimization loop, which enabled the agent to achieve policy generalization and self-evolution over long-term interactions. More recent works, such as UItron[297], Mobile-Agent-v3/GUI-Owl[279] and OpenCUA [237], represent a culmination of this research direction. Both approaches combine reinforcement learning with large multimodal models, allowing a single architecture to handle the three core functions of perception, planning, and execution. By continuously fine-tuning with online reward signals, these agents achieve adaptive, end-to-end capabilities across different platforms and tasks.

Overall, this line of research clearly illustrates the evolutionary path of online end-to-end GUI agents, progressing from being merely “interactive” to “optimizable,” and finally to “self-evolving.” This marks a fundamental shift in GUI automation, moving from static models to dynamic learning systems.

6.2.4 Summary and Discusion. The evolution of GUI Agents can be understood as two major leaps. The first was the shift from system-based workflow to prompt-based methods with the introduction of LLMs, which enabled agents to generate operational plans from natural language. The second, more significant leap, has been the transition to the model-native paradigm. This stage focuses on internalizing the “perceive-plan-act” loop through data-driven training, progressing from modular training of separate capabilities to fully end-to-end training. This has culminated in recent methods like UItron[297] and GUI-Owl[279], which leverage RL for continuous, real-world improvement.

Despite this significant progress, the field of GUI Agents is still in its early stages, and several fundamental challenges remain. The two core properties of GUI interaction, fine-grained/low-level and dynamic/non-stationary, give rise to the primary bottlenecks like the scarcity of high-quality labeled data, the insufficient prior knowledge of foundational models in GUI scenarios, and the lack of realistic interactive environments. As emphasized in Ultron[297], overcoming these issues requires robust data engineering and a sophisticated interaction infrastructure, both of which are still developing.

Fig. 9. Paradigm shift in AI: from human-designed rules to data-driven learning.

Future development will likely continue to focus on the model-native paradigm, with the objective of creating learning-driven, end-to-end GUI agents. Based on recent progress, key research directions are emerging to address the aforementioned challenges:

• Data efficiency and generalization: Future work will aim to mitigate data scarcity issues through techniques such as domain knowledge injection, dense reward design, self-supervised annotation, and superior exploration strategies. The goal is to enable agents to adapt to new interfaces with a minimal number of samples.
• Enhanced GUI perception: This involves developing more fine-grained visual-language encoders for the recognition and localization of complex UI elements, as seen in works like MobileFlow[162]. It also includes fusing screenshots, OCR, and structured metadata to achieve more robust GUI grounding.
• Scalable evaluation and realistic environment: The development of unified benchmarks, such as A3[16] and MLA-Trust[272], and automated evaluation pipelines is essential. These tools will allow for the systematic assessment of agent capabilities across multiple platforms and tasks, thereby promoting fair comparisons and rapid iteration among different models. Future work in this area will likely move beyond static benchmarks toward creating realistic environments that better simulate the dynamic and non-stationary nature of real-world applications.

7 Future Direction and Discussion

We have observed the shift in agentic capabilities and the two main forms of agent application, transitioning from pipeline-based to model-native paradigms. This transition actually coincides with the recurring trend from human-designed logic toward data-driven learning that has defined the progress of AI for decades.

As illustrated in Fig. 9, we have witnessed this pattern repeatedly: knowledge acquisition from expert-defined rules to a more automated, data-driven process through statistical machine learning; feature representation from hand-crafted filters in traditional machine learning to deeply layered, automatically learned features in deep learning. The recent evolution from supervised learning to reinforcement learning mirrors a similar pattern: the objective function transformed from fitting explicit, human-provided labels to self-guided exploration and optimization based on environmental feedback. The common drivers behind these repeated transitions are twofold: the inherent cost and scalability limitations of manual design, and the concurrent advancement of computational power and data infrastructure.

Fig. 10. A roadmap for the model-native internalization of agentic capabilities.

In this context, we now turn to the future of Agentic AI by discussing two questions: (1) What other core agentic capabilities are likely to become model-native? (2) As these capabilities are gradually internalized, how will the role of the system layer evolve within agentic AI?

7.1 Emerging Model-native Agentic Capabilities

As previously established, the ?????? + ???? + ???????? solution provides a promising, unified path to internalize a spectrum of agentic capabilities. Beyond planning, tool use, and memory, we envision a landscape of other agentic capabilities that will progressively transition from pipeline-based to model-native implementations.

To illustrate this evolution, Fig. 10 presents a predictive roadmap, categorizing these future capabilities into three groups based on their implementation difficulty and current model-native progress. This landscape spans from quick implementation of internalization (e.g., output formatting) to long-term challenges (e.g., safety/alignment). In the following, we focus on the critical mid-term research frontier, briefly reviewing recent attempts to internalize two key agentic capabilities of multi-agent collaboration and reflection.

7.1.1 Multi-agent Collaboration. Recent advances have transformed LLMs from generative text tools into agentic systems endowed with planning, tool use and memory capabilities. Organizing multiple LLM-based agents into collaborative systems enables the solution of tasks that exceed the capacity of any single agent. However, achieving transferability, robustness and controllability in such collaborations requires a shift from manual orchestration toward internalized, training-time learning. Multi-agent reinforcement learning (MARL) serves as a key catalyst for this transition. It can not only optimize individual agent policies but also learn system-level organizational structures and communication topologies, thereby enabling the shift from pipeline-based paradigm to modelnative paradigm.

Pipeline-based Paradigm. Methods in this category orchestrate multi-agent collaboration without modifying the underlying model parameters, instead relying on mechanisms such as prompt engineering, role assignment, and predefined workflow architectures. This hand-crafted orchestration is well-suited for low-cost, rapid-iteration applications. However, its fundamental limitation is the inability to internalize collaborative strategies into the model’s parameters, which constrains generalization.

Early multi-agent systems primarily depended on manually defined roles and static interaction protocols. While straightforward to implement for well-defined tasks, these approaches struggle with the variability and long-horizon nature of complex problems. For example, CAMEL [100] utilizes a role-playing framework where agents coordinate through dialogue, guided by inception prompting to maintain alignment. Similarly, MetaGPT [74] employs an assembly-line structure, decomposing tasks and assigning specialized roles to different agents to reduce error propagation in multi-step processes.

To mitigate the biases inherent in single-chain reasoning, a subsequent line of work introduced mechanisms such as debate and arbitration to foster multi-perspective thinking. While these methods substantially improve the quality of reasoning and question-answering tasks, their effectiveness remains bounded by the design of the prompts and the arbitration strategy itself. For instance, MAD [118] formalizes a debate where multiple agents argue iteratively before a judge aggregates the final outcome. MoA [227] employs a multi-layer agent structure where agents at each level exchange auxiliary inputs to refine the final output.

More recent pipeline-based methods have begun to explore the algorithmic generation of workflows to create more dynamic team structures. AFLOW [304], for example, formulates workflow optimization as a search problem, applying Monte-Carlo Tree Search to explore and refine candidate collaboration structures. Such approaches lay the methodological groundwork for treating the system’s architecture itself as an action space, setting the stage for the more adaptive strategies found in the model-native paradigm.

Of course. Here is a simplified and restructured version of the Model-native Paradigm section for multi-agent collaboration, followed by a concise summary, discussion, and future direction, as you requested.

Model-native Paradigm. To overcome the limitations of the pipeline-based paradigm, research has shifted toward incorporating reinforcement learning to internalize collaborative strategies directly into model parameters. This progression has generally occurred in two stages: first, by enhancing the core capabilities of individual agents within a multi-agent system, and second, by elevating the learning objective to the architectural level of the system itself.

Early efforts in this paradigm focused on improving the core reasoning and collaborative capabilities of the agents. For instance, MALT [157] uses an offline multi-agent RL approach to improve reasoning by decomposing tasks into distinct generator, verifier, and refiner roles. CORY [141] employs a two-role replication strategy within a cooperative MARL framework, where a single LLM plays both a “leader” and an “observer”, swapping them during training to improve stability. Building on this, more general frameworks have emerged. MARFT [119] and MAGRPO l, adapt algorithms like GRPO to the multi-agent setting to improve training stability without requiring individual value networks, while MATPO [155] introduces a principled credit assignment mechanism for training a single model to perform distinct roles.

A more advanced line of research has moved beyond optimizing agent behavior to learning the system’s construction and architecture. Works like MasHost [271], G-Designer [302], and ARG-Designer [107] treat the multi-agent system’s topology—including the number of agents, their roles, and their communication links—as a learnable action space. By using reinforcement learning or graph generation techniques, these methods can autonomously assemble a bespoke multi-agent system tailored to a specific task, elevating the learning objective from behavior to architecture.

A related approach involves distilling the emergent behaviors of a complex multi-agent system back into a single, powerful model. In this paradigm, collaboration data is first generated through multi-agent interaction and then used to fine-tune a student model. As demonstrated by MAGDi [21] and Chain-of-Agents [109], the resulting distilled model internalizes multi-role cognitive patterns, allowing it to execute complex, coordinated tasks at inference time without external orchestration.

Table 9. Representative studies on multi-agent collaboration.

Method		Task	Algorithm	Affliation	Access	Date
Pipeline-based Paradigm
CAMEL [100]MetaGPT[74]MAD [118]		GeneralProgrammingReasoning		Academia	Yes	23.03
				MetaGPT[74]MAD [118]MoA [227]AFlow [304]		Academia/IndustryAcademia/IndustryAcademia/IndustryAcademia/Industry	YesYesYesYes	23.08
								23.05
GeneralPlanning	24.0624.10
Model-native Paradigm
MALT[157]		Reasoning	SFT+DPOPPO-like RL	Academia	No	24.12
CORY [141]		Interaction	SFT+DPOPPO-like RL	Academia/Industry	Yes	24.10
MARFT [119]		General	MARFT	Academia/Industry	Yes	25.04
MAGRPO [124]		General	MAGRPO	Academia	No	25.08
RLCCF [290]MATPO [155]		Reasoning	GRPO	Academia	No	25.08
RLCCF [290]MATPO [155]		MATPO [155]		General	MATPO	Industry	Yes	25.10
MasHost [271]		Collaboration	HRPO	Academia	No	25.06
G-Designer [302]		Topology	VGAE	Academia	Yes	24.10
ARG-Designer [107]		Topology	ARG	Academia/Industry	Yes	25.07
MAGDi [21]		Reasoning	SFT	Academia	Yes	24.02
CoA [109]		General	SFT+RL	Industry	Yes	25.08

By leveraging multi-agent reinforcement learning, model-native multi-agent system allows for the direct internalization of collaborative behaviors. However, significant challenges remain. First, the credit assignment problem is exacerbated; with multiple agents acting concurrently, it becomes exceedingly difficult to attribute the final team outcome to the specific actions of any single agent, which can dilute or misdirect the learning signal. Second, the training environment becomes highly non-stationary. From the perspective of any one agent, the environment is constantly changing due to the evolving policies of all other agents. This dynamic instability can destabilize the learning process, as an optimal action at one point in training may become suboptimal as other agents adapt.

To tackle the credit assignment challenge, research will likely focus on more sophisticated attribution techniques, such as developing reward structures that better reflect individual contributions to the collective goal. To manage the non-stationary environment, a key direction will be the development of more robust MARL algorithms that can maintain stable learning amidst shifting policies, potentially incorporating opponent modeling or adaptive learning rates. Ultimately, progress in this domain will depend on creating training methodologies that can effectively navigate the complex interplay between individual contribution and collective success in dynamic, multi-agent settings.

7.1.2 Reflection. Agent reflection is also undergoing a systematic shift from reliance on handcrafted, run-time pipelines to the internalization of reflective processes. Early approaches used prompt engineering, role assignment, and fixed pipelines to trigger self-checks at inference time. These methods are easy to prototype but limited by templates. Subsequent work introduced supervised and reinforcement learning to improve the reliability of these external modules. Recent research focuses on making reflection an intrinsic behavior of policy so that models can autonomously assess, verify, and correct their outputs without external orchestration.

Table 10. Representative studies on agent reflection.

Method

Task Algorithm Data Source

Affiliation Access Date

Pipeline-based Paradigm

Reflexion [197]Self-Refine [145]

Interaction/Coding

1REINFORCESFTSFT+DPO

一

Academia

Yes

23.03

5

Text Gen./Reasoning

一1Self-gen.Self-gen. + Ext.Self-gen. + Ext.

Academia/Industry

Yes

23.03

Critic [60]Conf. Matters [103]SCoRe [91]CoV[70]DPSDP [292]

ReasoningReasoning Math/CodingOpen-domain QAMath/Reasoning

Academia/Industry

Academia

YesYes

23.05

AcademiaIndustryAcademia/IndustryAcademia

24.02

NoNoNo

24.09

24.10

25.06

Model-native Paradigm

Agent-R [288]AgentRefine [49]

Interaction/Agent Task

SFT

Self-gen.

Industry

Yes

25.01

Interaction/Agent Task

SFT

Ext.

Academia/Industry

Yes

25.01

STeP [25]KnowSelf [183]

Interaction/Agent TaskInteraction/Agent Task

SFTSFT+RPO

Self-gen. + Ext.Self-gen. + Ext.

Academia/IndustryAcademia/Industry

NoYes

25.05

25.04

Pipeline-based Paradigm. In early work on reflective behavior, researchers found that prompt engineering and fixed pipelines can elicit self-checking and iterative revision in LLMs without changing model parameters. Representative studies such as Reflexion [197], Self-Refine [145], Critic [60] and Confidence Matters [103] implement reflection by verbalizing the reflection process and iterating loops such as generate, self-evaluate and revise. These methods demonstrate that pipeline-based reflection can reduce errors and improve robustness on tasks such as question answering, code debugging, and factual verification. However, pipeline-based approaches often depend heavily on the base model’s internal evaluation abilities, e.g., the reliability of confidence estimates. They also rely on manually designed templates and templates scale poorly as task complexity or interaction length increases. Crucially, pipelines do not embed successful collaborative strategies into model parameters, which limits long-term generalization.

To address the robustness and generalization limits of early pipeline-based paradigms, researchers began to introduce training signals that improve a model’s ability to execute a prescribed pipeline. Typical approaches generate high-quality self-correction examples or reward signals via online or offline reinforcement learning, preference learning, or adversarial data synthesis. For instance. SCoRe [91] uses multi-turn RL and regularization to strengthen self-correction. Chain-ofVerification [70] augments retrieval-augmented generation with verify-and-rewrite chains to align retrieval and generation. DPSDP [292] formalizes multi-turn improvement as a Markov decision process and applies actor-critic style policy search. These studies show that training can substantially broaden the applicability and effectiveness of external pipelines. Still, such methods treat reflection as an external module: they remain constrained by the pipeline’s predefined structure, and they face practical bottlenecks in the cost and quality of the data required for training.

Model-native aradigm. Recently, some pioneering studies have begun to explore the internalization of reflection, aiming to make self-correction an intrinsic component of the model’s generation policy. For example, Agent-R [288] builds training examples by iteratively self-training and using tree search to convert erroneous trajectories into corrected ones. AgentRefine [49] and STeP [25] enhance generalization by fine-tuning on partially masked self-reflection trajectories, teaching the model to identify and fix its own mistakes. Furthermore, research on agentic selfawareness [183] emphasizes situational awareness, enabling the agent to dynamically regulate its use of knowledge and verification procedures based on the context of the task.

In summary, the model-native paradigm for reflection represents a critical shift toward greater agent autonomy. By internalizing the self-correction process, these methods have the potential to improve inference efficiency and long-term robustness. However, this direction also introduces significant challenges, including higher demands on the design of training data, the complexity of reward and value modeling, and the need to maintain interpretability. Future work will likely focus on developing more efficient methods for training reflective capabilities, potentially by exploring self-supervised techniques to reduce the dependency on explicit error labels. A key research frontier will be to investigate the trade-off between the computational cost of performing these internal self-checks and the resulting gains in task performance and reliability.

7.2 System vs. Model: Evolving Roles in Agentic AI

As we explore the ongoing shift from pipeline-based to model-native agentic systems, a fundamental divergence emerges between the optimistic visions within academic research and the engineering-centric industry. While researchers eagerly anticipate a future where agentic capabilities are fully internalized in models, with intelligence becoming intrinsic to the system, many industry practitioners still believe that production-ready agentic systems are “90% engineering, 10% intelligence.” [56] This pragmatic view reflects the current reality of practical systems, which treat the AI as a functional component or a specialized tool, rather than as the central driver of the system itself.

This conflict is actually a common phenomenon in the early stage of a technological revolution. History offers a compelling parallel. The early days of the Internet saw it evolve from a basic infrastructure for information access into a ubiquitous utility, with its power encapsulated in websites and applications. As illustrated in the top of Fig. 11, its architectural evolution clearly demonstrates a dynamic shift in the balance between engineering and intelligence: from an early, purely engineering stage of hand-coded HTML/CSS to a dynamic interaction stage where applications like search engines began to incorporate statistics-based intelligence. This has finally culminated in the present day, where cloud services and MLOps have standardized the underlying engineering, allowing AI-driven personalization and recommendation to become the core value of web applications.

The evolution of web applications suggests that the current engineering-heavy agentic AI is transitory. As the underlying technologies discussed in this survey mature, the applications built upon them will likely follow a similar path. As illustrated in the bottom of Fig. 11, we envision the development of these agent systems in three stages:

• Pipeline design (2023˘2025). In the current era, LLMs are often treated as components, which are callable functions within a human-designed pipeline. • Model-native transition (2025˘2027). We are now entering a transitional phase where the model is increasingly becoming the driving force behind core agentic functions like planning, tool use, and memory. At the same time, task automation is evolving from procedural process automation toward more sophisticated levels of intent-driven and role-level autonomy. During this period, scaffolding frameworks such as LangChain, AutoGen, and AgentCore will continue to play an important role. They will act as intermediate layers that bridge the gap between model-native intelligence and real-world deployment. • Autonomous evolution (2027−). This stage anticipates the maturation of the engineering stack into a standardized and automated practice, or AgentOps. The focus will shift to fostering highly autonomous agents, enabling agents exhibiting capabilities such as emergent task discovery, on-demand AI-generated software, and even architectural self-evolution.

As agentic models become more self-sufficient, the role of the system layer will undergo a critical transformation: from compensating for the model’s capability deficits to providing foundational system support. Today, system engineering is primarily focused on patching the limitations of agentic models. In the future, the system layer will provide the essential, high-level infrastructure required for a robust and scalable agent ecosystem. The possible functions include identity management, resource allocation of compute and tools, standardized agent protocols, and granular permissions management to ensure safe and compliant operation.

Fig. 11. Comparison of the evolution of web applications and agent systems

8 Conclusions

The evolution of agentic AI reflects a deeper transformation in how intelligence itself is conceived, trained, and deployed. From pipeline-based systems, where reasoning, memory, and action were orchestrated by external scaffolds, to model-native paradigms that internalize these capabilities, we are witnessing a fundamental redefinition of agentic AI. Reinforcement learning, acting as the engine of experience, bridges perception and action, turning static models into adaptive, goal-directed entities capable of learning from interactions with the environment.

Through this survey, we have reviewed how planning, tool use, and memory are progressively being absorbed into the model’s intrinsic policy. The unifying principle ?????? +???? +???????? is emerging as the methodological singularity of modern AI. This framework is transforming compute into intelligence through cycles of pretraining, post-training, and inference.

Ultimately, the trajectory of agentic AI is not merely toward greater autonomy, but toward a deeper synthesis between models and the environments they inhabit. Therefore, the paradigm shift from external pipeline to model-native marks a transition from building systems that use intelligence to systems that grow intelligence. The next era of AI will be defined less by how we design agents, and more by how we enable them to learn, collaborate, and evolve through experience.

References

[1] Pranjal Aggarwal and Sean Welleck. 2025. L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. arXiv:2503.04697 [cs.CL] https://arxiv.org/abs/2503.04697
[2] Simone Agostinelli, Marco Lupia, Andrea Marrella, and Massimo Mecella. 2022. Reactive synthesis of software robots in RPA from user interface logs. Computers in Industry 142 (2022), 103721. doi:10.1016/j.compind.2022.103721
[3] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, et al. 2022. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv preprint arXiv:2204.01691 (2022). https://arxiv.org/abs/2204.01691
[4] Perplexity AI. 2025. Perplexity: An AI-powered Search and Answer Engine. https://perplexity.ai. Accessed: 2025-09-XX.
[5] Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, et al. 2025. Open deep search: Democratizing search with open-source reasoning agents. arXiv preprint arXiv:2503.20201 (2025). https://arxiv.org/abs/2503.20201
[6] Amazon Science. 2025. MemInsight: AI agents that organize and retrieve their own memories. Amazon Science Blog. https://www.amazon.science/blog/ai-agents-that-organize-and-retrieve-their-own-memories
[7] Akari Asai, Rui Zhong, Alexander Fabbri, and Danqi Chen. 2023. Self-RAG: Learning to Retrieve, Generate, and Critique for Improved Language Modeling. arXiv preprint (2023). arXiv:2310.01342 [cs.CL] https://arxiv.org/abs/2310.01342
[8] SWiRL Authors et al. 2025. SWiRL: Structured Web Interaction with Reinforcement Learning. arXiv preprint arXiv:2504.04736 (2025). https://arxiv.org/abs/2504.04736
[9] Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani. 2016. Ringer: web automation by demonstration. In Proceedings of the 2016 ACM SIGPLAN international conference on object-oriented programming, systems, languages, and applications. 748–764.
[10] Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. In Proceedings (demo/resources) around NeurIPS 2020 / AI2 resources. TBD Prefer non-arXiv: AI2 page or NeurIPS resources if available.
[11] Michael Bolin, Matthew Webber, Philip Rha, Tom Wilson, and Robert C Miller. 2005. Automation and customization of rendered web pages. In Proceedings of the 18th annual ACM symposium on User interface software and technology. 163–172.
[12] Sebastian Borgeaud, Arthur Mensch, et al. 2021. Improving language models by retrieving from trillions of tokens. arXiv:2112.04426 (2021).
[13] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, et al. 2024. Video generation models as world simulators.
[14] Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, et al. 2025. SkyRL-v0: Train Real-World Long-Horizon Agents via Reinforcement Learning.
[15] Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. 2024. Web agents with world models: Learning and leveraging environment dynamics in web navigation. arXiv preprint arXiv:2410.13232 (2024).
[16] Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, and Hongsheng Li. 2025. A3: Android Agent Arena for Mobile GUI Agents. arXiv preprint arXiv:2501.01149 (2025). https://arxiv.org/abs/2501.01149
[17] Karthik Chandy. 2025. M+: Multi-Memory Multi-Tool AI Fabric for Building Complex LLM Applications. arXiv preprint (2025). arXiv:2502.11372 https://arxiv.org/abs/2502.11372
[18] Harrison Chase and Ankush Gola. 2023. Conversation Summary Memory: Maintaining Short-Term Memory in LLM-Based Agents via On-the-Fly Summarization. LangChain Technical Report.
[19] Rajat Chawla, Adarsh Jha, Muskaan Kumar, Mukunda NS, and Ishaan Bhola. 2024. GUIDE: Graphical User Interface Data for Execution. arXiv:2404.16048 [cs.HC] https://arxiv.org/abs/2404.16048
[20] Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs. arXiv:2412.18925 [cs.CL] https://arxiv.org/ abs/2412.18925
[21] Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. 2024. Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. arXiv preprint arXiv:2402.01620 (2024).
[22] Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, et al. 2025. ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning. arXiv:2503.19470 [cs.AI] https://arxiv.org/abs/2503.19470
[23] Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2025. GUICourse: From General Vision Language Model to Versatile GUI Agent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 21936–21959. doi:10.18653/v1/2025.acl-long.1065
[24] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. Transactions on Machine Learning Research (2023). https://openreview.net/forum?id=YfZ4ZPt8zd with Synthetic Self-Reflected Trajectories and Partial Masking. arXiv preprint arXiv:2505.20023 (2025).
[26] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. arXiv:2401.10935 [cs.HC] https://arxiv.org/abs/2401.1 0935
[27] Pengzhou Cheng, Zheng Wu, Zongru Wu, Tianjie Ju, Aston Zhang, Zhuosheng Zhang, and Gongshen Liu. 2025. OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 6701–6725. doi:10.18653/v1/2025.findings-acl.348
[28] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building ProductionReady AI Agents with Scalable Long-Term Memory. arXiv preprint (2025). arXiv:2504.19413 [cs.AI] https://arxiv.org/ abs/2504.19413
[29] Krzysztof Choromanski et al. 2020. Rethinking Attention with Performers. arXiv preprint (2020). TBD If you use a project page or official impl page, replace URL.
[30] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. 2025. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. arXiv:2501.17161 [cs.AI] https://arxiv.org/abs/2501.17161
[31] Hyung Won Chung. 2024. Don’t teach. Incentivize. MIT EI Seminar / OpenAI Talk. Video talk, https://www.youtube. com/watch?v=kYWUEVe2ss.
[32] Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks. Machine Learning 20, 3 (1995), 273–297. doi:10.1023/A:1022627411411
[33] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, et al. 2025. Process Reinforcement through Implicit Rewards. arXiv:2502.01456 [cs.LG] https://arxiv.org/abs/2502.01456
[34] Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. 2025. Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards. arXiv preprint arXiv:2506.11425 (2025). https://arxiv.org/abs/2506.11425
[35] Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. 2025. Advancing mobile gui agents: A verifier-driven approach to practical deployment. arXiv preprint arXiv:2503.15937 (2025).
[36] Tri Dao et al. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In NeurIPS. TBD Use NeurIPS paper page.
[37] Google DeepMind. 2025. Gemini 2.5. Official blog / product page. TBD
[38] DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 (2025). arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501.12948
[39] Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, and Yu Cheng. 2025. MoM: Linear Sequence Modeling with Mixture-ofMemories. arXiv:2502.13685 (2025). GitHub: https://github.com/OpenSparseLLMs/MoM.
[40] Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2024. SKVQ: Slidingwindow Key and Value Cache Quantization for Large Language Models. arXiv:2405.06219 [cs.LG] https://arxiv.org/ abs/2405.06219
[41] Darren Edge, Ha Trinh, Newman Cheng, et al. 2024. A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130 (2024). https://arxiv.org/abs/2404.16130 Code: https://github.com/microsoft/graphrag.
[42] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130 (2024). https://arxiv.org/abs/2404.16130
[43] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2025. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130 [cs.CL] https://arxiv.org/abs/2404.16130
[44] Chen Fang et al. 2025. RCR-Router: Retrieval-Critic-Rerank for Efficient Knowledge Retrieval. arXiv preprint (2025). arXiv:2509.08181 [cs.CL] https://arxiv.org/abs/2509.08181
[45] et al. Fei. 2023. MemoChat: Conversational Agents with Long-Term Memory. arXiv preprint (2023). arXiv:2308.02828 [cs.CL] https://arxiv.org/abs/2308.02828
[46] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. ReTool: Reinforcement Learning for Strategic Tool Use in LLMs. arXiv:2504.11536 [cs.CL] https://arxiv.org/abs/2504.11536
[47] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978 [cs.LG] https://arxiv.org/abs/2505.10978
[48] Richard E. Fikes and Nils J. Nilsson. 1971. Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence 2, 3 (1971), 189–208. doi:10.1016/0004-3702(71)90010-5 Xunliang Cai, and Weiran Xu. 2025. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint arXiv:2501.01702 (2025). [50] Junyi Gao et al. 2025. Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous Reinforcement Learning. arXiv preprint arXiv:2508.07976 (2025). https://arxiv.org/abs/2508.07976
[51] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: Program-aided Language Models. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 10764–10799. https://proceedings.mlr.press/v202/gao 23f.html
[52] Yifei Gao, Junhong Ye, Jiaqi Wang, and Jitao Sang. 2025. WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis. arXiv:2507.04370 [cs.AI] https://arxiv.org/abs/2507.04370
[53] Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. 2025. RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning. arXiv:2410.02089 [cs.CL] https://arxiv.org/abs/2410.02089 [54] Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jialong Wu, et al. 2025. WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent. arXiv preprint arXiv:2508.05748 (2025). https://arxiv.org/abs/2508.05748
[55] Malik Ghallab, Craig Knoblock, David Wilkins, Anthony Barrett, and Daniel Weld. 1998. PDDL - The Planning Domain Definition Language. (1998).
[56] Rakesh Gohel. 2024. AI Agents are about 90% software engineering and 10% AI. Do you agree? https://www.linkedin .com/posts/rakeshgohel01ai-agents-are-about-90-software-engineering-activity-7353405600610881536-D36M LinkedIn Post.
[57] Lorenzo Gomez, Iulian Neamtiu, Tanzirul Azim, and Todd Millstein. 2013. Reran: Timing-and touch-sensitive record and replay for android. In 2013 35th international conference on software engineering (ICSE). IEEE, 72–81. [58] Google. [n. d.]. Google Deep Research Agent. https://www.google.com/research/deepresearch. [59] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=kxnoqaisCT
[60] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738 (2023).
[61] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. In Proceedings of the International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=Sx038qxjek [62] Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. 2023. Leveraging pre-trained large language models to construct and utilize world models for model-based task planning. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 3459, 14 pages. [63] Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, et al. 2025. OpenThoughts: Data Recipes for Reasoning Models. arXiv:2506.04178 [cs.LG] https://arxiv.org/ abs/2506.04178
[64] Jiaqi Guo, Shuyue Li, Jian-Guang Lou, Zijiang Yang, and Ting Liu. 2019. Sara: self-replay augmented record and replay for android in industrial cases. In Proceedings of the 28th acm sigsoft international symposium on software testing and analysis. 90–100.
[65] Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856 (2023).
[66] Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2024. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=9JQtrumvg8
[67] Bingguang Hao, Maolin Wang, Zengzhuang Xu, Yicheng Chen, Cunyin Peng, Jinjie GU, and Chenyi Zhuang. 2025. Exploring Superior Function Calls via Reinforcement Learning. arXiv:2508.05118 [cs.LG] https://arxiv.org/abs/2508.0 5118
[68] Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Reasoning with Language Model is Planning with World Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 8154–8173. doi:10.18653/v1/2023.emnlp-main.507 [69]
[70] Bolei He, Nuo Chen, Xinran He, Lingyong Yan, Zhenkai Wei, Jinchang Luo, and Zhen-Hua Ling. 2024. Retrieving, rethinking and revising: The chain-of-verification can improve retrieval augmented generation. arXiv preprint arXiv:2410.05801 (2024).
[71] Guangxin He, Shen Nie, Fengqi Zhu, Yuankang Zhao, Tianyi Bai, Ran Yan, Jie Fu, Chongxuan Li, and Binhang Yuan. 2025. UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models. arXiv:2510.10481 [cs.CL] https://arxiv.org/abs/2510.10481
[72] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919 (2024). https://arxiv.org/abs/2401.13919
[73] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, et al. 2025. Skywork Open Reasoner Series. https://capricious-hydrogen-41c.notion.site/Skywork-Open-Reaonser-Series1d0bc9ae823a80459b46c149e4f51680. Notion Blog.
[74] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, et al. 2023. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations.
[75] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, et al. 2024. CogAgent: A Visual Language Model for GUI Agents. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 14281–14290. doi:10.1109/CVPR52733.2024.01354
[76] et al. Hu. 2024. HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Models. Preprint. https://github.com/OpenBMB/HiAgent
[77] Hugging Face. 2025. Open R1: A fully open reproduction of DeepSeek-R1. https://github.com/huggingface/open-r1
[78] Gautier Izacard, Patrick Lewis, Maria Lomeli, et al. 2022. Atlas: Few-shot Learning with Retrieval Augmented Language Models. arXiv:2208.03299 (2022). https://arxiv.org/abs/2208.03299
[79] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 13358–13376. doi:10.18653/v1/2023.emnlp-main.825
[80] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839 (2023). https://arxiv.org/abs/2310.06839
[81] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world GitHub Issues?. In Proceedings of the 12th International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=VTF8yNQM66
[82] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv:2503.09516 [cs.CL] https://arxiv.org/abs/2503.09516
[83] Jiajie Jin, Yutao Zhu, Yujia Zhou, and Zhicheng Dou. 2024. Bider: Bridging knowledge inconsistency for efficient retrieval-augmented llms via key supporting evidence. arXiv preprint arXiv:2402.12174 (2024). https://arxiv.org/abs/ 2402.12174
[84] King Julio. 2025. Memary. GitHub repository. https://github.com/kingjulio8238/Memary
[85] Dongwon Jung, Qin Liu, Tenghao Huang, Ben Zhou, and Muhao Chen. 2024. Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation. arXiv preprint arXiv:2409.12468 (2024). https://arxiv.org/abs/2409.12468
[86] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906 [cs.CL] https: //arxiv.org/abs/2004.04906
[87] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. Advances in Neural Information Processing Systems 36 (2023), 39648–39677.
[88] Kimi Team. 2025. Kimi K2: Open Agentic Intelligence. arXiv preprint arXiv:2507.20534 (2025). arXiv:2507.20534 [cs.CL] https://arxiv.org/abs/2507.20534
[89] Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-Augmented Dialogue Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 8460–8478. doi:10.18653/v1/2022.acl-long.579
[90] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS). 1097–1105. 2024. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917 (2024).
[92] Alibaba / Tongyi Lab. 2025. Tongyi DeepResearch: an open deep research agent and toolkit. https://github.com/AlibabaNLP/DeepResearch.
[93] Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. Llms get lost in multi-turn conversation. arXiv preprint arXiv:2505.06120 (2025). https://arxiv.org/abs/2505.06120
[94] Bespoke Labs. 2025. Bespoke-Stratos: The unreasonable effectiveness of reasoning distillation. https://www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation. Accessed: 2025-01-22.
[95] Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, and Insik Shin. 2024. Mobilegpt: Augmenting llm with human-like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 1119–1133.
[96] Gilly Leshed, Eben Haber, Tessa Lau, and Allen Cypher. 2007. CoScripter: Sharing ‘How-to’Knowledge in the Enterprise. GROUP 7 (2007), 4–7.
[97] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 [cs.CL]
[98] Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, and Zhiyuan Cheng. 2024. MixLoRA: Enhancing LLM Fine-Tuning with LoRA-based Mixture of Experts (overview). https://magazine.sebastianraschka.com/p/llm-researchinsights-instruction.
[99] Dawei Li, Zhen Tan, Chengshuai Zhao, Bohan Jiang, Baixiang Huang, Pingchuan Ma, Abdullah Alnaibari, Kai Shu, and Huan Liu. 2025. Who’s Your Judge? On the Detectability of LLM-Generated Judgments. arXiv:2509.25154 [cs.AI] https://arxiv.org/abs/2509.25154
[100] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems 36 (2023), 51991–52008.
[101] Jiayi Li et al. 2024. Enhancing Lifelong Model Editing with Mixture-of-LoRA. arXiv:2408.11869 (2024).
[102] Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye, Yida Zhao, Liwen Zhang, Litu Ou, Dingchu Zhang, et al. 2025. WebSailor: Navigating Super-human Reasoning for Web Agent. arXiv preprint arXiv:2507.02592 (2025). Tongyi Lab / Alibaba NLP.
[103] Loka Li, Zhenhao Chen, Guangyi Chen, Yixuan Zhang, Yusheng Su, Eric Xing, and Kun Zhang. 2024. Confidence matters: Revisiting intrinsic self-correction capabilities of large language models. arXiv preprint arXiv:2402.12563 (2024).
[104] Pengxiang Li, Zechen Hu, Zirui Shang, Jingrong Wu, Yang Liu, Hui Liu, Zhi Gao, Chenrui Shi, et al. 2025. Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation. arXiv:2509.23866 [cs.LG] https://arxiv.org/abs/2509.23866
[105] Peiji Li, Kai Lv, Yunfan Shao, Yichuan Ma, Linyang Li, Xiaoqing Zheng, Xipeng Qiu, and Qipeng Guo. 2025. FastMCTS: A Simple Sampling Strategy for Data Synthesis. arXiv:2502.11476 [cs.CL] https://arxiv.org/abs/2502.11476
[106] Shuangtao Li, Shuaihao Dong, Kexin Luan, Xinhan Di, and Chaofan Ding. 2025. Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search. arXiv:2501.01478 [cs.AI] https://arxiv.org/abs/2501.01478
[107] Shiyuan Li, Yixin Liu, Qingsong Wen, Chengqi Zhang, and Shirui Pan. 2025. Assemble your crew: Automatic multi-agent communication topology design via autoregressive graph generation. arXiv preprint arXiv:2507.18224 (2025).
[108] Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems. 6038–6049.
[109] Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, et al. 2025. Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic rl. arXiv preprint arXiv:2508.13167 (2025).
[110] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366 (2025). https: //arxiv.org/abs/2501.05366
[111] Xiaoxi Li, Zhicheng Dou, Yujia Zhou, and Fangchao Liu. 2024. Corpuslm: Towards a unified language model on corpus for knowledge-intensive tasks. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 26–37. https://arxiv.org/abs/2402.01176
[112] Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. 2025. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776 (2025). https://arxiv.org/abs/2504.21776
[113] Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yongkang Wu, Zhonghua Li, Qi Ye, and Zhicheng Dou. 2024. Retrollm: Empowering large language models to retrieve fine-grained evidence within generation. arXiv preprint arXiv:2412.11919 (2024). https://arxiv.org/abs/2412.11919
[114] Xuefeng Li, Haoyang Zou, Pengfei Liu, et al. 2025. ToRL: Tool-integrated Reinforcement Learning for LLM Agents. arXiv preprint arXiv:2503.23383 (2025). https://arxiv.org/abs/2503.23383
[115] Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. 2023. Compressing Context to Enhance Inference Efficiency of Large Language Models. arXiv:2310.06201 [cs.CL] https://arxiv.org/abs/2310.06201
[116] Zhiwei Li, Yong Hu, and Wenqing Wang. 2025. Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning. arXiv:2508.19598 [cs.LG] https://arxiv.org/abs/2508.19598
[117] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as Policies: Language Model Programs for Embodied Control. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 9493–9500. doi:10.1109/ICRA48891.2023.10160591
[118] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118 (2023).
[119] Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. 2025. Marft: Multi-agent reinforcement fine-tuning. arXiv preprint arXiv:2504.16129 (2025).
[120] Zhiyu Lin, Yifei Gao, Xian Zhao, Yunfan Yang, and Jitao Sang. 2025. Mind with Eyes: from Language Reasoning to Multimodal Reasoning. arXiv:2503.18071 [cs.CL] https://arxiv.org/abs/2503.18071
[121] Jiarun Liu, Jia Hao, Chunhong Zhang, and Zheng Hu. 2025. WEPO: web element preference optimization for LLMbased web navigation. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence (AAAI’25/IAAI’25/EAAI’25). AAAI Press, Article 2965, 9 pages. doi:10.1609/aaai.v39i25.34863
[122] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 (2023). https: //arxiv.org/abs/2307.03172
[123] Qianchu Liu, Sheng Zhang, Guanghui Qin, Timothy Ossowski, Yu Gu, Ying Jin, Sid Kiblawi, Sam Preston, Mu Wei, Paul Vozila, Tristan Naumann, and Hoifung Poon. 2025. X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains. arXiv:2505.03981 [cs.AI] https://arxiv.org/abs/2505.03981
[124] Shuo Liu, Zeyu Liang, Xueguang Lyu, and Christopher Amato. 2025. Llm collaboration with multi-agent reinforcement learning. arXiv preprint arXiv:2508.04652 (2025).
[125] Yang Liu, Jianfeng Gao, and Graham Neubig. 2024. A Survey on Context Engineering for Large Language Models. Comput. Surveys 57, 1, Article 23 (2024). doi:10.1145/3636363
[126] Yang Liu, Zhitao Wang, Zhe Hu, Zhiqiang Zhang, Yu Chen, Xiaohan Zhang, and Zhi Jin. 2023. LLM+P: Empowering Large Language Models with Optimal Planning Proficiency. arXiv:2306.07178 [cs.AI]
[127] Zhe Liu et al. 2023. Scissorhands: Exploiting the Persistence of Importance for Efficient Inference of Large Language Models. arXiv preprint (2023). arXiv:2305.17118 [cs.CL] https://arxiv.org/abs/2305.17118
[128] Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. 2025. ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay. arXiv:2505.16282 [cs.CV] https://arxiv.org/abs/2505.16282
[129] Fanbin Lu, Zhisheng Zhong, Ziqin Wei, Shu Liu, Chi-Wing Fu, and Jiaya Jia. 2025. STEVE: A Step Verification Pipeline for Computer-use Agent Training. arXiv preprint arXiv:2503.12532 (2025).
[130] Keer Lu, Chong Chen, Bin Cui, Huang Leng, and Wentao Zhang. 2025. PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning. arXiv:2508.00344 [cs.CL] https://arxiv.org/abs/2508.0 0344
[131] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In Advances in Neural Information Processing Systems, Vol. 36. https://proceedings.neurips.cc/paperfiles/paper/2023/file/871ed095b734818c fba48db6aeb25a62-Paper-Conference.pdf
[132] Rui Lu, Zhenyu Hou, Zihan Wang, Hanchen Zhang, Xiao Liu, Yujiang Li, Shi Feng, Jie Tang, and Yuxiao Dong. 2025. Deepdive: Advancing deep search agents with knowledge graphs and multi-turn rl. arXiv preprint arXiv:2509.10446 (2025). https://arxiv.org/abs/2509.10446
[133] Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203 (2024).
[134] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. 2025. UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning. arXiv preprint arXiv:2503.21620 (2025).
[135] Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. 2025. O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning. arXiv:2501.12570 [cs.CL] https://arxiv.org/abs/2501.12570
[136] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, et al. 2024. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv:2406.06592 [cs.CL] https://arxiv.org/abs/2406.06592
[137] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, et al. 2025. DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL. https://pretty-radio-b75.notion.site/DeepScaleRSurpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2. Notion Blog.
[138] Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, and Xiaobo Xia. 2025. GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents. arXiv:2504.10458 [cs.CV] https://arxiv.org/abs/2504.10458
[139] Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. 2025. Agent Lightning: Train ANY AI Agents with Reinforcement Learning. arXiv preprint arXiv:2508.03680 (2025). https://arxiv.org/abs/2508.03680
[140] Ziyang Luo, Zifan Cheng, Yiyang Cai, Shangxi Wu, Zhiqing Xiao, Caiming Xiong, and Shafiq Joty. 2025. MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers. arXiv:2508.14704 [cs.CL] https://arxiv.org/abs/2508.14704
[141] Hao Ma, Tianyi Hu, Zhiqiang Pu, Liu Boyin, Xiaolin Ai, Yanyan Liang, and Min Chen. 2024. Coevolving with the other you: Fine-tuning llm with sequential cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems 37 (2024), 15497–15525.
[142] Xuezhi Ma et al. 2025. CARE: Conflict-Aware Retrieval-Augmented Generation. arXiv preprint (2025). arXiv:2508.21345 [cs.CL] https://arxiv.org/abs/2508.21345
[143] Xinbei Ma, Yeyun Gong, Pengcheng He, Nan Duan, et al. 2023. Query rewriting in retrieval-augmented large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/2023. emnlp-main.322
[144] Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. 2024. CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation. In Findings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 9097–9110. doi:10.18653/v1/2024.findings-acl.539
[145] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2023), 46534–46594.
[146] Sathwik Tejaswi Madhusudhan, Shruthan Radhakrishna, Jash Mehta, and Toby Liang. [n. d.]. Millions scale dataset distilled from R1-32b. https://huggingface.co/datasets/ServiceNow-AI/R1-Distill-SFT.
[147] Xinji Mai, Haotian Xu, Xing W, Weinong Wang, Yingying Zhang, Wenqiang Zhang, Jian Hu, et al. 2025. Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving. arXiv preprint arXiv:2505.07773 (2025). https://arxiv.org/abs/2505.07773
[148] Justus Mattern, Sami Jaghouar, Manveer Basra, Jannik Straube, Matthew Di Ferrante, Felix Gabriel, Jack Min Ong, Vincent Weisser, and Johannes Hagemann. 2025. SYNTHETIC-1: Two Million Collaboratively Generated Reasoning Traces from Deepseek-R1. https://www.primeintellect.ai/blog/synthetic-1-release
[149] Jiaqi Mei et al. 2025. O2-Searcher: A Searching-based Agent Model for Open-domain QA via Reinforcement Learning. arXiv preprint arXiv:2505.16582 (2025). https://arxiv.org/abs/2505.16582
[150] Lang Mei, Zhihan Yang, and Chong Chen. 2025. AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning. arXiv:2508.20368 [cs.AI] https://arxiv.org/abs/2508.20368
[151] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and Editing Factual Associations in GPT. In NeurIPS. https://arxiv.org/abs/2202.05262
[152] Kevin Meng and Arnold P. et al. 2022. Mass-Editing Memory in a Transformer. arXiv preprint arXiv:2210.07229 (2022). https://arxiv.org/abs/2210.07229
[153] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A Benchmark for General AI Assistants. In Proceedings of the 12th International Conference on Learning Representations, ICLR 2024. https://openreview.net/forum?id=r1gG0T0a0P
[154] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. 2022. Memory-Based Model Editing at Scale. In ICML. https://proceedings.mlr.press/v162/mitchell22a/mitchell22a.pdf
[155] Zhanfeng Mo, Xingxuan Li, Yuntao Chen, and Lidong Bing. 2025. Multi-Agent Tool-Integrated Policy Optimization. arXiv preprint arXiv:2510.04678 (2025).
[156] Moonshot AI. 2025. MoBA. https://github.com/MoonshotAI/MoBA. GitHub repository.
[157] Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip HS Torr, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. 2024. Malt: Improving reasoning with multi-agent llm training. arXiv preprint arXiv:2412.01928 (2024).
[158] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv:2501.19393 [cs.CL] https://arxiv.org/abs/2501.19393
[159] Hyungho Na, Yunkyeong Seo, and Il-Chul Moon. 2024. Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning. In ICLR. https://arxiv.org/abs/2403.01112
[160] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. WebGPT: Browser-assisted question-answering with human feedback. CoRR abs/2112.09332 (2021). arXiv:2112.09332 https://arxiv.org/abs/2112.09332
[161] Xuan-Phi Nguyen, Shrey Pandit, Revanth Gangi Reddy, Austin Xu, Silvio Savarese, Caiming Xiong, and Shafiq Joty. 2025. Sfr-deepresearch: Towards effective reinforcement learning for autonomously reasoning single agents. arXiv preprint arXiv:2509.06283 (2025). https://arxiv.org/abs/2509.06283
[162] Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. 2024. MobileFlow: A Multimodal LLM For Mobile GUI Agent. arXiv:2407.04346 [cs.CV] https://arxiv.org/abs/2407.04346
[163] OpenAI. [n. d.]. OpenAI Deep Research Agent (model-native, based on o3). https://openai.com/deepresearch.
[164] OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt
[165] OpenAI. 2023. Function calling and other API updates. OpenAI Blog. https://openai.com/blog/function-calling-andother-api-updates
[166] OpenAI. 2024. Learning to reason with LLMs. https://openai.com/index/learning-to-reason-with-llms/.
[167] OpenAI. 2025. Introducing OpenAI o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/.
[168] Sophie Ostmeier, Brian Axelrod, Maya Varma, Michael E. Moseley, Akshay Chaudhari, and Curtis Langlotz. 2025. LieRE: Lie Rotational Positional Encodings. arXiv:2406.10322 [cs.CV] https://arxiv.org/abs/2406.10322
[169] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, et al. 2022. Training language models to follow instructions with human feedback. CoRR abs/2203.02155 (2022). https://arxiv.org/abs/2203.02155
[170] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560 (2023). GitHub: https://github.com/letta-ai/letta.
[171] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560 [cs.AI] https://arxiv.org/abs/2310.08560
[172] Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Hassan Awadallah. 2025. Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 6300–6323. doi:10.18653/v1/2025.findings-acl.326
[173] Panasonic Connect. 2024. RAP. GitHub repository. https://github.com/PanasonicConnect/rap
[174] Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2025. BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation. arXiv:2502.03860 [cs.CL] https://arxiv.org/abs/2502.03860
[175] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. arXiv:2304.03442 (2023). doi:10.1145/3586183.3606763
[176] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by selfsupervised prediction. In International conference on machine learning. PMLR, 2778–2787. https://arxiv.org/abs/1705 .05363
[177] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient Context Window Extension of Large Language Models. In International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2309 .00071
[178] Ofir Press et al. 2022. Measuring and Narrowing the Compositionality Gap in Language Models. arXiv preprint arXiv:2210.03350 (2022). https://arxiv.org/abs/2210.03350 Self-Ask with Search.
[179] Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv:2108.12409 [cs.CL] https://arxiv.org/abs/2108.12409
[180] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 5687–5711. doi:10.18653/v1/2023.findings-emnlp.378 ToolRL: Reinforcement Learning for Tool-Use in Large Reasoning Models. arXiv preprint arXiv:2504.13958 (2025). https://arxiv.org/abs/2504.13958
[182] Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. 2025. MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation. arXiv:2409.05591 [cs.CL] https://arxiv.org/abs/2409.05591
[183] Shuofei Qiao, Zhisong Qiu, Baochang Ren, Xiaobin Wang, Xiangyuan Ru, Ningyu Zhang, Xiang Chen, Yong Jiang, et al. 2025. Agentic knowledgeable self-awareness. arXiv preprint arXiv:2504.03553 (2025).
[184] Zile Qiao, Guoxin Chen, Xuanzhong Chen, Donglei Yu, Wenbiao Yin, Xinyu Wang, Zhen Zhang, Baixuan Li, et al. 2025. WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents. arXiv preprint arXiv:2509.13309 (2025).
[185] Yujia Qin, Shiguang Liu, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, et al. 2025. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv preprint arXiv:2501.12326 (2025). arXiv:2501.12326 [cs.AI] https://arxiv.org/abs/2501.12326
[186] Qulox. 2025. SeCom. GitHub repository. https://github.com/Qulox/SeCom
[187] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [cs.CL] https://arxiv.org/abs/2412.15115
[188] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative PreTraining. OpenAI preprint (2018). arXiv:1801.10198.
[189] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290 (2023). https://arxiv.org/abs/2305.18290
[190] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, et al. 2024. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv:2405.14573 arXiv preprint.
[191] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347 (2017). https://arxiv.org/abs/1707.06347
[192] Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, et al. 2025. rStar2-Agent: Agentic Reasoning Technical Report. arXiv:2508.20722 [cs.CL] https://arxiv.org/abs/2508.20722
[193] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300 (2024). https://arxiv.org/abs/2402.03300
[194] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. In Advances in Neural Information Processing Systems, Vol. 36. https://proceedings.neurips.cc/paperfiles/paper/2023/file/77c33e6a367922d003ff102ffb92b658-PaperConference.pdf
[195] Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, Zhaoxiang Liu, and Shiguo Lian. 2025. DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models. arXiv:2503.04472 [cs.LG] https://arxiv.org/abs/2503.04472
[196] Yifan Sheng et al. 2023. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv:2311.03285 (2023).
[197] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 8634–8652. https://proceedi ngs.neurips.cc/paperfiles/paper/2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf
[198] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016), 484–489. doi:10.1038/nature16961
[199] David Silver and Richard S. Sutton. 2025. Welcome to the Era of Experience. Preprint / Chapter in *Designing an Intelligence* (MIT Press) (2025).
[200] Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. 2025. Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning. arXiv:2505.01441 [cs.AI] https://arxiv.org/abs/2505.01441
[201] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592 (2025). https://arxiv.org/abs/2503.05592
[202] Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning. arXiv preprint arXiv:2505.17005 (2025). 1g RoPE; use official project page if preferred.
[204] Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. 2025. Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains. arXiv:2503.23829 [cs.CL] https://arxiv.org/abs/2503.23829 [205] Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. 2025. ZeroSearch: Incentivize the Search Capability of LLMs without Searching. arXiv:2505.04588 [cs.CL] https://arxiv.org/abs/2505.04588
[206] Haoran Sun and Shaoning Zeng. 2025. H-MEM: Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents. arXiv preprint (2025). arXiv:2507.22925 [cs.AI] https://arxiv.org/abs/2507.22925
[207] Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, and Zhiyong Wu. 2025. OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 5555–5579. doi:10.18653/v1/2025.acl-long.277
[208] Shuang Sun, Huatong Song, Yuhao Wang, Ruiyang Ren, Jinhao Jiang, Junjie Zhang, Fei Bai, Jia Deng, et al. 2025. SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis. arXiv:2505.16834 [cs.CL] https://arxiv.org/abs/2505.16834
[209] Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. 2025. GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 19477–19486. doi:10.1109/CVPR52734.2025.01814
[210] Suno. 2023. Suno: AI Music Generation Platform. https://suno.ai [211] G. Surís, S. Adeli, D. Bashkirova, J. Carreira, and C. Vondrick. 2023. ViperGPT: Visual Inference via Python Execution for Reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). https://openaccess .thecvf.com/content/ICCV2023/html/SurisViperGPTVisualInferenceviaPythonExecutionforReasoningI CCV2023paper.html [212] Liujian Tang, Shaokang Dong, Yijia Huang, Minqi Xiang, Hongtao Ruan, Bin Wang, Shuo Li, Zhiheng Xi, et al. 2025. MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning. arXiv:2508.03700 [cs.HC] https://arxiv.org/abs/2508.03700
[213] Yimin Tang, Yurong Xu, Ning Yan, and Masood Mortazavi. 2024. Enhancing Long Context Performance in LLMs Through Inner Loop Query Mechanism. arXiv:2410.12859 [cs.CL] https://arxiv.org/abs/2410.12859 [214] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, et al. 2025. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599 [cs.AI] https://arxiv.org/abs/2501.12599
[215] Qwen Team. 2025. Qwen2.5-1M Collection (Hugging Face). https://huggingface.co/collections/Qwen/qwen25-1m679325716327ec07860530ba. Models and tech report links.
[216] Qwen Team. 2025. QwQ-32B: Embracing the Power of Reinforcement Learning. https://qwenlm.github.io/blog/qwq32b/
[217] Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. ReFT: Reasoning with Reinforced Fine-Tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 7601–7614. doi:10.18653/v1/2024.acl-long.410
[218] Unknown. 2025. Nemori: Self-Organizing Agent Memory Inspired by Cognitive Science. arXiv preprint (2025). arXiv:2508.03341 [cs.AI] https://arxiv.org/abs/2508.03341 [219] Kumar Valmeekam et al. 2024. COPPER: Cooperative Planning with Preference Elicitation and Reinforcement. In NeurIPS 2024. https://openreview.net/forum?id=9xJPCjB8rc
[220] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS). Curran Associates, Inc., 5998–6008.
[221] Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, et al. 2023. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214 (2023). https://arxiv.org/abs/2310.03214 [222] et al. Wan. 2023. Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory. arXiv preprint (2023). arXiv:2311.09227 [cs.CL] https://arxiv.org/abs/2311.09227 [223] Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. 2025. SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution. arXiv preprint arXiv:2505.20732 (2025).
[224] ng, Ji ng, Ji g， ng LL M Agents via Stepwise Progress Attribution. arXiv:2505.20732 [cs.CL] https://arxiv.org/abs/2505.20732
[225] Han Wang, Ziwei Wang, Xinyi Yang, Keming Lu, Qi Liu, Shumin Deng, Wenxuan Zhang, and Minlie Huang. 2025. Acting Less is Reasoning More: Teaching Models Optimal Tool Calls via Reinforcement Learning. arXiv preprint arXiv:2504.14870 (2025). https://arxiv.org/abs/2504.14870
[226] Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, et al. 2025. UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning. arXiv:2509.02544 [cs.AI] https://arxiv.org/abs/2509.02544
[227] Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2024. Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692 (2024).
[228] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. arXiv preprint arXiv:2406.01014 (2024). https://arXiv.org/abs/2406.01014
[229] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. MobileAgent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv preprint arXiv:2401.16158 (2024). https://arXiv.org/abs/2401.16158
[230] Junyang Wang, Haiyang Xu, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Jitao Sang. 2025. Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automation. arXiv preprint arXiv:2505.13887 (2025).
[231] Kun Wang and Shuicheng Yan. 2025. G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems. ResearchGate Preprint. https://www.researchgate.net/publication/392531451
[232] Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678 (2023). https://arxiv.org/abs/2303.07678
[233] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. 2024. MathShepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv:2312.08935 [cs.AI] https: //arxiv.org/abs/2312.08935
[234] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of ACL-IJCNLP. GitHub: https://github.com/microsoft/K-Adapter.
[235] Xidong Wang, Yue Geng, Zihan Zhang, Hong Liu, and Zhen Lei. 2025. Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding. In Findings of NAACL. https://arxiv.org/abs/2502.06020
[236] Xiao Wang, Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2023. Generative query reformulation for effective adhoc search. arXiv preprint arXiv:2308.00415 (2023). https://arxiv.org/abs/2308.00415
[237] Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, et al. 2025. OpenCUA: Open Foundations for Computer-Use Agents. arXiv:2508.09123 [cs.AI] https://arXiv.org/abs/2508.09123
[238] Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, et al. 2024. MEMORYLLM: Towards Self-Updatable Large Language Models. arXiv preprint arXiv:2402.04624 (2024). arXiv:2402.04624 [cs.CL] doi:10.48550/arXiv.2402.04624
[239] Zihao Wang, Yibo Jiang, Jiahao Yu, and Heqing Huang. 2025. The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them). arXiv preprint arXiv:2505.00626 (2025). https://arxiv.org/abs/2505.00626
[240] Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. 2024. Agent Workflow Memory. arXiv preprint (2024). arXiv:2409.07429 [cs.AI] https://arxiv.org/abs/2409.07429
[241] Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, et al. 2025. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv:2504.20073 [cs.LG] https://arxiv.org/abs/2504.20073
[242] Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. 2025. StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization. arXiv:2505.15107 [cs.CL] https://arxiv.or g/abs/2505.15107
[243] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516 [cs.CL] https://arxiv.org/abs/2504.12516
[244] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL]
[245] Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. 2025. SIM-CoT: Supervised Implicit Chain-of-Thought. arXiv:2509.20317 [cs.CL] https://arxiv.org/abs/2509.20317
[246] Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, and Li Du. 2025. AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning. arXiv preprint arXiv:2507.21836 (2025). https://arxiv.org/abs/2507
[247] Jiong Wu et al. 2025. MMSearch-R1: Incentivizing LMMs to Search. arXiv preprint arXiv:2506.20670 (2025). https: //arxiv.org/abs/2506.20670
[248] Jinyang Wu, Mingkuan Feng, Shuai Zhang, Fangrui Lv, Ruihan Jin, Feihu Che, Zengqi Wen, and Jianhua Tao. 2025. Boosting Multimodal Reasoning with Automated Structured Thinking. arXiv:2502.02339 [cs.CL] https: //arxiv.org/abs/2502.02339
[249] Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, et al. 2025. WebDancer: Towards Autonomous Information Seeking Agency. arXiv:2505.22648 [cs.CL] https: //arxiv.org/abs/2505.22648
[250] Mingyuan Wu, Jingcheng Yang, Jize Jiang, Meitang Li, Kaizhuo Yan, Hanchao Yu, Minjia Zhang, Chengxiang Zhai, and Klara Nahrstedt. 2025. VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use. arXiv:2505.19255 [cs.LG] https://arxiv.org/abs/2505.19255
[251] Qingyun Wu et al. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv preprint arXiv:2308.08155 (2023). https://arxiv.org/abs/2308.08155
[252] Weiqi Wu, Xin Guan, Shen Huang, Yong Jiang, Pengjun Xie, Fei Huang, Jiuxin Cao, Hai Zhao, and Jingren Zhou. 2025. MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability. arXiv:2505.20285 [cs.CL] https://arxiv.org/abs/2505.20285
[253] Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Yong Jiang, et al. 2025. ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization. arXiv preprint arXiv:2509.13313 (2025). https://arxiv.org/abs/2509.13313
[254] Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. 2022. Memorizing Transformers. In ICLR. arXiv:2203.08913 Implementation: https://github.com/lucidrains/memorizing-transformers-pytorch.
[255] Yongliang Wu, Yizhou Zhou, Ziheng Zhou, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Mingzhuo Yang, and Xu Yang. 2025. On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification. ArXiv abs/2508.05629 (2025). https://api.semanticscholar.org/CorpusID:280545885
[256] et al. Xia. 2025. COLA: A Multi-agent Framework for Generating Large-Scale UI Automation Datasets on Windows. arXiv preprint (2025). https://arxiv.org/abs/2503.17086
[257] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453 [cs.CL] https://arxiv.org/abs/2309.17453
[258] Han Xiao, Guozhi Wang, Yuxiang Chai, Zimu Lu, Weifeng Lin, Hao He, Lue Fan, Liuyang Bian, et al. 2025. UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents. arXiv preprint arXiv:2505.21496 (2025).
[259] Zhiyou Xiao, Qinhan Yu, Binghui Li, Geng Chen, Chong Chen, and Wentao Zhang. 2025. M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generation. arXiv preprint arXiv:2508.06328 (2025). https://arxiv.org/abs/2508.06328
[260] Tao Xiong, Xavier Hu, Yurun Chen, Yuhang Liu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, and Shengyu Zhang. 2025. GUI-PRA: Process Reward Agent for GUI Tasks. arXiv preprint arXiv:2509.23263 (2025).
[261] Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408 (2023). https://arxiv.org/abs/2310.04408
[262] Jing Xu, Arthur Szlam, and Jason Weston. 2022. Beyond Goldfish Memory: Long-Term Open-Domain Conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 5180–5197. doi:10.18653/v1/2022.acl-long.356
[263] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-MEM: Agentic Memory for LLM Agents. arXiv preprint (2025). arXiv:2502.12110 [cs.CL] https://arxiv.org/abs/2502.12110
[264] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2025. Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=PlihOwfx4r
[265] et al. Yan. 2025. MemAgent: Reshaping Long-Context LLM with Multi-Conversation Memory and Reinforcement Learning. arXiv preprint (2025). arXiv:2507.02259 [cs.CL] https://www.alphaxiv.org/overview/2507.02259v1
[266] Shuo Yan et al. 2025. Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning. arXiv preprint (2025). arXiv:2508.19828 [cs.CL] https://arxiv.org/abs/2508.19828
[267] Yang Yan, Yu Lu, Lizhi Ma, and Zhenzhong Lan. 2024. Planning with MCTS: Enhancing Problem-Solving in Large Language Models. https://openreview.net/forum?id=sdpVfWOUQA
[268] An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Jingren Zhou, et al. 2025. Qwen2.5-1M Technical Report. arXiv preprint arXiv:2501.15383 (2025). arXiv:2501.15383 [cs.CL] doi:10.48550/arXiv.2501.15383
[269] Chenyu Yang, Shiqian Su, Shi Liu, Xuan Dong, Yue Yu, Weijie Su, Xuehui Wang, Zhaoyang Liu, et al. 2025. ZeroGUI: Automating Online GUI Learning at Zero Human Cost. arXiv:2505.23762 [cs.AI] https://arxiv.org/abs/2505.23762
[270] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://papers.nips.cc/paperfiles/paper/2024/file/5a7c947568c1b1328ccc523017 2e1e7c-Paper-Conference.pdf
[271] Kuo Yang, Xingjie Yang, Linhui Yu, Qing Xu, Yan Fang, Xu Wang, Zhengyang Zhou, and Yang Wang. 2025. MasHost Builds It All: Autonomous Multi-Agent System Directed by Reinforcement Learning. arXiv preprint arXiv:2506.08507 (2025).
[272] Xiao Yang, Jiawei Chen, Jun Luo, Zhengwei Fang, Yinpeng Dong, Hang Su, and Jun Zhu. 2025. MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments. arXiv preprint arXiv:2506.01616 (2025). https://arXiv.org/abs/2506.01616
[273] Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. 2025. Aria-UI: Visual Grounding for GUI Instructions. In Findings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 22418–22433. doi:10.18653/v1/2025.findings-acl.1152
[274] Shunyu Yao. 2025. The Second Half of AI. Blog post, https://ysymyth.github.io/The-Second-Half/.
[275] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://proceedings.neurips.cc/paperfiles/paper/2022/hash/82ad13ec01f9fe44c01cb91814fd7b8c-PaperConference.pdf
[276] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS) 2023. 11809–11822.
[277] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=WE_vluYUL-X
[278] Yao-Hung Hubert Yao et al. 2024. HippoRAG: Memory-based Retrieval for LLMs. arXiv preprint (2024). https: //arxiv.org/abs/2404. Exact identifier TBD; see project repo https://github.com/oyooyo/hipporag.
[279] Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, et al. 2025. Mobile-Agent-v3: Foundamental Agents for GUI Automation. arXiv preprint arXiv:2508.15144 (2025). https://arxiv.or g/abs/2508.15144
[280] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. LIMO: Less is More for Reasoning. arXiv:2502.03387 [cs.CL] https://arxiv.org/abs/2502.03387
[281] Yufan Ye, Ting Zhang, Wenbin Jiang, and Hua Huang. 2025. Process-Supervised Reinforcement Learning for Code Generation. arXiv:2502.01715 [cs.SE] https://arxiv.org/abs/2502.01715
[282] Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 183–192.
[283] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. Demystifying Long Chain-of-Thought Reasoning in LLMs. arXiv:2502.03373 [cs.CL] https://arxiv.org/abs/2502.03373
[284] Yaobo Yin et al. 2024. Optimus-1: On-device Data of Task Execution and Decision Making for LLM-based Agents. arXiv preprint (2024). arXiv:2410.05623 [cs.AI] https://arxiv.org/abs/2410.05623
[285] Q. Yu et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning Framework. arXiv preprint arXiv:2503.14476 (2025). https://arxiv.org/abs/2503.14476
[286] Zhuohao Yu, Weizheng Gu, Yidong Wang, Zhengran Zeng, Jindong Wang, Wei Ye, and Shi-Bo Zhang. 2024. Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation. https://api.semanticscholar.org/ CorpusID:274859836
[287] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. 2025. Free Process Rewards without Process Labels. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267), Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (Eds.). PMLR, 73511–73525. https://proceedings.mlr.press/v267/yuan25c.html
[288] Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. 2025. Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training. arXiv preprint arXiv:2501.11425 (2025).
[289] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning. https://arxiv.org/ abs/2401.10020
[290] Wenzhen Yuan, Shengji Tang, Weihao Lin, Jiacheng Ruan, Ganqu Cui, Bo Zhang, Tao Chen, Ting Liu, et al. 2025. Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedback. arXiv preprint arXiv:2508.12338 (2025).
[291] Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, et al. 2025. Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning. arXiv preprint arXiv:2505.12370 (2025).
[292] Yurun Yuan and Tengyang Xie. 2025. Reinforce LLM Reasoning through Multi-Agent Reflection. arXiv preprint arXiv:2506.08379 (2025).
[293] Sizhe Yuen, Francisco Gomez Medina, Ting Su, Yali Du, and Adam J. Sobey. 2025. Intrinsic Memory Agents: Heterogeneous Multi-Agent LLM Systems through Structured Contextual Memory. arXiv preprint arXiv:2508.08997 (2025). https://arxiv.org/abs/2508.08997
[294] Manzil Zaheer et al. 2020. Big Bird: Transformers for Longer Sequences. In NeurIPS. TBD Use NeurIPS proceedings page if possible.
[295] Guangtao Zeng, Maohao Shen, Delin Chen, Zhenting Qi, Subhro Das, Dan Gutfreund, David Cox, Gregory Wornell, et al. 2025. Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering. arXiv preprint arXiv:2505.23604 (2025). https://arxiv.org/abs/2505.23604
[296] Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. 2025. Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment. arXiv:2505.11821 [cs.LG] https://arxiv.org/ abs/2505.11821
[297] Zhixiong Zeng, Jing Huang, Liming Zheng, Wenkang Han, Yufeng Zhong, Lei Chen, Longrong Yang, Yingjie Chu, Yuzhi He, and Lin Ma. 2025. UItron: Foundational GUI Agent with Advanced Perception and Planning. arXiv:2508.21767 [cs.CV] https://arxiv.org/abs/2508.21767
[298] Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Jinpeng Wang, Zaiyuan Wang, Yang Yang, et al. 2025. FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction. arXiv preprint arXiv:2508.11987 (2025). https://arxiv.org/abs/2508.11987
[299] Zep Inc. 2025. Zep: The AI Memory Server. Documentation. https://docs.getzep.com/
[300] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. AppAgent: Multimodal Agents as Smartphone Users. arXiv preprint arXiv:2312.13771 (2023). https://arxiv.org/abs/2312.13771
[301] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search. arXiv:2406.03816 [cs.CL] https://arxiv.org/abs/2406.03816
[302] Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. 2024. G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782 (2024).
[303] Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024. Android in the Zoo: Chain-of-Action-Thought for GUI Agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 12016–12031. doi:10.18653/v1/2024.findings-emnlp.702
[304] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, et al. 2024. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762 (2024).
[305] Kaiyan Zhang et al. 2025. Tool-N1: Training General Tool-Use in Large Reasoning Models. arXiv preprint arXiv:2505.00024 (2025). https://arxiv.org/abs/2505.00024
[306] Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. 2025. Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization. arXiv:2504.05812 [cs.LG] https://arxiv.org/abs/2504 .05812
[307] Shouyuan Zhang et al. 2024. Efficient Language Modeling with Lightning Attention. In ICML / PMLR. TBD Prefer PMLR paper page.
[308] Shouyuan Zhang et al. 2024. Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths. arXiv preprint (2024). TBD
[309] Taolin Zhang, Dongyang Li, Qizhou Chen, Chengyu Wang, Longtao Huang, Hui Xue, Xiaofeng He, and Jun Huang. 2024. R4: Reinforced Retriever-Reorder-Responder for Retrieval-Augmented Large Language Models. arXiv:2405.02659 [cs.CL] https://arxiv.org/abs/2405.02659
[310] Yanfei Zhang. 2025. Agent-as-Tool: A Study on the Hierarchical Decision Making with Reinforcement Learning. arXiv:2507.01489 [cs.AI] https://arxiv.org/abs/2507.01489
[311] Yuxiang Zhang, Jiangming Shu, Ye Ma, Xueyuan Lin, Shangxi Wu, and Jitao Sang. 2025. Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Tasks. arXiv:2510.12635 [cs.AI] https://arxiv.org/abs/2510 .12635
[312] Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. 2024. o1-Coder: an o1 Replication for Coding. arXiv:2412.00154 [cs.SE] https://arxiv.org/abs/2412.00154
[313] Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Yuhang Wang, Jinlin Xiao, and Jitao Sang. 2024. OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning. arXiv:2412.16849 [cs.AI] https://arxiv.org/abs/2412.16849
[314] Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Xinyan Wen, and Jitao Sang. 2025. Agent models: Internalizing Chain-ofAction Generation into Reasoning models. arXiv:2503.06580 [cs.AI] https://arxiv.org/abs/2503.06580
[315] Zeyu Zhang, Quanyu Dai, Rui Li, Xiaohe Bo, Xu Chen, and Zhenhua Dong. 2025. Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework. arXiv preprint (2025). arXiv:2508.16629 [cs.LG] https: //arxiv.org/abs/2508.16629
[316] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In NeurIPS 2023. https://arxiv.org/abs/2306.14048
[317] Zhuosheng Zhang and Aston Zhang. 2024. You Only Look at Screens: Multimodal Chain-of-Action Agents. arXiv:2309.11436 [cs.CL] https://arxiv.org/abs/2309.11436
[318] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The Lessons of Developing Process Reward Models in Mathematical Reasoning. arXiv:2501.07301 [cs.CL] https://arxiv.org/abs/2501.07301
[319] Linxi Zhao, Sofian Zalouk, Christian K. Belardi, Justin Lovelace, Jin Peng Zhou, Ryan T. Noonan, Dongyoung Go, Kilian Q. Weinberger, Yoav Artzi, and Jennifer J. Sun. 2025. Pre-training Large Memory Language Models with Internal and External Knowledge. arXiv:2505.15962 (2025).
[320] Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, and Limin Liu. 2025. R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning. arXiv preprint arXiv:2506.04185 (2025). https://arxiv.org/abs/ 2506.04185
[321] Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2024. Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions. arXiv:2411.14405 [cs.CL] https://arxiv.org/abs/2411.14405
[322] Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. ProcessBench: Identifying Process Errors in Mathematical Reasoning. arXiv:2412.06559 [cs.AI] https://arxiv.org/abs/2412.06559
[323] Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. 2024. Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control. In ICLR. arXiv:2306.07863
[324] Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments. arXiv:2504.03160 [cs.AI] https://arxiv.org/abs/2504.03160
[325] Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. 2025. DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning. arXiv:2505.14362 [cs.CV] https: //arxiv.org/abs/2505.14362
[326] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2023. MemoryBank: Enhancing Large Language Models with Long-Term Memory. arXiv:2305.10250 [cs.CL] https://arxiv.org/abs/2305.10250
[327] Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. 2025. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents. arXiv preprint (2025). arXiv:2506.15841 [cs.AI] https://arxiv.org/abs/2506.15841

迈向模型原生智能体式人工智能的范式转变综述.pdf免费PDF下载