Post-Training Verifiable Agents - Technical Guide

This guide distills the core concepts and methodologies necessary for understanding and building post-training verifiable agentic AI systems, contrasting them with traditional models and outlining the critical training steps. It is based on a lecture by Jiantao Jiao’s lecture video in the 2025 Agentic AI MOOC by Berkeley RDI.

Furthermore, you can watch our lecture summary video here.

1. The Defining Shift: Agentic vs. Traditional LLMs

Agentic models represent a fundamental evolution from earlier large language models (LLMs) by changing the primary axis of alignment from human preference to verifiable environmental feedback.

LLMvsAgent

1.1. Maximizing Human Preference (Earlier Models)

Earlier chat models were primarily Human Aligned Models, optimized for conversation and designed to provide interactions that maximize human preference. These models focused on becoming great chat-bots, often using Reinforcement Learning from Human Feedback (RLHF) and sophisticated reward models to rank responses based on subjective human feeling.

1.2. Maximizing Verifiable Rewards (Agentic Models)

Agentic Models are Environment Feedback Aligned Models. They are designed to provide interactions that maximize verifiable rewards in addition to human preference. This shift is necessary because conversation is not verifiable and often lacks a single correct answer. Many real-world tasks (especially enterprise applications) require complex interaction with tools and environments and have one (or a few) correct answers, making correctness a verifiable mandate.

“Agentic Models: Environment Feedback Aligned Models, designed to provide interactions that maximize verifiable rewards (in addition to human preference).”

This focus on maximizing correctness means agentic models are trained to accomplish specific tasks, requiring interaction with many entities (the user, the environment, tools) to understand the full state before executing precisely.

2. Step 1: Getting Good Verifiable Training Data

The journey to building verifiable agents begins with acquiring high-quality verifiable training data. Agentic models maximize verifiable reward by training on three core components: the Environment, Tools, and the Verifier. Diversity across all three components is considered critical.

tool-env-verifier

2.1. Environment

The environment provides context and state information.

Definition: Tokens consumed by the model that inform it about the environment’s current state and the user’s intention.
Examples: A code repository, a web browser, a database, or sales assistant scenarios.
Importance: The environment defines the scope within which the agent operates and must encode the full state, including files, scripts, and system prompts.

2.2. Tools

Tools allow the agent to gather information and manipulate the environment.

Definition: Software tools (APIs) consume tokens generated by the model to provide additional relevant information or change the environment’s state.
Examples: Software API/MCP tools (weather, news), Database tools (search_database, run_sql), Browsing tools (search_google), and Coding tools (run_python).
Importance: Strong agents must be able to decompose a task, figure out the right tools to call, and execute precise API calls, ensuring interaction with the rigidly formatted software systems is successful.

2.3. Verifier

The verifier is responsible for assessing correctness and maximizing the reward.

Definition: Mechanisms used to check whether the agent has succeeded in accomplishing the task. Ideally, verifiers should be viewed as a vector evaluating quality from multiple perspectives, not just a single binary check.
Examples: Unit tests for code, math checkers for proofs, DOM scripts for web tasks, or ground truth state verifiers.
Verifier Quality Insight: Verifier quality is extremely important, especially for difficult prompts. A high-quality verifier must be able to reward all paths that reach the correct ground truth, while strictly penalizing wrong answers. For instance, if an instruction demands the “simplest form” of an answer, the verifier must enforce that nuance. Errors like false positives and false negatives must be minimized as they can produce catastrophic results.

3. Step 2: Ensuring Holistic Evaluation

Building evaluation systems is necessary to define what “works” and ensure the agent generalizes beyond the limited scope of the training data.

3.1. Requirements for Agent Intelligence

A good agent must satisfy several key requirements for success in the wild, which must be tracked quantitatively:

Tool and Usecase Diversity: Agents are expected to work with many different tools and for many different use cases.
Vaguely Specified Instructions: Agents must handle instructions that are vaguely specified, requiring the system to understand human intentions proactively.
Software System Robustness: Agents must be robust across different software systems (different harnesses) without breaking when APIs or environments are slightly modified.
General Robustness: Agents are expected to debug issues and deal with complex errors that emerge in daily life.

3.2. The Holistic Approach to Measurement

Evaluation needs to be holistic to truly measure intelligence.

Benchmark Diversity: Utilize a wide variety of benchmarks (many tasks and verifiers) covering different capabilities, such as SWEBench (coding), AIME (math), HLE (knowledge), and BrowserComp.
Harness Testing: Evaluation must test the agent’s ability to explore different environments by swapping out the harnesses (the specific set of tools/APIs available in the sandbox). A drop in accuracy when harnesses are swapped indicates a failure in generalization.
Tool Testing: One each task, testing agents on many tools to ensure they can handle different action spaces.

“When we choose a benchmark suite, we are defining what intelligence is. We MUST make sure it is a holistic and good measure!”

3.3. Quantifying Benchmark Quality

To prevent overfitting and track genuine progress, benchmarks themselves must be continuously monitored for quality.

Hardness: Measures if the benchmark is challenging enough to differentiate models. If all LLMs score highly (e.g., saturated benchmarks like GSM8K), improvement metrics become unclear.
Separability: Checks if the benchmark can reliably separate the capabilities of stronger LLMs from weaker ones. If weak and strong models produce similar scores, the benchmark loses credibility.
Diversity: Ensures the benchmark covers a wide variety of tasks and prompt styles, preventing high scores based on similarity within a small dataset.

4. Step 3: Training the Agent Well (SFT and RL Recipes)

Training an agent means maximizing its ability to use tools to achieve correctness. This process is typically divided into two key stages to balance imitation and exploration.

Two-stage training

4.1. Stage 1: Supervised Fine-Tuning (SFT)

SFT serves as imitation learning to “jump-start” the system.

Goal: Imitate demonstration trajectories that achieve correctness to minimize any silly or non-meaningful attempts (effective reducing wasteful compute).
Requirement: SFT should be light and use diverse demonstration samples. Heavy SFT (over-SFT’ing) restricts the model’s ability to explore and generalize later, especially on tasks outside the training distribution.

4.2. Stage 2: Reinforcement Learning (RL)

RL allows the model to explore its environment and reinforce intelligent behavior.

Goal: Let the model explore and reinforce diverse, great attempts while discouraging unsuccessful ones. RL is here to truly reinforce intelligence.
Core Challenge: Training attempts consume compute. The algorithm must strike a balance: attempts should be very different from each other to generalize well, but they should not be “very silly attempts that has no change of success”.

4.3. The Three Pillars of Good RL

Good RL algorithms must focus on: Train Longer, Train Harder, and Sample Better.

4.3.1. Train Longer

Core Idea & Importance: Crucial goal is to prevent entropy collapse—where the model saturates and only produces the exact same answers, lacking diversity and exploration. High entropy means higher uncertainty in token generation, leading to diverse trajectories.

Training Interventions:

Reducing Biased Updates: Favoring on-policy learning, where the model learns best from its own mistakes.
Balance Update Strength: Decoupling clipping thresholds in algorithms like GRPO (epsilon_high should be larger than epsilon_low to encourage increasing low-probability tokens for exploration).
Encourage Entropy Directly: Adding a specific loss function to maintain a target entropy level.

4.3.2. Train Harder

Core Idea & Importance: Training only on easy tasks limits learning. However, blindly throwing in super hard prompts also fails. The key is improving the Learning Signal by ensuring the model’s confidence is meaningfully correlated with the reward.

Training Interventions:

Rewarding Harder Prompts More: To reflect their difficulty.
Penalizing Long Generations: For simple answers helps drive efficiency.

4.3.3. Sample Better

Core Idea & Importance: Aims to generate better, more diverse responses during the rollout stage while managing limited compute.

Training Interventions:

Parallel Reasoning, (GenSelect): Scaling up compute to sample many trajectories (e.g., 32+ answers).
Beam Search Over Reasoning, (DeepConf): Confidence-based majority voting to discard trajectories with very low generation confidence before aggregation, ensuring only meaningful outputs are considered.

DeepConf

5. The Path Forward: Unresolved Challenges

The development of verifiable agentic AI is still very immature, lacking convergence on the “right algorithm”. The central challenge remains bridging the gap between human learning and system learning.

Core Open Questions

Environment Collection Design: How can we mimic human learning by exposing models to many different environments? This requires defining the right design for a collection of open-source environments, evaluations, and algorithms.
Stability and Prioritization: Humans focus on some lessons more than others. What is the right algorithm that allows the AI to do the same without compromising training stability?
Balancing Imitation and Exploration: Humans learn from both their own exploration and from teachers. What is the correct balance between SFT (teaching) and RL (exploration), and can they be effectively integrated simultaneously?
Defining Intelligence: How can we effectively compare different definitions of intelligence encapsulated in various benchmark suites?

The collective community effort—including crowdsourcing environments, verifiers, and recipes—is essential to scale up RL and define intelligence holistically.

Following along: I post these on LinkedIn — that’s where the conversation happens now.