Learning from Real-World Experience

How users interact with our products in the real world is a valuable source for model training. The data is abundant, reflects the actual distribution of user tasks, and captures user corrections and tool failures that synthetic environments may miss.

A common way to learn from real-world data is rejection sampling fine-tuning: judge each session’s outcome, keep the successful ones, and train the model to imitate them. But a successful outcome does not mean every step was correct, so imitating the whole trajectory risks reinforcing bad intermediate behaviours in addition to good ones. Discarding unsuccessful sessions also loses critical evidence of where the model falls short.

We combine rejection sampling fine-tuning with hint-guided self-distillation to learn from both successful and unsuccessful sessions. A hint is a short corrective instruction grounded in information the model already had when it made the mistake. Useful steps from successful sessions remain imitation targets, while grounded hints turn avoidable mistakes into correction targets.

User corrections and tool errors add supervision about what went wrong along the way, on top of whether the task ultimately succeeded. We apply this stage after reinforcement learning in synthetic environments to help close the gap between those environments and real-world use.

In live use, a later post-trained GLM-5.2 checkpoint reduced tool-call failures by 21.2% relative to an earlier trained checkpoint. For GLM-5.3, post-training reduced tool-call failures by 33.0% and the average estimated probability of medium-or-high user dissatisfaction by 17.2% relative to the stock model.

Synthetic tasks let us create controlled problems with outcomes that are easy to verify, but they are typically limited to a single prompt and can only cover the scenarios we explicitly design.

In contrast, users in real-world sessions bring diverse instructions, context, and tool combinations into sessions that unfold over multiple turns of interaction. Such real-world usage naturally yields two valuable signals of mistakes: user corrections reveal where the agent failed to deliver what they wanted, and tool errors show where its actions broke down in real environments. Our sampling pipeline respects users’ privacy and training choices by excluding sessions with personally identifiable information (PII) and from users who have opted out.

Real-world data therefore gives us failure cases that a synthetic environment may not expose and often comes with where the failure occurred already baked in.

When training on real-world data, standard rejection sampling fine-tuning (RFT) selects successful sessions based on their final outcomes, without explicitly using user and tool feedback to distinguish good intermediate actions from bad ones. An agent may recover from a bad tool call or correct a mistake after user feedback and still deliver the right output at the end.

Discarding failed sessions also limits sample efficiency. A session that never reaches a good result could still reveal a clear, avoidable mistake as useful training signals.

We address these limitations by separating two decisions: which sessions contain behaviour worth imitating, and which turns contain mistakes worth correcting. The data pipeline makes these decisions separately, then assigns model turns one of three treatments:

Imitate: non-error turns in successful sessions receive ordinary Cross-Entropy (CE) loss, raising the probability of recorded tokens.
Correct: in both successful and unsuccessful sessions, error turns with a valid hint receive a Kullback–Leibler (KL) divergence loss, moving the model toward the next-token predictions it makes when it sees the hint.
Keep as context: turns in unsuccessful sessions that lack a validated hint, and marked bad turns without a validated hint in either session type, serve as context but don’t receive either loss.

As a result, successful sessions can supply both imitation and correction targets, while unsuccessful sessions supply only correction targets. All remaining turns, including system, user, and tool output, stay as context.

Session-routing diagram: successful sessions provide CE imitation and eligible hinted corrections; unsuccessful sessions provide only eligible corrections. — Current session routing and assistant-content masks. Successful sessions contribute imitation targets; retained feedback records can contribute hinted corrections even when the session is unsuccessful. Other content stays as context. Incomplete sessions are excluded. The non-final yield/stop-token CE exception is not drawn, and this diagram does not establish the mixture used by an earlier checkpoint.

The imitation part uses the standard RFT. The corrective part uses On-Policy Self-Distillation (OPSD): the same model acts as teacher and student, with the teacher given privileged information. Here, that privileged information is a short hint explaining an avoidable mistake. We train the student to match the teacher’s next-token predictions without being handed the hint.

Both parts matter. Correction-only training can expose a shortcut: teacher and student can agree by ignoring context rather than learning the correction. The CE term anchors training to useful behaviour, helping discourage agreement that comes from ignoring context.

The post-trained student then becomes a model option in Computer. Fresh sessions from live use pass through the same eligibility and privacy filters before entering another round of the training pipeline.

Next, we walk through a concrete example that contains both imitation and correction.

Consider an eventual successful session where the user asks for a comparison of asyncio changes in Python 3.13 using official documentation. Our tool documentation states that search.web returns a list of hits, each with a url field.

The model first runs a valid search for one version as part of gathering evidence for the comparison, which is a good behaviour to imitate.

However, the next code-execution call incorrectly treats that list as a dictionary. The code tool returns TypeError: list indices must be integers or slices, not str. Because the return type was documented before the call, it is a mistake the model should have been able to avoid. A hint with the right call can be generated to correct that action.

Illustrative coding session: system and user input and tool errors remain context; useful recorded search and recovery calls receive CE in a successful session; a marked erroneous list lookup receives KL toward a teacher given a grounded hint. — An illustrative coding session shows three training roles: context-only inputs and tool outputs (gray), CE imitation of useful recorded calls (dark), and KL correction of a marked mistake (teal). Here, the model incorrectly treats a list of search hits as a dictionary; a teacher-only hint grounded in earlier tool documentation guides the correction. CE targets apply only when the completed session is judged successful, while an unsuccessful session can still supply an eligible KL correction target.

The training pipeline has four stages: select sessions and judge their outcomes, identify mistakes and create hints from user feedback and tool errors, generate next-token probabilities from both the teacher and the student, and train the model with a combined CE and KL loss.

The sampling pipeline draws from a training-eligible subset of real-world Computer sessions served by GLM 5.2. Training opt-outs and the pipeline’s PII-related exclusions are applied before selecting examples for this workflow.

The initial screening process selects for candidates rated 4 or 5 on a five-point task-difficulty scale by an LLM judge and splits them into 2 groups:

The user-feedback group contains sessions with a genuine user follow-up or a medium-to-high dissatisfaction flag. We find these high-intent follow-ups especially useful for learning: users are actively trying to complete a task, and their corrections pinpoint what the model missed.
The tool-error group contains sessions with failed calls flagged by rule-based checks, excluding recognized infrastructure failures and declined authorisation.

Two LLM judges then assess the final delivery against the user’s request; both judges must return a positive verdict to label the session as “successful”. For cases that deliver artefacts, the outcome stage can reconstruct and inspect the delivery, falling back to its text-based judgments if that stage fails. Sessions classified as incomplete, such as work waiting on the user, are excluded.

This step identifies potential sources of learning signals; however, not every error is attributable to the model, as some arise from ambiguous instructions or infrastructure failures. Later annotation determines which candidate mistakes are actually avoidable and suitable for training.

Consider a sample session in the user-feedback group, where the user asks, “How do I find my w3 on paychex”. The model treats “w3” as a typo for W-2, searches for W-2 instructions, and explains how to find that form. The user’s follow-up, “I meant the W-3 transmittal form,” identifies the mistake, even if the session ultimately succeeds.

The complaint helps locate the mistake, but the root cause is the earlier decision to reinterpret “w3” as W-2. The original request makes the mistake avoidable: the model could have preserved W-3 or asked for clarification. It turns out that the last assistant turn before the complaint is the root cause only about half of the time.

Our annotation pipeline maps cause and effect systematically within each user-feedback session. The annotator first screens for a missed requirement or an intention supported by earlier context. Three LLM judges then identify the responsible turns, requiring at least two to agree on a given turn. Sometimes an earlier statement explicitly establishes the requirement. In other cases, no single prior statement does so, but multiple pieces of the preceding conversation jointly make the intended behaviour inferable. This design lets them connect a complaint to an earlier root mistake instead of attributing it solely to the last answer.

Before accepting a hint, the pipeline also checks that its corrective claims follow from information available before the marked turn. These checks are intended to reduce hindsight bias, where a new preference revealed in the complaint is treated as something the model should already have known.

In this example, the hint tells the model to search for the W-3 transmittal form on Paychex rather than W-2, or ask which form the user means if the request is ambiguous. The correction targets the earlier interpretation and search decision, not just the final answer. The original turn remains unchanged; the annotation adds is_bad_turn: true and a nonempty hint.

A W-3 request is misinterpreted as W-2; a later complaint is traced to the earlier search decision and checked against prior evidence. — A real session showing how a complaint is traced back to an earlier mistake, with the annotation procedure overlaid. The user asked for “w3” on Paychex, but the model treated it as a typo for W-2 and searched for the wrong form. Proposer consensus locates the root cause, while two grounding checks assess whether the evidence predates the mistake and the hint’s claims follow from information available at that point. Accepted annotations preserve the original text and add marks and hints, rather than rewritten targets.

Consider another session in the tool-error group with a search call where the agent searches with recency_filter: "year", but the tool rejects it because the allowed values are “day,” “week,” or “month”. This is a clear schema violation.

The tool-error path uses deterministic rules for known mistakes and model judgment for ambiguous cases. An error message alone is also not enough to blame the model; a judged correction must point to something the model could have known from the tool schema, system instructions, or earlier conversation.

A search call uses the invalid recency_filter value year; a grounded hint instructs the model to use a listed value or omit an optional field. — The report’s invalid search argument illustrates the current tool-error procedure. The schema makes this mistake avoidable; an outage alone would not. The original call stays in the record and receives a grounded hint. The hint allows a listed value or omission if the field is optional; it does not require “month” for every request.

Here, the hint names the failed call, includes the validation error, and instructs the model to use an allowed value or omit the field if it is optional.

Once a turn is marked, the trainer runs the same checkpoint twice: once as the teacher and once as the student. The teacher sees the hint; the student does not.

Both passes use teacher forcing, where we feed the model the recorded session instead of letting it choose new tokens. The forward pass still produces a full next-token distribution at each step, but it is always conditioned on the fixed prefix from the recording, not on rollout tokens.

For the search example, the same model assigns much higher probability to “month” and less to “year” when given the hint. Training uses this distributional shift as a soft target.

The same model scores a logged turn with and without a hint; detached teacher probabilities supply KL targets while CE anchors imitation. — Two passes, one set of weights. Both passes score the logged continuation; only the teacher receives hints. Its detached next-token probabilities guide the student through forward KL. Separate CE targets anchor imitation. Both losses feed one weight update. Earlier hints may remain in the teacher’s current segment; neither pass generates a replacement training answer.

The teacher’s predictions are detached, or stop-gradient: they supply a target but do not receive an update through the teacher pass. After the student’s pass, one optimizer step updates the shared model.

CE loss treats the recorded tokens selected for imitation as ground-truth targets and penalizes the model for assigning them low probabilities. This creates a hard imitation target.

At token position $i$ , let $p_i(v)$ be the student’s probability of token $v$ in the full vocabulary $V$ . The recorded next token in the response or tool call is $y_i$ , the one-hot target distribution is $r_i$ .

r_i(v)=\mathbf{1}[v=y_i]

Cross-entropy measures how well the student predicts that recorded token. With a one-hot target, it reduces to the negative log-probability assigned to $y_i$ .

H(r_i,p_i)=-\sum_{v\in V}r_i(v)\log p_i(v)=-\log p_i(y_i)

The CE mask $m_i^{\mathrm{CE}}$ is $1$ where the token should be imitated and $0$ otherwise. Summing over the selected positions gives the imitation loss.

\mathcal{L}_{\mathrm{CE}}=\sum_i m_i^{\mathrm{CE}}H(r_i,p_i)

KL loss instead uses the teacher’s full next-token probability distribution $q_i$ as a soft correction target. The teacher’s context includes the current hint and any earlier hints retained in that training segment; the student’s context is always hint-free.

The KL mask $m_i^{\mathrm{KL}}$ is $1$ where the token should be corrected and $0$ otherwise; the CE and KL masks do not overlap. At these positions, forward KL penalises disagreement between the teacher and student across the full vocabulary:

\mathcal{L}_{\mathrm{KL}}=\sum_i m_i^{\mathrm{KL}}D_{\mathrm{KL}}(q_i\Vert p_i)=\sum_i m_i^{\mathrm{KL}}\sum_{v\in V}q_i(v)\log\frac{q_i(v)}{p_i(v)}

Both terms share one denominator rather than being normalised separately. This denominator counts the total tokens selected for imitation across the global training batch. The weight $\lambda$ controls the contribution of corrective training relative to imitation.

This shared normalisation fits our training infrastructure’s single-denominator interface and recovers the standard SFT loss when $\lambda=0$ , simplifying baseline comparisons. Because CE tokens are much more abundant than KL tokens, using a shared denominator also avoids giving individual KL tokens disproportionately large weights in batches with few marked tokens.

\mathcal{L}=\frac{\mathcal{L}_{\mathrm{CE}}+\lambda\mathcal{L}_{\mathrm{KL}}}{\sum_i m_i^{\mathrm{CE}}}

Applied to the earlier examples, the turns containing the W-3/W-2 mix-up and the invalid search call receive KL. Other unmarked assistant content in a successful session still receives CE. Other assistant content in an unsuccessful session is retained as context but receives neither CE nor KL.

Our evaluations answer two separate questions. First, does a hint actually help the model correct an error when the hint is present? Second, does training help the model avoid that error later, without the hint? The first tests whether hints provide useful training signals. The second tests whether the model learns from them.

We test the first question before the training runs to make sure that hints can indeed steer the model away from the mistakes.

We regenerated marked user-feedback turns with and without a hint, using the same GLM-5.2 base model at temperature zero. A blind judge sees the later complaint, but neither the hint nor which condition produced the answer.

In the W-3 example, the hint shifts next-token probability toward “3” and away from “2” after the prefix “W-”.

Next-token probabilities after W- shift toward 3 rather than 2 when the unchanged base model receives a hint. — A hint preserves the requested W-3 form. Bars show mean next-token probabilities across eight forward passes; whiskers show the observed range, not a confidence interval. Both conditions use the unchanged base model.

On a larger 40-turn held-out sample backed by explicitly stated earlier evidence, the share judged fixed or on track rises from 40.0% without a hint to 75.0% with one. On another 40 turns backed by inferred intent, it rises from 32.5% to 80.0%.

Hint validation: fixed or on-track rates increase from 40.0% to 75.0% for explicit evidence and 32.5% to 80.0% for inferred intent. — Hint validation, not post-training evaluation. The same 40 marked turns in each evidence group are regenerated with and without a hint using unchanged GLM-5.2 at temperature zero. The judge sees the complaint but not the hint or condition. Counts are derived from the report’s rates and sample sizes; confidence intervals are not reported.

In the search tool error example, a hint indeed shifts probability mass from the year token to the month token.

A hint shifts the unchanged base model's next-token probabilities away from year and toward the allowed value month. — A hint shifts probability toward an allowed value. Bars show mean next-token probabilities across eight forward passes; whiskers show the observed range, not a confidence interval. Both conditions use the same base model; no trained checkpoint is compared.

On a larger 985-turn held-out sample, we regenerate each turn four times with and without a hint using the same base model. A blind judge evaluates whether each regeneration avoids the original failure. Hints raise failure avoidance from 75.1% to 93.7%, a paired gain of 18.6 percentage points, and increase the share taking the corrected action from 60.6% to 82.3%. Gains are largest on errors that repeated sampling alone struggles to fix.

On 985 held-out marked tool-error turns, hints raise failure avoidance from 75.1% to 93.7% and corrected-action rates from 60.6% to 82.3%. — Tool-error hint validation on 985 held-out marked turns, with four regenerations per condition. Bars show failure avoidance and corrected-action rates for the unchanged base model, with and without hints. Avoiding the targeted mistake does not establish tool-call or task success.

We next test whether the trained model makes fewer mistakes without receiving hints. We evaluate the stock model and various post-trained checkpoints on offline benchmark tasks and in live user sessions, checking for fewer tool failures and less user dissatisfaction.

The offline evaluation uses BrowseComp, Finch, WideSearch, DRACO, DSQA, SpreadsheetBench, HLE, a representative Computer task set, and checks for skill loading, document review, and inline citations. We measure the share of tool calls whose results are explicitly flagged as errors across the evaluation trajectories.

In these evaluations, the recorded tool-error rate was 2.79% for stock GLM 5.2, 1.35% for the RFT-only checkpoint, and 0.87% for a RFT-plus-OPSD checkpoint. These checkpoints were trained on different data, so this is not a matched-data ablation of OPSD. Task-level benchmark results were mixed, so fewer tool errors alone does not necessarily translate into broad improvement in task success.

Offline recorded tool-error rates: 2.79% for stock GLM 5.2, 1.35% for RFT-only, and 0.87% for RFT plus OPSD; training data differ. — Offline tool-error rates for stock GLM-5.2, RFT-only, and RFT-plus-OPSD checkpoints. The training data differ across checkpoints; this is not a controlled ablation.

For GLM-5.2, we ran two separate A/B tests with about one hundred thousand users in each condition. In the first, an early RFT-plus-OPSD checkpoint had a slightly lower tool-failure rate than stock GLM 5.2, 2.82% versus 2.94%, but the difference was not statistically significant.

In the later test, we compared two RFT-plus-OPSD checkpoints: an earlier one (checkpoint one) and a later one (checkpoint two). Tool-call failures fell from 2.24% to 1.77% of calls with a recorded status. This was a statistically significant 21.2% relative reduction. We did not directly compare checkpoint two with stock GLM 5.2 online.

The user-feedback results were less conclusive. In the second GLM-5.2 experiment, the average probability of strong dissatisfaction (medium or high) was 2.58% for checkpoint one and 2.54% for checkpoint two, with no statistically significant difference.

Two separate online A/B tests; only the later checkpoint comparison has a significant tool-failure reduction, with no significant MID+ dissatisfaction change. — Two separate GLM-5.2 live A/B tests. Checkpoint 1 is compared with stock GLM-5.2; checkpoint 2 is compared with checkpoint 1. Within these two tests, only the later tool-failure reduction is statistically significant. MID+ rates average estimated probabilities of medium-or-high dissatisfaction, not counts of complaints.

For GLM-5.3, we compared a post-trained checkpoint directly with stock GLM-5.3. Tool-call failures fell from 3.56% to 2.38% of calls with a recorded status, a 33.04% relative reduction. The MID+ rate, the average estimated probability of medium-or-high dissatisfaction, fell from 4.12% to 3.41%, a 17.2% relative reduction. Both reductions were statistically significant.

GLM-5.3 A/B test: tool-failure rate falls from 3.56% to 2.38% and MID+ rate, the average estimated probability of medium-or-high dissatisfaction, falls from 4.12% to 3.41%. CUPED-adjusted relative reductions are 33.04% and 17.16%, respectively. Both 95% sequential intervals exclude zero; the primary engagement metric is not significant. — Post-trained GLM-5.3 versus stock GLM-5.3 in a separate online A/B test. Tool-failure and MID+ rates, relative changes, and 95% sequential confidence intervals use CUPED adjustment. Both reductions are statistically significant. MID+ rates average estimated probabilities of medium-or-high dissatisfaction over eligible scored follow-up threads, not counts of complaints. Tool-failure rates exclude calls without a recorded success or failure status.

The central idea is to learn at the level of the decision, not just the session.

A successful outcome can hide mistakes, and an unsuccessful session can still reveal how to improve. We keep useful behaviour from successful sessions as an anchor, use complaints and tool errors to identify avoidable failures, and train the model toward the predictions it makes when those mistakes and corrections are made explicit. Success provides behaviour to preserve; failure provides direction for improvement.

Real-world use makes this approach especially useful. Real users bring goals, constraints, and workflows that are difficult to anticipate in a synthetic environment. Their corrections reveal requirements the model missed, while tool errors expose gaps between what the model attempted and what the environment allows. These signals complement outcome-based training: instead of asking only whether a task succeeded, we can ask where the model went wrong and what information could have helped it make a better decision.

The limits are equally important. Judges can be wrong, and correcting one turn does not guarantee that the full task succeeds. The goal is not merely an agent that recovers after a correction. It is an agent that needs correction less often.

BibTeX

@online{perplexity2026LearningFromRealWorldExperience,
  author = {{Perplexity} and Senzeyu Zhang and Chuan Wang and Hang Wu and Hao Zhang and Yuan Liang and Yunfan Zhong and Phi Hoang and Claire Beamer and Jeremy Yang and Andrew Wilson and Denis Yarats},
  title = {{Learning from real-world experience}},
  date = {2026-09-21},
  url = {https://www.perplexity.ai/hub/blog/learning-from-real-world-experience}
}

From synthetic to real-world data

Imitate the good steps, correct the bad ones

An example: one session can teach both lessons

How the training works

Select sessions, then judge the outcome

Find the mistakes from user feedback

Find the mistakes from tool errors

Two forward passes

The loss function

The CE loss

KL loss

Combined loss

What the evaluations show

Hints can steer the model away from the mistakes

User feedback

Tool errors

Tool reliability and user feedback after training

Offline evaluation

Online evaluation: GLM-5.2

Online evaluation: GLM-5.3

Conclusion

Citation