Alignment: Training the Trajectory

Overview

As we have seen, a conversation creates a trajectory — and at any point, many different responses could validly continue it. What separates valuable computational thought from mere storytelling is that the LLM produces outputs humans find valuable — which usually means it is performing work on behalf of a human being. In this lesson, we look at what alignment is, how alignment is performed at a high level as part of model training, and some of the important ways alignment shapes conversations.

The conversation shapes both participants

Earlier we saw the loop from the model's side: its outputs become part of the context that conditions its next outputs. But there are two participants in that loop. The LLM's responses influence what you write next; what you write next influences the LLM. Neither participant is steering alone — the conversation is shaping both of you, turn by turn.

Any continuation is "valid"

At any point in a conversation, the LLM could take the trajectory in many, many different directions. As pure text prediction, there is no inherent constraint — any fluent continuation is a continuation:

All five are "valid" as language. But they send the conversation down very different trajectories, and most of them waste your time. What we want is a model that tends to produce continuations that move the conversation in a direction useful to humans. That tendency has to come from somewhere — and it is built in before you ever start typing.

Alignment: learning which trajectories humans prefer

That is what alignment is. During training — long before your conversation exists — the model is tuned, through reinforcement learning, to produce outputs that humans tend to prefer.

The simplified mechanism: take a conversation, let the model produce multiple candidate continuations, and have humans score or rank them.

In practice, humans can't rank every response the model will ever produce, so their rankings are used to train a scoring model that stands in for human judgment at scale, and the LLM's weights are then adjusted so that continuations like B become more probable and continuations like A and C less so. The details vary across labs and generations of models, but the shape is stable: human preferences over alternative trajectories are turned into weight updates.

Connect this to the recursion from earlier: the model's next output conditions the whole future of the conversation. Alignment trains the model so that the trajectories it tends to launch are ones humans find useful — before any particular human shows up.

What "preferred" quietly includes

Humans obviously prefer answers they perceive as correct. But preference is broader than correctness, and everything raters respond to can creep into the defaults: how verbose the answer is, how it hedges, whether it opens with agreement, how much it formats with lists and headers, how eager it is to help with everything. When you notice that every answer starts with "Great question!" or arrives as five bullet points — that is not the model's personality. It is the residue of what scored well.

But preference has another side. Raters are humans, and humans do not respond to outputs with judgment alone:

You can always push against the defaults inside a conversation — your messages condition the trajectory too. But the defaults themselves live in the weights, set at training time.

Ramification: can the user correct the model?

Here is where alignment meets self-conditioning most sharply. A few years ago, models were notorious for defending their own earlier outputs — take a position, then argue for it, insistently:

Frontier models today are aligned to accept corrections far more readily. That sounds like pure progress — until you see the same willingness pointed the wrong way:

One more correction, in a setting where the stakes are real:

This is a genuinely hard problem, and it is worth saying precisely. In every turn, the model is implicitly weighing its own earlier outputs against the user's pushback when producing the next output. How much a user's correction can redirect the trajectory is a tunable property of the model — tuned not by you, but by the alignment process. If the model has it right, we want it to hold its ground; if it made a mistake, we want it to let go. But the model has no oracle for which case it is in — and neither did the raters whose preferences set the dial.

So alignment is not a detail of training trivia. It sets the defaults for everything the conversation does next: what a "good" continuation looks like, how the model weighs itself against you, and how steerable the whole trajectory is.