Constraining the Coding Agent: The Ralph Loop and Why Determinism Matters
Abstract
In late 2025, agentic coding tools went from impressive demos to daily infrastructure. The problem nobody talked about enough: when an LLM agent has write access to a codebase and no formal constraints, reproducibility breaks down. The Ralph Loop is a story-driven harness intended to bound that variability through one call to the coding agent per attempt, scoped writes, and atomic state. These constraints make runs easier to inspect; they do not make an unpinned model deterministic.
Contents
Ralph now lives in the
rae repository as the TypeScript
package in packages/ralph; the standalone ralph-loop repository this post
originally linked is no longer available. This post is the design rationale,
and the sections below describe the current package.
Literature and calculations checked through 24 October 2026; implementation as described on 3 October 2026.
December 2025
It happened fast. In the twelve months before I am writing this, agentic coding went from a niche research topic to the default mode for several categories of software engineering task. Codex runs code in a sandboxed container and submits pull requests. Claude Code works through a task list in your terminal while you make coffee. Cursor’s agent mode rewrites a file, runs the tests, reads the failures, and tries again — automatically, without waiting for you to press a button.
The demos are impressive. The production reality is messier.
The problem is not that these systems do not work. They work well enough, often enough, to be genuinely useful. The problem is that “works” means something different when an agent is executing than when a human is. A human who makes a mistake can tell you what they were thinking. An agent that produces a subtly wrong result leaves you with a diff and no explanation. And an agent run that worked last Tuesday might not work today, because the model changed, or the context window filled differently, or the prompt-to-output mapping is, at bottom, a stochastic function.
This is the problem the Ralph Loop is designed to address: not “make agents more capable” but “make the harness-controlled parts of agent runs bounded and auditable.” Strong reproducibility additionally requires pinned model, runner, repository, tools, configuration, and inputs.
The name has a precedent. In July 2025 Geoffrey Huntley described a technique he called Ralph: “In its purest form, Ralph is a Bash loop,” one that pipes the same prompt file into a coding agent again and again, under the rule “one item per loop” [[1]]. In his version the agent picks the next item itself, and what he calls back pressure rejects bad work: type systems, builds, tests the agent is told to write, and “security scanners, it could be static analysers, it could be anything.” The harness described here also does one item per call, but the runner, not the model, selects the story and accepts or rejects the attempt. Its acceptance check, described below, is far weaker than that back pressure; the point is only that the decision is not the model’s.
The Reproducibility Problem, Formally
An LLM tool call is usefully modelled as a stochastic function. Given a prompt $p$ and a particular model, version, decoding configuration, tool interface, and context, it samples from a distribution over possible outputs:
$$T : \mathcal{P} \to \Delta(\mathcal{O})$$where $\mathcal{P}$ is the space of prompts, $\mathcal{O}$ is the space of outputs, and $\Delta(\mathcal{O})$ denotes the set of probability distributions on $\mathcal{O}$. For a coding agent the context includes the repository snapshot $R$ that the call can read; I write $T(p \mid R)$ where that dependence matters.
At temperature zero — the most deterministic setting most systems support — this collapses toward a point mass:
$$T_0(p) \approx \delta_{o^*}$$where $o^*$ is the greedy decode: the output obtained by taking the most probable next token at every step. That is not necessarily the most probable output: if A (probability 0.6) is followed by x (0.55) or y (0.45), and B (0.4) by x (0.9), greedy decoding returns Ax (0.33) and misses Bx (0.36).
“Approximately” because even $T_0$ is not strictly reproducible across runs, environments, or model versions. Floating-point addition is not associative, and the order in which an inference kernel adds can depend on the batch size, which on a shared server depends on the load. He and colleagues drew 1,000 completions of one prompt from Qwen3-235B-A22B-Instruct-2507 at temperature zero and obtained 80 distinct texts, the commonest of them 78 times; with batch-invariant kernels, all 1,000 were identical [[2]].
A naive agentic loop composes these calls. If an agent takes $k$ sequential tool calls to complete a task, each prompt is built from the outputs before it:
$$o_j \sim T(p_j), \qquad p_j = f(p_0, o_1, \ldots, o_{j-1}), \qquad j = 1, \ldots, k \tag{1}$$where $p_0$ is the task as first stated and $f$ is whatever the loop does with earlier results: appending tool output, summarising, re-planning. I first wrote $o_k = T(T(\cdots T(p_0) \cdots))$, which does not type-check: $T$ returns a distribution, not a prompt, and $f$ is the missing step.
The variance does not simply add up, as it would for independent steps — it propagates through the dependencies. Early outputs condition later prompts; a small deviation at step 2 can shift the trajectory of step 5 substantially. This is not a theoretical concern. It is the practical experience of anyone who has tried to reproduce a multi-step agent run.
The Ralph Loop does not solve the stochasticity of $T$. One call bounds the harness-controlled interaction history for a story attempt; it does not make an unpinned external model run reproducible in the strong sense.
The Ralph Loop as a State Machine
The system’s state at any point in a run is a quadruple:
$$\sigma = (Q,\; S,\; C,\; L)$$where:
- $Q = (s_1, s_2, \ldots, s_n)$ is the ordered story queue — the PRD (product requirements document) — with stories sorted by priority, then by ID
- $S \in \lbrace \texttt{open}, \texttt{passing}, \texttt{skipped} \rbrace^n$
is the status vector, one entry per story (stored as the
passesandskippedfields of each story inprd.json) - $C \in \mathbb{N}^n$ counts the failed runs of each story; it persists between runs and is what the skip threshold below is compared with
- $L \in \lbrace \texttt{free}, \texttt{held} \rbrace$ is the single-run
lock (
.runtime/.run.lock) that keeps a second run from writing $S$ at the same time
Let $m$ be the run’s mode (audit, linting or fixing) and $n_m \le n$
the number of stories of that mode that are open when the run starts. The
transition function $\delta$ at each step is:
- Select: $i^* = \min\lbrace i : S[i] = \texttt{open},\ \operatorname{mode}(s_i) = m \rbrace$ — deterministic by construction, since $Q$ has a fixed ordering and only stories of the active mode are considered
- Build: $p = \pi(s_{i^*},\; I,\; \theta)$ — built from the story
definition, the static policy document $I$ (
INSTRUCTIONS.md) and run parameters $\theta$ such as the run’s mode, report path and sandbox setting and the current UTC date; the runner copies no earlier tool output into it - Execute: $o \sim T(p \mid R)$ — in the design described here, exactly one Codex CLI call per attempt, output captured; $R$ is the repository the call works in, an isolated copy in fixing mode
- Accept: $\alpha(o) \in \lbrace \top, \bot \rbrace$ — the call must
exit successfully and, in fixing mode, its changes must pass the scope
check described below; the runner then writes the returned report body to
the report path named by the story’s
Created <path>.md ...acceptance criterion - Commit: if $\alpha(o) = \top$, set $S[i^*] \leftarrow \texttt{passing}$.
Otherwise, with a persistent-failure threshold $A_{\max} \ge 1$
(
RALPH_SKIP_AFTER_FAILURES), set $C[i^*] \leftarrow C[i^*] + 1$ and mark the story skipped once $C[i^*] = A_{\max}$; a failure that does not skip the story stops the run. State is written while the run lock $L$ is held
Fig. 1 draws one pass through these steps, with acceptance split into its two checks.
The next state is $\sigma' = (Q, S', C', L)$, where $S'$ and $C'$ differ
from $S$ and $C$ at most in position $i^*$: a pass or a skip changes
$S[i^*]$, a counted failure below the threshold changes only $C[i^*]$. The
loop continues until no open story of mode $m$ remains, a failure stops it,
or a story limit $N$ is reached (the count passed on the command line, or
defaults.max_stories_default in prd.json).
Termination. Each step either closes a story of mode $m$, because it
passes or is skipped, or ends the run, so one run takes at most
$\min(N, n_m)$ steps. The counter bounds the work across runs as well, if
every failed attempt is counted, scope violations included. Then, with
$A_{\max} \ge 1$, a story can fail at most $A_{\max}$ times before it is
skipped, so restarting the loop after every stop leaves every story of the
mode passing or skipped after at most $n_m A_{\max}$ steps in all. Under the
default of 0 nothing is counted or skipped: a story that keeps failing
stops every run that reaches it, and only the operator can break that cycle.
If every open story of the mode passes before its counter reaches
$A_{\max}$ — the outcome the constraints in INSTRUCTIONS.md are designed to
encourage — the loop ends
with all $n_m$ stories passing after exactly $n_m$ successful transitions,
possibly spread over several runs, and at most $n_m (A_{\max} - 1)$ failed
ones.
Replay. The trajectory $\sigma_0 \to \sigma_1 \to \cdots \to \sigma_k$
is determined by $\sigma_0$ and the sequence of call results (exit status and
report body, and in fixing mode the workspace diff). That is the
state-machine argument Lamport used for distributed synchronisation:
processes that apply the same transition function to “the same sequence of
commands” pass through the same states [[3]]. Here the commands are the call
results. The package writes lifecycle events to .runtime/events.log and,
when RALPH_CAPTURE_TOOL_OUTPUT=true, redacted provider output to
.runtime/run.log; the event log records story starts, completions and
failures, not the outputs themselves. If every input and tool output were
deterministic, replay could be deterministic; this article does not supply a
pinned-run replay test. If they are not — as in practice they will not be —
the runner at least copies no earlier output into a later prompt, so it does
not build the chain of Eq. (1) itself. Earlier outputs can still reach later
calls through the repository, as the next section shows.
The One-Tool-Call Invariant
The most important constraint in the Ralph Loop is also the simplest: exactly one tool call per story attempt.
This is not the natural design. A natural agentic loop would let the model plan, execute, observe, reflect, and re-execute within a single story. Some frameworks call this “inner monologue” or “chain-of-thought with tool use.” The model emits reasoning tokens, calls a tool, reads the result, emits more reasoning, calls another tool, and eventually produces the final output.
This is more capable for complex tasks. It is also what makes reproducibility hard. Each additional tool call in the chain is a fresh draw from $T$, conditioned on the previous outputs. After five tool calls, the prompt for the fifth includes four previous outputs — each of which varied slightly from the last run. The fifth output is now conditioned on a different input.
Formally: let the multi-call policy use $k$ sequential calls per story, the chain of Eq. (1) with $p_0$ built from the story $s_{i^*}$. I first wrote the two-term law of total variance for $o_k$ itself. Text has no variance, and that law holds for any two random variables, so it said nothing about the chain. For a scalar outcome $g(o_k)$, such as whether the final change passes the tests, applying the law once per step does. Write $h_j = \mathbb{E}[g(o_k) \mid o_1, \ldots, o_j]$ for the expected outcome once the first $j$ outputs are known, so that $h_0 = \mathbb{E}[g(o_k)]$ and $h_k = g(o_k)$; then
$$\operatorname{Var} g(o_k) \;=\; \sum_{j=1}^{k} \mathbb{E}\!\left[\, \operatorname{Var}\!\left(h_j \mid o_1, \ldots, o_{j-1}\right) \right] \tag{2}$$Each term is a conditional variance: how far the $j$-th draw moves the expected outcome, given the draws before it. No term is negative; for $k = 2$ the sum is the familiar split into an explained part and a residual. Eq. (2) is an identity, so it does not show that variance grows with $k$. A term $j < k$ is non-zero only if $o_j$ actually moves the expected final outcome; if the last call ignored everything before it, only the last term would survive. Conditioning later prompts on earlier outputs makes those terms possible, not certain. In a well-designed, low-variance chain they may stay small; in practice, LLM outputs have non-trivial variance at each step, and that variance propagates through the conditioning chain.
The one-call constraint collapses $k$ to 1:
$$o_i \sim T\!\bigl(\pi(s_i,\; I,\; \theta) \bigm| R_i\bigr)$$The runner builds the prompt from the story definition, the static policy document and the run parameters. It copies in no previous tool output. At the level of the harness, Eq. (2) has a single term per attempt: the spread of one call.
Two dependencies remain (Fig. 2). The first is inside the call. One Codex CLI call is one run of codex exec, and Codex CLI
is “a coding agent from OpenAI that runs locally on your computer” [[4]];
while codex exec runs, it “streams progress to stderr and prints only the
final agent message to stdout” [[5]]. The chain of Eq. (1) therefore still
exists inside the call; the harness fixes its input and judges only its
result. The second runs through the repository. $R_i$ is not fixed: a
passing story leaves its report, and in fixing mode its changes, in the
files that later calls read, including any file the prompt is built from. No
earlier output is copied into a later prompt. Earlier outputs can still reach
a later call, but only through the repository.
The stories are designed to be atomic enough that one call is sufficient. If a story requires more, it should be split into two stories in the PRD. This is a forcing function toward better task decomposition, which I consider a feature rather than a limitation.
Scope as a Locality Constraint
In fixing mode, each story carries a scope[] field listing the files
or directories the agent is permitted to modify. The runner gives the
agent an isolated writable workspace and keeps a separate immutable baseline
of the repository state:
where $h(f)$ is a hash of the file contents. After the tool call:
$$F_{\text{after}} = \lbrace (f,\; h(f)) : f \in \text{repo} \rbrace$$The changed paths are those whose entry appears in only one of the two snapshots, the symmetric difference:
$$\Delta = \lbrace f : (f, h) \in F_{\text{before}} \mathbin{\triangle} F_{\text{after}} \text{ for some } h \rbrace$$$\Delta$ holds every file the call added, modified or deleted, and it must satisfy:
$$\forall\, f \in \Delta \;:\; f \in \text{scope}(s_{i^*})$$where $\text{scope}(s_{i^*})$ is the set of paths the story’s scope declaration admits. I first wrote $\Delta = F_{\text{after}} \setminus F_{\text{before}}$, which holds only added and modified files: by that formula a story could delete any file outside its scope and pass.
This is a locality constraint: everything the agent changes, deletions included, must lie in the part of the repository that the story’s scope declares. Writes that escape it are a story failure, regardless of whether they look correct.
The motivation is containment. When a fixing agent makes a “small repair”
to one file but also helpfully tidies up three adjacent files it noticed
while reading, you have three undocumented changes outside the story’s
intent. In a system with many stories running sequentially, out-of-scope
changes accumulate silently. The scope constraint prevents this.
Crucially, prompt instructions alone are not sufficient — an agent told
“only modify files in scope” can still modify out-of-scope files if the
instructions are interpreted loosely or the context is long. The runner
checks scope at the file-system level after the call: it diffs the isolated
workspace against the baseline (transactionDiff and pathMatchesScope in
packages/ralph/src/runner.ts), and on a violation discards the workspace and
leaves the live repository unchanged. Changes are promoted to the live
repository only after that check passes, and multi-path promotion is
recoverable but, as the package README notes, not globally atomic. That is an
implementation property of a particular release, not a consequence of the
formal sketch alone.
Acceptance Criteria: Grounding Evaluation in Deterministic Checks
Each story carries exactly one acceptance criterion of the form
Created <path>.md ... — the path where the report should appear. The model
returns only the report body, and the runner writes it to that path after
validating the path (under strict mode it must stay below
defaults.report_dir).
This is intentionally minimal. The alternative — semantic acceptance criteria (“did the agent identify all relevant security issues?”) — would require another model call to evaluate, reintroducing stochasticity at the evaluation layer and creating the infinite regress of “who checks the checker.” A successful run that yields a report at the right path is a necessary condition for a valid run. It is not a sufficient condition for correctness, but necessary conditions that can be checked deterministically are already more than most agentic pipelines provide.
The quality of the outputs — whether the audit findings are accurate, whether the fix is correct — depends on the model and the prompt quality. The Ralph Loop gives you a framework for running agents safely and repeatably. Verifying that the agent was right is a different problem and, arguably, a harder one.
From Bash to TypeScript
The original harness was Bash plus jq, chosen to keep the dependency surface
small inside agent sandboxes. The current package in packages/ralph is
TypeScript ("type": "module", built with tsc) and requires Node.js 24 or
newer plus the repository’s native @rae/fs-bridge module, so that argument
no longer describes the code. The reason for the move is not documented in
the package, so I will not invent one; what the code shows is how much safety
machinery the loop now carries. Fixing mode uses a journaled filesystem
transaction with no-clobber renames (src/transaction.ts), the PRD is
validated against prd.schema.json with Ajv, and each provider call runs under
process-group supervision with a deadline and output limits (src/supervisor.ts).
The orchestration idea is unchanged: select a story, build a prompt from a
template, call one external tool, write one report, update one state field.
What This Is Not
The Ralph Loop is not an agent. It is a harness for agents. It does not decide what tasks to run, does not reason about a codebase, and does not write code. It sequences discrete, pre-specified stories, enforces the constraints on each execution, and records the outcomes. The intelligence is in the model and in the story design; the framework contributes only discipline.
This distinction matters because the current wave of agentic tools conflates two things that are worth keeping separate: the capability to reason and act (what the model provides) and the infrastructure for doing so safely and repeatably (what the harness provides). Improving the model does not automatically improve the harness — and a better model in a poorly constrained harness just fails more impressively.
The code is at
github.com/sebastianspicker/rae
under packages/ralph: the TypeScript implementation, prd.schema.json, and
the INSTRUCTIONS.md policy document.
A later post, The Day Claude Decided to Delete My Whole Codebase, comes back to this post’s closing claim, that a better model in a poorly constrained harness fails more impressively, after a test run by a coding subagent deleted every checkout in my projects directory.
References
[1] Huntley, G. (2025, July 14). Ralph Wiggum as a “software engineer”. ghuntley.com (blog post, accessed 2026-10-24). https://ghuntley.com/ralph/
[2] He, H., & Thinking Machines Lab (2025, September 10). Defeating nondeterminism in LLM inference. Thinking Machines Lab: Connectionism (blog post). DOI: 10.64434/tml.20250910
[3] Lamport, L. (1978). Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7), 558–565. DOI: 10.1145/359545.359563
[4] OpenAI (n.d.). Codex CLI. README of the
openai/codex repository (accessed 2026-10-24).
https://github.com/openai/codex
[5] OpenAI (n.d.). Non-interactive mode. Codex documentation (accessed 2026-10-24). https://learn.chatgpt.com/docs/non-interactive-mode
Changelog
- 2026-07-11: Qualified the formal sketch: one tool call limits interaction history but does not make an unpinned model, tool, repository snapshot, and runner version deterministic.
- 2026-10-03: Updated the repository links and implementation details after
the code moved from
ralph-loopto theraemonorepo (packages/ralph, now TypeScript rather than Bash andjq): policy fileINSTRUCTIONS.mdinstead ofCODEX.md, report written by the runner from the returned body, scope checked against an isolated workspace, failure and skip semantics, and the “Why Bash” section, now “From Bash to TypeScript”. - 2026-10-24: Corrected the formal sketch. At temperature zero $o^*$ is the greedy decode, not the most probable sequence, and run-to-run variation is now traced to batch-size dependence, with He and colleagues' measurement [[2]]. $\Delta(\mathcal{O})$ is the set of distributions on $\mathcal{O}$, not a “probability simplex”. Rewrote the multi-call chain as Eq. (1), since $T(T(\cdots))$ applied $T$ to an output. The prompt was said to depend “only on the story definition and the static policy document”; it is now $\pi(s, I, \theta)$ with the run parameters, and the claim is that the runner copies no earlier output into it. Added the persistent failure counter $C$ and the mode filter to the state machine; acceptance now names the scope check, so a scope violation is a counted failure. Restated termination for the $n_m$ open stories of the active mode: one run takes at most $\min(N, n_m)$ steps and, if every failure is counted, all runs together at most $n_m A_{\max}$. The text had said that $S$ changes in exactly one position per step and that the loop converges in exactly $n$ successful transitions. Replaced the two-term variance display, which was applied to text and holds for any two random variables, with the per-step decomposition of a scalar outcome, Eq. (2). Corrected the scope diff from $F_{\text{after}} \setminus F_{\text{before}}$, which misses deleted files, to the symmetric difference; the headings now say “locality” instead of the undefined “topological”, and “deterministic checks” instead of “filesystem events”. Qualified the one-call invariant: a Codex call runs its own agent loop [[4]] [[5]], and earlier outputs reach later calls through the repository; the summary now says “one call to the coding agent” instead of “one model invocation”. Added Figs. 1–2, Huntley’s Ralph technique [[1]], a citation for Lamport (1978) [[3]], which was listed but never cited, and a link to the October 2026 follow-up. The header gave 11 July 2026 as the implementation cutoff although the implementation description was updated on 3 October; it now gives both dates.