Squad: a postmortem

Squad was one of our most ambitious attempts at building agents.

If Oracle tried to let one agent receive a goal and keep advancing the work, Squad went a step further. It asked whether a group of agents could organize around a long-term goal, create their own division of labor, act in parallel, exchange results, and keep adapting as the environment changed.

This was not merely a sequence of model calls. What we imagined was a small AI organization: a persistent mission, roles with different capabilities, state that survived many interactions, and the ability to keep working while the user was away.

Squad ultimately did not work. Its failure was not that the system could not run. It could do a great many things. The real problem was that we could not reliably determine whether all that activity was moving it toward the right outcome.

What made Squad compelling

Squad’s most important idea was to bring time into the design of the agent product.

Most agents organize themselves around one task: understand the request, call tools, and return a result. Squad was intended for goals that might continue for weeks or months. No initial plan could anticipate every change, and no context window could contain the entire history. The system therefore had to preserve tasks, replan, pause and recover, and continue when new information appeared.

We designed different roles around that work. Some interpreted the goal and organized action. Others carried out concrete tasks. Other responsibilities revisited stages and assembled results. Tasks could proceed in parallel, plans could evolve, and the user could review, add information, or change direction at important boundaries.

These qualities made Squad feel less like a workflow. It resembled an organization that could keep adjusting: members understood what they owned, the system understood the current stage, and new results could change what happened next.

That direction still interests me. A true long-running agent may need organizational structure, not merely a longer chain of reasoning.

Activity easily impersonates progress

Squad could continuously generate activity: create plans, divide tasks, assign roles, produce material, report progress, complete stages, and replan around the results.

From the product interface, it looked alive. New messages, states, and deliverables kept appearing. Every local step could appear reasonable.

But activity is not progress.

A system completing ten tasks proves only that ten tasks were completed. It does not prove that the long-term goal is closer. A more detailed plan, a more extensive discussion between roles, or a larger collection of artifacts may improve the internal coherence of the system without changing anything outside it.

Short tasks often have natural stopping conditions. Code runs or it does not. A file exists or it does not. A question has been answered or it has not. Long-horizon tasks are less generous. The consequence of a decision made today may not become visible for weeks or months.

When feedback takes that long to arrive, the thing agents do best—continuing to act—can itself become a risk.

Evaluation is the hard part of long-horizon tasks

Long-horizon tasks are difficult for reasons beyond having more steps, longer context, or more complicated tools. Their deepest difficulty is evaluation.

First, feedback is delayed. When the system makes a decision, the outcome does not yet exist. By the time it appears, many later actions may already depend on that choice.

Second, feedback is sparse and ambiguous. The success or failure of a long-term project rarely has a single metric. Human judgment, environmental change, collaboration quality, and chance can all shape the result.

Attribution is harder still. Even after the result is visible, it may be impossible to identify which plan, role, or intermediate decision caused it. The environment has changed, and the goal itself may have changed with it.

Squad therefore lacked a learning loop that was both fast enough and real enough. It could replan when new information arrived, but it did not know what evidence was sufficient to invalidate the old plan. It could review its own work, but it lacked a standard outside its own narrative for determining whether that review was trustworthy.

Without evaluation, replanning easily becomes a new presentation of the same assumptions.

Multi-agent systems do not correct themselves automatically

We placed considerable hope in role specialization. Different agents could bring different perspectives. One could execute while another reviewed. It seemed less likely to fail than a single agent working alone.

More roles, however, increase capacity before they improve judgment.

When every role sees similar information, shares similar assumptions, and ultimately judges results against the same vague standard, discussion and review merely give the error more consensus. A blind spot held by one agent can become a well-structured plan written by several.

Context, memory, and better coordination can make a system more consistent. They cannot prove that its direction is correct. Remembering more means continuing the past more faithfully. If the initial judgment was wrong, continuity can make the error accumulate more reliably.

Multi-agent systems address how action is organized. Evaluation determines whether that action deserves to continue.

Why the bee perspective did not solve it

We even discussed a more radical idea: simulating long-horizon tasks from the perspective of a bee.

The survival, gathering, division of labor, and continuity of a colony can be compressed into a much shorter observation period. The environment also offers clearer feedback: whether resources increase, whether an individual survives, and whether the colony continues. Perhaps a question that normally takes months to evaluate could become a simulation that runs repeatedly in a short time.

The idea was attractive because it attacked the cost of evaluation directly. It quickly exposed a deeper problem: shortening time does not preserve the problem.

When a real goal is rewritten as survival, resources, or reproduction, the evaluation standard has already been replaced. Real long-term tasks contain changing human judgment, irreversible choices, slowly earned trust, and environmental shifts that cannot be listed in advance. The easier a compressed simulation is to score, the more likely it is to have removed what mattered in the original problem.

It could tell us whether an agent learned to adapt to that simulation. It could not tell us whether Squad would make decisions that remained correct months later in the real world.

Faster wrong feedback is still wrong feedback.

Why Squad failed

Squad did not lack plans, roles, state, tools, or the ability to keep running. We invested heavily in those areas, and many of the resulting ideas still shape how I think about agent systems.

It failed further upstream. We built a system capable of acting over the long term before defining a system capable of judging it over the long term.

When outcomes cannot be verified in time, autonomous execution expands uncertainty. More activity creates more paths. More roles make attribution harder. Longer operation raises the cost of correcting a mistake.

We could observe that Squad was running. We could not prove that it was improving.

This is not a problem solved by adding tests at the end. Evaluation determines how a task should be divided, which decisions can be delegated, where a person must intervene, and how long the system can safely act on its own. It is not a score attached after the agent is finished. It defines the boundary of the product.

If I were starting again

If I were rebuilding Squad today, I would not begin by asking how many roles it needed or how many stages a plan should contain. I would ask: what is the latest point at which the system can discover that it is wrong?

A long-term goal can still exist, but it needs evidence at different timescales:

• immediately verify that an action was valid and its result actually exists;
• at the end of a stage, verify that external state changed as expected;
• bring in human judgment before irreversible decisions;
• for outcomes that genuinely require months of observation, acknowledge that current signals are proxies rather than disguising them as answers.

An agent should not receive months of unsupervised action merely because its goal takes months. A better approach is to organize the long-term goal as a series of judgments that produce evidence, then expand the system’s room to act as that evidence accumulates.

Squad showed me the possible shape of a long-running agent: persistent, collaborative, aware of time, and open to human intervention. It also showed me the real constraint on that future earlier than I might otherwise have seen it.

How long an agent can run does not determine how far it can go.

How quickly it can discover that it is wrong does.