The model attempts behind a successful request

An AI product receives a request, tries one route, encounters an error, falls back to another, and eventually returns a valid result.

From the user’s point of view, the request succeeded. From the infrastructure’s point of view, something may already be degrading.

If we only record the final response, both interpretations collapse into a single green data point. The product appears healthy until fallback capacity is exhausted, latency becomes unacceptable, or the last working route fails as well.

The missing unit of observation is the model attempt.

A request and an attempt answer different questions

A request describes the user outcome:

• Did the user receive a result?
• How long did the complete experience take?
• Was the final result billable and usable?

An attempt describes one execution decision inside that request:

• Which route was selected?
• Did it succeed?
• How long did it occupy capacity?
• What kind of failure occurred?
• Was another attempt allowed to continue?

Neither unit replaces the other. Request-level metrics tell us whether the product worked. Attempt-level metrics tell us why it worked, how close it came to failing, and which part of the system needs attention.

Fallback converts failure into hidden debt

Fallback is an important reliability mechanism, but it can make a dashboard look healthier than the underlying system.

Imagine that the first route fails thirty percent of the time while the second route reliably recovers it. Final request success may remain close to perfect. Meanwhile:

• users are waiting through an additional timeout;
• two pieces of capacity are consumed for one result;
• cost and latency distributions are shifting;
• the fallback route is becoming a single point of failure;
• the primary route may remain broken for days because no user-facing alarm fires.

Fallback did its job. Observability failed to show the debt it created.

This is why “the request eventually succeeded” is not sufficient evidence that the serving path is healthy.

The smallest useful attempt record

Attempt telemetry does not need to reproduce the entire request. A compact record is usually more useful:

• a request identifier shared by all attempts;
• the model or capability requested by the product;
• the selected route;
• the attempt index and number of available candidates;
• success or failure;
• duration;
• a normalized error category;
• whether another fallback remained;
• whether this attempt determined the final user outcome.

The distinction between attempt status and finality matters. A failed attempt followed by a successful fallback is still a real failure for infrastructure analysis. A successful attempt is not necessarily the final result if later validation rejects its output.

The record should capture facts that support an operational decision. Raw prompts, generated content, credentials, and complete provider responses usually add risk and cardinality without improving that decision.

Error categories should imply different actions

A single failure counter is almost as weak as no attempt telemetry at all.

Different failures ask for different responses:

• Rate limits may require capacity changes or temporary routing adjustments.
• Timeouts may indicate saturation, an unrealistic deadline, or an unhealthy dependency.
• Invalid requests usually point to a contract or parameter problem and should not be retried blindly.
• Policy rejections need product-specific handling rather than being mixed with availability failures.
• Network errors may be transient, but repeated retries can amplify an outage.
• Invalid outputs mean transport succeeded while the product contract did not.

The categories do not need to be academically perfect. They need to be stable enough that a change in the chart leads to a different investigation or response.

Classification should also happen close to the attempt. Once several failures have been compressed into one final error, the information needed to distinguish them is often gone.

Two dashboards, two truths

I prefer to keep user outcome and execution health visibly separate.

The product view answers:

• request success rate;
• end-to-end latency;
• user-visible error distribution;
• completion and delivery rate.

The infrastructure view answers:

• attempt success by capability and route;
• fallback frequency;
• first-attempt success;
• error category and latency by attempt position;
• how often final success depended on recovery.

Putting both into one blended success rate produces a number that is easy to report and difficult to act on.

The relationship between the two views is often more useful than either view alone. Stable user success with falling first-attempt success means resilience is compensating for degradation. Falling user success with stable attempt health may point somewhere after inference: validation, persistence, settlement, or delivery.

Telemetry should not become an automatic router too early

Once attempt data exists, the next temptation is to let a score choose routes automatically.

That can work, but a recent success rate is not a complete routing policy. Small samples fluctuate. Different request shapes have different difficulty. A fast route may produce worse outputs. A route that looks cheap may trigger more retries. Some failures are correlated across all candidates.

Before telemetry controls production decisions, the system needs explicit answers for:

• the minimum sample required;
• the time window and freshness policy;
• which failure categories count against availability;
• how quality and cost enter the decision;
• which limits remain fixed regardless of the score;
• how the router behaves when evidence is missing.

Observability should first make decisions legible. Automation should follow only when the measurement semantics are trustworthy.

Measure the recovery, not only the result

Reliable AI products do more than return successful responses. They understand the path that produced them.

A fallback can protect the user and still reveal a serious operational problem. An individual attempt can fail without making the product fail. A final success can be expensive, slow, or one dependency away from disappearing.

Request metrics describe the promise made to the user. Attempt metrics describe the machinery keeping that promise.

We need both truths before we can call the system healthy.