Errors and Recovery#

Failure is a typed outcome, not a silence. Every binding gives you enough structure to decide, per failure, whether to retry, reroute, or stop — and the rule underneath all of them is the same.

Retry only when the typed result and capability contract make another attempt safe.

A pre-execution backpressure result can permit a delayed retry when it establishes that the provider did not admit the work. Retrying a serialization failure reruns the bug. Retrying a denial re-asks a question already answered. Retrying a deadline may run the work twice.

The five things a caller has to be able to witness#

An authority boundary you cannot observe is not an authority boundary. These five are the minimum an agent needs to behave correctly, and each fragment below demonstrates the ones its binding exposes.

1. An unauthorized invocation is denied. The caller gets a distinct, typed refusal — not a timeout, not a generic internal error. Your code must be able to tell you may not from nobody answered, because the correct response differs: one may need a credential, while the other requires a routing, availability, or deadline decision.

2. A grant expires or is revoked, and the next call stops working. Authority is checked at call time against the authenticated caller origin. There is no cached permission to go stale and no announcement to wait for — the call after the revocation fails, and it fails as an authorization error rather than a routing one.

3. A deadline elapses. You learn no answer arrived in time. You do not learn whether the work ran. See below.

4. A typed remote error crosses the binding boundary. A handler that fails with a status and a body delivers that status and body to the caller, in the caller's language, rather than a stringified exception. A handler that fails with an untyped error delivers Internal — which is a decision the handler author made, usually without noticing.

5. Execution is ambiguous, and the binding says so. A call that timed out, was cancelled, or lost its transport mid-flight has an unknown outcome.

Ambiguous execution is the one that costs money#

When a deadline elapses the caller emits a cancel for that call id, so a cooperating provider can drop the in-flight handler. "Can drop" is not "did not run." The request may have arrived, executed, had an effect, and failed only to get its reply back to you.

A retry after a timeout can duplicate an external effect. Use one of these contracts:

1. Make the operation idempotent. Then a duplicate is harmless and this whole section stops mattering.
2. Carry an idempotency key the provider deduplicates on.
3. Accept the duplicate, deliberately, and write down that you did.

What does not work is treating a timeout as a failure and retrying, then treating the second timeout the same way. That is how one intended effect becomes three.

Retry, hedge, and circuit breaking#

All four bindings offer the same three strategies over a raw call, and they solve different problems:

• Retry — bounded attempts with backoff, on retryable failures only. Fixes a transient.
• Hedge — race several providers when latency matters more than duplicate work. Costs duplicate execution by design.
• Circuit breaker — fast-fail a provider that is failing, instead of paying its deadline on every call. Fixes a sick provider, not a sick request.

Calling by service or tool name rather than a pinned node id is what makes failover possible at all: the mesh picks a provider, so a substitutable capability survives the loss of the primary. Pinning a node id opts out of that.

The end-to-end patterns are in Recover a Failed Workflow, and the wire-level codes every binding's taxonomy wraps are in Error Codes.

The one rule, expanded#

Retry backpressure, and a transient no-route briefly. Treat serialization and config failures as bugs. Treat authorization failures as "get a new credential," never as "try again." Treat shutdown, not-connected and closed streams as state changes that retrying will not undo. And treat a deadline as unknown, not as failure.

Handle it — Go#

Go surfaces failures as error values you check on every call. There is no exception path, and — because every call crosses the cgo boundary — there are more error returns than you would expect from the equivalent pure-Go API.

The five witnesses#

RpcError carries a stable Kind discriminator. This is the cleanest of the four taxonomies to branch on, because the kind is a real field rather than a class or a substring:

var re *net.RpcError
if errors.As(err, &re) {
    switch re.Kind {
 
    // WITNESS 1 & 2 — denied; revocation lands here on the next call.
    case net.RpcKindCapabilityDenied:
        // Get a credential. A retry re-asks a question already answered.
 
    // WITNESS 3 & 5 — deadline elapsed. Outcome UNKNOWN, not failed.
    case net.RpcKindTimeout:
        // The work may or may not have run.
 
    // WITNESS 4 — a typed remote error crossed the boundary.
    case net.RpcKindServerError:
        log.Printf("provider returned: %s", re.Message)
 
    case net.RpcKindNoRoute:
        // Retry briefly — topology may settle.
    case net.RpcKindCodecEncode, net.RpcKindCodecDecode:
        // Your bug. Encode never left; Decode means the work ran.
    case net.RpcKindTransport:
        // Retry with backoff.
    case net.RpcKindUnknown:
        // The Rust formatter emitted a kind this binding does not know.
    }
}

Go splits codec failures into two kinds where Rust uses one variant with a direction field. RpcKindCodecEncode means the request never reached the wire; RpcKindCodecDecode means the reply arrived and could not be read — so the work ran.

The status=0x4001 value for a server error is inside re.Message, not a field.

Sentinels and errors.Is#

The binding exports 49 sentinel errors. The returned error wraps the sentinel with context, so == will not match — use errors.Is:

if errors.Is(err, net.ErrBackpressure) {
    // the one case a blind retry fixes
}

Bus and lifecycle: ErrBackpressure, ErrIngestionFailed, ErrPollFailed, ErrInitFailed, ErrShuttingDown, ErrInvalidJSON, ErrNullPointer, ErrBufferTooSmall, ErrUnknown.

Mesh and streams: ErrMeshInit, ErrMeshHandshake, ErrMeshTransport, ErrNotConnected, ErrStreamEnded, ErrStreamTimeout, ErrChannel, ErrChannelAuth.

Storage: ErrRedex, ErrNetDb, ErrCortexClosed, ErrCortexFold.

Tokens — each one implies a different fix, which is why they are separate rather than one authorization error: ErrTokenExpired, ErrTokenNotYetValid, ErrTokenInvalidFormat, ErrTokenInvalidSignature, ErrTokenNotAuthorized, ErrTokenDelegationNotAllowed, ErrTokenDelegationExhausted, plus ErrIdentity.

ErrTokenExpired is witness 2 on the token path: a grant that has run out reports expiry rather than a generic refusal, so your code can renew instead of escalating.

NAT traversal — all mean "the direct path did not open," none break correctness, and the routed path still works: ErrTraversalUnsupported, ErrTraversalPunchFailed, ErrTraversalPeerNotReachable, ErrTraversalReflexTimeout, ErrTraversalPortMapUnavailable, ErrTraversalRendezvousNoRelay, ErrTraversalRendezvousRejected, ErrTraversalTransport.

Organizations: ErrOrgAdmissionDenied, ErrOrgCredentials, ErrOrgDiscovery, ErrOrgProvision, ErrOrgRPC, ErrOrgAlreadyServing, ErrOrgClosed, ErrOrgUnclassified.

MeshOS: ErrMeshOs, ErrMeshOsInvalidArg, ErrMeshOsCallFailed, ErrMeshOsAlreadyShutdown.

Typed error structs#

Where a failure has structure rather than just an identity, the binding returns a struct with a typed Kind — use errors.As:

var ge *net.GroupError
if errors.As(err, &ge) {
    switch ge.Kind {
    // ... GroupErrorKind values
    }
}

GroupError (GroupErrorKind), MigrationError (MigrationErrorKind), RegistryClientError (RegistryErrorKind), FoldQueryClientError (FoldQueryErrorKind), plus DaemonError, DeckError, McpError, MeshOsSdkError, OrgError, RpcError, RpcCallStatusError and DuplicateKindError.

Returning a typed error from a handler#

return SummarizeResp{}, net.AppError(net.NrpcTypedBadRequest, body)

Any other error a handler returns surfaces to the caller as status=0x4001 (Internal). A handler that returns a bare fmt.Errorf has thrown away the distinction the caller needs to decide whether to retry — this is the most common way a Go provider degrades its callers' error handling.

Recover a call#

Retry, hedge and circuit-breaker strategies apply as in the other bindings, and calling by tool or service name lets the mesh pick a provider so a substitutable capability fails over. The end-to-end patterns are in Recover a Failed Workflow.

Next: back to the SDK index.