Errors and Recovery#

Failure is a typed outcome, not a silence. Every binding gives you enough structure to decide, per failure, whether to retry, reroute, or stop — and the rule underneath all of them is the same.

Retry only when the typed result and capability contract make another attempt safe.

A pre-execution backpressure result can permit a delayed retry when it establishes that the provider did not admit the work. Retrying a serialization failure reruns the bug. Retrying a denial re-asks a question already answered. Retrying a deadline may run the work twice.

The five things a caller has to be able to witness#

An authority boundary you cannot observe is not an authority boundary. These five are the minimum an agent needs to behave correctly, and each fragment below demonstrates the ones its binding exposes.

1. An unauthorized invocation is denied. The caller gets a distinct, typed refusal — not a timeout, not a generic internal error. Your code must be able to tell you may not from nobody answered, because the correct response differs: one may need a credential, while the other requires a routing, availability, or deadline decision.

2. A grant expires or is revoked, and the next call stops working. Authority is checked at call time against the authenticated caller origin. There is no cached permission to go stale and no announcement to wait for — the call after the revocation fails, and it fails as an authorization error rather than a routing one.

3. A deadline elapses. You learn no answer arrived in time. You do not learn whether the work ran. See below.

4. A typed remote error crosses the binding boundary. A handler that fails with a status and a body delivers that status and body to the caller, in the caller's language, rather than a stringified exception. A handler that fails with an untyped error delivers Internal — which is a decision the handler author made, usually without noticing.

5. Execution is ambiguous, and the binding says so. A call that timed out, was cancelled, or lost its transport mid-flight has an unknown outcome.

Ambiguous execution is the one that costs money#

When a deadline elapses the caller emits a cancel for that call id, so a cooperating provider can drop the in-flight handler. "Can drop" is not "did not run." The request may have arrived, executed, had an effect, and failed only to get its reply back to you.

A retry after a timeout can duplicate an external effect. Use one of these contracts:

1. Make the operation idempotent. Then a duplicate is harmless and this whole section stops mattering.
2. Carry an idempotency key the provider deduplicates on.
3. Accept the duplicate, deliberately, and write down that you did.

What does not work is treating a timeout as a failure and retrying, then treating the second timeout the same way. That is how one intended effect becomes three.

Retry, hedge, and circuit breaking#

All four bindings offer the same three strategies over a raw call, and they solve different problems:

• Retry — bounded attempts with backoff, on retryable failures only. Fixes a transient.
• Hedge — race several providers when latency matters more than duplicate work. Costs duplicate execution by design.
• Circuit breaker — fast-fail a provider that is failing, instead of paying its deadline on every call. Fixes a sick provider, not a sick request.

Calling by service or tool name rather than a pinned node id is what makes failover possible at all: the mesh picks a provider, so a substitutable capability survives the loss of the primary. Pinning a node id opts out of that.

The end-to-end patterns are in Recover a Failed Workflow, and the wire-level codes every binding's taxonomy wraps are in Error Codes.

The one rule, expanded#

Retry backpressure, and a transient no-route briefly. Treat serialization and config failures as bugs. Treat authorization failures as "get a new credential," never as "try again." Treat shutdown, not-connected and closed streams as state changes that retrying will not undo. And treat a deadline as unknown, not as failure.

Handle it — Rust#

The five witnesses#

RpcError is the type that carries all five. Match on it and each one has its own variant:

use net_sdk::mesh_rpc::{RpcError, CodecDirection};
 
match caller.call_service_typed::<Req, Resp>("summarize", &req, opts).await {
    Ok(resp) => { /* 1 — succeeded */ }
 
    // WITNESS 1 & 2 — denied, and revocation takes effect here too.
    Err(RpcError::CapabilityDenied { target, capability }) => {
        // The target does not authorize `nrpc:<capability>` for this caller.
        // Get a credential. Retrying re-asks a question already answered.
        eprintln!("denied: {target:#x} does not authorize nrpc:{capability}");
    }
 
    // WITNESS 3 & 5 — deadline elapsed. Outcome UNKNOWN, not failed.
    Err(RpcError::Timeout { elapsed_ms }) => {
        eprintln!("no answer in {elapsed_ms}ms — the work may or may not have run");
    }
 
    // WITNESS 4 — a typed remote error crossed the boundary.
    Err(RpcError::ServerError { status, message, headers }) => {
        eprintln!("provider returned {status:#06x}: {message}");
        // `headers` is the structured sidecar — e.g. a `net-failure-schematic`
        // verdict beside the human diagnostic.
    }
 
    Err(RpcError::NoRoute { target, reason }) => { /* retry briefly */ }
    Err(RpcError::Codec { direction: CodecDirection::Encode, .. }) => { /* your bug */ }
    Err(RpcError::Cancelled) => { /* you did this */ }
    Err(e) => return Err(e.into()),
}

CapabilityDenied is raised before the request hits the wire by the caller-side gate, and again on receipt of a CapabilityDenied status from the callee's defence-in-depth path. Both arrive as the same variant, which is what lets you handle authority once.

Codec carries a direction: Encode means it never left, Decode means the reply landed and you could not read it. The second one means the work ran.

The bus error surface#

Bus and lifecycle calls return net_sdk::error::SdkError:

use net_sdk::error::SdkError;
 
match node.emit(&event) {
    Ok(receipt) => { /* accepted into the ring buffer */ }
    Err(SdkError::Backpressure) => { /* the only blindly-retry-safe case */ }
    Err(SdkError::Unrouted) => { /* topology settling — retry briefly */ }
    Err(SdkError::Serialization(_)) | Err(SdkError::Config(_)) => { /* a bug */ }
    Err(SdkError::Shutdown) => { /* state change — stop */ }
    Err(e) => return Err(e.into()),
}

Unrouted is deliberately distinct from Backpressure. Backpressure means the destination is full; unrouted means there is no destination right now. Both retry, with different shapes.

Subsystem error types#

Each surface has its own enum, all implementing std::error::Error, so source() chains and ? composes:

Recover a call#

use net_sdk::mesh_rpc_resilience::{
    RetryPolicy, HedgePolicy, CircuitBreaker, CircuitBreakerConfig,
};
 
let resp: Resp = caller
    .call_typed_with_retry(node_id, "svc", &req, opts, &RetryPolicy::default())
    .await?;
 
let resp: Resp = caller
    .call_service_with_hedge("svc", &req, opts, &HedgePolicy::default())
    .await?;
 
let breaker = CircuitBreaker::new(CircuitBreakerConfig::default());

The default retry predicate deliberately skips Codec, CapabilityDenied and Cancelled — none of them get better on a second attempt.

Next: back to the SDK index.