Errors and Recovery#

Failure is a typed outcome, not a silence. Every binding gives you enough structure to decide, per failure, whether to retry, reroute, or stop — and the rule underneath all of them is the same.

Retry only when the typed result and capability contract make another attempt safe.

A pre-execution backpressure result can permit a delayed retry when it establishes that the provider did not admit the work. Retrying a serialization failure reruns the bug. Retrying a denial re-asks a question already answered. Retrying a deadline may run the work twice.

The five things a caller has to be able to witness#

An authority boundary you cannot observe is not an authority boundary. These five are the minimum an agent needs to behave correctly, and each fragment below demonstrates the ones its binding exposes.

1. An unauthorized invocation is denied. The caller gets a distinct, typed refusal — not a timeout, not a generic internal error. Your code must be able to tell you may not from nobody answered, because the correct response differs: one may need a credential, while the other requires a routing, availability, or deadline decision.

2. A grant expires or is revoked, and the next call stops working. Authority is checked at call time against the authenticated caller origin. There is no cached permission to go stale and no announcement to wait for — the call after the revocation fails, and it fails as an authorization error rather than a routing one.

3. A deadline elapses. You learn no answer arrived in time. You do not learn whether the work ran. See below.

4. A typed remote error crosses the binding boundary. A handler that fails with a status and a body delivers that status and body to the caller, in the caller's language, rather than a stringified exception. A handler that fails with an untyped error delivers Internal — which is a decision the handler author made, usually without noticing.

5. Execution is ambiguous, and the binding says so. A call that timed out, was cancelled, or lost its transport mid-flight has an unknown outcome.

Ambiguous execution is the one that costs money#

When a deadline elapses the caller emits a cancel for that call id, so a cooperating provider can drop the in-flight handler. "Can drop" is not "did not run." The request may have arrived, executed, had an effect, and failed only to get its reply back to you.

A retry after a timeout can duplicate an external effect. Use one of these contracts:

1. Make the operation idempotent. Then a duplicate is harmless and this whole section stops mattering.
2. Carry an idempotency key the provider deduplicates on.
3. Accept the duplicate, deliberately, and write down that you did.

What does not work is treating a timeout as a failure and retrying, then treating the second timeout the same way. That is how one intended effect becomes three.

Retry, hedge, and circuit breaking#

All four bindings offer the same three strategies over a raw call, and they solve different problems:

• Retry — bounded attempts with backoff, on retryable failures only. Fixes a transient.
• Hedge — race several providers when latency matters more than duplicate work. Costs duplicate execution by design.
• Circuit breaker — fast-fail a provider that is failing, instead of paying its deadline on every call. Fixes a sick provider, not a sick request.

Calling by service or tool name rather than a pinned node id is what makes failover possible at all: the mesh picks a provider, so a substitutable capability survives the loss of the primary. Pinning a node id opts out of that.

The end-to-end patterns are in Recover a Failed Workflow, and the wire-level codes every binding's taxonomy wraps are in Error Codes.

The one rule, expanded#

Retry backpressure, and a transient no-route briefly. Treat serialization and config failures as bugs. Treat authorization failures as "get a new credential," never as "try again." Treat shutdown, not-connected and closed streams as state changes that retrying will not undo. And treat a deadline as unknown, not as failure.

Handle it — Python#

Read this before you branch on any exception type#

When the extension is built without the nRPC feature, every RPC exception class is aliased to RpcError, which is itself aliased to Exception. So except RpcTimeoutError silently becomes except Exception and swallows everything, including the bugs you wanted to surface.

from net.mesh_rpc import RpcError, RpcTimeoutError
 
assert RpcTimeoutError is not RpcError, "nRPC feature not compiled in"

Confirm the feature before writing per-type handlers, or write one handler for RpcError and accept the coarser branch. This is the single sharpest gotcha in the Python binding.

The five witnesses#

import re
from net.mesh_rpc import (
    RpcError, RpcCapabilityDeniedError, RpcTimeoutError,
    RpcServerError, RpcCodecError, RpcCancelledError,
    NRPC_TYPED_BAD_REQUEST,
)
 
try:
    rpc.call_service("summarize", req, {"deadline_ms": 500})
 
# WITNESS 1 & 2 — denied; revocation lands here on the next call.
except RpcCapabilityDeniedError:
    ...   # get a credential — a retry re-asks a question already answered
 
# WITNESS 3 & 5 — deadline elapsed. Outcome UNKNOWN, not failed.
except RpcTimeoutError:
    ...   # the work may or may not have run
 
# WITNESS 4 — a typed remote error crossed the boundary.
except RpcServerError as e:
    m = re.search(r"status\s*=?\s*0x([0-9a-fA-F]+)", str(e))
    status = int(m.group(1), 16) if m else None
    if status == NRPC_TYPED_BAD_REQUEST:
        ...   # the provider rejected the request shape
 
except RpcCodecError:
    ...   # your bug — do not retry
except RpcCancelledError:
    ...   # you did this

RpcServerError has no status attribute. The status is embedded in the message as status=0xNNNN and the only parser is private (_parse_status_from_message), so branching on a status means the regex above. TypeScript exposes .status as a property; Python does not, and that asymmetry is real rather than an omission in this page.

Every message the binding raises begins with a stable nrpc:<kind>: prefix shared with the Node and Go bindings — match on that when you only need the kind.

The nRPC family#

All derive from RpcError. Import from net, not net_sdk:

RpcAppError is the one class that does carry its status as an argument — it is what a handler raises to signal an application status, and it is constructed RpcAppError(status, body).

Classed exceptions on the mesh#

from net_sdk import MeshNode, BackpressureError, NotConnectedError
 
try:
    node.send_on_stream(stream, [payload])
except BackpressureError:
    ...   # window full — the only blindly-retry-safe case
except NotConnectedError:
    ...   # connection lost — a state change, not a retry

These are raised by the reliable mesh-stream send path (send_on_stream, send_blocking), not by the bus emit, which drops under load rather than raising. MeshNode.send_with_retry(...) retries BackpressureError for you with a 5 ms → 200 ms backoff; prefer it over hand-rolling that loop.

Organizations, channels, everything else#

Organizations derive from OrgError: OrgAdmissionDeniedError, OrgCredentialsError, OrgDiscoveryError, OrgUnclassifiedError. Channels — ChannelAuthError derives from ChannelError.

MigrationError and GroupError are flat in Python, where TypeScript nests both under DaemonError. Catch them individually here.

Recover a call#

call_with_retry and call_with_hedge_to are methods on TypedMeshRpc. The default retry predicate treats a RpcServerError whose status will not parse as retryable and emits a RuntimeWarning — fail-open by design, so a formatter change cannot silently disable retry. If you see that warning, the message format has drifted, not your code.

Next: back to the SDK index.