Errors and Recovery#

Failure is a typed outcome, not a silence. Every binding gives you enough structure to decide, per failure, whether to retry, reroute, or stop — and the rule underneath all of them is the same.

Retry only when the typed result and capability contract make another attempt safe.

A pre-execution backpressure result can permit a delayed retry when it establishes that the provider did not admit the work. Retrying a serialization failure reruns the bug. Retrying a denial re-asks a question already answered. Retrying a deadline may run the work twice.

The five things a caller has to be able to witness#

An authority boundary you cannot observe is not an authority boundary. These five are the minimum an agent needs to behave correctly, and each fragment below demonstrates the ones its binding exposes.

1. An unauthorized invocation is denied. The caller gets a distinct, typed refusal — not a timeout, not a generic internal error. Your code must be able to tell you may not from nobody answered, because the correct response differs: one may need a credential, while the other requires a routing, availability, or deadline decision.

2. A grant expires or is revoked, and the next call stops working. Authority is checked at call time against the authenticated caller origin. There is no cached permission to go stale and no announcement to wait for — the call after the revocation fails, and it fails as an authorization error rather than a routing one.

3. A deadline elapses. You learn no answer arrived in time. You do not learn whether the work ran. See below.

4. A typed remote error crosses the binding boundary. A handler that fails with a status and a body delivers that status and body to the caller, in the caller's language, rather than a stringified exception. A handler that fails with an untyped error delivers Internal — which is a decision the handler author made, usually without noticing.

5. Execution is ambiguous, and the binding says so. A call that timed out, was cancelled, or lost its transport mid-flight has an unknown outcome.

Ambiguous execution is the one that costs money#

When a deadline elapses the caller emits a cancel for that call id, so a cooperating provider can drop the in-flight handler. "Can drop" is not "did not run." The request may have arrived, executed, had an effect, and failed only to get its reply back to you.

A retry after a timeout can duplicate an external effect. Use one of these contracts:

1. Make the operation idempotent. Then a duplicate is harmless and this whole section stops mattering.
2. Carry an idempotency key the provider deduplicates on.
3. Accept the duplicate, deliberately, and write down that you did.

What does not work is treating a timeout as a failure and retrying, then treating the second timeout the same way. That is how one intended effect becomes three.

Retry, hedge, and circuit breaking#

All four bindings offer the same three strategies over a raw call, and they solve different problems:

• Retry — bounded attempts with backoff, on retryable failures only. Fixes a transient.
• Hedge — race several providers when latency matters more than duplicate work. Costs duplicate execution by design.
• Circuit breaker — fast-fail a provider that is failing, instead of paying its deadline on every call. Fixes a sick provider, not a sick request.

Calling by service or tool name rather than a pinned node id is what makes failover possible at all: the mesh picks a provider, so a substitutable capability survives the loss of the primary. Pinning a node id opts out of that.

The end-to-end patterns are in Recover a Failed Workflow, and the wire-level codes every binding's taxonomy wraps are in Error Codes.

The one rule, expanded#

Retry backpressure, and a transient no-route briefly. Treat serialization and config failures as bugs. Treat authorization failures as "get a new credential," never as "try again." Treat shutdown, not-connected and closed streams as state changes that retrying will not undo. And treat a deadline as unknown, not as failure.

Handle it — TypeScript#

Every error the SDK throws is a subclass of Error, so instanceof is always the discriminator.

The five witnesses#

import {
  RpcError,
  RpcCapabilityDeniedError,
  RpcTimeoutError,
  RpcServerError,
  RpcCancelledError,
  RpcCodecError,
} from '@net-mesh/core/errors';
 
try {
  await clientRpc.callService('summarize', req, { deadlineMs: 500 });
} catch (e) {
  // WITNESS 1 & 2 — denied; revocation lands here on the next call.
  if (e instanceof RpcCapabilityDeniedError) {
    // Get a credential. A retry re-asks a question already answered.
  }
  // WITNESS 3 & 5 — deadline elapsed. Outcome UNKNOWN, not failed.
  else if (e instanceof RpcTimeoutError) {
    // The work may or may not have run. Idempotency, or accept the duplicate.
  }
  // WITNESS 4 — a typed remote error crossed the boundary.
  else if (e instanceof RpcServerError) {
    console.error(`provider returned ${e.status}`);
  }
  else if (e instanceof RpcCodecError)   { /* your bug — do not retry */ }
  else if (e instanceof RpcCancelledError) { /* you did this */ }
  else if (e instanceof RpcError)        { /* the rest of the family */ }
  else throw e;
}

RpcServerError has a status property in TypeScript. That is worth naming because Python's does not — there the status is only inside the message string.

classifyError(e) maps a raw thrown value to the right subclass when you are catching something that came through an untyped path:

import { classifyError } from '@net-mesh/core/errors';
import { NRPC_TYPED_BAD_REQUEST } from '@net-mesh/core/mesh_rpc';
 
const typed = classifyError(e);
if (typed instanceof RpcServerError && typed.status === NRPC_TYPED_BAD_REQUEST) {
  // the provider rejected the request shape — a caller bug
}

The nRPC family#

All extend RpcError, so catching the base handles the whole family:

Classed errors on the mesh#

import { BackpressureError, NotConnectedError } from '@net-mesh/sdk';
 
try {
  await node.sendOnStream(stream, payloads);
} catch (e) {
  if (e instanceof BackpressureError) {
    // window full — the only blindly-retry-safe case (or use sendWithRetry)
  } else if (e instanceof NotConnectedError) {
    // connection lost — a state change, not a retry
  } else {
    throw e;
  }
}

Note that the bus emit does not throw under backpressure — it returns null. BackpressureError is the reliable mesh-stream path only.

Organizations, compute, everything else#

Organizations all extend OrgError: OrgAdmissionDeniedError, OrgCredentialsError, OrgDiscoveryError, OrgUnclassifiedError.

Compute — MigrationError and GroupError both extend DaemonError.

Recover a call#

import { RetryPolicy, HedgePolicy, CircuitBreaker } from '@net-mesh/core/mesh_rpc';
 
await clientRpc.callWithRetry(nodeId, 'summarize', req,
  new RetryPolicy({ maxAttempts: 4, initialBackoffMs: 50 }));
 
await clientRpc.callWithHedgeTo([nodeA, nodeB, nodeC], 'summarize', req,
  new HedgePolicy({ maxParallel: 3, hedgeDelayMs: 50 }));
 
const breaker = new CircuitBreaker({ failureThreshold: 5, resetAfterMs: 1000 });
await breaker.call(() => clientRpc.call(nodeId, 'summarize', req, { deadlineMs: 500 }));

Next: back to the SDK index.