Blob Storage with Dataforts#

Dataforts stores and transfers content-addressed payloads that should not travel inline in events: model weights, datasets, video segments, generated artifacts, and directory trees. Events and RPC calls carry a BlobRef; consumers fetch the bytes only when they need them.

A blob is referenced by its content hash. Producers store bytes and receive a BlobRef; consumers use the reference to read locally or fetch from a holder. Caching and placement policy determine which nodes retain copies.

Putting and getting#

The simplest possible blob flow:

use net_sdk::transport::{
    store_blob_reader, fetch_blob_discovered, Encoding, MeshBlobAdapter, BlobRef,
};
 
// A per-node blob adapter over a Redex manager (`redex: Arc<Redex>`).
let adapter = MeshBlobAdapter::new("blobs", redex.clone());
 
// Put: stream bytes through the adapter; returns a BlobRef identifying
// the content by its hash. The reader is any `AsyncRead` — a File,
// stdin, or an in-memory slice.
let blob_ref: BlobRef = store_blob_reader(
    Some(&adapter),
    &bytes[..],
    "mem://artifact",   // source URI, recorded on the ref as metadata
    Encoding::Replicated,
).await?;
 
// Get: fetch the bytes for a BlobRef from the nearest holder on the mesh.
let bytes = fetch_blob_discovered(&mesh, &blob_ref).await?;

store_blob_reader is content-addressed. Storing the same bytes twice yields the same BlobRef; the runtime deduplicates automatically and never stores the same content twice on the same node. The hash is BLAKE3, chunked with content-defined chunking — so editing the middle of a file doesn't invalidate the chunks at the start or end, and the unchanged chunks dedupe across versions.

fetch_blob_discovered is location-aware. It probes connected peers and pulls the verified bytes from the first holder that serves them; when the holder is already known, fetch_blob(&mesh, source, &blob_ref) names it directly and skips discovery. Manifest fetches can request chunks concurrently. Completion time depends on holder availability, link capacity, scheduling, disk behavior, and configured concurrency rather than simply adding one round trip per chunk.

How the transfer actually works#

Discovery and transfer ride the substrate's existing primitives — there's no separate transfer process, no broker, no out-of-band protocol.

Discovery is fold-based. A node holding a chunk advertises it through the capability fold with a causal:<blake3-hex> tag. A node that wants the chunk consults the fold in memory, finds the nearest holder, and opens a transfer stream to it. The 256-bit BLAKE3 digest is treated as an unguessable bearer token — anyone who learns it can fetch from any holder, so sensitive-content callers must treat the hash as a secret or layer channel / capability auth above the transport.

Transfer rides a scheduled stream. The dedicated blob-transfer subprotocol opens a fair-scheduled reliable stream between the requester and the holder. The holder chunks the blob into ≤8108-byte reliable events terminated by FIN; the receiver concatenates by arrival order and verifies the BLAKE3 digest matches the request. Because the stream is scheduled, multiple in-flight transfers share the link fairly — a 16-GB model download doesn't starve interactive RPC.

Auto-store + heat bump on fetch. Once a blob's bytes arrive, the local Dataforts adapter automatically stores them and bumps the heat counter for the receiver. The next request for the same blob hits the local cache; the next request in the deployment for the same blob can pull from this node instead of crossing the original boundary.

Memory footprint#

Both sides of a transfer stream chunk-at-a-time, so peak memory for a single transfer is roughly one chunk (4 MiB) regardless of total blob size. The receive path writes each verified chunk straight to disk via an atomic-rename writer — it opens an <out>.partial file, appends each chunk as it lands, and renames into place once the manifest is fully consumed. The send path reads through store_blob_reader, hashing and persisting each chunk as it's pulled from the source (a file, stdin, or any AsyncRead). Large leaves inside a directory tree get the same treatment inside fetch_dir — anything above one chunk streams to disk rather than buffering.

The only remaining per-chunk cap is TRANSFER_MAX_CHUNK_BYTES (16 MiB), which guards against a misbehaving holder claiming an absurd chunk size. Total transfer size is primarily bounded by storage and configured limits rather than requiring the entire blob in memory. Process and protocol overhead still vary with chunk count, concurrency, manifests, and filesystem behavior.

The CLI surfaces this directly: net-mesh transfer recv-blob shows a determinate byte-progress bar driven from the per-chunk loop, so an operator watching a long transfer sees byte-count and percentage rather than a generic spinner.

Passing blobs through events#

A BlobRef is small (32 bytes plus a few framing bytes). It's small enough to put in an event payload, store in a CortEX state, or pass as an RPC argument. The pattern that makes Dataforts useful in practice is putting the bytes in Dataforts and the reference in the bus:

// Producer
let bytes = generate_artifact();
let blob_ref = store_blob_reader(
    Some(&adapter),
    &bytes[..],
    "mem://artifact",
    Encoding::Replicated,
).await?;
 
let event = Event::from_str(&serde_json::to_string(&ArtifactReady {
    job_id,
    artifact: blob_ref,
})?)?;
bus.ingest(event)?;
 
// Consumer
let event: ArtifactReady = parse(&payload);
let bytes = fetch_blob_discovered(&mesh, &event.artifact).await?;
process_artifact(&bytes);

The producer puts the bytes once. The reference fans out through the bus to every consumer. Each consumer pulls the bytes from the nearest holder — sometimes that's the producer, sometimes it's a peer that read it earlier and cached it. Network traffic scales with how many distinct consumers actually want the blob, not with how many subscribers the channel has.

publish_with_blob — the two steps as one#

Storing and then publishing by hand has a race: a consumer can receive the reference before the bytes are durable. publish_with_blob does both with the ordering pinned:

let receipt = publish_with_blob(
    &mesh, &adapter, &publisher,
    "mem://artifact",
    bytes,
    BlobDurability::DurableOnLocal,
).await?;

BlobDurability picks how hard the store commits before the event goes out — BestEffort or DurableOnLocal. Retrying a failed call must keep the same durability: dropping from DurableOnLocal to BestEffort on a retry after a partial sync publishes an event whose consumer can race the substrate's flush of the remaining chunks.

One failure mode is worth knowing. If the store succeeds and only the mesh publish fails, you get BlobError::Backend — but the blob is stored and durable. Don't re-store it; republish with MeshNode::publish directly, using the receipt's blob_ref.encode() as the payload.

Directory trees with store_dir and fetch_dir#

Directory transfer is a first-class operation on top of the blob primitive. store_dir walks a local directory, hashes each file and each subtree, and writes a manifest blob that references them all; fetch_dir consumes a manifest blob and materializes the tree on the receiving side.

use net_sdk::transport::{store_dir, fetch_dir};
use std::path::Path;
 
// Producer side — walk the tree, hash it, write the manifest blob.
let root_ref: BlobRef = store_dir(&adapter, Path::new("./workspace")).await?;
publish_event(WorkspaceReady { root: root_ref });
 
// Consumer side — pull the manifest from `source` and materialize the tree.
// The trailing `0` picks the default fetch concurrency.
fetch_dir(&mesh, source, &root_ref, Path::new("./materialized"), 0).await?;

fetch_dir materializes the tree in a sibling temporary path and renames it into place after all entries complete. This gives the destination the filesystem's same-filesystem rename semantics. Applications should still account for platform differences, cleanup after abrupt process or host failure, and external readers that do not follow the same path discipline.

If dest already exists, the runtime preserves the previous tree during the swap and removes the backup after completion. Recovery behavior follows the underlying filesystem and the point at which an abrupt failure occurred.

Caching#

Dataforts maintains a greedy-LRU cache on every node. The cache evicts cold content first; heat counters on each blob bias eviction so frequently-read blobs stay cached longer than their LRU position alone would imply. The cache size is configurable per node — the default is a fraction of available memory, with the rest left to the operating system.

When the cache evicts a blob, the BlobRef remains valid as a content identity but a later fetch still requires an authorized reachable holder or colder tier. If no holder retains the bytes, the reference cannot be materialized.

Data gravity#

The runtime tracks per-blob, per-node read counts. When a blob is repeatedly read from a particular node, the placement layer biases toward landing a copy of the blob on that node — not on every read, but enough that the workload-equilibrium state has popular content near its readers without an operator drawing the placement map by hand.

This is the "data gravity" model the existing literature talks about: heavily-read data migrates toward heavy readers, and lightly-read data stays where it was written. The migration is async, doesn't slow down reads, and doesn't require coordination — each node makes local decisions about what to cache, based on the heat counters it's seen.

For workloads where placement matters more than the default would give you, pin a blob on the nodes that must always hold it. Pinning is a local, per-node decision — each node exempts the pinned content from its own eviction and gravity sweeps:

let now_ms = /* current unix time in ms */;
if let Some(hash) = blob_ref.small_hash() {
    adapter.pin(*hash, now_ms);
}

A pinned blob is exempt from that node's eviction and gravity sweeps until you unpin it. (pin is keyed on a chunk hash; a multi-chunk manifest blob is pinned by pinning each of its chunk hashes.)

Durability#

A blob lives in two places: the local cache (memory, fast, unreliable) and the persistent tier (disk, slower, durable). New puts land in both by default; reads hit the cache when warm, fall back to local disk on miss, fall back to a peer on miss-miss.

The persistent tier uses BLAKE3 as the file name, content-defined chunking to split large blobs into deduplication-friendly pieces, and (for Phase C deployments) Reed-Solomon erasure coding to reduce storage cost across the cluster. The erasure-coding piece is optional; the default is full replication.

adapter.sync_blob(&blob_ref).await? forces a blob's chunks to the persistent tier. You won't usually call it — the runtime persists on its own schedule — but it's available when you need a hard durability barrier (e.g. before acknowledging an upload to an external caller).

The transport SDK#

Five language tiers — Rust (net_sdk::transport), C (net.h extensions), Python (pyo3), TypeScript (napi-rs), Go (CGO over C) — expose the same three operations:

// Rust (net_sdk::transport)
let bytes = fetch_blob(&mesh, source, &blob_ref).await?;
let root  = store_dir(&adapter, Path::new("./src")).await?;
fetch_dir(&mesh, source, &root, Path::new("./out"), 0).await?;

// TypeScript — the ops hang off the Mesh handle
const bytes = await mesh.fetchBlob(holderId, blobRef);
const root = await mesh.storeDir(adapter, "./src");
await mesh.fetchDir(sourceId, root, "./out");

# Python — synchronous module functions (they block internally)
data = fetch_blob(mesh, holder_id, blob_ref)
root = store_dir(mesh, adapter, "./src")
fetch_dir(mesh, source_id, root, "./out")

The SDK stays deliberately thin — no retry policy, no rollback machinery beyond the substrate's own atomicity, no directory-sync primitives. Substrate primitives are exposed; applications compose policy above. The DirManifest and DirEntry introspection types are also re-exported so applications that want to walk a manifest before materializing it (build systems, dependency resolvers, agent delegators) can do so without reaching into substrate internals.

Operator surface#

The runtime exposes per-blob, per-node, per-cluster counters:

• Cache hit rate, evictions, average dwell time.
• Bytes ingested, served from cache, fetched from peers, fetched from disk.
• Heat counters per blob (top-N readers, top-N writers).
• Replication coverage per blob (which nodes hold full copies, which hold partials).
• Bandwidth used by replication-sync vs. user-driven fetches.

The metrics are exposed in the same Prometheus shape as the rest of Net. For one-off inspection, the net-blob CLI (behind the cli feature) wraps the persistent-tier operations directly:

net-blob put ./model.bin
net-blob get <ref> > ./local-copy.bin
net-blob stat <ref>
net-blob ls --tag "version=v3"
net-blob pin <ref> --nodes a,b,c
net-blob gc --max-age 30d

When Dataforts is the right tool#

The rule of thumb is around tens of kilobytes. If your payload fits comfortably in an event (small JSON, short messages, encoded structs), put it in the event. If it's larger — and especially if it's content you'll want to deduplicate, cache, or fetch from arbitrary readers — put it in Dataforts and pass a BlobRef.

The model is composable. The bus moves the small references at high frequency; Dataforts moves the large payloads on demand. CortEX folds can hold BlobRefs in their state; NetDB queries can join against them; nRPC can return them. The blob's identity, like everything else in Net, is content-bound and verifiable — the hash is the reference, so a forged BlobRef won't decode to the wrong bytes.

Use Dataforts when content identity, mesh-local discovery, on-demand transfer, and Net's artifact workflow belong together. Keep an external object store when it is the system of record, its access model already fits, or independent object-store operations are more important than mesh-local placement.