Durable Logs with RedEX#

RedEX provides an append-only log for channel events. Persistence and crash-loss bounds depend on with_persistent and the selected FsyncPolicy; the default policy does not fsync each append. Consumers can read retained ranges or follow a tail subscription.

The mental model is unromantic. A RedEX file is a named log. Producers append; consumers read; consumers can subscribe to the tail and receive new events as they land. There's no schema, no compaction, no transaction. The simplicity is the point — RedEX does one thing and the higher layers compose against that one thing.

Opening a log#

use net::adapter::net::redex::{Redex, RedexFileConfig, FsyncPolicy};
use net::adapter::net::channel::ChannelName;
use std::sync::Arc;
 
let redex = Arc::new(Redex::new());
 
let cfg = RedexFileConfig::default()
    .with_persistent(true)
    .with_fsync_policy(FsyncPolicy::EveryN(100));
 
let file = redex.open_file(&ChannelName::new("sensors/lidar/front")?, cfg)?;

Redex is the top-level manager: it owns the open files, handles the on-disk layout, and exposes per-channel handles. open_file either opens an existing log or creates a fresh one. The same call shape applies whether the channel is in-memory or disk-backed; the difference is the with_persistent(true) flag and a base directory configured on the Redex manager.

A persistent log is two on-disk files: an index file (20-byte records, one per event, fixed-size) and a data file (variable-size, packed). Recovery on reopen is bounded — the manager truncates any partial write at the tail of the data file and rebuilds the in-memory sequence map from the index — so a crash mid-write costs you nothing structural.

Appending#

Append is the only write path. Events go in monotonically by sequence; the log is single-writer, and the writer is the channel's authoritative publisher.

let seq = file.append(payload)?;

payload is opaque bytes (&[u8]). RedEX doesn't decode it, doesn't validate it, doesn't transform it. Append is synchronous — it returns the assigned sequence number, which is what consumers reference when they want to resume from a specific point.

For higher-throughput append paths, use the batched variant — it amortizes the index write and the fsync. It takes a slice of Bytes and returns the sequence of the first event in the batch (None for an empty batch):

let first_seq = file.append_batch(&payloads)?;

Reading#

Three reading patterns cover the cases that come up.

Read a specific range. Useful for inspection, audit, and re-derivation:

let events = file.read_range(100, 200);
for event in events {
    process(event.entry.seq, &event.payload);
}

read_range is synchronous and returns the retained events in the half-open range [start, end) directly — not a Result. Evicted seqs are silently skipped.

Tail subscription. A long-lived stream that delivers events as they land. tail(from_seq) is synchronous — it returns the stream immediately — and each item is a Result<RedexEvent, RedexError> (the Err surfaces a lagged or closed subscriber). Pass 0 to start from the beginning of the log and then follow the tail live. This is the path consumers use for reactive workloads:

use futures::StreamExt;
 
let mut tail = file.tail(0);
while let Some(Ok(event)) = tail.next().await {
    process(event.entry.seq, &event.payload);
}

Resume from a cursor. The tail-from-cursor flavor; useful for consumers that crash and restart and need to pick up where they left off. Same call, with the resume sequence in place of 0:

let mut tail = file.tail(last_processed_seq);

The cursor is a sequence number. Persist it only after the consumer has completed the corresponding work. On restart, resume from that checkpoint and make the consumer idempotent because an event may be processed again around the checkpoint boundary.

Choosing durability#

FsyncPolicy controls when the log is fsynced to disk. Four options, each with a different trade-off:

Two invariants are worth committing to memory:

• close() always fsyncs, regardless of policy. A clean shutdown loses nothing.
• file.sync() always fsyncs. It's the explicit barrier when you need a hard durability point — for example, before acknowledging a write to a caller that's relying on persistence.

Pick the loosest policy you can tolerate. The default (Never) is fastest and is the right answer for the large class of channels where loss-on-crash of the most recent events is acceptable.

Retention#

Logs grow without bound by default. To put a ceiling on size, configure retention on the file config:

let cfg = RedexFileConfig::default()
    .with_retention_max_bytes(1024 * 1024 * 1024)  // 1 GB
    .with_retention_max_age(Duration::from_secs(86400 * 7));  // 7 days

Retention runs on append and on demand (file.sweep_retention()). Events past the retention horizon are evicted from the in-memory index — disk-backed events stay on disk until the next compaction pass, which is a separate concern. If you need hard caps, configure both bytes and age; the runtime applies whichever cuts first.

Retention only affects the head of the log. Active subscribers and recent readers don't lose events out from under them; the eviction targets data that's older than the cutoff and not actively held.

Replication#

A RedEX log on a single node lives and dies with that node's disk. To put a channel on more than one node, enable replication on the file config:

use net::adapter::net::redex::{ReplicationConfig, PlacementStrategy};
 
// First, install the replication plumbing on the Redex manager:
redex.enable_replication(mesh.clone());
 
// Then open the channel with replication configured:
let cfg = RedexFileConfig::default()
    .with_replication(Some(
        ReplicationConfig::new()
            .with_factor(3)
            .with_heartbeat_ms(500),
    ));
let file = redex.open_file(&ChannelName::new("sensors/lidar/front")?, cfg)?;

enable_replication installs the per-Redex router on the mesh's replication subprotocol. After that, opening a channel with replication: Some(_) spawns a per-channel replication coordinator: leader election (deterministic by RTT and health), heartbeat-based liveness, and sync requests that bring replicas up to date.

Replication has its own reference page with placement strategies, bandwidth budgets, and failure modes. A configured factor still depends on eligible nodes, reachable routes, catch-up, and enough retained history to bring a replica current.

When to reach for RedEX directly#

Most application code doesn't open RedEX files by hand — it uses CortEX or NetDB, which open RedEX files on its behalf and give you a higher-level surface. You reach for RedEX directly in three cases.

The first is append-only data that doesn't need a fold. Audit logs, event sourcing for systems that don't have a current-state question to ask, raw telemetry that downstream consumers will fold themselves.

The second is custom domain models. CortEX ships with tasks and memories; if you need a different model with the same fold-driven shape, you implement RedexFold<State> against a RedEX log directly.

The third is operational tooling. Replaying a log, inspecting a range, exporting to another store, validating chain integrity — all of these are RedEX-level operations.

In all three cases, the surface is the same: open a file, append, read, subscribe. RedEX is small enough that the entire API fits on one screen.