MA
← Back to selected work

Architecture · Storage · Scale

Designing a Memory-Efficient Data Distribution Architecture

I started by investigating OOM failures in a data export path. While tracing where the generated artifact went next, I found that the serving side made the same full-dataset assumption. That led me to evaluate alternative designs, prototype LMDB with data representative of production scale, and propose a local snapshot architecture for serving lookups.

StatusDesign · Validation · Architecture review

At a glance

Problem

The exporter was running out of memory as the dataset grew. Tracing the artifact downstream also exposed full-dataset materialization in the serving path.

My role

I investigated the end-to-end flow, evaluated alternatives, built and measured LMDB POCs, and designed the proposed serving integration.

Current status

The storage direction has been validated through POCs. Snapshot distribution, lifecycle, and failure behavior are still under architecture review.

The problem

Tracing the artifact exposed the same memory assumption twice.

The original issue was on the producer side. A service scanned a distributed datastore, built the full dataset in memory, serialized it, and uploaded the resulting artifact to object storage. As the dataset grew, that process started running into memory pressure and OOM failures.

I followed the artifact beyond the exporter to understand how it was consumed. The serving path periodically downloaded it and also materialized the complete dataset in application memory. That had not become an operational incident on the serving side, but it showed that the same scaling assumption existed at both ends of the flow.

At that point, fixing only the exporter felt too narrow. I started looking at how the data was represented, distributed, and queried end to end.

Existing data flow

The existing design produced a serialized snapshot and later rebuilt the same logical dataset inside the serving process.

Existing systemTwo points materialize the complete dataset

Options

Options I considered

Once local membership lookup became the main requirement, I compared a few ways to change the flow. The main constraint was keeping a remote dependency out of a latency-sensitive serving path.

  1. 01

    Direct remote lookups

    Query the upstream datastore directly from serving. This would simplify distribution, but every membership check would add a network dependency to a high-volume request path. I ruled this out early.

    Rejected
  2. 02

    Sharded serialized artifacts

    Split the existing artifact into smaller pieces. This would relieve some pressure on the exporter and avoid relying on one continually growing serialized message, but serving would still rebuild the dataset in application memory. I kept this as a possible incremental fallback.

    Fallback
  3. 03

    Local LMDB snapshot

    LMDB was already used for another local lookup on the same latency-sensitive serving path, which made it a practical candidate to evaluate for this workload. I built POCs to understand how well it fit the membership use case and what its storage characteristics looked like at representative scale.

    The useful property was that the distributed artifact could become the queryable local structure itself, instead of something the serving process had to deserialize into another full in-memory representation.

    Selected direction

Proposed architecture

The proposed flow builds a read-only LMDB snapshot, distributes it to the serving environment, and performs membership checks against the local database. Remote lookups stay outside the request path.

Proposed architectureLocal reads keep remote lookups outside the hot path

Boundary: snapshot distribution and failure semantics remain under architecture review.

The storage model is the better-understood part of the design today. Snapshot publication, distribution, activation, rollback, and failure behavior still need to be finalized as part of the architecture review.

Validation

What the POC showed

The workload only needed membership checks, so I also tested whether the original identifier representation was necessary inside the database. One POC used a fixed-width 8-byte key representation to explore the storage trade-off.

With data representative of production scale, that experiment reduced the average key representation from roughly 27 bytes to 8 bytes. The resulting LMDB footprint was about 43% smaller, and the compressed snapshot about 31% smaller.

The 8-byte representation was an experiment for the POC. Selecting a production representation would still require explicit decisions around stability and collision behavior.

~27 B → 8 B
Average identifier/key representation in the experiment
~43%
POC LMDB footprint reduction
~31%
Compressed snapshot reduction

Trade-offs

Snapshot distribution and failure behavior

Local reads simplify the serving path and avoid adding a remote lookup, but they move complexity into snapshot management. The system still needs clear behavior for publishing a new version, distributing it safely, switching versions, rolling back, and handling an invalid or unavailable snapshot.

Those details are still part of the architecture work. They matter because a local database only helps if every serving instance can move between versions predictably.

My contribution

From the exporter issue to the serving design

I started with the exporter OOM investigation and traced the data through the serving path. From there, I evaluated the main alternatives, built and measured the LMDB POCs, explored the key representation trade-off, and proposed how a local read-only snapshot could fit into serving.

The work grew from investigating an immediate memory problem into evaluating a different way to represent, distribute, and query the dataset end to end.

Current status

Where the design stands

The POCs support the local LMDB snapshot direction. The remaining work is mainly around snapshot distribution, lifecycle, and failure semantics, which are still being refined through architecture review.

The measurements above come from the POCs. The architecture is still under review and hasn't been deployed to production.