Modern applications can generate enormous quantities of telemetry data. Metrics, traces, logs, audit events, performance measurements, security signals, and business events can easily reach millions or even billions of events per day. At that scale, seemingly harmless implementation details—such as allocating a small object for every event—can become a significant performance and operational problem.
A telemetry pipeline designed for massive traffic volumes must therefore do more than simply process events quickly. It needs to control memory usage, minimize garbage collection pressure, avoid unnecessary copying, provide predictable latency, and remain resilient when downstream systems become slow or temporarily unavailable.
One particularly effective design principle is to avoid per-event memory allocation. Instead of creating a new object, string, map, or buffer for every telemetry event, the pipeline can reuse memory through preallocated buffers, object pools, ring buffers, batching, and carefully designed serialization strategies.
This article explains how to design such a pipeline, why per-event allocation becomes expensive at scale, and how to implement a high-throughput architecture using practical coding patterns.
Why Per-Event Allocation Becomes a Problem
Consider a telemetry producer receiving one million events per second.
Suppose each event causes only 200 bytes of temporary heap allocation:
1,000,000 events/sec × 200 bytes
= 200 MB/secOver one minute:
200 MB/sec × 60
= 12 GBThe system may not retain all of that memory simultaneously, but the allocation rate is still enormous. The runtime must continuously allocate objects, track them, and eventually reclaim them.
In a garbage-collected language, this creates additional pressure on the garbage collector. In a manually managed environment, excessive allocation and deallocation can increase allocator overhead and fragmentation.
The problem becomes especially serious when each event contains multiple objects:
TelemetryEvent
├── timestamp
├── metadata object
├── attributes map
├── string fields
├── payload object
└── serialized representationA logically small event can therefore result in many individual allocations.
At low traffic levels, this may be invisible. At billions of events per day, it can become one of the dominant performance costs.
The Core Architecture
A high-throughput telemetry pipeline can be structured around a small number of stages:
Application
|
v
Event Capture
|
v
Preallocated Ring Buffer
|
v
Batch Aggregator
|
v
Serializer
|
v
Compression
|
v
Network Transport
|
v
Telemetry BackendEach stage should have a clear ownership model for memory.
The key principle is:
Allocate memory for the pipeline rather than allocating memory for individual events.
Instead of thinking:
event -> allocate -> process -> freethink:
fixed memory -> reuse -> reuse -> reuseThis approach makes memory consumption more predictable.
Choosing the Right Data Representation
The first step is to avoid unnecessarily object-heavy event structures.
An object-oriented representation might look like this:
class TelemetryEvent {
long timestamp;
String service;
String operation;
Map<String, String> attributes;
byte[] payload;
}This is convenient but potentially expensive at very high event rates.
Every event may involve:
- an object allocation,
- string allocations,
- a map allocation,
- map entry allocations,
- a byte-array allocation,
- serialization allocations.
A more efficient representation can use fixed-size structures where appropriate:
final class EventRecord {
long timestamp;
long traceId;
int serviceId;
int operationId;
int statusCode;
long durationNanos;
}The event now consists primarily of primitive values.
Instead of storing repeated strings such as:
"checkout"
"checkout"
"checkout"
"checkout"the pipeline can assign an integer identifier:
serviceId = 42A separate dictionary can map:
42 -> checkoutThis reduces both memory consumption and serialization overhead.
Reusing Objects With Preallocated Buffers
A straightforward optimization is to allocate a reusable byte buffer once.
For example, in Java:
final class TelemetryBuffer {
private final byte[] buffer;
private int position;
TelemetryBuffer(int capacity) {
this.buffer = new byte[capacity];
}
void reset() {
position = 0;
}
void writeLong(long value) {
buffer[position++] = (byte) (value >>> 56);
buffer[position++] = (byte) (value >>> 48);
buffer[position++] = (byte) (value >>> 40);
buffer[position++] = (byte) (value >>> 32);
buffer[position++] = (byte) (value >>> 24);
buffer[position++] = (byte) (value >>> 16);
buffer[position++] = (byte) (value >>> 8);
buffer[position++] = (byte) value;
}
byte[] data() {
return buffer;
}
int size() {
return position;
}
}The buffer is allocated once:
TelemetryBuffer buffer = new TelemetryBuffer(1024 * 1024);The application can then repeatedly write telemetry data into it and reset the position after a batch has been transmitted.
This eliminates a large class of short-lived byte-array allocations.
Batch Events Instead of Sending Individual Events
One of the most important design decisions is batching.
Sending every event individually creates excessive network overhead:
Event 1 -> network
Event 2 -> network
Event 3 -> network
Event 4 -> networkA batching strategy changes this to:
Event 1 \
Event 2 \
Event 3 -> Batch -> network
Event 4 /
Event 5 /For example:
final int MAX_BATCH = 4096;
EventRecord[] batch = new EventRecord[MAX_BATCH];
int count = 0;
void add(EventRecord event) {
batch[count++] = event;
if (count == MAX_BATCH) {
flush();
}
}
void flush() {
send(batch, count);
count = 0;
}However, this example still allocates EventRecord objects if each event is constructed separately.
A more allocation-conscious approach stores primitive fields in reusable arrays.
Using Structure-of-Arrays Storage
Instead of:
Event[]use:
timestamps[]
traceIds[]
serviceIds[]
durations[]
statusCodes[]For example:
final class EventBatch {
private final long[] timestamps;
private final long[] traceIds;
private final int[] serviceIds;
private final int[] statusCodes;
private final long[] durations;
private int size;
EventBatch(int capacity) {
timestamps = new long[capacity];
traceIds = new long[capacity];
serviceIds = new int[capacity];
statusCodes = new int[capacity];
durations = new long[capacity];
}
void add(
long timestamp,
long traceId,
int serviceId,
int statusCode,
long duration
) {
int index = size++;
timestamps[index] = timestamp;
traceIds[index] = traceId;
serviceIds[index] = serviceId;
statusCodes[index] = statusCode;
durations[index] = duration;
}
void reset() {
size = 0;
}
}The arrays are allocated once.
Every subsequent event simply writes values into existing memory.
This is particularly effective for telemetry because telemetry records frequently contain a predictable collection of numeric fields.
Ring Buffers for High-Throughput Producers
When multiple threads produce telemetry simultaneously, a ring buffer can provide a fixed-size memory region for communication between producers and consumers.
Conceptually:
+-----+-----+-----+-----+-----+-----+
| 0 | 1 | 2 | 3 | 4 | 5 |
+-----+-----+-----+-----+-----+-----+
^ ^
read writeWhen the writer reaches the end, it wraps around:
0 -> 1 -> 2 -> 3 -> 4 -> 5 -> 0 -> ...A simplified implementation might look like:
final class RingBuffer {
private final EventBatch[] slots;
private final int mask;
RingBuffer(int capacityPowerOfTwo) {
slots = new EventBatch[capacityPowerOfTwo];
mask = capacityPowerOfTwo - 1;
for (int i = 0; i < capacityPowerOfTwo; i++) {
slots[i] = new EventBatch(4096);
}
}
EventBatch get(long sequence) {
return slots[(int) sequence & mask];
}
}The important characteristic is that the memory is allocated during initialization.
The hot path does not continually create new batches.
Handling Concurrency Without Creating Objects
Telemetry systems often run across many application threads.
A naive implementation might create a queue node for every event:
queue.add(new TelemetryEvent(...));At high throughput, this generates a continuous allocation stream.
A more efficient design can use preallocated slots.
Each slot contains storage for one batch:
Producer 1 ----\
Producer 2 -----+--> Preallocated Queue --> Consumer
Producer 3 ----/The producers reserve slots, write telemetry into them, and publish them to the consumer.
The consumer processes the same memory and eventually marks the slot reusable.
This is the fundamental idea behind many high-performance queue designs.
Avoiding String Allocation
Strings are one of the most common hidden allocation sources.
Consider:
String message = "latency=" + duration + ",status=" + status;This can create temporary objects during concatenation and formatting.
At massive scale, avoid constructing human-readable strings on the hot path unless they are genuinely required.
Instead, write primitive values directly into a buffer:
buffer.writeLong(timestamp);
buffer.writeLong(duration);
buffer.writeInt(statusCode);If textual output is required, use a reusable encoder.
For example:
final class IntegerEncoder {
private final byte[] buffer;
IntegerEncoder(byte[] buffer) {
this.buffer = buffer;
}
int writeInt(int value, int offset) {
return Integer.toString(value)
.getBytes(java.nio.charset.StandardCharsets.UTF_8,
java.nio.ByteBuffer.wrap(buffer, offset, buffer.length - offset));
}
}For production systems, a specialized numeric encoder that writes directly into the destination byte array is preferable because even helper methods can introduce temporary allocations depending on the implementation.
Dictionary Encoding for Repeated Metadata
Telemetry often contains repeated values.
For example:
service=payments
service=payments
service=payments
service=paymentsRather than repeatedly storing the complete string, maintain a dictionary:
1 -> payments
2 -> checkout
3 -> inventory
4 -> authenticationEvents can then store:
serviceId=1A dictionary can be implemented using a concurrent map during initialization or when new metadata is discovered:
ConcurrentHashMap<String, Integer> dictionary =
new ConcurrentHashMap<>();
AtomicInteger nextId = new AtomicInteger();
int getId(String value) {
return dictionary.computeIfAbsent(
value,
ignored -> nextId.getAndIncrement()
);
}However, this lookup should ideally not happen for every event when the same metadata repeatedly appears. Cache the resulting numeric identifier in the application or instrumentation layer whenever possible.
Designing the Serialization Layer
Serialization is another place where allocation can quietly become expensive.
A common but allocation-heavy pattern is:
String json = objectMapper.writeValueAsString(event);
byte[] payload = json.getBytes(StandardCharsets.UTF_8);This may involve multiple intermediate representations.
A high-throughput pipeline should serialize directly into a reusable buffer.
Conceptually:
void serialize(EventBatch batch, ByteBuffer output) {
for (int i = 0; i < batch.size(); i++) {
output.putLong(batch.timestamp(i));
output.putLong(batch.traceId(i));
output.putInt(batch.serviceId(i));
output.putInt(batch.statusCode(i));
output.putLong(batch.duration(i));
}
}The exact format can vary. Binary formats are often more compact and faster to generate than verbose textual formats, especially when the schema is known.
Compression Should Happen After Batching
Compression can dramatically reduce network bandwidth, but compressing every individual event is usually inefficient.
Instead:
events
|
v
batch
|
v
serialize
|
v
compress
|
v
sendFor example:
void flush(EventBatch batch) {
serializer.serialize(batch, outputBuffer);
compressor.compress(
outputBuffer,
compressedBuffer
);
transport.send(compressedBuffer);
batch.reset();
}Compression operates much more efficiently when it receives a larger contiguous block of related data.
Repeated field names, timestamps, identifiers, and structural patterns can also compress effectively.
Applying Backpressure
A telemetry pipeline must account for downstream failure.
Suppose the application generates:
1,000,000 events/secbut the telemetry backend temporarily accepts only:
500,000 events/secIf the pipeline continuously allocates memory for the remaining events, memory usage can grow until the application becomes unstable.
A fixed-capacity queue provides a hard memory boundary.
For example:
Queue capacity = 100 MBOnce the queue is full, the system must choose a policy.
Possible policies include:
- Drop the oldest telemetry.
- Drop the newest telemetry.
- Sample low-priority events.
- Preserve critical events.
- Block producers temporarily.
- Reduce telemetry detail.
For application telemetry, blocking business-critical application threads indefinitely is usually undesirable. A bounded, non-blocking telemetry path with controlled loss is often preferable.
Sampling Is a Capacity-Control Mechanism
Sampling should not be viewed solely as a storage optimization.
It is also a memory and CPU protection mechanism.
For example:
boolean shouldSample(long traceId) {
return (traceId & 0xFF) == 0;
}This deterministic approach samples approximately one out of every 256 trace IDs.
More sophisticated strategies can sample based on event type:
Errors -> 100%
Slow requests -> 100%
Security events -> 100%
Normal requests -> 1%
Debug diagnostics -> 0.1%This allows the pipeline to prioritize information that is operationally valuable.
Separating Hot and Cold Paths
Not every telemetry event needs identical processing.
A useful architecture separates the hot path from the cold path.
The hot path should do as little as possible:
capture timestamp
capture identifiers
write primitive values
publish batch
returnThe cold path can perform more expensive work:
serialization
compression
network transmission
enrichment
indexing
retry handlingThis keeps application request latency independent from many telemetry operations.
For example:
void recordRequest(
long traceId,
int status,
long duration
) {
EventBatch batch = localBatch();
batch.add(
System.nanoTime(),
traceId,
SERVICE_ID,
status,
duration
);
if (batch.isFull()) {
publisher.publish(batch);
}
}The request thread does not need to serialize the event into JSON, compress it, or perform network I/O.
Thread-Local Batches
For applications with many producer threads, thread-local batching can further reduce synchronization.
Conceptually:
private static final ThreadLocal<EventBatch> LOCAL_BATCH =
ThreadLocal.withInitial(
() -> new EventBatch(1024)
);An event can then be written directly into the producer’s local batch:
void record(long traceId, long duration) {
EventBatch batch = LOCAL_BATCH.get();
batch.add(
System.nanoTime(),
traceId,
SERVICE_ID,
200,
duration
);
if (batch.isFull()) {
publish(batch);
batch.reset();
}
}This can significantly reduce contention because producers are not competing for the same mutable batch.
The trade-off is that each thread owns some memory. Therefore, thread counts and batch sizes should be controlled carefully.
Object Pools: Useful but Not Always Necessary
Object pools are frequently proposed as a solution to allocation overhead.
For example:
Event event = pool.borrow();
try {
event.populate(...);
process(event);
} finally {
pool.release(event);
}Pooling can help for expensive objects, but it should not automatically be applied to everything.
A pool introduces:
- lifecycle complexity,
- contention,
- ownership concerns,
- stale state risks,
- pool exhaustion behavior.
For simple telemetry records containing primitive fields, a preallocated array or ring buffer is often simpler and faster than an object pool.
The preferred hierarchy is generally:
1. Avoid the object entirely
2. Reuse a buffer
3. Reuse a fixed slot
4. Pool expensive objects when necessary
5. Allocate only when unavoidableMonitoring the Pipeline Itself
A telemetry pipeline needs telemetry of its own.
Important metrics include:
events_received_total
events_dropped_total
events_sampled_total
batches_created_total
batches_flushed_total
queue_depth
queue_capacity
serialization_duration
compression_duration
network_duration
flush_failures
retry_countMemory-related metrics are especially important:
heap_used
heap_committed
allocation_rate
gc_pause_duration
gc_frequency
buffer_utilizationA useful signal is queue utilization:
double utilization =
(double) queueSize / queueCapacity;You can then expose:
0.0 - 0.5 -> healthy
0.5 - 0.8 -> increasing pressure
0.8 - 0.95 -> serious pressure
0.95 - 1.0 -> overloadThese thresholds should be treated as operational examples rather than universal values.
A Complete Simplified Pipeline
The following example combines the major concepts:
final class TelemetryPipeline {
private final EventBatch batch;
private final ByteBuffer serializationBuffer;
private final ByteBuffer compressionBuffer;
TelemetryPipeline(int capacity) {
this.batch = new EventBatch(capacity);
this.serializationBuffer =
ByteBuffer.allocateDirect(1024 * 1024);
this.compressionBuffer =
ByteBuffer.allocateDirect(1024 * 1024);
}
void record(
long traceId,
int serviceId,
int statusCode,
long duration
) {
batch.add(
System.nanoTime(),
traceId,
serviceId,
statusCode,
duration
);
if (batch.isFull()) {
flush();
}
}
void flush() {
serializationBuffer.clear();
serialize(batch, serializationBuffer);
compressionBuffer.clear();
compress(
serializationBuffer,
compressionBuffer
);
send(compressionBuffer);
batch.reset();
}
private void serialize(
EventBatch batch,
ByteBuffer output
) {
for (int i = 0; i < batch.size(); i++) {
output.putLong(batch.timestamp(i));
output.putLong(batch.traceId(i));
output.putInt(batch.serviceId(i));
output.putInt(batch.statusCode(i));
output.putLong(batch.duration(i));
}
}
private void compress(
ByteBuffer input,
ByteBuffer output
) {
// Compression implementation.
}
private void send(ByteBuffer buffer) {
// Network transport implementation.
}
}This is intentionally simplified, but it demonstrates the architecture.
The pipeline allocates its primary memory during initialization. The event path primarily writes primitive values into existing memory.
Common Mistakes To Avoid
A high-throughput telemetry implementation can still accidentally reintroduce allocation.
Watch for patterns such as:
new HashMap<>()inside an event handler.
Also avoid:
String.format(...)for every event.
Be careful with:
stream.map(...)when the implementation creates temporary objects on a critical path.
Another common mistake is logging the telemetry system itself excessively:
logger.info("Processed event {}", event);If this occurs millions of times per second, the diagnostic system becomes part of the allocation problem.
Use counters and sampled diagnostic logs instead.
Capacity Planning
A fixed-memory design requires capacity calculations.
Suppose:
Peak event rate: 2,000,000 events/sec
Average event size: 80 bytes
Batch size: 8,192 events
Number of buffers: 64The approximate raw memory required for event storage is:
8,192 × 80 × 64
= 41,943,040 bytesor approximately:
40 MBAdditional memory must be reserved for serialization, compression, networking, metadata, and operating-system buffers.
The goal is not to eliminate memory usage. The goal is to make it bounded and predictable.
Testing Under Realistic Load
A telemetry pipeline should be tested at and beyond its expected peak rate.
Measure:
events/sec
CPU utilization
allocation rate
GC pauses
p99 latency
queue depth
network throughput
compression ratio
dropped eventsA particularly useful test is an overload test.
For example:
Normal:
1M events/sec
Peak:
2M events/sec
Backend degradation:
500K events/sec
Recovery:
2M events/secThe pipeline should remain bounded during backend degradation and recover without causing an uncontrolled memory spike.
A successful system should demonstrate that queue growth reaches a predefined limit rather than continuing indefinitely.
Graceful Shutdown
Telemetry systems also need careful shutdown behavior.
When the application terminates, producers should stop accepting new events:
acceptingEvents = false;Then the pipeline should flush outstanding batches:
while (!batch.isEmpty()) {
flush();
}Finally, network resources and buffers can be released.
The shutdown timeout should be bounded so that telemetry cannot prevent the application from terminating indefinitely.
Designing for Failure
No telemetry backend is permanently reliable.
Networks fail. Brokers become unavailable. DNS fails. Authentication expires. Storage systems throttle requests.
The telemetry pipeline should therefore assume that delivery failures are normal operational conditions.
A robust architecture might look like:
Application
|
v
Local bounded buffer
|
+----> primary collector
|
+----> controlled retry
|
+----> sampling/drop policyRetries should also be bounded.
An unlimited retry queue is simply another form of unbounded memory growth.
The Fundamental Design Principle
The most important principle can be summarized as:
Make memory proportional to configured capacity, not proportional to traffic duration.
If your system receives twice as much traffic for ten seconds, memory usage should remain within the configured bounds.
If the backend is unavailable for five minutes, memory usage should still remain bounded.
If the application generates billions of events per day, the system should not require billions of simultaneously allocated objects.
Instead, the pipeline should continuously recycle a relatively small number of memory regions.
Putting Everything Together
A production-grade telemetry architecture can therefore follow this pattern:
+--------------------+
| Application |
+---------+----------+
|
v
+--------------------+
| Lightweight Event |
| Capture |
+---------+----------+
|
v
+--------------------+
| Thread-Local / |
| Preallocated Batch |
+---------+----------+
|
v
+--------------------+
| Bounded Ring |
| Buffer |
+---------+----------+
|
v
+--------------------+
| Serializer |
+---------+----------+
|
v
+--------------------+
| Compression |
+---------+----------+
|
v
+--------------------+
| Network Transport |
+---------+----------+
|
v
+--------------------+
| Telemetry Backend |
+--------------------+The application-facing portion remains extremely lightweight.
The expensive work is moved to asynchronous consumers operating on batches.
Memory is allocated during initialization, reused during processing, and released only when the application or pipeline itself is shut down.
Conclusion
Building a telemetry pipeline capable of handling massive daily traffic volumes is fundamentally a resource-management problem as much as it is a networking or observability problem. Once event rates reach hundreds of thousands or millions of events per second, seemingly insignificant allocations can become a major source of CPU consumption, garbage-collection pressure, latency variation, and memory instability.
The central strategy is to stop treating each telemetry event as an independent heap object.
Instead, design the pipeline around reusable memory.
Preallocated arrays can store event fields without creating a new object for every record. Ring buffers can provide bounded communication between producers and consumers. Thread-local batches can reduce synchronization and contention. Reusable serialization buffers can eliminate repeated byte-array creation. Dictionary encoding can replace repetitive strings with compact identifiers. Batching can reduce network overhead and make compression more effective. Finally, bounded queues, sampling, and explicit backpressure policies can ensure that a slow downstream system does not turn into an unbounded memory problem.
The distinction between an allocation-oriented design and a reuse-oriented design is significant.
An allocation-oriented pipeline behaves conceptually like this:
receive event
↓
allocate object
↓
allocate metadata
↓
allocate payload
↓
serialize
↓
allocate serialized buffer
↓
send
↓
garbage collectionA reuse-oriented pipeline instead looks like:
receive event
↓
write into existing slot
↓
append to existing batch
↓
serialize into reusable buffer
↓
compress reusable output
↓
send
↓
reset and reuseThe second model does not eliminate all allocations from the application. That is neither realistic nor necessarily desirable. Instead, it removes unnecessary allocations from the hottest and most frequently executed parts of the telemetry path.
That distinction is crucial.
The goal should not be to achieve a theoretical “zero-allocation” system at any cost. The goal is to create a system whose memory behavior is predictable, measurable, bounded, and appropriate for its workload.
This also changes how the system should be tested. Benchmarking only average throughput is insufficient. A production telemetry pipeline must be tested under peak traffic, downstream throttling, network failures, consumer slowdowns, retry conditions, and recovery scenarios. Memory usage should remain bounded throughout these situations. Queue depth, allocation rate, garbage-collection behavior, dropped events, batch sizes, serialization time, compression time, and network latency should all be observable.
Backpressure is particularly important. If the producer can generate telemetry faster than the consumer can transmit it, the system needs an explicit policy. A bounded buffer combined with sampling, prioritization, and controlled dropping is generally safer than allowing an ever-growing queue to consume application memory. Critical signals such as errors or security events can receive different treatment from high-volume informational events.
There is also an important architectural lesson here: the application should not perform unnecessary telemetry work synchronously. Capturing a timestamp, identifiers, status codes, and a few numeric measurements should be inexpensive. Serialization, compression, batching, retries, and network communication should preferably occur asynchronously. This separation allows the application’s critical request path to remain stable even when the telemetry backend experiences temporary problems.
Data representation matters as well. Repeated strings, maps, nested objects, and generic serialization frameworks can be convenient, but convenience can become expensive at extreme throughput. A compact schema using primitive values and identifiers can dramatically reduce memory and CPU overhead. This is especially effective when the telemetry structure is well understood and changes relatively infrequently.
Ultimately, a scalable telemetry pipeline should be designed around a simple question:
How much memory are we willing to dedicate to telemetry, and how does the system behave when that capacity is exhausted?
Once that question has a concrete answer, many design decisions become clearer. Buffer sizes can be calculated. Queue limits can be defined. Sampling thresholds can be established. Backpressure behavior can be tested. Failure scenarios can be simulated. Operational alerts can be built around measurable limits.
The result is a telemetry system that can handle enormous daily event volumes without allowing event count itself to dictate memory consumption.
The strongest architecture is therefore not the one that merely processes the greatest number of events in a benchmark. It is the one that can sustain its expected workload while maintaining predictable latency, bounded memory usage, controlled failure behavior, and sufficient observability into its own operation.
By preallocating memory, reusing buffers, batching events, avoiding unnecessary strings and objects, using bounded queues, separating capture from delivery, and explicitly designing for overload, it is possible to build a telemetry pipeline that remains efficient even as traffic grows from millions to billions of events per day.
That is the essential principle behind high-throughput telemetry engineering:
allocate for the pipeline, reuse for the events, batch for the network, and bound everything that can grow.