Log Shipper Offline Buffering: How to Keep Security Logs Safe When the Destination Goes Down

Offline buffering is a log shipper's ability to temporarily store events when the destination is unavailable and automatically forward them once connectivity is restored. Without effective buffering, a SIEM outage, network interruption, or agent restart can create permanent gaps in security telemetry.

Every SIEM environment will eventually experience an interruption. The same applies to WAN connections, central collectors, and the infrastructure running your log management stack. These incidents should not result in lost security data, but the outcome depends heavily on how your log collection technology handles events when delivery is interrupted.

This article explains how offline buffering works, how NXLog Agent protects logs during destination outages, how to size and test buffers, and how common log shippers compare in terms of resilience and data protection.

What Is Offline Buffering in a Log Shipper?

A log shipper sits between event sources such as Windows Event Log, syslog, files, and applications and destinations such as a SIEM, data lake, database, or central collector.

When the destination processes events more slowly than they are generated—or becomes completely unavailable—the log shipper needs temporary storage for the growing backlog. This storage is commonly referred to as a buffer, while the process of retaining events until the destination becomes available again is known as offline buffering.

For security operations teams, this capability is critical. Missing logs can prevent correlation rules from triggering and leave threat hunters without the evidence they need for investigations. The same gaps can weaken audit trails and compliance evidence.

Effective buffering changes the situation from data loss to delayed delivery.

Figure 1. Where a log pipeline stores events when the destination becomes unavailable.

Where Log Pipelines Can Lose Data

Several common failure scenarios can result in lost log data. Each requires the right protection mechanism.

The destination becomes unavailable

SIEM maintenance, ingestion throttling, service failures, licensing limits, or quota restrictions can prevent the destination from accepting new events. The log shipper needs sufficient queue or buffer capacity to retain those events until the destination recovers.

The network connection fails

A broken VPN tunnel, WAN outage, routing problem, or saturated network link can have the same effect as a failed SIEM. In these situations, both buffer capacity and backpressure handling become important.

The agent or host restarts

Software updates, system crashes, power failures, or planned maintenance can remove anything that exists only in process memory. Persistent disk-based queues are therefore essential for scenarios where data must survive an unexpected restart.

The source cannot be paused

Some log sources, particularly UDP syslog and local /dev/log sockets, cannot simply wait for the collector to become available. If the receiver stops accepting data, the operating system or sending application may discard events before the log shipper can process them.

Memory vs. Disk Buffering

Neither approach is universally optimal. Memory buffering provides high performance and is well suited to short interruptions. Persistent queues and disk-based buffers provide stronger protection against data loss but increase disk I/O.

For production environments, a combination of both approaches is often the most practical solution: memory handles short disruptions, while persistent disk buffering protects critical sources during longer outages.

How NXLog Agent Buffers Logs When the Destination Is Offline

NXLog Agent provides several mechanisms for handling temporary destination failures, including built-in log queues, flow control, persistent queues, dedicated disk buffers, and acknowledged transport between agents.

Log Queues and Flow Control

Every processor and output module instance in NXLog Agent has an input log queue that temporarily stores events waiting to be processed. The LogqueueSize directive controls the queue capacity.

For memory-based queues, the default size is 2 MiB, with a minimum of 512 KiB. When the queue approaches its configured capacity, NXLog Agent uses flow control to apply backpressure to upstream modules.

For pausable sources such as files, Windows Event Log, and TCP connections, this allows the system to temporarily slow or suspend collection rather than continuously generating an unmanageable backlog.

NXLog Agent also preserves queued records during a clean shutdown by writing them to disk and processing them after restart before accepting new incoming events.

In routes with multiple destinations, flow control can also be configured independently so that a blocked destination does not necessarily prevent healthy destinations from continuing to receive events.

Persistent Queues for Crash Protection

Clean shutdowns can preserve queued events, but a sudden power failure or process crash requires stronger protection. NXLog Agent supports persistent log queues through PersistLogqueue.

For the strongest protection against hard crashes, SyncLogqueue can be enabled to synchronize records to disk before processing continues.

<Input windows_events>
    Module             im_msvistalog
</Input>

<Output siem>
    Module             om_elasticsearch
    URL                https://siem.example.com:9200/_bulk
    LogqueueSize       4194304
    PersistLogqueue    TRUE
    SyncLogqueue       TRUE
</Output>

<Route r1>
    Path               windows_events => siem
</Route>

SyncLogqueue TRUE provides the strongest queue-level protection against a hard crash, but it also has the highest performance cost. It should therefore be reserved for sources where losing even individual events is unacceptable.

Dedicated Buffers for Longer Outages

Log queues are primarily designed to provide backpressure and absorb short periods of congestion. Longer outages may require a dedicated Buffer processor module.

The Buffer processor module can maintain either memory or disk-based buffers and allows administrators to define maximum capacity using MaxSize. The optional WarnLimit setting generates a warning before the buffer reaches its maximum size.

<Input syslog_udp>
    Module        im_udp
    ListenAddr    0.0.0.0:514
</Input>

<Processor disk_buffer>
    Module        pm_buffer
    Type          Disk
    MaxSize       512000
    WarnLimit     409600
</Processor>

<Output siem>
    Module        om_http
    URL           https://siem.example.com:8080/
</Output>

<Route r1>
    Path          syslog_udp => disk_buffer => siem
</Route>

Organizations that experience frequent short interruptions can also combine a small memory buffer with a larger disk buffer. Routine network disruptions can remain in memory, while prolonged outages spill over into persistent storage.

Handling Sources That Cannot Be Paused

Flow control works by applying backpressure to sources that can safely be suspended. This is not appropriate for every log source.

UDP is connectionless, so the receiver must accept incoming packets immediately. NXLog Agent provides the SockBufSize directive to increase the operating system socket buffer and absorb temporary bursts.

Local /dev/log sockets require additional consideration. Suspending the reader can block the syslog() call for applications across the host. In such scenarios, flow control can be disabled while downstream queues or dedicated buffers are increased.

<Extension syslog>
    Module          xm_syslog
</Extension>

<Input dev_log>
    Module          im_uds
    UDS             /dev/log
    Exec            parse_syslog();
    FlowControl     FALSE
</Input>

<Output siem>
    Module          om_elasticsearch
    URL             https://siem.example.com:9200/_bulk
    LogqueueSize    4194304
</Output>

<Route r1>
    Path            dev_log => siem
</Route>

Disabling flow control creates an explicit trade-off: the input will continue accepting data until the downstream queue fills. Once the queue reaches capacity, events may be discarded. Queue and buffer sizes should therefore be calculated based on the maximum outage the environment is expected to tolerate.

Buffering Is Not the Same as Delivery Confirmation

Buffering protects events that are already held by the log shipper, but it does not automatically guarantee that a remote receiver has successfully received every event.

TCP provides reliable packet delivery, but the application layer may still encounter situations where a connection closes before the sender knows exactly which events were processed by the destination.

For higher delivery assurance, NXLog Agent supports the NXLog Transport module pair, which provides application-level acknowledgment between NXLog agents. With persistent queues, unacknowledged batches can remain queued and be retransmitted.

This approach can provide at-least-once delivery. If the destination is sensitive to duplicate events, appropriate duplicate detection should also be implemented.

Managing these settings manually across hundreds or thousands of systems can quickly become impractical. NXLog Platform enables centralized configuration management, agent grouping, and monitoring, allowing buffering policies to be deployed consistently across the fleet.

How Large Should a Log Buffer Be?

Buffer capacity should be based on the amount of data your environment generates and the maximum outage period you need to survive.

A simple calculation is:

Required buffer size = events per second × average event size × outage duration

For example, an agent processing 2,000 events per second with an average event size of 800 bytes produces approximately 1.6 MB of data per second, or around 5.8 GB per hour.

To survive a four-hour SIEM outage, the environment would therefore need approximately 23 GB of buffer capacity, before accounting for additional safety margin.

When calculating capacity, consider the following:

• Measure actual event sizes. Windows security events can be substantially larger than individual firewall syslog messages.
• Set an early warning threshold. A WarnLimit around 80% of MaxSize provides time to react before the buffer is exhausted.
• Use dedicated storage where possible. A full root filesystem can turn a log collection outage into a broader host availability problem.
• Include additional headroom. Unexpected spikes in event volume can consume a buffer much faster than average traffic.

Test Your Buffer Before an Outage Does

A buffering strategy should be tested before it becomes necessary during a real incident.

NXLog Agent provides tools that can simulate a failed destination. The Blocker output module can act as an unavailable destination, while the Blocker processor module can block or unblock a route on demand or according to a schedule.

<Input app_logs>
    Module       im_file
    File         '/var/log/app/*.log'
</Input>

<Processor disk_buffer>
    Module       pm_buffer
    Type         Disk
    MaxSize      512000
    WarnLimit    409600
</Processor>

<Output blackhole>
    Module       om_blocker
</Output>

<Route outage_drill>
    Path         app_logs => disk_buffer => blackhole
</Route>

During a test, verify three key behaviors:

1. The buffer fills at approximately the rate predicted by your capacity calculations.
2. WarnLimit warnings appear in the NXLog Agent logs as the buffer approaches its threshold.
3. The complete backlog drains successfully and in the expected order after the destination becomes available again.

You can also use the Blocker processor module in front of the real destination to simulate an outage without changing the production output configuration.

A buffer that has never been tested is an assumption rather than a proven resilience control.

How Popular Log Shippers Handle Offline Buffering

Log shippers differ significantly in their default buffering behavior, persistent storage options, and response when buffers reach capacity.

Behavior and defaults can change between software releases. Always verify the configuration and documentation for the specific version deployed in your environment.

Several patterns are clear. Memory is commonly used as the default because it provides high performance, but it does not inherently protect queued data against crashes or power failures.

Full-buffer behavior is also important. Some technologies pause upstream sources, while others discard events when their available storage is exhausted. These differences can have a significant impact on security visibility during an extended outage.

Keep Collecting When Your SIEM Is Down

Log collection resilience is an important part of a modern security architecture. A destination outage should not automatically become a permanent loss of security telemetry.

NXLog Agent provides built-in queues and flow control, persistent queues and dedicated disk buffers for longer interruptions, and acknowledged transport between NXLog agents. NXLog Platform adds centralized configuration and fleet management, making it possible to apply consistent buffering policies across large environments.

As an official NXLog partner, Softprom can help organizations evaluate their current log collection architecture, identify potential data-loss scenarios, and design a resilient telemetry pipeline for SIEM, security, compliance, and observability use cases.

Ready to strengthen your log collection resilience? Contact Softprom to explore NXLog solutions and determine the right buffering strategy for your environment.

Frequently Asked Questions

Does a log shipper lose logs when the SIEM is down?

It can. Memory-only queues can eventually fill or disappear following an agent restart. Properly configured disk-based buffering allows events to accumulate locally and be forwarded when the SIEM becomes available again.

What is the difference between NXLog Agent log queues and the Buffer processor module?

Log queues are built into processor and output module instances and primarily handle short periods of backpressure. The Buffer processor module provides a dedicated memory or disk buffer that can be sized for longer outages.

Should I use memory or disk buffering?

Memory is appropriate for short interruptions where performance is the priority. Disk buffering is preferable when events must survive agent restarts or when longer outages are possible. Because persistent buffering increases disk I/O, it should be applied where the additional resilience justifies the overhead.

Does offline buffering guarantee delivery?

No. Buffering protects events held by the log shipper. Stronger delivery guarantees require application-level acknowledgment. NXLog Agent's NXLog Transport module pair can acknowledge compressed batches between agents and retransmit unacknowledged data.

How can I test offline buffering?

Use the NXLog Agent Blocker output module to simulate an unavailable destination, or use the Blocker processor module to temporarily block a route. Monitor buffer growth, warning thresholds, and complete backlog recovery after the destination becomes available.