Pre-SIEM Log Filtering and Log Reduction: How to Cut Ingestion Costs Without Losing Detections

Security information and event management (SIEM) platforms provide centralized visibility into security events, but ingesting large volumes of telemetry can significantly increase operational costs. Many SIEM solutions calculate pricing based on the amount of data ingested, making unnecessary or repetitive log data an avoidable expense.

Pre-SIEM log filtering and log reduction address this challenge by filtering, deduplicating, trimming, or routing events before they cross the SIEM ingestion boundary. Processing data at the collection agent or relay can reduce network traffic, infrastructure requirements, and SIEM ingestion volume.

NXLog Agent enables organizations to apply these techniques at the source, helping security teams control log volume while preserving the events required for threat detection, investigation, and compliance.

This guide explains what to retain, what to reduce, and how to implement practical log optimization strategies with NXLog Agent.

What Is Pre-SIEM Log Filtering and Log Reduction?

Pre-SIEM log reduction encompasses techniques that reduce the amount of telemetry sent to a SIEM. Four approaches are particularly useful:

Where reduction occurs matters. Filtering at the endpoint with NXLog Agent can reduce bandwidth consumption, relay processing, and SIEM ingestion simultaneously. Filtering on a centralized relay simplifies policy management but still requires transmitting the original data from the source to the relay.

Filtering at the SIEM ingestion boundary may reduce billable ingestion in some services, but it does not eliminate the cost of transporting data from endpoints to that boundary. The rules may also be tied to a particular SIEM platform.

For Microsoft Sentinel's data lake tier, review Microsoft's billing documentation to understand when data processing charges apply. Moving data into a cheaper storage tier after ingestion is not the same as reducing telemetry before it reaches the platform.

Why Filter Logs Before They Reach the SIEM?

1. Control Ingestion Costs

Many SIEM pricing models depend on data volume. Microsoft Sentinel offers pay-as-you-go and commitment-tier pricing for eligible analytics data, while Splunk offers several pricing models, including ingestion-based and workload-based options.

The financial benefit of filtering depends on your contract and billing model. Before estimating savings, confirm which meter applies to your deployment and whether any minimum commitments or other charges affect the calculation.

For reference, consult the official Microsoft Sentinel billing documentation and Splunk ingestion pricing.

2. Improve Detection Efficiency

High-volume, low-value events can increase search latency, complicate investigations, and make it harder for analysts to identify meaningful security signals. Thoughtful reduction helps prioritize useful telemetry without eliminating the context required to investigate suspicious activity.

3. Support Compliance and Audit Requirements

Log reduction must be governed by an approved logging policy. Security teams should document which events are collected, which are retained, which are routed elsewhere, and which are discarded.

The joint Best Practices for Event Logging and Threat Detection guidance emphasizes enterprise-approved logging policies and centralized log collection. NIST's SP 800-92, Guide to Computer Security Log Management, provides additional background on log generation, transmission, storage, access, and disposal.

A filtering rule is part of your log management and disposal policy. Each rule should have a documented purpose, an owner, and a review process.

What Should You Keep, Reduce, or Route?

Start with detection coverage, investigation workflows, and retention requirements—not just a chart showing which sources generate the most data.

Use Windows Event Monitoring Guidance

Microsoft's Appendix L: Events to Monitor provides a useful baseline for identifying potentially important Windows Security events. Use it alongside the specific event IDs consumed by your SIEM analytics rules and threat-hunting queries.

Some events produce substantial volume but remain essential for particular detections. For example, Windows Security event 4662 can be relevant to monitoring directory-service access and detecting DCSync activity. Rather than indiscriminately discarding this event, review audit policy and SACL configuration on domain controllers to control unnecessary volume while preserving required visibility.

Map Events to Detection Requirements

Map every important log source to the detection rules, investigations, and retention policies that depend on it. The MITRE ATT&CK data sources catalogue can help identify telemetry relevant to specific adversary techniques.

Potential reduction candidates in some environments include:

• High-volume Windows Filtering Platform events, such as 5156 and 5158.
• Handle-close events, such as event 4658, where they are not required by detection content.
• Machine-account network logons, after reviewing lateral-movement detection requirements.
• Syslog messages at informational or debug severity.
• Firewall allow-rule events that are not required for investigations or policy monitoring.
• Load-balancer health checks and other repetitive operational events.

These are candidates for assessment, not universal recommendations. Their value depends on your threat model, audit requirements, and detection logic.

Rule of thumb: Route data you may need for forensics. Drop events only when you have confirmed that no detection, investigation, operational, or compliance requirement depends on them.

Five NXLog Agent Techniques for Log Reduction

NXLog Agent can apply filtering and transformation rules on endpoints or relays before events are sent to the SIEM. The following examples illustrate common approaches. Test and adapt each configuration to your environment and the version of NXLog Agent you deploy.

1. Filter Events by Syslog Severity or Field

A severity threshold is one of the simplest ways to reduce syslog volume. In the following example, NXLog Agent parses incoming syslog messages and drops events with a severity value greater than 4. Syslog severity ranges from 0 (emergency) to 7 (debug), so the rule retains warning-level events and more severe messages.

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input udp_listen>
    Module        im_udp
    ListenAddr    0.0.0.0:514
    <Exec>
        parse_syslog();
        if ($SyslogSeverityValue > 4) {
            drop();
        }
    </Exec>
</Input>

Review the meaning of each severity level for your applications before implementing a blanket threshold. Security-relevant activity can be logged at informational severity, and some sources do not consistently classify events.

For additional guidance, see the NXLog documentation on filtering events.

2. Filter Windows Events at the Source

Windows Event Log filtering can be performed in two stages. First, use an XPath query to limit the events collected by NXLog Agent. Then apply field-based conditions in an Exec block for additional filtering logic.

The example below illustrates filtering by event level, suppressing selected high-volume event IDs, and dropping specific machine-account network logons and process-creation events.

<Input windows_security>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Security">*[System[(Level<=4)]]</Select>
                <Suppress Path="Security">
                    *[System[(EventID=4658 or EventID=5156 or EventID=5158)]]
                </Suppress>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        if $EventID == 4624 and $LogonType == 3 and $TargetUserName =~ /\$$/ {
            drop();
        }
        if $EventID == 4688 and $NewProcessName =~ /^C:\\Windows\\System32\\conhost\.exe$/i {
            drop();
        }
    </Exec>
</Input>

XPath queries have length and clause-count limitations. For complex conditions, use a second filtering stage instead of trying to express every condition in the subscription query.

NXLog Agent's Windows Event Log input module exposes structured fields such as EventID, LogonType, and TargetUserName. Comparing these fields is generally more reliable than searching the entire raw message.

Important: Machine-account logons can contribute to lateral-movement detections. Review your analytics rules before suppressing them. Do not indiscriminately exclude event 4662 from domain controllers, where directory-service access details can be essential to DCSync detection.

See the NXLog Windows Event Log filtering guide for additional details.

3. Deduplicate Repeated Events

Repeated messages can overwhelm a SIEM without providing much additional context. Deduplication reduces repetitive records while preserving a summary of how many times an event occurred.

The following example uses the NXLog Agent De-Duplicator processor to compare consecutive events based on selected fields.

<Extension syslog>
    Module         xm_syslog
</Extension>

<Input udp_listen>
    Module         im_udp
    ListenAddr     0.0.0.0:514
    Exec           parse_syslog();
</Input>

<Processor norepeat>
    Module         pm_norepeat
    CheckFields    Hostname, Message, SourceName
</Processor>

<Output agent_relay>
    Module         om_batchcompress
    Host           relay.example.com:2514
</Output>

<Route r>
    Path           udp_listen => norepeat => agent_relay
</Route>

The selected fields determine which events are treated as duplicates. Avoid comparing fields such as EventTime when they change for every occurrence.

The De-Duplicator module is being phased out according to the source documentation. For new deployments, review the current NXLog Agent reference manual and plan for an implementation using supported language variables or other recommended functionality.

Deduplication is not appropriate for every source. Brute-force detection, for example, may depend on the number and timing of repeated authentication failures. Preserve raw events where frequency and sequence are security signals.

Refer to the NXLog De-Duplicator documentation and the current NXLog language variables reference.

4. Trim Unnecessary Fields and Repetitive Text

Some events need to be retained, but they do not need every field or the full descriptive text included in the original message. Trimming reduces event size while preserving the fields required for detection and investigation.

The following example uses the Rewrite extension module to retain a defined set of fields. This allow-list is deliberately minimal and must be expanded to include the fields your detection content requires.

<Extension allowlist>
    Module    xm_rewrite
    Keep      AccountName, Channel, EventID, EventReceivedTime, EventTime, \
              Hostname, Severity, SeverityValue, SourceName
</Extension>

<Input eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path='Security'>*[System/Level<=4]</Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec      allowlist->process();
</Input>

This example removes event-specific fields such as account name details, logon type, and source address unless they are explicitly added to the allow-list. In a real deployment, preserve fields such as TargetUserName, LogonType, and IpAddress wherever detection rules or investigations require them.

You can also remove repetitive descriptive text that adds little analytical value. For example, the following rule trims boilerplate from a specific Windows Security event and rebuilds the raw output:

<Exec>
    if ($Channel == 'Security') and ($EventID == 4769) {
        $Message =~ s/\s*This event is generated every time access is.*$//s;
    }
    $raw_event = $EventTime + ' ' + $Message;
</Exec>

Rebuilding $raw_event is important when using stream-oriented output modules that send the raw event rather than the parsed fields. Removing fields from the internal event alone does not necessarily reduce the size of the transmitted record.

Actual savings depend on the event type and output format. NXLog's Windows Event Log guidance documents examples where shortening the message or removing redundant fields significantly reduced event size. The results are event-specific and should not be interpreted as a guaranteed fleet-wide reduction rate.

Risk to consider: Removing fields or shortening messages can break SIEM parsers, detection rules, and dashboards. A modified event is also no longer an exact copy of the original record, which may matter for audit and forensic requirements. Validate the transformed output and preserve a complete archive where necessary.

Learn more in the NXLog guides to trimming events and trimming Windows events.

5. Route High-Value Events to the SIEM and Archive the Rest

Routing is often the safest reduction strategy because it allows organizations to reduce SIEM ingestion without discarding the entire event stream. Detection-critical events go to the SIEM, while the full stream is retained in a lower-cost archive.

The following example routes Windows Security events to a TLS-protected SIEM destination and Amazon S3. The event ID list is illustrative and must be adapted to the detection requirements of your environment.

define CERTDIR %CONFDIR%/cert

<Extension json>
    Module    xm_json
</Extension>

<Input windows_security>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Security">*</Select>
            </Query>
        </QueryList>
    </QueryXML>
</Input>

<Output siem>
    Module    om_ssl
    Host      siem.example.com:6514
    CAFile    %CERTDIR%/ca.pem
    <Exec>
        if $EventID NOT IN (4624, 4625, 4648, 4672, 4688, 4697, 4698,
                            4720, 4728, 4732, 4756, 4768, 4769, 4776) {
            drop();
        }
        to_json();
    </Exec>
</Output>

<Output archive>
    Module    om_amazons3
    Region    eu-west-1
    Bucket    security-archive
    Server    SRV01
    Exec      to_json();
</Output>

<Route split>
    Path      windows_security => siem, archive
</Route>

The drop() condition in the SIEM output affects only that destination. The archive output continues to receive the full event stream, allowing analysts to retrieve events that were not indexed by the SIEM.

Use an approved event list derived from Microsoft's Windows event monitoring guidance and the rules your SIEM actually runs.

Archiving does not remove the need for appropriate security controls. Protect the archive with access restrictions, encryption, retention policies, and monitoring. Also validate delivery guarantees, buffering, and failure handling for both destinations.

For more information, see the NXLog guide to routing events to different destinations and the Amazon S3 output module documentation.

Manage Filtering Rules Across Thousands of Agents with NXLog Platform

Maintaining individual nxlog.conf files becomes difficult as an environment grows. NXLog Platform provides centralized agent management, enabling administrators to create configurations, assign them to selected agents, and standardize collection and routing policies across distributed infrastructure.

Key capabilities include:

• Centralized configuration management: Create and assign filtering rules without manually editing each endpoint.
• Configuration templates: Reuse common patterns for Windows Event Log, syslog, and other sources.
• Configuration version history: Track changes, identify who made them, and restore an earlier version when necessary.
• Data flow visualization: Review which agents send events to which destinations and verify that the intended archive and SIEM routes are configured.

NXLog has published a customer case study describing how QNB Finansbank used NXLog to filter up to 80% of events on endpoints and forward security-relevant data to security systems. This is a customer-reported result for its particular environment, not a guaranteed or independently controlled benchmark.

Explore NXLog Platform and its documentation on creating configurations, assigning configurations, and configuration version history.

How to Validate Log Reduction Without Breaking Detection

A filtering rule determines which events your detection rules can see. An incorrect suppression can weaken security monitoring just as much as a faulty detection rule. Treat filtering changes as part of the detection engineering lifecycle.

1. Measure event counts before and after. Use NXLog Agent metrics to compare volumes with the rule enabled and disabled. Track dropped-event counters and monitor event counts by source and route.
2. Replay known security scenarios. Trigger controlled tests for high-priority detections and confirm that the expected rules still fire after filtering is applied.
3. Version and document every change. Record the purpose of each suppression, the owner, the affected sources, and the detections reviewed. Use NXLog Platform configuration history to compare or restore versions.
4. Maintain an exception list. Document events that must not be filtered, including events needed for forensic investigations, compliance, and critical detections.
5. Review rules when sources change. New audit policies, operating system upgrades, application releases, and changes to SIEM detections can alter which events are necessary.

NXLog Agent's internal metrics can help monitor collection and processing behavior. Consult the Internal Metrics input module documentation and the applicable remote management documentation for your agent version.

Common Pre-SIEM Filtering Mistakes

• Filtering the raw message unnecessarily: Use parsed fields where available. Field-based conditions are generally easier to maintain and less fragile than string matching against the entire raw event.
• Dropping events instead of routing them: Once an event is discarded, it cannot be recovered during an incident unless another copy exists.
• Filtering only at the SIEM boundary: This may reduce ingestion charges but does not eliminate the cost of transporting data from the endpoint to the ingestion boundary.
• Removing fields without checking dependencies: Parsers, detection rules, dashboards, and reports may rely on fields that appear redundant.
• Leaving filters without an owner: Unreviewed rules can silently become outdated as security requirements and data sources evolve.
• Ignoring compliance requirements: Reduced ingestion must remain consistent with approved logging, retention, and evidence-preservation policies.

Conclusion: Reduce SIEM Costs While Preserving Security Visibility

Pre-SIEM log reduction helps organizations control SIEM costs by reducing unnecessary data before it reaches the ingestion boundary. Filtering, deduplication, trimming, and routing each have a role, but every technique must be evaluated against detection coverage, incident response, and compliance requirements.

For many environments, the safest starting point is to route events rather than discard them. Measure the volume reduction, test high-priority detections, and document every filtering rule before expanding the policy to more systems.

NXLog Agent enables organizations to apply log collection and reduction policies at the source, while NXLog Platform helps manage configurations and data flows centrally across large agent deployments.

Softprom is an official NXLog distributor. We help organizations evaluate NXLog solutions, optimize log collection architectures, and integrate security telemetry with SIEM platforms. Contact Softprom to discuss your log reduction strategy and identify opportunities to lower ingestion costs without compromising detection coverage.