Kubernetes DaemonSet Log Collection: A Practical Guide for SecOps

Kubernetes DaemonSet log collection is one of the most practical ways to collect container logs across an entire Kubernetes cluster. A DaemonSet runs one log collector pod on every node, allowing the collector to read container logs from the node filesystem and forward them to a SIEM or centralized log platform.

The main advantage is operational simplicity: applications do not need to be modified, and when Kubernetes adds a new node, the DaemonSet automatically schedules another collector.

For security teams, this architecture is especially important because container logs are often short-lived. If logs remain only on the node, a pod restart, node replacement, eviction, or log rotation can permanently remove valuable security telemetry.

This guide explains how Kubernetes logging works, how to deploy NXLog Agent as a DaemonSet, how to handle modern container runtimes such as containerd and CRI-O, and how to forward container and Kubernetes audit logs to platforms such as Microsoft Sentinel and Splunk.

What Is DaemonSet-Based Log Collection in Kubernetes?

A DaemonSet is a Kubernetes controller that ensures a copy of a pod runs on every node, or on a selected group of nodes.

When a new node joins the cluster, Kubernetes automatically schedules the DaemonSet pod there. When the node is removed, its DaemonSet pod is removed as well.

This makes DaemonSets a natural deployment model for node-level infrastructure agents such as log collectors, monitoring agents, security sensors, and other host-level services.

Kubernetes does not provide a centralized log storage system by default. Container logs are initially stored locally on the node, which means organizations need a separate cluster-level logging architecture if they want logs to remain available after workloads or nodes disappear.

Three common collection patterns are used in Kubernetes:

Pattern	How It Works	Advantages	Trade-offs
DaemonSet node agent	One collector pod per node reads container log files from the host filesystem.	Cluster-wide coverage, no application changes, and one collector per node regardless of pod count.	Requires host log mounts and appropriate node-level permissions.
Sidecar collector	A dedicated collector container runs alongside the application in the same pod.	Can collect application logs that are written exclusively to files inside the container.	Higher resource consumption and per-application configuration.
Application-level logging	The application sends its own logs directly to a central backend.	No separate collection infrastructure.	Logging logic becomes part of the application, with limited coverage for Kubernetes infrastructure and possible gaps when applications fail.

For most Kubernetes environments, a DaemonSet node agent is the preferred default. A sidecar is better suited to applications that write important logs directly to files rather than stdout or stderr.

How Kubernetes Writes Container Logs

A DaemonSet collector is fundamentally a node-level log reader. To configure it correctly, you need to understand where Kubernetes stores container logs and which format the container runtime uses.

The basic pipeline is:

1. The application writes to stdout or stderr.
2. The container runtime captures the output.
3. The runtime writes the records to files on the Kubernetes node.
4. Kubernetes exposes convenient log-file paths through /var/log/containers.
5. The DaemonSet collector reads those files and forwards the events to a central destination.

Container logs are stored under paths such as:

/var/log/pods/ /var/log/containers/

Files under /var/log/containers are typically symbolic links that expose pod, namespace, container, and container ID information through their filenames.

This allows a log collector to enrich events without necessarily querying the Kubernetes API.

CRI Logging vs. Legacy Docker JSON Logs

The container runtime is an important part of the architecture.

Kubernetes removed dockershim in version 1.24. Modern Kubernetes clusters typically use a Container Runtime Interface (CRI) runtime such as containerd or CRI-O.

These runtimes use the Kubernetes CRI logging format. A typical record looks like this:

2026-08-14T09:26:31.123456789Z stderr F ERROR: connection to db-svc:5432 refused

The record contains:

• a timestamp;
• the stream, such as stdout or stderr;
• a completion tag, such as F or P;
• the actual log message.

Legacy Docker json-file logging uses a different structure:

{"log":"ERROR: connection to db-svc:5432 refused\n","stream":"stderr","time":"2026-08-14T09:26:31.123456789Z"}

This difference is important when deploying a collector. Configurations written specifically for Docker JSON logs may fail to parse logs correctly on modern containerd or CRI-O clusters.

What Do the CRI P and F Tags Mean?

The third field in a CRI log record indicates whether the runtime considers the record complete.

• F indicates a complete log entry.
• P indicates that the record is a partial fragment and additional data follows.

Long application messages may therefore arrive as multiple fragments. A production logging pipeline should account for this behavior when applications generate very large or multiline messages.

Container Log Rotation

Kubernetes also rotates container logs. The kubelet controls rotation through settings such as:

• containerLogMaxSize
• containerLogMaxFiles

Typical defaults limit how much historical container log data remains on a node. Once old files are rotated and deleted, a collector cannot recover them.

This is one of the strongest reasons to forward security-relevant logs off the node as soon as possible.

Why Node-Level Log Collection Matters for SecOps

Kubernetes workloads are highly dynamic. Pods can be restarted, rescheduled, evicted, or removed as part of normal cluster operations.

For SecOps teams, this means that node-local logs should be treated as temporary telemetry rather than a long-term evidence store.

A security event may become unavailable when:

• a compromised pod is terminated;
• a node is removed by the cluster autoscaler;
• a workload is evicted;
• container logs are rotated;
• a node fails;
• a cluster is rebuilt or migrated.

Centralized collection moves the logs away from this volatile layer and makes them available for detection, investigation, threat hunting, compliance, and forensic analysis.

Security Principles for a Kubernetes Log Collector

The collector itself should follow a least-privilege architecture.

• Mount host log directories as read-only. The collector should read logs without being able to modify the host filesystem.
• Minimize Kubernetes RBAC. If metadata can be extracted from filenames and environment variables, a cluster-wide API permission may not be necessary.
• Encrypt traffic to the SIEM. Use TLS whenever the destination supports it.
• Use persistent buffering. Temporary SIEM or network outages should not automatically become permanent log gaps.
• Protect collector credentials. Store SIEM tokens, certificates, and other secrets using appropriate Kubernetes mechanisms.

NXLog Agent may require root privileges inside the collector container to read log files belonging to all workloads. This should be combined with read-only host mounts and the minimum Kubernetes permissions required by the deployment.

Deploy NXLog Agent as a Kubernetes DaemonSet

NXLog Agent can be deployed as a DaemonSet so that every Kubernetes node receives its own log collector.

The following architecture is designed for modern CRI-based Kubernetes environments and can collect container logs without requiring application changes.

Step 1: Build the NXLog Agent Image

Before deploying the DaemonSet, create an NXLog Agent container image using the appropriate NXLog installation procedure.

When the agent must read logs generated by all workloads, configure the container to run with the permissions required to access the host log files. Push the resulting image to a container registry accessible by the Kubernetes cluster.

For a production deployment, avoid relying on locally built images. Use a versioned image in a registry and define an explicit image tag rather than using latest.

Step 2: ServiceAccount and RBAC

A Kubernetes ServiceAccount and RBAC permissions are only required if the collector needs to query the Kubernetes API, for example to enrich events with additional pod metadata.

If the collector extracts pod and namespace information directly from log filenames, this additional API access can be avoided.

When API enrichment is required, a minimal example is:

apiVersion: v1 kind: ServiceAccount metadata: name: nxlog namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: nxlog rules: - apiGroups: [""] resources: ["pods", "namespaces"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: nxlog roleRef: kind: ClusterRole name: nxlog apiGroup: rbac.authorization.k8s.io subjects: - kind: ServiceAccount name: nxlog namespace: kube-system

If API access is not required, omit the ServiceAccount and RBAC objects and run the collector with the default service account.

Step 3: Deploy the DaemonSet

A simplified DaemonSet manifest can look like this:

apiVersion: apps/v1 kind: DaemonSet metadata: name: nxlog namespace: kube-system labels: k8s-app: nxlog-logging spec: selector: matchLabels: name: nxlog template: metadata: labels: name: nxlog spec: serviceAccountName: nxlog tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: nxlog image: registry.example.com/nxlog:1.0 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: - name: varlog mountPath: /var/log readOnly: true - name: nxlog-cache mountPath: /var/lib/nxlog volumes: - name: varlog hostPath: path: /var/log - name: nxlog-cache hostPath: path: /var/lib/nxlog type: DirectoryOrCreate terminationGracePeriodSeconds: 30

Several details are important in this manifest.

Control-plane nodes

Control-plane nodes may have a NoSchedule taint. If the collector should also run on these nodes, the DaemonSet needs a matching toleration.

Read-only log mounts

The host /var/log directory should be mounted as readOnly: true. This gives the collector access to container and system logs without granting it write access to the host log filesystem.

Persistent agent state

The writable /var/lib/nxlog mount provides persistent storage for the agent's cache and read positions. Without persistent state, a restarted collector may re-read previously processed log files.

Step 4: Configure NXLog Agent for Kubernetes Container Logs

The most important configuration detail is supporting both the modern CRI format and legacy Docker JSON logs if the cluster contains a mixed runtime environment.

CacheDir /var/lib/nxlog envvar NODE_NAME <Extension json> Module xm_json </Extension> <Input k8s_containers> Module im_file File '/var/log/containers/*.log' <Exec> # CRI format: # timestamp stream P|F message if $raw_event =~ /^(\S+) (stdout|stderr) ([FP]) (.*)$/ { $EventTime = parsedate($1); $stream = $2; $Message = $4; } else { # Legacy Docker JSON format parse_json(); $EventTime = parsedate($time); $Message = $log; delete($log); delete($time); } $log_type = "k8s_container"; $log_file = file_name(); $k8s_node = '%NODE_NAME%'; # Extract pod, namespace and container information # from the Kubernetes container-log filename. if $log_file =~ /\/.*\/(.+)_(.+)_(.+)-(.+).log$/ { $k8s_pod = $1; $k8s_namespace = $2; $k8s_container = $3; $k8s_container_id = $4; } </Exec> </Input>

This configuration reads files under /var/log/containers, identifies the CRI format used by modern container runtimes, and retains a JSON parsing branch for legacy Docker-based environments.

It also extracts useful Kubernetes context from the filename:

• Pod name
• Namespace
• Container name
• Container ID
• Node name

This enrichment can be performed without granting the collector broad access to the Kubernetes API.

Forwarding Kubernetes Logs to Microsoft Sentinel

NXLog Agent can forward collected Kubernetes events to Microsoft Sentinel using the Azure Monitor Logs Ingestion API.

A simplified output configuration is:

<Output sentinel> Module om_azuremonitor API LogsIngestion ClientId <entra-app-client-id> ClientSecret <entra-app-secret> TenantId <entra-tenant-id> URL https://<your-dce>.ingest.monitor.azure.com DcrImmutableId dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx StreamName Custom-K8sContainers-Stream Exec to_json(); </Output>

The event structure should be aligned with the columns defined in the Microsoft Sentinel Data Collection Rule and stream.

In production, credentials should be stored securely rather than embedded directly in the image or configuration repository.

Forwarding Kubernetes Logs to Splunk

For Splunk environments, NXLog Agent can send Kubernetes events to the Splunk HTTP Event Collector (HEC).

<Output splunk_hec> Module om_http URL https://splunk.example.com:8088/services/collector/event AddHeader Authorization: Splunk <hec-token> HTTPSCAFile %CERTDIR%/splunk-ca.pem <Exec> $raw_event = '{"event":' + to_json() + '}'; </Exec> </Output>

For production deployments, include the appropriate Splunk metadata such as time, host, and sourcetype so that Kubernetes events can be efficiently searched and correlated.

Step 5: Deploy and Verify the DaemonSet

Apply the manifest:

$ kubectl apply -f nxlog-daemonset.yml

Then verify the DaemonSet:

$ kubectl -n kube-system get daemonset nxlog

The DESIRED count should correspond to the nodes that the DaemonSet is intended to cover.

Also check the individual collector pods:

$ kubectl -n kube-system get pods -l name=nxlog -o wide

On the SIEM side, a correctly enriched event should contain information similar to:

{ "EventTime": "2026-10-05T09:26:31.123456+00:00", "stream": "stderr", "Message": "ERROR: connection to db-svc:5432 refused", "log_type": "k8s_container", "k8s_node": "node003", "k8s_pod": "payments-api-7f6d9c5b8-x2kkq", "k8s_namespace": "prod", "k8s_container": "payments-api", "k8s_container_id": "bdc9da43d94800e213477bbddbc9f96fa7a32b680509580cfce97fa4186d63e5" }

For SecOps, this context makes the event significantly more useful because analysts can immediately identify the workload, namespace, node, and container associated with an event.

Collecting Kubernetes Audit Logs with NXLog Agent

Container logs show what workloads are doing. Kubernetes audit logs show what users and services are doing to the Kubernetes API.

Audit records can reveal:

• who accessed the Kubernetes API;
• which resources were accessed;
• which actions were performed;
• whether requests were allowed or denied;
• which service account initiated an operation.

This makes Kubernetes audit logs particularly valuable for security monitoring and incident response.

After Kubernetes auditing is enabled, the API server can write audit events to a file such as:

/var/log/kubernetes/audit.log

NXLog Agent can collect that file through an additional input:

<Input k8s_audit> Module im_file File '/var/log/kubernetes/audit.log' BufferSize 150000 <Exec> parse_json(); $log_type = "k8s_audit"; $k8s_node = '%NODE_NAME%'; </Exec> </Input>

Audit events can be significantly larger than ordinary application log records. Increase the input and output buffer sizes as necessary to prevent large audit records from being truncated.

Audit log rotation should also be considered. Centralized collection prevents the API server's local rotation policy from becoming the effective security log-retention policy.

Managing Kubernetes Log Collectors with NXLog Platform

A DaemonSet solves the deployment problem: it ensures that a collector runs on the required nodes.

It does not, however, solve all operational challenges.

Large Kubernetes environments also need to answer questions such as:

• Which agents are currently healthy?
• Which nodes have stopped sending logs?
• How can a parsing rule be updated across hundreds of nodes?
• How can new nodes automatically receive the correct configuration?
• How can different SIEM destinations be managed centrally?

NXLog Platform provides centralized management for NXLog agents, allowing organizations to manage configurations and monitor agent status from a central interface.

When Kubernetes autoscaling adds new nodes, centralized agent management can help ensure that the newly deployed collector receives the correct configuration without requiring manual configuration on each node.

This is particularly valuable when the logging architecture evolves—for example, when a new CRI parsing rule, SIEM destination, filtering policy, or security control needs to be deployed across the entire cluster.

Common Kubernetes Logging Pitfalls

1. parse_json() Fails on Container Logs

If the cluster uses containerd or CRI-O, container logs normally use the CRI text format rather than Docker's JSON format.

A configuration that blindly calls parse_json() on every record can therefore fail to parse modern Kubernetes logs correctly.

The solution is to identify the CRI format first and retain JSON parsing only as a compatibility branch for legacy Docker environments.

2. Multiline Stack Traces Become Separate Events

Kubernetes container logging is line-oriented. A Java stack trace, Python traceback, or similar multiline application event can therefore arrive as multiple records.

If the SIEM needs the entire stack trace as a single event, use an appropriate multiline parsing strategy in the collector.

Long CRI records can also be split into partial records marked with P until the final F fragment arrives. The collection pipeline should account for this when applications produce very long messages.

3. The DaemonSet Does Not Run on Control-Plane Nodes

If the number of DaemonSet pods is lower than expected, inspect the taints on the missing nodes:

$ kubectl describe node <node-name>

Control-plane nodes commonly use a NoSchedule taint, so the DaemonSet must include the corresponding toleration when those nodes need to be monitored.

4. Kubernetes Audit Events Are Truncated

Audit events can be considerably larger than ordinary application messages.

If the collector's input or output buffer is too small, large records can be truncated. Increase BufferSize on the relevant modules and test with realistic audit events before deploying the configuration across the cluster.

5. The Agent Re-Sends Old Events After a Restart

A file-based collector needs to maintain read positions.

If the agent's cache is stored only on the container's ephemeral filesystem, restarting the DaemonSet pod can cause the collector to lose its state and re-read previously processed files.

Use persistent storage for the NXLog Agent cache when your operational requirements demand reliable resume behavior.

Kubernetes Logging Security Checklist

Control	Recommendation
Host log mounts	Mount host log directories as read-only.
RBAC	Grant Kubernetes API permissions only when API-based enrichment is required.
Transport	Use TLS when sending logs to external SIEM or centralized collectors.
Buffering	Use persistent queues or buffering to protect against temporary destination outages.
Log rotation	Ship important logs before Kubernetes rotates and deletes local files.
Multiline logs	Implement multiline processing for applications that produce stack traces or similar events.
Audit logs	Collect Kubernetes API audit events centrally and monitor access to sensitive resources.
Credentials	Keep SIEM tokens, certificates, and secrets outside container images.
Agent state	Use persistent storage for collector state where duplicate or missed events must be minimized.

Why NXLog Agent Fits Kubernetes SecOps

Kubernetes logging requires more than simply reading files. A production collection layer must understand container runtime formats, preserve useful metadata, handle log rotation, support secure transport, survive temporary outages, and integrate with the organization's SIEM.

NXLog Agent addresses these requirements through:

• native collection from Kubernetes node log files;
• support for modern CRI-based logging environments;
• event parsing and enrichment;
• Syslog, HTTP and other output options;
• TLS-secured log delivery;
• persistent queues and buffering;
• collection from Linux, Windows and other enterprise sources;
• centralized management through NXLog Platform.

This makes NXLog particularly useful when Kubernetes is only one part of a broader security monitoring environment and organizations need a consistent log collection architecture across multiple operating systems and platforms.

Conclusion: Ship Kubernetes Logs Before the Cluster Deletes Them

A Kubernetes cluster can generate enormous volumes of security and operational telemetry, but much of that data is initially stored only on individual nodes.

A DaemonSet-based log collector provides a scalable way to move this data off the node without modifying application pods. As the cluster grows, the logging layer grows with it.

The key is to design the pipeline around the way modern Kubernetes actually writes logs: CRI-based formats, node-level log rotation, short-lived workloads, control-plane taints, multiline events, and temporary destination failures.

NXLog Agent provides a flexible collection layer for Kubernetes container and audit logs, while NXLog Platform can centralize configuration and monitoring across the agent fleet.

Softprom is an official NXLog partner and helps organizations design and implement centralized log collection architectures, integrate Kubernetes telemetry with SIEM platforms, and build reliable security monitoring pipelines.

Contact Softprom to discuss your Kubernetes logging requirements and evaluate the right NXLog architecture for your environment.