Troubleshooting Fluent Bit

Start with the integration overview. Inspect each affected node and destination: process readiness and aggregate test counters do not establish complete log delivery.

Elasticsearch credentials and certificate changes

Check that logging-ek-es-elastic-user and logging-ek-es-http-certs-public exist in fluentbit and contain the expected password and CA. If Kyverno synchronization is used, verify the GeneratingPolicy API/controller and source Secret updates in logging. A reconciled HelmRelease alone does not prove successful Secret synchronization.

The password is consumed as an environment variable. If Fluent Bit cannot reconnect after credential or certificate updates, an authorized operator can restart the DaemonSet after verifying the replacement Secrets:

kubectl rollout restart daemonset/fluentbit-fluent-bit -n fluentbit

This command mutates the selected cluster. Allow the rollout to finish, inspect connection errors and confirm records arrive at the backend. Do not publish Secret contents in troubleshooting logs. See the maintainer procedure for the existing manual test workflow.

DaemonSet host paths are unavailable

The package defaults expect /var/log, /var/lib/docker/containers and /etc/machine-id. k3d nodes do not run kubelet as a systemd service. The umbrella CI values replace the entire input configuration and host volume/mount lists to collect only container logs, retaining filesystem buffering through the separate buffer mount.

For Bottlerocket or other node layouts, inspect the actual runtime log paths and journal availability before adopting that container-only pattern. Configure node-appropriate inputs and volumes through the upstream parent described in the overview. Replacing an input string or mount list is not additive; preserve every input/mount the deployment needs. A container-only override does not collect host journal logs.

If pods fail admission or mounts are denied, inspect node permissions, SELinux denials and platform admission policy. Do not assume that blanket privileged mode is required or disable policy cluster-wide.

Too many open files

If the tail input reports errno=24 or Too many open files, inspect both process file-descriptor limits and the affected node’s inotify limits:

cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches

Manage any required node-level changes through your approved host configuration process. Do not treat example test-environment limits as production sizing. Bottlerocket persists kernel settings through its API or user data rather than conventional /etc/sysctl.d files; consult its kernel settings reference.

The upstream tail input documentation describes watcher behavior. If testing a stat watcher instead of inotify, replace the complete input configuration, retain required journal collection and buffering, and measure memory and log-rotation behavior before using it operationally.

Records are collected but not delivered

Inspect the final output configuration, record tag matches, Secret references, TLS verification and writable buffer mount. Verify explicit NetworkPolicy egress and mesh ServiceEntries for external destinations. Watch retries, dropped records and node disk capacity during outages, then query the backend for representative records. See destinations, storage and mesh integration.