Skip to content

Troubleshooting📜

Fluent Bit did not restart after an Elasticsearch redeployment or secret update📜

If Fluent Bit cannot connect to Elasticsearch after an Elasticsearch redeployment or certificate update, restart the Fluent Bit DaemonSet:

kubectl rollout restart daemonset/fluentbit-fluent-bit -n fluentbit

DaemonSet host paths are unavailable📜

The package defaults retain upstream host volumes for /var/lib/docker/containers and /etc/machine-id. Big Bang’s exact-head CI jobs use k3d nodes, which do not run kubelet as a systemd service. The umbrella tests/test-values.yaml replaces the complete input, volume, and mount values so CI collects Kubernetes container logs without requiring the systemd input or /etc/machine-id:

fluentbit:
  values:
    upstream:
      config:
        inputs: |
          [INPUT]
              Name tail
              Path /var/log/containers/*.log
              Exclude_Path /var/log/containers/*fluent*.log
              Parser containerd
              Tag kube.*
              Mem_Buf_Limit 50MB
              Skip_Long_Lines On
              storage.type filesystem
      daemonSetVolumes:
        - name: varlog
          hostPath:
            path: /var/log
      daemonSetVolumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: true

Helm replaces config.inputs and both lists rather than merging their entries, so include every input, volume, and mount the deployment requires. This CI override does not collect or validate host journal logs.

The same pattern can be used on Bottlerocket or other nodes where the default host paths are unavailable, provided host journal collection is not required. Deployments that require host journal logs must supply node-appropriate inputs, volumes, and mounts instead.

Too many open files📜

In some environments you may see errors such as the one below:

fluent-bit [2022/07/05 16:20:10] [error] [build/plugins/in_tail/CMakeFiles/flb-plugin-in_tail.dir/compiler_depend.ts:304 errno=24] Too many open files

Check the node’s current inotify limits:

cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches

For example, Big Bang test environments use 1024 instances and 1048576 watches. On a conventional Linux node, update the limits temporarily with:

sudo sysctl fs.inotify.max_user_instances=1024
sudo sysctl fs.inotify.max_user_watches=1048576

Persist the settings through a reboot with a file under /etc/sysctl.d:

sudo tee /etc/sysctl.d/fluent-bit-inotify.conf <<EOF
fs.inotify.max_user_instances=1024
fs.inotify.max_user_watches=1048576
EOF
sudo sysctl --system

This filesystem-based approach does not apply to Bottlerocket. Configure persistent Bottlerocket values through its API or user data instead:

[settings.kernel.sysctl]
"fs.inotify.max_user_instances" = "1024"
"fs.inotify.max_user_watches" = "1048576"

See the Bottlerocket kernel settings and the Fluent Bit tail input documentation for additional details.

If increasing the limits does not resolve the issue, test the stat watcher by setting Inotify_Watcher to false:

upstream:
  config:
    inputs: |
      [INPUT]
          Name tail
          Path /var/log/containers/*.log
          Exclude_Path /var/log/containers/*fluent*.log
          Parser containerd
          Tag kube.*
          Mem_Buf_Limit 50MB
          Skip_Long_Lines On
          storage.type filesystem
          Inotify_Watcher false

upstream.config.inputs replaces the complete input configuration, so include every input the deployment still requires. There have been reports of possible memory issues with the stat watcher. If using this method, ensure that Fluent Bit memory usage is monitored.