Skip to content

Troubleshooting📜

Fluentbit did not restart after Elasticsearch re-deployment/secret update.📜

If you are having issues with Fluentbit connectivity to Elastic (caused by an Elastic redeployment/cert update) you should be able to resolve them by restarting Fluentbit. For example, run the below to cycle all pods in the Fluentbit daemonset: 

kubectl rollout restart daemonset/logging-fluent-bit -n logging

Too many open files📜

In some environments you may see errors such as the one below:

fluent-bit [2022/07/05 16:20:10] [error] [build/plugins/in_tail/CMakeFiles/flb-plugin-in_tail.dir/compiler_depend.ts:304 errno=24] Too many open files

Currently the best way to resolve this issue is to update each of your nodes’ number of inotify resources and file watches:

/proc/sys/fs/inotify/max_user_watches
/proc/sys/fs/inotify/max_user_instances

For example, BigBang testing environments have the following values set:

cat /proc/sys/fs/inotify/max_user_instances
1024
cat /proc/sys/fs/inotify/max_user_watches
1048576

You can update the limits temporarily, with the following commands (setting the values to 1024 and 1048576 respectively in this example):

sudo sysctl fs.inotify.max_user_instances=1024
sudo sysctl fs.inotify.max_user_watches=1048576
sudo sysctl -p

In order to make the changes permanent, i.e. to persist through a node reboot:

echo "fs.inotify.max_user_instances=1024" > /etc/sysctl.d/fs-inotify-max_user_instances.conf
sysctl -w fs.inotify.max_user_instances=1024
echo "fs.inotify.max_user_watches=1048576" > /etc/sysctl.d/fs-inotify-max_user_watches.conf
sysctl -w fs.inotify.max_user_watches=1048576
sysctl -p

Checkout https://github.com/fluent/fluent-bit/issues/1777#issuecomment-562178857 for more details

If updating the above does not help to resolve the issue you can test switching to stat watch for the input config. To do this set Inotify_Watcher to false, which can be done via the below values:

config:
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        # -- Excluding fluentbit logs from sending to ECK, along with gatekeeper-audit logs which are shipped by clusterAuditor.
        Exclude_Path /var/log/containers/*fluent*.log
        Parser containerd
        Tag kube.*
        Mem_Buf_Limit 50MB
        Skip_Long_Lines On
        storage.type filesystem
        Inotify_Watcher false

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
        storage.type filesystem

Additional details on this configuration value can be reviewed on the tail plugin configuration page. Note that there has been report of possible memory issues with stat watcher. If using this method make sure you have an alert/adequate visibility into Fluentbit memory usage.