Change the number of generated Elasticsearch indexes

When Logstash_Format is enabled, Fluent Bit uses Logstash_DateFormat to build the date portion of each Elasticsearch index name. Adjust this setting to change the index rollover interval. See strftime for the supported time formatting syntax.

For example, %Y.%m.%d-%H generates one index per hour, while %Y.%m.%d-%P generates separate AM and PM indexes. Use %H, rather than the space-padded %k, to avoid spaces in index names.

See the Fluent Bit Elasticsearch output documentation for additional settings.

Note: These package values must be nested under upstream: because this chart wraps the upstream Fluent Bit chart. When configuring the package through Big Bang, place them under fluentbit.values.upstream.

Example

upstream:
  config:
    outputs: |
      [OUTPUT]
          Name es
          Match kube.*
          Host logging-ek-es-http.logging
          Port 9200
          HTTP_User elastic
          HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
          Logstash_Format On
          # Generate a new index for each hour of the day.
          Logstash_DateFormat %Y.%m.%d-%H
          Suppress_Type_Name On
          Retry_Limit False
          Replace_Dots On
          tls On
          tls.verify On
          tls.ca_file /etc/elasticsearch/certs/ca.crt
          storage.total_limit_size 10G

The example assumes Big Bang is also deploying Elasticsearch/Kibana, which supplies the password environment variable and certificate mount. Because upstream.config.outputs replaces the complete output configuration, include any Loki, S3, Fluentd, or additional Elasticsearch outputs that the deployment still requires.