Fluent Bit

Overview

This package wraps the upstream Fluent Bit chart with Iron Bank images, node-log collection and buffering defaults, output credential resources, and Big Bang networking integration. Fluent Bit is a collector and forwarder, not a log storage or query service. Elasticsearch and Loki own retention and queries; Alloy is a separately configured collector. Avoid enabling overlapping collection pipelines unless duplicate ingestion is intentional.

Use the upstream product documentation for Fluent Bit concepts and plugins. This guide describes the package’s Big Bang integration.

Configuration entry points

The package currently wraps fluent-bit chart 0.58.3, not fluent-bit-collector. The upstream parent passes values to the bundled chart. Use its versioned upstream values for the complete configuration surface and package values for Big Bang overrides. The bundled chart uses classic configuration text; do not apply a different chart’s structured pipeline examples.

For umbrella installations, use packages.fluentbit with packageConfiguration.version: v1. The legacy entry point remains fluentbit; canonical overrides take precedence when v1 is selected. Package-owned integration settings belong under packages.fluentbit.values, while upstream settings belong beneath its upstream parent. For a direct package Helm installation, omit the umbrella nesting.

# Big Bang umbrella values; selects the collector, not a storage backend.
packageConfiguration:
  version: v1
packages:
  fluentbit:
    enabled: true
    values:
      additionalOutputs:
        disableDefault: false

The umbrella creates the fluentbit namespace, pull credentials, Flux source and HelmRelease, using release name fluentbit. It supplies common values, generated defaults, then user overlays. Configuration strings and volume/mount lists replace previous values rather than appending entries: preserve every stanza and mount still required when overriding them.

Big Bang touch points

Logging and destinations

Package defaults tail /var/log/containers/*.log with the containerd parser and kube.* tags. They also read the kubelet.service systemd journal with host.* tags. The active exclusion is /var/log/containers/*fluent*.log; it does not exclude Gatekeeper audit logs. Removing that exclusion can increase ingestion and create feedback from the collector’s own error logs. Replace the full input configuration if customizing it, including the journal input when required.

No output or filter is configured directly in the package’s upstream defaults. The umbrella supplies Kubernetes metadata enrichment, a cluster-label Lua filter, and outputs according to selection:

• Elasticsearch/Kibana enabled: Elasticsearch outputs for container and host records, using the generated password environment variable and certificate mount.
• Loki enabled: Loki outputs for both tags. The endpoint follows the selected monolith, scalable, or distributed strategy. Enabling Loki does not by itself disable Elasticsearch outputs; both can be present.
• additionalOutputs provides Big Bang-owned Elasticsearch, Loki, Fluentd and S3 credential/certificate helpers. Inspect the generated configuration and mounts before relying on a helper: configuring a destination does not automatically enable its network egress or guarantee that an output stanza was generated.

additionalOutputs.disableDefault: true suppresses the umbrella’s generated output/filter and related credential/mount block. Use it only with a complete replacement pipeline and its own Secret references and mounts. See index customization for replacement-output cautions and the umbrella logging-stack guide for cross-package selection.

Elasticsearch credentials

The umbrella’s default Elasticsearch connection expects logging-ek-es-elastic-user (key elastic) and logging-ek-es-http-certs-public in the collector namespace. The package renders cluster-scoped Kyverno GeneratingPolicy resources only when Elasticsearch/Kibana integration is enabled and policies.kyverno.io/v1 is available. These policies copy the named Secrets from logging, generating existing resources and synchronizing updates.

Without that API/controller, arrange secure Secret provisioning in fluentbit through your GitOps process. Never put the password in a ConfigMap or commit copied credentials. Custom Elasticsearch names do not automatically change the policies’ hard-coded source Secret names. The ECK Operator and Elasticsearch workload are separately owned packages; enabling the collector does not install either. After certificate/password changes, verify synchronization and connection recovery; see troubleshooting.

Storage

The DaemonSet reads host logs through read-only mounts for /var/log, /var/lib/docker/containers and /etc/machine-id. The package also mounts a writable hostPath at /var/log/flb-storage/, created with DirectoryOrCreate, and configures filesystem buffering for both inputs. It does not request a PVC by default, but it does use persistent node-local files. The buffer is neither replicated storage nor a substitute for a backend.

Changing the buffer location requires coordinated service configuration and volume/mount changes under the upstream parent; the old storage_buffer.path example is not an active chart setting. Check the final merged lists: umbrella-generated certificate or S3 mounts can replace package buffer lists. Keep the buffer mounted and writable when filesystem buffering is configured.

Size and monitor node disk capacity for destination interruptions. The umbrella’s generated outputs normally use a 10G logical queue limit per output, configurable through the Big Bang-owned storage.total_limit_size setting. Queue limits are not a global disk quota; reaching a limit can discard older records. See upstream buffering documentation for durability and backpressure behavior.

Node security and scheduling

The package’s container defaults run as UID 0, drop all capabilities, use a read-only root filesystem, set privileged: false, and request SELinux type spc_t. Host paths, SELinux policy and admission rules must be evaluated for your nodes. Enforcing SELinux is not a universal requirement to enable privileged mode. Review permissions and denials before approving a narrowly scoped security exception; do not disable security controls cluster-wide.

The DaemonSet collects only on nodes where scheduling and admission allow it to run. See scheduling considerations and host-path troubleshooting. OpenShift integration includes SCC-related resources and requires appropriate platform permissions.

Networking and mesh

The package uses bb-common for NetworkPolicies and Istio resources. The umbrella maps API access, selected backend egress and metrics ingress. External Elasticsearch, Loki and Fluentd egress is explicit and disabled by default; restrict destination CIDRs/ports rather than enabling broad internet access. S3 also needs suitable storage-subnet access.

When Istio restricts outbound traffic, configure appropriate ServiceEntries as well as NetworkPolicies; an output helper alone does not permit traffic. See package Istio guidance. The umbrella conditionally orders reconciliation after selected backends, policy, mesh and monitoring releases; this ordering does not prove credential readiness or successful delivery.

High availability

The default workload is a DaemonSet, not a replicated pool of interchangeable collectors. Filesystem buffering can retain queued data across a pod restart on the same node, but does not protect against node loss, disk exhaustion or every crash scenario. The default tail input does not configure a persistent offset database. Validate restart, rotation and destination-outage behavior for your workload rather than assuming exactly-once delivery or lossless recovery.

Health checks and metrics

The package enables the HTTP server on port 2020, with liveness at / and readiness at /api/v2/health. The umbrella enables a ServiceMonitor and dashboard when its metrics-scraping helper is active (monitoring Prometheus metrics or Alloy metrics). ServiceMonitor rendering also requires the monitoring.coreos.com/v1 API; direct package defaults do not enable it. The umbrella aligns service/monitor port names through post-rendering.

The optional bbtests script checks HTTP availability, deployed version, positive aggregate input and processed-output counters, and zero aggregate output errors. It does not query the backend or validate every node, parser, output or host journal. For install/upgrade acceptance, confirm scrape targets, inspect retries/errors and buffer capacity, then query the configured backend for representative container and host records. See upstream monitoring documentation and the maintainer test procedure.

k3d CI compatibility

The umbrella’s k3d test values replace the complete input configuration with container-tail collection and replace the DaemonSet host volumes/mounts with only read-only /var/log. These nodes do not run kubelet as a systemd service; the override avoids the /etc/machine-id dependency. It is a CI-specific override, not the package default, and does not validate host journal collection. Deployments needing journal logs must supply node-appropriate inputs and mounts.