Skip to content

Troubleshooting

Read-only workload status

Start with CR status and namespace events before changing configuration:

kubectl -n logging get elasticsearch,kibana
kubectl -n logging get pods,pvc
kubectl -n logging get events --sort-by=.lastTimestamp

These commands inspect the active kubeconfig; they do not deploy or repair anything. Share only redacted diagnostics. Compare desired versus observed generations, Elasticsearch phase/health, Kibana association/health, Pending PVCs, and resource pressure. The wait job gates reconciliation but does not validate end-to-end ingestion or every shard’s allocation.

Safe Elasticsearch API access

Follow ECK service access and CA retrieval. With default umbrella naming, the HTTPS endpoint is logging-ek-es-http.logging.svc:9200; the bootstrap password Secret is logging-ek-es-elastic-user in logging. Prefer an approved read-only monitoring credential over elastic. Do not print the password or include it in command-line arguments, transcripts, or tickets. Disable shell tracing around sensitive access.

Use trusted CA verification and a hostname matching the certificate’s SANs. A local port-forward does not make https://localhost match the service certificate; retain the service hostname through an appropriate local connection override or use an approved in-cluster client. Never bypass trust with curl -k to make a health example succeed. Query cluster health and shard allocation through that verified connection; do not blindly apply allocation-setting changes from an old recipe.

Failed log delivery and HTTP 429

Fluent Bit retries and Elasticsearch rejection responses can indicate indexing pressure, resource constraints, unavailable shards, or transport problems. Correlate collector errors with Elasticsearch metrics and allocation diagnostics before adjusting capacity. Package resource and heap inputs live under elasticsearch.<node-set>.resources / heap; umbrella v1 overrides wrap them in packages.elasticsearchKibana.values. The chart renders heap min/max into ES_JAVA_OPTS; plan equal bounds with sufficient memory for non-heap usage. Do not use fixed large heap examples as a universal fix.

Collector buffer/filter changes belong to Fluent Bit, not this workload chart. A custom collector may use different index names from the logstash-* ingestion test. See log access.

Node, network, and SSO failures

  • For mmap bootstrap errors, check node prerequisites rather than applying the obsolete privileged DaemonSet recipe.
  • For Pending pods/PVCs, review storage and placement constraints before changing counts or deleting volumes.
  • For connection failures, distinguish NetworkPolicy, mesh registry/authorization, and TLS. Preserve Elastic’s transport exception.
  • For OIDC login failures, check provider endpoints, CA, and role mappings. Successful authentication without roles is not successful authorization.

Use Elastic troubleshooting for upstream shard, memory, and cluster-health diagnosis. Storage/topology or runtime changes require separate clean-install/upgrade evidence; this guide is not authorization to mutate a production cluster.