Grafana Mimir

Grafana Mimir provides Prometheus-compatible, long-term metrics storage. Big Bang deploys the mimir-distributed chart as the mimir release in the mimir namespace. Configure the package through the umbrella addons.mimir entry; package-specific values are supplied beneath addons.mimir.values.

Configuration

The package passes addons.mimir.values.upstream to the upstream mimir-distributed chart. Refer to the upstream chart values for version 6.2.1 for its complete configuration surface.

Big Bang defaults to the classic gRPC ingest path. To use the Kafka-backed ingest-storage path, configure the umbrella-owned addons.mimir.kafkaIngestStorage settings, including the broker endpoint and, when network policies are enabled, its network CIDR. The package enables ingest storage and disables the classic gRPC push path together. Use a production Kafka service for production workloads; the bundled Kafka image is intended for testing only.

Storage and availability

Mimir uses filesystem storage by default, which is suitable only for temporary or evaluation use. For durable metrics, configure the umbrella-owned addons.mimir.s3ObjectStorage settings for an S3-compatible backend. The umbrella creates and supplies storage credentials when access and secret keys are provided; workload-identity users can leave those keys blank and configure the service-account annotations through addons.mimir.serviceAccountAnnotations.

The package supports upstream zone-aware replication through the passthrough values. Its rollout operator protects zone-aware workloads, but Big Bang sets the webhook failure policy to Ignore to prevent GitOps upgrade deadlocks. When that webhook is unavailable, its zone-aware safety checks are bypassed, so operators should keep it healthy and follow the package 6.x upgrade guidance before changing replication or upgrading from earlier releases.

Integrations and network access

Prometheus and Alloy can send metrics to the Mimir API, and Grafana can query Mimir as a datasource. When the corresponding Big Bang packages are enabled, the umbrella configures their required network-policy access and enables Mimir monitoring and Grafana dashboard integration.

For Alloy remote writes through a separately deployed Cortex Tenant proxy, enable addons.mimir.values.networkPolicies.ingress.to.mimir-distributor:8080.from.k8s.cortex-tenant/cortex-tenant. The package defines this distributor rule with the correct pod selector but keeps it disabled by default. Cortex Tenant must forward writes with X-Scope-OrgID.

With Istio enabled, Big Bang adds the package resources required for Mimir’s internal gRPC and memberlist traffic. Add external object-storage or Kafka destinations through the package’s istio.serviceEntries.custom value. For Kafka configured through addons.mimir.kafkaIngestStorage, the umbrella can also render the required broker routes from its broker host and cluster-domain settings. Ensure any external endpoint is permitted by both Istio and the selected network-policy configuration.

Mimir’s gateway route is disabled by default. Enable and configure routes.inbound.mimir-gateway only when an external API endpoint is required; Mimir does not provide a general-purpose web UI.

Granting Egress to Blocked Services

Configuring object storage does not grant network access to its API. Block, ruler, and alertmanager storage can use different endpoints; S3 virtual-hosted bucket requests and workload identity can also require hosts beyond the base storage endpoint. Kafka clients need the advertised broker destinations, not only their bootstrap host. Review the actual selected backend and identity path.

In sidecar mode, traffic captured by a proxy with effective REGISTRY_ONLY policy can be denied if its destination is unregistered. The chart exposes istio.sidecar.outboundTrafficPolicyMode; rendering a Sidecar does not inject a proxy. Check namespace enrollment, pod injection, and capture exceptions first.

Discovering Blocked Hosts

Confirm the intended kubeconfig and list Mimir pods using their default label:

kubectl -n mimir get pods -l app.kubernetes.io/name=mimir --show-labels

Use the failing component’s application errors to identify the endpoint. Select a pod making that request, such as an ingester or store-gateway for block storage, rather than assuming the gateway or rollout operator makes it:

SOURCE_POD="replace-with-failing-component-pod-name"
kubectl -n mimir get pod "$SOURCE_POD" \
  -o jsonpath='{.spec.containers[*].name}{"\n"}{.spec.initContainers[*].name}{"\n"}'

If it has an istio-proxy, inspect recent proxy logs during the failed request:

kubectl -n mimir logs "$SOURCE_POD" -c istio-proxy --since=10m \
  | grep -i 'BlackHoleCluster'

With proxy access logging enabled, BlackHoleCluster entries can help identify unregistered destinations. No match does not establish reachability: check DNS, NetworkPolicies, TLS trust, storage credentials, and backend permissions too. In ambient mode there may be no pod proxy; inspect the relevant ztunnel/waypoint path using the umbrella’s ambient guidance. Redact credentials, signed URLs, and sensitive metric metadata before sharing logs.

Registering External Destinations

These umbrella values register one HTTPS object-storage endpoint. Replace the example with the host actually used by the client; register additional required bucket or identity hosts deliberately, not with a broad cloud wildcard:

addons:
  mimir:
    values:
      istio:
        enabled: true
        serviceEntries:
          custom:
            - name: object-storage
              spec:
                hosts:
                  - storage.example.com
                exportTo: ["."]
                location: MESH_EXTERNAL
                resolution: DNS
                ports:
                  - number: 443
                    name: https
                    protocol: TLS

The mesh and its APIs must already be available; this does not enable injection or configure Mimir’s storage backend. Direct chart values start at istio. The custom array replaces existing entries, so preserve those still needed. For Kafka, prefer the umbrella broker-route integration described above and verify its rendered hosts against broker-advertised addresses and ports.

Mesh registration does not grant destination IP/port access through enabled NetworkPolicies. Review the affected component’s networkPolicies.egress rules, DNS access, and endpoint IP changes separately. Do not change to ALLOW_ANY or broad internet egress as a routine fix. Commit narrowly scoped changes through GitOps, then verify backend operations and ingestion/query health with synthetic metrics; a rendered ServiceEntry is not proof of storage durability.

See the bundled bb-common 1.6.1 Istio and NetworkPolicy guides; match the version to Chart.yaml.

Further reading

• Grafana Mimir documentation
• Mimir distributed Helm chart documentation
• Big Bang 6.x upgrade guidance