Migrating from Istio Operator to Helm based Istio📜

Timeline:📜

The new Istio Helm packages istio-core and istio-gateway are Beta in Big Bang 2.51
These packages will be generally available and stable for production use in 2.52 (or 2.53)
The istio-operator and istio-controlplane packages will no longer be present in Big Bang 3.0
Therefore, migrate from Istio Operator to Istio Helm in BB 2.52 (or 2.53) before upgrading BB to 3.0

Considerations:📜

The helm packages update Istio from 1.23 to 1.25
The Istio Operator is End of Life and does not support versions of Istio after 1.23
Istio 1.23 is only supported through April 2025

Migration Process📜

Istio can be migrated from the old operator packages to the new helm-based packages in-place with a few steps.

Step 1 : Swap `istio` for `istioCRDs` and `istiod`📜

Disable the old istio package and enable the new istioCRDs and istiod packages:

istioOperator:
  enabled: true
istio:
  enabled: false

istioCRDs:
  enabled: true
istiod:
  enabled: true
istioGateway:
  enabled: false

Give the cluster a few minutes for all helm releases to become ready.

Step 2 : Disable `istioOperator` and enable `istioGateway`📜

Removal of the operator and the enablement of the new gateway package reinstantiates cluster gateways.

When migrating gateway configurations, see the examples here as a reference to format values and configure postRenderers.

istioOperator:
  enabled: false
istio:
  enabled: false

istioCRDs:
  enabled: true
istiod:
  enabled: true
istioGateway:
  enabled: true

After all helm releases become ready once again, verify gateway(s) recieves an external IP:

kubectl get svc -n istio-gateway
NAME                  TYPE         CLUSTER-IP    EXTERNAL-IP  PORT(S)                                    
public-ingressgateway LoadBalancer 10.43.110.109 172.16.88.88 15021:31155/TCP,80:31302/TCP,443:31046/TCP

The migration process is now complete.

Troubleshooting📜

Below are a few tips for troubleshooting if the migration did not go as smoothly as expected.

Services are unreachable📜

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end

To resolve this issue, cycle all Istio injected pods allowing their reconnection.

The below bash script iterates across all istio-injected namespaces and recycles all pods:

# in istio-injected namespaces, recycle pods
for namespace in `kubectl get ns -o custom-columns=:.metadata.name --no-headers -l istio-injection=enabled`
do
    echo -e "\n♻️ recycling pods in namespace: $namespace"
    for pod in `kubectl get pods -o custom-columns=:.metadata.name --no-headers -n $namespace`
    do 
        kubectl delete pod $pod -n $namespace
    done
done

Pods should return to ready within a few minutes.

Reconcile Helm Releases📜

If may be necessary to synchronize helm releases managed by Flux when they become out of sync.

The flux CLI must be installed to use this bash script that iterates across all helm releases initiating a reconciliation:

# reconcile all of big bang's helm releases w/ flux
for hr in `kubectl get hr --no-headers -n bigbang | awk '{ print $1 }'`
do
    echo -e '\n☸️ reconciling hr:' $hr
    flux reconcile hr $hr -n bigbang --with-source
done

All services in the cluster should once again be reachable.