Zero-downtime Migration from Ingress-Nginx to Istio in a Multi-Cluster Kubernetes Platform at Bloomberg

The need for Multi-Cluster deployments at Bloomberg

Bloomberg needs multi-region and multi-cluster orchestration for business continuity. Meaning, application down equals no clicks equals no $$$

So they have multiple management clusters, and multiple client clusters. One holds the config, the other the applications.

They used Karmada to do this. The way this works is: you deploy Karmada, then talk to Karmada instance, which configures your client clusters.

Managed Karmada and a custom operator

They deployed the Karmada operator, but they needed more control. So they developed their own operator to extend the Karmada capabilities.

This was first deployed as a standalone deployment.

High Availability

The standalone deployment can handle failover and recovery of an outage of one or more client clusters.

However, if the management cluster goes down, the client clusters are also gone.

So they figured out how to do multi-karmada deployments to circumvent this.