· javascript
LinkedIn Replaces Argo CD with a Custom GitOps Engine for Kubernetes Deployments
This new system deploys applications across over 250 Kubernetes clusters.
It supports more than 4,000 services and manages over 100,000 deployments each week. According to a post, deployment infrastructure failures fell from 925 to 63 despite a 3.7x increase in deployment volume. The median time to detect and conclude a failed deployment dropped from 16.1 to 2.45 minutes.
LinkedIn started with one Argo CD instance per cluster, a topology it chose because Argo CD already managed its cluster add-ons that way. Clusters run up to 5,000 nodes each. A central deployment orchestrator spawned an ephemeral pod per deployment, with Argo CD hidden behind it.
The team lists seven problems that emerged at scale. Argo CD doesn't have an observedGeneration, so you can't link sync and health status to a specific deployment. LinkedIn created a Lua health check that uses a tracking annotation to address this.
A Synced and Healthy application could also resolve to an unintended revision when stale repo-server cache entries were used. Two of the issues were architectural. Argo CD maintains one watch per group-version-resource across Applications.
LinkedIn's clusters have low resource-type diversity. This means etcd churn from one app can slow syncs for all apps.
