Karpenter + EKS: ghost Target Groups and a zero-downtime fix

By the QSTools team — composite field notes from production EKS work. Details generalized on purpose.

In multi-account AWS setups with EKS + Karpenter, a recurring failure mode shows up after a hasty EC2→EKS move: every node recycle drops traffic. Karpenter is rarely the villain. The data path still pins static node IPs inside classic Target Groups behind a legacy ALB.

At a glance

Signal	Root cause	Fix
Outages on scale / deploy	Target Groups with static node IPs	TargetGroupBinding + targetType: ip
Support ticket storms	API Gateway → legacy ALB without LBC	Keep ALB/Gateway; automate the TG
Fear of touching prod	No maintenance window	Phased cutover, near-zero downtime
Remaining debt	EKS nodes on public subnets	Safer once targets no longer depend on node IPs

The pattern: growth faster than network governance

Teams often start well—account segregation, Terraform, standardized EKS templates—then product velocity outruns the exposure layer:

1. Services still fronted by manual ALBs (originally built for EC2).
2. EKS migration without modernizing that ALB/TG model.
3. API Gateway (VPC Link) still glued to those ALBs.
4. Target Groups with hardcoded node IPs.
5. Karpenter arrives for cost/scale… and recycles those nodes.

Day-to-day: every deploy or scale event means lost targets (“ghost nodes”). Remediation usually has a hard constraint: no downtime on critical paths.

Diagnosis in one line

Karpenter did its job. The Target Group kept pointing at IPs that no longer existed.

Three-phase fix (zero downtime as a hard constraint)

Phase 1 — Decouple with TargetGroupBinding

Use the TargetGroupBinding CRD from the AWS Load Balancer Controller (LBC) so the controller manages the existing Target Group—without recreating ALBs or rewriting API Gateway routes.

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: tgb-critical-service
  namespace: production
spec:
  serviceRef:
    name: svc-critical-service
    port: 8080
  targetGroupARN: arn:aws:elasticloadbalancing:region:123456789012:targetgroup/tg-critical/abcdef123456
  targetType: ip

(Example ARN — use your real Target Group in production.)

Phase 2 — Ports and health checks

Align Service ports and health-check paths so LBC registers pods (targetType: ip). In the incident we documented, reconciliation happened live with near-zero service interruption.

Phase 3 — Helm governance

Once production is stable, bake LBC annotations into chart Service manifests so future deploys do not slip back to manual TG edits:

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

Before vs after

Initial situation	After the change
Traffic loss when nodes rotated	Dynamic endpoint registration via LBC
Tickets on every deploy / scale	Stable deploys without hand-editing TGs
Manual Target Group upkeep	Pod IP targeting (targetType: ip)
Freeze from downtime fear	Transparent production change
Same fragile pattern copied across accounts	Repeatable chart/TGB pattern per cluster

What this teaches (and what stays open)

• Autoscaling is not a networking free lunch. If the data path assumes node IPs, Karpenter will “break” something that was already fragile.
• TargetGroupBinding is an excellent bridge when you cannot redesign API Gateway + ALB in one window.
• Common next step: move EKS nodes from public to private subnets—after targets no longer depend on node IPs.

If you are in this mess (EKS + legacy ALB + Karpenter), order matters: stabilize the TG first, standardize Helm second, private subnets once traffic no longer depends on node IPs.