Three years ago, I helped a fintech team investigate a cloud bill that had grown by roughly 40 percent over six months despite no meaningful change in workload volume. We went through the usual suspects: oversized instances, idle resources, storage lifecycle gaps. Everything looked fine. It was not until we pulled apart the networking line items that we found it. Their Kubernetes cluster, spread across three availability zones as it should be, was routing almost all service-to-service traffic across zone boundaries. The application had hundreds of microservices. Each service call crossed an AZ. The charge per gigabyte is small. The aggregate across millions of requests per second is not.
The fix took about two hours to deploy. The bill dropped noticeably on the next cycle. That experience turned cross-AZ traffic into the first thing I audit whenever a Kubernetes cluster looks expensive.
Why This Problem Is Invisible
Cross-availability-zone data transfer is priced differently from internet egress, and it does not appear prominently in most cloud dashboards. On AWS, GCP, and Azure, traffic that crosses availability zone boundaries within the same region typically costs around $0.01 per gigabyte in each direction. That sounds trivial. For a cluster handling ten terabytes per day of service-to-service traffic with the typical cross-zone distribution, it adds up quickly. Add both directions and you are paying for a substantial chunk of data movement that never leaves your region.
The invisible part is that Kubernetes does not route around this by default. When a pod calls a Service, kube-proxy or eBPF-based implementations like Cilium pick an endpoint from the Service’s EndpointSlice. Without topology awareness, that selection is random across all healthy endpoints, regardless of which availability zone the calling pod lives in. If your cluster is evenly distributed across three zones, roughly two-thirds of your traffic crosses a zone boundary on every call.
The traditional answer was to use pod affinity rules to co-locate services, but that creates brittle scheduling constraints, reduces bin-packing efficiency, and completely falls apart during rolling deployments when pods from multiple versions exist simultaneously. What you actually want is a routing preference, not a hard constraint. That is exactly what the topology-aware routing features in Kubernetes provide.

The Evolution: From Service Topology to TrafficDistribution
The Kubernetes community has been working on this problem for several years, and the API surface has gone through a few iterations that are worth understanding before you dive in.
The first attempt was Service Topology, which allowed you to specify routing preferences through a topologyKeys field on the Service object. It was added as an alpha feature in Kubernetes 1.17, deprecated in 1.21, and removed in 1.22. The problem was complexity: you had to enumerate topology preferences explicitly, and the semantics were difficult to reason about. It was replaced by Topology Aware Hints in 1.21 and eventually removed.
Topology Aware Hints, controlled via the service.kubernetes.io/topology-mode: Auto annotation, addressed the same problem through a different mechanism. Rather than putting topology logic in the Service spec, the approach moved it into the EndpointSlice controller. When the annotation is set, the controller adds per-endpoint zone hints to the EndpointSlice objects. kube-proxy on each node reads those hints and prefers same-zone endpoints when routing traffic.
The hints approach works, but it has a few rough edges. The controller only generates hints when it judges the zone distribution to be safe, specifically when each zone has proportional endpoint coverage relative to its share of cluster capacity. If your pods are unevenly spread or if one zone has substantially more cluster capacity than endpoints, hints may not be generated at all, and traffic silently falls back to cluster-wide routing. The annotation-based interface is also not especially discoverable.
Kubernetes 1.33 (released April 2025 and named Octarine) graduated a cleaner solution to general availability: the trafficDistribution field on the Service spec. Setting it to PreferSameZone tells kube-proxy to prefer same-zone endpoints, falling back to any healthy endpoint when no same-zone option is available. The semantics are clear: this is a best-effort preference, not a hard constraint. If the only healthy pod for a service is in another zone, your traffic still gets there.
This is the API you should be using on any cluster running 1.33 or later. The hints annotation still works, but trafficDistribution is the direction the project is moving.
How EndpointSlice Topology Works
Understanding what actually happens when you enable topology-aware routing helps you debug the cases where it does not work as expected.
Every Service in Kubernetes is backed by EndpointSlice objects. Each EndpointSlice tracks a subset of endpoints and, for each endpoint, includes a nodeName, zone, and other topology information when that data is available. This topology information is populated by the EndpointSlice controller based on node labels, specifically the topology.kubernetes.io/zone label that cloud providers apply to nodes automatically.
When you use trafficDistribution: PreferSameZone, kube-proxy on each node builds its forwarding rules to prefer endpoints whose zone label matches the node’s own zone. The fallback is the full set. With the older hints mechanism, the EndpointSlice controller adds an explicit hints section to each endpoint, and kube-proxy reads those hints instead of doing zone matching itself.
The practical implication: your nodes must have topology.kubernetes.io/zone labels for any of this to work. On EKS, GKE, and AKS, these labels are applied automatically. If you are running self-managed nodes or using a custom provisioner, verify the labels are present before spending time debugging topology routing.
Another implication: the zone information flows through the EndpointSlice, not the pod. If your workload uses headless Services (where DNS resolves directly to pod IPs instead of a cluster IP), topology-aware routing does not apply the same way. The Service IP routing path is what kube-proxy controls. For headless Services, you need DNS-level topology awareness, which is a different problem.
Enabling It in Practice
For a Service running on Kubernetes 1.33 or later, enabling topology preference is a one-line change:
apiVersion: v1
kind: Service
metadata:
name: checkout-api
namespace: payments
spec:
trafficDistribution: PreferSameZone
selector:
app: checkout-api
ports:
- port: 80
targetPort: 8080
That is the entire change. No application code change, no pod restart, no network policy modification. The EndpointSlice controller sees the field and adjusts its behavior. kube-proxy picks up the updated routing preference on its next sync cycle.
For clusters that have not yet reached 1.33, you can use the hints annotation instead:
apiVersion: v1
kind: Service
metadata:
name: checkout-api
namespace: payments
annotations:
service.kubernetes.io/topology-mode: "Auto"
If you manage Services through Helm charts, you can template the field based on cluster version. If you use Kustomize, a patch across all services in a namespace takes about ten minutes to write.
Before rolling this out cluster-wide, test it on a non-critical service and verify the routing behavior. The best way to confirm it is working is to watch the traffic in your cloud provider’s flow logs or VPC metrics, or to instrument your services with zone-aware request tagging and check whether the requesting zone matches the responding zone in your distributed traces.

The Cases Where It Does Not Work
Topology-aware routing is a preference, and there are several situations where it will not behave as you expect.
The most common issue is uneven zone distribution. If you have 30 pods in us-east-1a and 10 in us-east-1b, the routing preference will be honoured for traffic within each zone, but the uneven distribution means pods in us-east-1b will occasionally receive traffic from us-east-1a when their local pool is saturated. For the hints-based mechanism, uneven distribution may cause the controller to generate no hints at all rather than partial hints, which means you get no topology benefit. The trafficDistribution field handles this more gracefully because it is a kube-proxy-level preference rather than a controller-generated annotation.
Deployments with PodAntiAffinity rules set to RequiredDuringScheduling will spread pods but may prevent you from having more than one pod per node per zone. If your pods are very few and your cluster is very large, you may have only one or zero pods in some zones. In that case, PreferSameZone will frequently fall back to cross-zone routing because there are no local endpoints to prefer.
NodePort and LoadBalancer Services with externalTrafficPolicy: Cluster route traffic through kube-proxy on the receiving node and then potentially across zones to reach the backend pod. If you care about the topology of incoming traffic from outside the cluster, externalTrafficPolicy: Local ensures that traffic hits a pod on the same node that received it. This is a separate setting from trafficDistribution and addresses a different traffic flow: external-to-cluster rather than internal service-to-service.
Cilium in native routing mode replaces kube-proxy entirely and implements its own topology-aware routing via the loadBalancer.serviceTopology Helm value. The behavior is conceptually the same but controlled through Cilium’s configuration rather than the Service spec. If you are running Cilium, check its documentation for the equivalent configuration, as the Kubernetes-level trafficDistribution field may or may not be honoured depending on your Cilium version and configuration.
Istio with its sidecar model intercepts traffic before it reaches kube-proxy, so topology-aware routing via kube-proxy does not apply to mesh-managed traffic. Istio has its own locality load balancing feature, controlled through DestinationRule.trafficPolicy.loadBalancer.localityLbSetting. If you are running a service mesh, that is the right lever to pull instead, and it is the subject of the Istio ambient mesh article on this site.
DNS Traffic Is a Separate Problem
One thing that consistently surprises engineers when they first dig into cross-AZ costs is how much of the traffic is DNS. Every Service discovery lookup goes to kube-dns (backed by CoreDNS), and by default kube-dns pods are not necessarily in the same zone as the calling pods. Each DNS query is tiny, but high-throughput services that resolve Service names repeatedly accumulate real DNS traffic costs.
The right answer here is NodeLocal DNSCache, which runs a caching DNS agent on each node and intercepts DNS queries before they leave the node. For most workloads, this eliminates the cross-AZ DNS traffic entirely because the cache on the local node handles the resolution. The Kubernetes DNS and CoreDNS production guide on this site covers NodeLocal DNSCache in depth, including the ndots configuration that determines how many DNS queries your pods generate per request.
For topology-aware routing to have its full effect, deploy both: trafficDistribution: PreferSameZone for Service traffic, and NodeLocal DNSCache for DNS traffic. Missing either one leaves money on the table.
Measuring the Impact
Before enabling topology routing, establish a baseline. The measurement approach depends on your cloud provider and your observability stack.
On AWS with EKS, CloudWatch Network Flow Monitor can show you the volume of traffic crossing AZ boundaries. Alternatively, enable VPC Flow Logs and query them through Athena or CloudWatch Logs Insights. Filter for traffic between subnets in different availability zones to get the cross-AZ volume before and after the change. The cloud network observability article covers this in detail.
On GCP with GKE, GKE’s topology-aware routing has been a first-party feature for longer than the upstream Kubernetes equivalent, and GCP’s network observability tools give you per-zone traffic breakdowns in the Cloud Console.
In your Kubernetes metrics, the EndpointSlice controller exposes a discovery_endpointslice_controller_endpoints_added_per_slice_total metric and related counters that tell you whether hints are being generated. kube-proxy does not natively expose a “how often did I fall back to cross-zone routing” counter, but you can instrument this by tagging your service calls with the requesting and responding zone (via downward API environment variables injected into pods) and tracking mismatches in your distributed tracing data.
A rough sanity check: in a balanced three-zone cluster with topology routing enabled, you should see approximately 70-90 percent of service-to-service traffic staying within zone (depending on how evenly your services are distributed). If you are seeing much less than that, something is misconfigured: either the zone labels are missing, the pods are unevenly distributed, or a service mesh is intercepting traffic before topology hints are applied.

Cloud Provider Specifics
Each major cloud provider has nuances worth knowing about.
On AWS EKS, the zone labels are applied by the AWS cloud controller manager. All managed node groups and Fargate profiles include them automatically. The key caveat is that NAT Gateway charges also contribute to cross-AZ costs: if your pods hit a NAT Gateway in a different zone for internet egress, that traffic crosses a zone boundary to reach the gateway. This is separate from Service routing and requires placing NAT Gateways in each zone, which the cloud egress costs article covers. For topology routing specifically, EKS clusters on 1.33 and later can use trafficDistribution: PreferSameZone without any additional configuration.
On GCP GKE, the topology awareness features are implemented through GKE’s own layer and have been available longer. GKE Autopilot supports topology-aware routing natively. Standard mode works through the same EndpointSlice mechanism as upstream Kubernetes. GCP also offers Traffic Director for service mesh use cases, which has its own locality load balancing configuration.
On Azure AKS, zone labels are applied by the Azure cloud controller. The behavior of trafficDistribution: PreferSameZone on AKS clusters running 1.33 or later is consistent with upstream Kubernetes. Azure’s Carbon Aware KEDA Operator (a separate topic) interacts with zone selection too, but that is about carbon intensity scheduling rather than traffic routing.
Multi-cloud clusters or clusters that span cloud providers are unusual but exist. In those environments, the zone topology from one provider may not map cleanly to another, and topology routing may behave unexpectedly. If you are managing a multi-cloud control plane through tools like Karmada (covered in the Kubernetes multi-cluster management article), each member cluster handles topology routing independently. The federation layer does not add topology intelligence.
Interaction with Karpenter and Node Provisioners
Topology-aware routing only saves money if you have pods in every zone. If your cluster uses Karpenter for node provisioning, the topologySpreadConstraints on your Deployments and StatefulSets determine whether pods land in the right zones.
The minimum useful configuration for topology-aware routing: a topologySpreadConstraint that spreads pods across zones with whenUnsatisfiable: DoNotSchedule for pods that see real production traffic, and maxSkew: 1 to keep the distribution close to even. Without this, Karpenter may legitimately pack all your replicas into a single zone for cost efficiency reasons, leaving nothing for the in-zone routing preference to prefer.
The interaction goes the other way too: Karpenter’s consolidation logic can remove nodes and move pods around, temporarily collapsing a multi-zone deployment to a single zone during low-traffic periods. If your cluster scales down aggressively at night, check whether topology routing is delivering the benefit you measured during the day. Scale-down and scale-up windows affect your cross-AZ ratio.
If you are using pod disruption budgets alongside topology spread constraints, make sure the PDB does not prevent Karpenter from maintaining zone balance during rolling deployments. I have seen teams add PDBs that require a minimum of two pods per deployment, combined with slow rolling update settings, that result in imbalanced zone coverage for fifteen-minute windows during every deploy cycle. During those windows, topology-aware routing routes more traffic cross-zone. For high-frequency deployments, the imbalance windows accumulate.
Putting It Together: A Practical Rollout
The rollout I recommend:
First, instrument. Add Prometheus alerts on cross-AZ traffic volume using VPC flow metrics or cloud provider billing anomaly detection. The FinOps cost anomaly detection article describes the alerting setup. Establish a baseline.
Second, enable NodeLocal DNSCache if you have not already. It has no downside beyond a small memory footprint on each node and eliminates DNS as a source of cross-AZ cost.
Third, audit your topology spread constraints. Any Deployment serving substantial traffic should have a spread constraint across topology.kubernetes.io/zone. Without even distribution, topology routing cannot do its job.
Fourth, enable trafficDistribution: PreferSameZone on your highest-traffic services first. Use kubectl get endpointslices -n <namespace> -o yaml to verify that zone information is populated on the endpoints. Use your distributed tracing data to confirm that call graphs show same-zone responses.
Fifth, roll it out to remaining services. For services managed by Helm, add it to your chart values. For raw manifests, a targeted kubectl patch across all Services in a namespace is fast.
The whole process can usually be done in a day for a medium-sized cluster. The impact on the bill becomes visible within a billing cycle or two as the cross-AZ traffic volume drops.
This is one of the cheapest wins in cloud networking: a single field change per Service, no application modification, no downtime, and measurable cost reduction. After twenty years of debugging cloud bill surprises, I still enjoy finding the ones where the fix is this clean.
For related reading, see the articles on Kubernetes DNS and CoreDNS in production, cloud egress costs and architecture, and Kubernetes pod scheduling and topology spread constraints.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.
