Repository navigation
Description
Describe the bug
I went through setting up Istio GatewayApi looking to move from ingress-nginx.
I noticed that occasionally, the Istio GatewayApi will get incorrect values for X-Forwarded-For header and downstream_local_address in the logs.
I've traced this back to the Kubernetes Service crated by the Gateway will have externalTrafficPolicy: Cluster and not Local as suggested by this doc https://learn.microsoft.com/en-us/azure/aks/configure-load-balancer-standard?tabs=create-cluster-ip-based%2Ccreate-cluster-managed-outbound-ips%2Ccreate-cluster-custom-ips%2Ccreate-cluster-custom-ip-prefixes%2Ccreate-cluster-outbound-ports-ips%2Ccreate-cluster-idle-timeout#maintain-the-clients-ip-on-inbound-connections
This seems to result in:
- if the Azure LoadBalancer sends the request to a Node that has a Gateway Pod, it will show the correct X-forwarded-for IP, (edit: sometimes it will still return the Node Ip i am finding out)
- But if it goes to a Node without such a pod, it will still serve the request, but X-Forwarded-For headers and logs will show the IP of the Node. I believe it hits the
kube-proxyservice running on the node?
I tinkered with setting the service's externalTrafficPolicy: Local, however this causes issues. The Service has built-in annotations that seem to manage the health-probe_port for the load balancer. I tried to override these with the value of the service .spec.healthCheckNodePort after setting externalTrafficPolicy: Local on the Gateway using spec.infrastructure.annotations."service.beta.kubernetes.io/port_443_health-probe_port" but it did not seem to take. I can set other annotations like service.beta.kubernetes.io/azure-load-balancer-ipv4 and that seems to work correctly so there seems to be something overriding that.
service.beta.kubernetes.io/port_443_health-probe_port: "10256"
service.beta.kubernetes.io/port_443_health-probe_protocol: http
service.beta.kubernetes.io/port_443_health-probe_request-path: /healthzTo Reproduce
- Create AKS cluster with Istio GatewayApi ( we're using 1-28 )
- Setup a gateway following docs https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api
- scale out the node pool to have more than 2 nodes so some nodes will not have istio gateway deployment pods on them
Expected behavior
Would like the proper client IP in the X-Forwarded-For header and logs.
Screenshots
If applicable, add screenshots to help explain your problem. Make sure not to include sensitive or personal information.
Environment (please complete the following information):
- Kubernetes version [e.g. 1.33.7]
- istio service mesh addon 1-28
Additional context
Add any other context about the problem here.
I found this article https://learn.microsoft.com/en-us/azure/aks/shared-health-probes?tabs=existing-cluster-shared-probe-mode which seemed related, but this option is not enabled on the cluster currently.
Activity
defenestration commentedon Mar 20, 2026
My ugly workaround for this at the moment is to bascially copy the original deployment/service/serviceaccount under a new name and use the Istio setting in the gateway to manually setup those things without the annotations described above at https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/#manual-deployment
I can see a template setup in the helmchart in aks-istio-system that suggests these annotations in the service aren't currently overridable.
defenestration commentedon Mar 20, 2026
I found something better that might work and plays nicer than rolling our own deployment/service.
in the configmap
istio-gateway-class-defaultsin namespace aks-istio-system you can set defaults for all gateways.I set the value for the problem annotations to blank, this seems to not have it render in the service config, which causes the default setup for the loadbalancer to kick in. spec.externalTrafficPolicy can also be set here.
This applies to every gateway, but in our case that is what we want. For port 443 we probably want similar. but this seems promising.
more notes:
There seems to be per-gateway way to set this with a config map but i feel like i tried that and it did not work.
Setting spec.infrastructure.annotations on the gateway for this does not work for these either, though you can add other annotations.
defenestration commentedon Mar 23, 2026
Noting that setting those blank annotations using a per-gateway configmap seems to work as well.
https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#configure-settings-for-a-specific-gateway
linnea-s commentedon Mar 30, 2026
I'm not sure if it's relevant for this issue but I found that there is the following default (reported on startup of gateway pods):
This means that there is one trusted proxy in front of the gateway which can send the client IP address in the X-Forwarded-For header. This is appropriate when using Azure Application Gateway but if you only have a TCP load balancer in front of the gateway then I believe clients can pass anything they want in the X-Forwarded-For header and it will be trusted. See https://istio.io/latest/docs/ops/configuration/traffic-management/network-topologies/
I think that should be documented somewhere.
By the way, using Gateway API with Application Gateway in front and following instructions at https://learn.microsoft.com/en-us/azure/application-gateway/rewrite-http-headers-url#remove-port-information-from-the-x-forwarded-for-header, I get the correct remote client IP despite using externalTrafficPolicy Cluster.
linnea-s commentedon Apr 1, 2026
I am also evaluating istio as a replacement for ingress-nginx and found a use-case where I need externalTrafficPolicy: Local. I need to be able to restrict the client IP either in NetworkPolicy or in AuthorizationPolicy. That is only possible when using externalTrafficPolicy: Local. I can confirm that it works to use a ConfigMap gateway-options like this (thanks Alan B (@defenestration)):
And the Gateway then contains:
As far as I can tell the Gateway always receives the correct source IP otherwise my NetworkPolicy would not work.
However there's an important missing piece: According to https://blog.aks.azure.com/2025/04/04/optimize-aks-traffic-with-externaltrafficpolicy-local#best-practices-for-rolling-updates-and-pod-rotation we must set minReadySeconds to avoid disruption during rolling upgrades. That is not possible because it is not allowlisted: https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#deployment-customization-allow-list-fields
In summary AKS team, please:
nshankar13 commentedon Apr 15, 2026
Yes - for ETP: cluster, we needed to set the health probe port annotations directly due to a preview issue with Azure Load Balancer's health probing logic: #3646. So for Gateway API, we need to set it to the port for kube-proxy, so that SLB checks the health of the kube-proxy instance running on that node before forwarding traffic to that node - then kube-proxy will forward it to the correct backend instance depending.
For ETP: local, SLB will probe the healthCheckNodePort directly on the nodes where the backend pods are running.
However, right now we don't have
healthCheckNodePortas part of our allowlist for the Gateway service customization, which is possibly why your changes haven't taken effect: https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#service-customization-allow-list-fields. We recently added a change for to allowlist this, but it hasn't rolled out yet - we are trying to get this out in the next AKS release or so.Linnea Skogtvedt (@linnea-s) Alan B (@defenestration) to clarify - if you're able to set healthCheckNodePort with etp: local, will you still need the hardcoded health probe logic unset from the Service spec? We may be able to get rid of the health probe settings down the road (it's not required after a certain AKS version - trying to determine that right now) - but not sure the exact ETA on this.
As for deployment
spec.minReadySeconds- yes we can look into adding this - will keep you posted.defenestration commentedon Apr 17, 2026
My install of ingress-nginx, the service did not have any of the service.beta.kubernetes.io/port_*_health-probe settings, which seems to work fine with
externalTrafficPolicy: local. So far with GatewayAPI with my setup as above, the service is similar, and seems to work fine so far.Perhaps there is some value to setting those properly when ETP is local, but i cannot configure it otherwise currently, so have no way of testing. Seems like the LB would be able to check the health port of the gateway pod perhaps, which is probably more optimal?
linnea-s commentedon Apr 17, 2026
I would like it to just work with
etp: local. Because it's now necessary to override the hardcoded health probe logic using the annotations, I suspect that the hardcoded health probe logic just does not work withetp: localand must either be removed or made conditional onetp: cluster.nshankar13 commentedon Apr 30, 2026
Alan B (@defenestration) Linnea Skogtvedt (@linnea-s) apologies for the delayed response, just discussed with the team today. The change to allowlist "healthCheckNodePort" is going to be rolled out in the next AKS release - we will track this and keep you posted, but expect this to fully roll out by mid / end May.
We want to avoid removing the current health probe logic since for ETP cluster, that is the correct logic and we want to avoid users tampering with that directly. But being able to set healthCheckNodePort for ETP local should take precedence over the hardcoded health probe logic (for ETP cluster).
Please give this a shot once the release rolls out and if this still doesn't work, we can consider tweaking the hardcoded logic.
defenestration commentedon May 1, 2026
Let us know and the implementation details. I think Ideally i'd like to just define
externalTrafficPolicy: Localand the chart would just not set theservice.beta.kubernetes.io/port_*_health-probe_*annotations due to that.Hashiraee commentedon May 6, 2026
Niranjan Shankar (@nshankar13) Something related to users using
externalTrafficPolicy: Local: I see the default setting fornumTrustedProxiesis1, which is not something we can overwrite, as stated here:So, taking a look at the default with
kubectl -n aks-istio-system get cm istio-asm-1-29 -o yaml:This means, even if I apply this:
the default will still be
numTrustedProxies: 1, even after a rollout restart (which, given what the documentation states, makes sense).However, this is not desired if someone uses the Azure L4 LB without an L7 in front of it. Also, given that in Istio's own documentation, the default is stated to be zero:
I also tried adding this to the annotations of the pod template (
deployment.spec.template.metadata.annotations):which also does not work (both when configured for a specific gateway and/or when configured for gateway-class-level settings), per the documented instructions here: https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#configure-gatewayclass-level-settings
So, I can imagine scenario's where it'd be nice to configure numTrustedProxies to be
0, or even2(depending on the setup). It would be nice if this option would be configurable as well.nshankar13 commentedon May 14, 2026
Hashiraee do you mind creating a separate issue for the meshconfig customization issue?
In the meanwhile, we started discussing this internally and are trying to find a solution
Tazmaniac commentedon May 19, 2026
Semi off topic but linked to the azmk8s.io automation too:
With the Following workaround,
externalTrafficPolicy: Local is fully functional on an Internet facing gatewayAPI gateway/Azure LB.
We don't want to publicly expose the istio-gateway status-port (15021).
The official documented way is to add a $patch statement to spec.port section of the gateway-options customization config map:
But we get:
error: gateways.gateway.networking.k8s.io "istio-gateway-public" could not be patched: admission webhook "istio-kube-gateway-validating-webhook.azmk8s.io" denied the request: ConfigMap aks-istio-ingress/gw-options for Gateway istio-gateway-public is invalid: ConfigMap validation failed: service validation failed: field Spec.Ports is not allowed but has 1 elementsThe workaround is to deploy the gateway without the Spec.Ports section in the customization confimap and later edit/patch the configmap to add Spec.Ports $patch: delete section, effectively bypassing azmk8s.io admission webhook.
But any later modification of the gateway resource will fail and the same dance will be need.
Could your revise the admission webhook to allow to remove a port ?
nshankar13 commentedon Jun 2, 2026
Linnea Skogtvedt (@linnea-s) Alan B (@defenestration) the
healthCheckNodePortallowlist change has been fully rolled out, would you be able to test this out now with etp: Local?nshankar13 commentedon Jun 2, 2026
Tazmaniac created a separate issue here to track that: #5797
Tazmaniac commentedon Jun 2, 2026
Niranjan Shankar (@nshankar13) Thank you !
nshankar13 commentedon Jun 16, 2026
Linnea Skogtvedt (@linnea-s) Alan B (@defenestration) we looked into this issue a bit more on our end and realized allowlisting the
healthCheckNodePortisn't sufficient, we need to add some custom logic to completely unset the hardcoded probes / protocols when ETP is Local. I thought before thathealthCheckNodePortwould take precedence over the hardcoded annotations when ETP is set to Local, but apparently that's not how Azure SLB works - SLB still uses the annotated health probes overhealthCheckNodePorteven when ETP is local.We're discussing some potential ways to fix this on our end right now. But for now, your solution of using the ConfigMap to unset the hardcoded annotations should work.
defenestration commentedon Jul 27, 2026
Commenting to keep this open for now