Skip to content

[BUG] Istio GatewayApi will get incorrect values for X-Forwarded-For headerΒ #5667

Closed

Description

Describe the bug

I went through setting up Istio GatewayApi looking to move from ingress-nginx.

I noticed that occasionally, the Istio GatewayApi will get incorrect values for X-Forwarded-For header and downstream_local_address in the logs.

I've traced this back to the Kubernetes Service crated by the Gateway will have externalTrafficPolicy: Cluster and not Local as suggested by this doc https://learn.microsoft.com/en-us/azure/aks/configure-load-balancer-standard?tabs=create-cluster-ip-based%2Ccreate-cluster-managed-outbound-ips%2Ccreate-cluster-custom-ips%2Ccreate-cluster-custom-ip-prefixes%2Ccreate-cluster-outbound-ports-ips%2Ccreate-cluster-idle-timeout#maintain-the-clients-ip-on-inbound-connections

This seems to result in:

  • if the Azure LoadBalancer sends the request to a Node that has a Gateway Pod, it will show the correct X-forwarded-for IP, (edit: sometimes it will still return the Node Ip i am finding out)
  • But if it goes to a Node without such a pod, it will still serve the request, but X-Forwarded-For headers and logs will show the IP of the Node. I believe it hits the kube-proxy service running on the node?

I tinkered with setting the service's externalTrafficPolicy: Local, however this causes issues. The Service has built-in annotations that seem to manage the health-probe_port for the load balancer. I tried to override these with the value of the service .spec.healthCheckNodePort after setting externalTrafficPolicy: Local on the Gateway using spec.infrastructure.annotations."service.beta.kubernetes.io/port_443_health-probe_port" but it did not seem to take. I can set other annotations like service.beta.kubernetes.io/azure-load-balancer-ipv4 and that seems to work correctly so there seems to be something overriding that.

service.beta.kubernetes.io/port_443_health-probe_port: "10256"
service.beta.kubernetes.io/port_443_health-probe_protocol: http
service.beta.kubernetes.io/port_443_health-probe_request-path: /healthz

To Reproduce

Expected behavior

Would like the proper client IP in the X-Forwarded-For header and logs.

Screenshots
If applicable, add screenshots to help explain your problem. Make sure not to include sensitive or personal information.

Environment (please complete the following information):

  • Kubernetes version [e.g. 1.33.7]
  • istio service mesh addon 1-28

Additional context
Add any other context about the problem here.

I found this article https://learn.microsoft.com/en-us/azure/aks/shared-health-probes?tabs=existing-cluster-shared-probe-mode which seemed related, but this option is not enabled on the cluster currently.

Activity

  1. defenestration commented on Mar 20, 2026

    @defenestration
    Author

    My ugly workaround for this at the moment is to bascially copy the original deployment/service/serviceaccount under a new name and use the Istio setting in the gateway to manually setup those things without the annotations described above at https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/#manual-deployment

    I can see a template setup in the helmchart in aks-istio-system that suggests these annotations in the service aren't currently overridable.

    helm get manifest -n aks-istio-system  azure-service-mesh-istio-discovery | grep health-probe
            {{- $serviceAnnotations = mergeMaps $serviceAnnotations (strdict (printf "service.beta.kubernetes.io/port_%d_health-probe_protocol" $port) "http" (printf "service.beta.kubernetes.io/port_%d_health-probe_port" $port) "10256" (printf "service.beta.kubernetes.io/port_%d_health-probe_request-path" $port) "/healthz") }}
  2. defenestration commented on Mar 20, 2026

    @defenestration
    Author

    I found something better that might work and plays nicer than rolling our own deployment/service.

    in the configmap istio-gateway-class-defaults in namespace aks-istio-system you can set defaults for all gateways.

    I set the value for the problem annotations to blank, this seems to not have it render in the service config, which causes the default setup for the loadbalancer to kick in. spec.externalTrafficPolicy can also be set here.

    This applies to every gateway, but in our case that is what we want. For port 443 we probably want similar. but this seems promising.

    service: |
      metadata:
        annotations:
          stuff: things
          service.beta.kubernetes.io/port_80_health-probe_port:
          service.beta.kubernetes.io/port_80_health-probe_protocol: 
          service.beta.kubernetes.io/port_80_health-probe_request-path:
      spec:
        externalTrafficPolicy: Local
    

    more notes:

    There seems to be per-gateway way to set this with a config map but i feel like i tried that and it did not work.

    Setting spec.infrastructure.annotations on the gateway for this does not work for these either, though you can add other annotations.

  3. defenestration commented on Mar 23, 2026

    @defenestration
    Author

    Noting that setting those blank annotations using a per-gateway configmap seems to work as well.
    https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#configure-settings-for-a-specific-gateway

  4. linnea-s commented on Mar 30, 2026

    @linnea-s

    I'm not sure if it's relevant for this issue but I found that there is the following default (reported on startup of gateway pods):

    gatewayTopology:
      numTrustedProxies: 1
    

    This means that there is one trusted proxy in front of the gateway which can send the client IP address in the X-Forwarded-For header. This is appropriate when using Azure Application Gateway but if you only have a TCP load balancer in front of the gateway then I believe clients can pass anything they want in the X-Forwarded-For header and it will be trusted. See https://istio.io/latest/docs/ops/configuration/traffic-management/network-topologies/

    I think that should be documented somewhere.

    By the way, using Gateway API with Application Gateway in front and following instructions at https://learn.microsoft.com/en-us/azure/application-gateway/rewrite-http-headers-url#remove-port-information-from-the-x-forwarded-for-header, I get the correct remote client IP despite using externalTrafficPolicy Cluster.

  5. linnea-s commented on Apr 1, 2026

    @linnea-s

    I am also evaluating istio as a replacement for ingress-nginx and found a use-case where I need externalTrafficPolicy: Local. I need to be able to restrict the client IP either in NetworkPolicy or in AuthorizationPolicy. That is only possible when using externalTrafficPolicy: Local. I can confirm that it works to use a ConfigMap gateway-options like this (thanks Alan B (@defenestration)):

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: gateway-options
      namespace: gateways
    data:
      service: |
        spec:
          externalTrafficPolicy: Local
        metadata:
          annotations:
            service.beta.kubernetes.io/port_80_health-probe_port:
            service.beta.kubernetes.io/port_80_health-probe_protocol: 
            service.beta.kubernetes.io/port_80_health-probe_request-path:
            service.beta.kubernetes.io/port_443_health-probe_port:
            service.beta.kubernetes.io/port_443_health-probe_protocol: 
            service.beta.kubernetes.io/port_443_health-probe_request-path:

    And the Gateway then contains:

    spec:
      gatewayClassName: istio
      infrastructure:
        parametersRef:
          group: ""
          kind: ConfigMap
          name: gateway-options
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: 'true'

    As far as I can tell the Gateway always receives the correct source IP otherwise my NetworkPolicy would not work.

    However there's an important missing piece: According to https://blog.aks.azure.com/2025/04/04/optimize-aks-traffic-with-externaltrafficpolicy-local#best-practices-for-rolling-updates-and-pod-rotation we must set minReadySeconds to avoid disruption during rolling upgrades. That is not possible because it is not allowlisted: https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#deployment-customization-allow-list-fields

    In summary AKS team, please:

    1. Remove the need for the hack to override the faulty health probe
    2. Allow to set deployment spec.minReadySeconds
  6. nshankar13 commented on Apr 15, 2026

    @nshankar13
    Contributor

    I believe it hits the kube-proxy service running on the node?

    Yes - for ETP: cluster, we needed to set the health probe port annotations directly due to a preview issue with Azure Load Balancer's health probing logic: #3646. So for Gateway API, we need to set it to the port for kube-proxy, so that SLB checks the health of the kube-proxy instance running on that node before forwarding traffic to that node - then kube-proxy will forward it to the correct backend instance depending.

    For ETP: local, SLB will probe the healthCheckNodePort directly on the nodes where the backend pods are running.

    However, right now we don't have healthCheckNodePort as part of our allowlist for the Gateway service customization, which is possibly why your changes haven't taken effect: https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#service-customization-allow-list-fields. We recently added a change for to allowlist this, but it hasn't rolled out yet - we are trying to get this out in the next AKS release or so.

    Linnea Skogtvedt (@linnea-s) Alan B (@defenestration) to clarify - if you're able to set healthCheckNodePort with etp: local, will you still need the hardcoded health probe logic unset from the Service spec? We may be able to get rid of the health probe settings down the road (it's not required after a certain AKS version - trying to determine that right now) - but not sure the exact ETA on this.

    As for deployment spec.minReadySeconds - yes we can look into adding this - will keep you posted.

  7. defenestration commented on Apr 17, 2026

    @defenestration
    Author

    My install of ingress-nginx, the service did not have any of the service.beta.kubernetes.io/port_*_health-probe settings, which seems to work fine with externalTrafficPolicy: local. So far with GatewayAPI with my setup as above, the service is similar, and seems to work fine so far.

    Perhaps there is some value to setting those properly when ETP is local, but i cannot configure it otherwise currently, so have no way of testing. Seems like the LB would be able to check the health port of the gateway pod perhaps, which is probably more optimal?

  8. linnea-s commented on Apr 17, 2026

    @linnea-s

    @linnea-s @defenestration to clarify - if you're able to set healthCheckNodePort with etp: local, will you still need the hardcoded health probe logic unset from the Service spec? We may be able to get rid of the health probe settings down the road (it's not required after a certain AKS version - trying to determine that right now) - but not sure the exact ETA on this.

    I would like it to just work with etp: local. Because it's now necessary to override the hardcoded health probe logic using the annotations, I suspect that the hardcoded health probe logic just does not work with etp: local and must either be removed or made conditional on etp: cluster.

  9. nshankar13 commented on Apr 30, 2026

    @nshankar13
    Contributor

    Alan B (@defenestration) Linnea Skogtvedt (@linnea-s) apologies for the delayed response, just discussed with the team today. The change to allowlist "healthCheckNodePort" is going to be rolled out in the next AKS release - we will track this and keep you posted, but expect this to fully roll out by mid / end May.

    We want to avoid removing the current health probe logic since for ETP cluster, that is the correct logic and we want to avoid users tampering with that directly. But being able to set healthCheckNodePort for ETP local should take precedence over the hardcoded health probe logic (for ETP cluster).

    Please give this a shot once the release rolls out and if this still doesn't work, we can consider tweaking the hardcoded logic.

  10. defenestration commented on May 1, 2026

    @defenestration
    Author

    Let us know and the implementation details. I think Ideally i'd like to just define externalTrafficPolicy: Local and the chart would just not set the service.beta.kubernetes.io/port_*_health-probe_* annotations due to that.

  11. Hashiraee commented on May 6, 2026

    @Hashiraee

    Niranjan Shankar (@nshankar13) Something related to users using externalTrafficPolicy: Local: I see the default setting for numTrustedProxies is 1, which is not something we can overwrite, as stated here:

    ... the aks-istio-system namespace, and then the Istio control plane will merge this with the default ConfigMap, with the default settings taking precedence.

    So, taking a look at the default with kubectl -n aks-istio-system get cm istio-asm-1-29 -o yaml:

    apiVersion: v1
    data:
      mesh: |-
        defaultConfig:
          discoveryAddress: istiod-asm-1-29.aks-istio-system.svc:15012
          gatewayTopology:
            numTrustedProxies: 1
          image:
            imageType: distroless
        defaultProviders:
          metrics:
          - prometheus
        enablePrometheusMerge: true
        rootNamespace: aks-istio-system
        trustDomain: cluster.local
      meshNetworks: 'networks: {}'
    kind: ConfigMap
    ...
    

    This means, even if I apply this:

    cat <<EOF > istio-shared-configmap-asm-1-29.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: istio-shared-configmap-asm-1-29
      namespace: aks-istio-system
    data:
      mesh: |-
        defaultConfig:
          gatewayTopology:
            numTrustedProxies: 0
    EOF
    kubectl apply -f istio-shared-configmap-asm-1-29.yaml
    

    the default will still be numTrustedProxies: 1, even after a rollout restart (which, given what the documentation states, makes sense).

    However, this is not desired if someone uses the Azure L4 LB without an L7 in front of it. Also, given that in Istio's own documentation, the default is stated to be zero:

    ... The default value of numTrustedProxies is 0. See Envoy XFF header handling for more details.

    I also tried adding this to the annotations of the pod template (deployment.spec.template.metadata.annotations):

    ...
      metadata:
        annotations:
          "proxy.istio.io/config": '{"gatewayTopology" : { "numTrustedProxies": 0 } }'
    

    which also does not work (both when configured for a specific gateway and/or when configured for gateway-class-level settings), per the documented instructions here: https://learn.microsoft.com/en-us/azure/aks/istio-gateway-api#configure-gatewayclass-level-settings

    So, I can imagine scenario's where it'd be nice to configure numTrustedProxies to be 0, or even 2 (depending on the setup). It would be nice if this option would be configurable as well.

  12. nshankar13 commented on May 14, 2026

    @nshankar13
    Contributor

    Hashiraee do you mind creating a separate issue for the meshconfig customization issue?

    In the meanwhile, we started discussing this internally and are trying to find a solution

  13. Tazmaniac commented on May 19, 2026

    @Tazmaniac

    Semi off topic but linked to the azmk8s.io automation too:
    With the Following workaround,

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: gateway-options
      namespace: gateways
    data:
      service: |
        spec:
          externalTrafficPolicy: Local
        metadata:
          annotations:
            service.beta.kubernetes.io/port_80_health-probe_port:
            service.beta.kubernetes.io/port_80_health-probe_protocol: 
            service.beta.kubernetes.io/port_80_health-probe_request-path:
            service.beta.kubernetes.io/port_443_health-probe_port:
            service.beta.kubernetes.io/port_443_health-probe_protocol: 
            service.beta.kubernetes.io/port_443_health-probe_request-path:
    

    externalTrafficPolicy: Local is fully functional on an Internet facing gatewayAPI gateway/Azure LB.

    We don't want to publicly expose the istio-gateway status-port (15021).
    The official documented way is to add a $patch statement to spec.port section of the gateway-options customization config map:

    ...
        spec:
          externalTrafficPolicy: Local
          ports:
          - $patch: delete
            port: 15021
    ...
    

    But we get:
    error: gateways.gateway.networking.k8s.io "istio-gateway-public" could not be patched: admission webhook "istio-kube-gateway-validating-webhook.azmk8s.io" denied the request: ConfigMap aks-istio-ingress/gw-options for Gateway istio-gateway-public is invalid: ConfigMap validation failed: service validation failed: field Spec.Ports is not allowed but has 1 elements

    The workaround is to deploy the gateway without the Spec.Ports section in the customization confimap and later edit/patch the configmap to add Spec.Ports $patch: delete section, effectively bypassing azmk8s.io admission webhook.
    But any later modification of the gateway resource will fail and the same dance will be need.

    Could your revise the admission webhook to allow to remove a port ?

  14. nshankar13 commented on Jun 2, 2026

    @nshankar13
    Contributor

    Linnea Skogtvedt (@linnea-s) Alan B (@defenestration) the healthCheckNodePort allowlist change has been fully rolled out, would you be able to test this out now with etp: Local?

  15. nshankar13 commented on Jun 2, 2026

    @nshankar13
    Contributor

    Tazmaniac created a separate issue here to track that: #5797

  16. Tazmaniac commented on Jun 2, 2026

    @Tazmaniac
  17. nshankar13 commented on Jun 16, 2026

    @nshankar13
    Contributor

    Linnea Skogtvedt (@linnea-s) Alan B (@defenestration) we looked into this issue a bit more on our end and realized allowlisting the healthCheckNodePort isn't sufficient, we need to add some custom logic to completely unset the hardcoded probes / protocols when ETP is Local. I thought before that healthCheckNodePort would take precedence over the hardcoded annotations when ETP is set to Local, but apparently that's not how Azure SLB works - SLB still uses the annotated health probes over healthCheckNodePort even when ETP is local.

    We're discussing some potential ways to fix this on our end right now. But for now, your solution of using the ConfigMap to unset the hardcoded annotations should work.

  18. defenestration commented on Jul 27, 2026

    @defenestration
    Author

    Commenting to keep this open for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions