Skip to content

[BUG] LoadBalancer services use incorrect readiness logic with appProtocol=httpΒ #3646

Closed

Description

Describe the bug
Service YAML:

apiVersion: v1
kind: Service
metadata:
  name: server
spec:
  type: LoadBalancer
  selector:
    app: server
  ports:
    - name: http
      appProtocol: http
      port: 80

Per Kubernetes spec, endpoints of the service should be determined by the Endpoints/EndpointSlice health, which is determined by the pod's readinessProbe.

However, per https://cloud-provider-azure.sigs.k8s.io/topics/loadbalancer/#custom-load-balancer-health-probe, this is not the case for this service. Instead, HTTP requests to / will be sent to port 80 to determine health. This seems to violate the Service spec.

From a practical standpoint, this breaks usage of Istio Gateway-API, which create Services like this. However, we do not respond to GET / on port 80 (unless the user configures a route matching that, of course). If we do respond to it, we are likely forwarding it to some other service, which is fairly incorrect.

The end result is that users of Istio Gateway on Azure will see traffic not work due to failing health check. We can recommend users manually set a bunch of annotations to make things work, but this is tedious for users and leads to different usage for Azure vs other platforms. We could automatically detect + insert these for azure users, but we prefer to avoid vendor specific workarounds.

Activity

  1. keithmattix commented on May 9, 2023

    @keithmattix
    Contributor
  2. feiskyer commented on May 10, 2023

    @feiskyer
    Member

    Please follow https://cloud-provider-azure.sigs.k8s.io/topics/loadbalancer/#custom-load-balancer-health-probe to set the correct request path with annotations if the default request path doesn't work.

  3. howardjohn commented on May 10, 2023

    @howardjohn
    Author

    Thanks Pengfei Ni (@feiskyer). I am aware of how to fix it as a user. However, I am not a user but a maintainer of Istio (which, btw, now is shipped with Azure), and would like Istio to work out the box on Azure, as it does on other K8s platforms without relying on azure-specific annotations.

  4. biefy commented on May 10, 2023

    @biefy
    Contributor

    We are taking a look. Thanks John Howard (@howardjohn) for reporting this issue.

  5. AndrewFarley commented on Aug 4, 2023

    @AndrewFarley

    Just had this hit me as well and wasted about a day debugging specifically why.

    I was experimenting with Kubernetes on Azure and tried my preferred ingress controller Ingress-Nginx and it would never work and there were no event logs or logs of any kind as to why this failure occurred just an inaccessibly not actually public Service.

    Frustratingly, I eventually took a step back and tried a tutorial from Microsoft with Terraform here which seemed to work, eventually diff-ing and distilling it down to the fact that that controller and many modern controllers now specify this appProtocol=http by default. Thankfully, that controller has a flag to disable it so I can now use it without issue, but this is a fairly aggravating stain and blocker to adopt and utilize Azure AKS and alas has caused me wasted time and I'm sure will continue to cause others until this is fixed.

    Please fix! At the very least just ignore this new option and don't completely block traffic, that should be a bare minimum.

    Azure Kubernetes version (if it matters): 1.25.11

  6. jtv8 commented on Aug 30, 2023

    @jtv8

    If it helps anyone, here's a signpost to the relevant source code in the provider.

    Does this need a corresponding issue at kubernetes-sigs/cloud-provider-azure?

  7. akshayjnambiar commented on Jan 5, 2024

    @akshayjnambiar

    Do we have any update on this issue?

  8. biefy commented on Jan 12, 2024

    @biefy
    Contributor

    We are testing an internal fix from AKS.

  9. feiskyer commented on Jan 16, 2024

    @feiskyer
    Member

    Update: a new shared LB health probe would be introduced in cloud provider azure v1.29.0: feat: support shared load balancer health probe mode. By setting clusterServiceLoadBalancerHealthProbeMode to shared, all cluster services will share one health probe targeting the kube-proxy port 10256 and /healthz by default. The health check port and path can be configured by clusterServiceSharedLoadBalancerHealthProbePort and clusterServiceSharedLoadBalancerHealthProbePort.

    Refer kubernetes-sigs/cloud-provider-azure#4891 for the details.

  10. nathanweeks commented on Apr 19, 2024

    @nathanweeks

    Does this update to AKS k8s >= 1.28.5 imply the Gateway API should work with the Istio-based service mesh add-on for AKS as well? The docs currently mention the following limitation:

    • Gateway API for Istio ingress gateway or managing mesh traffic (GAMMA) are currently not yet supported with Istio addon.
  11. 7 remaining items

  12. Stevenjin8 commented on Jun 9, 2025

    @Stevenjin8

    Not stale. Can this be stale proof please

  13. sjwaight commented on Jun 10, 2025

    @sjwaight
    Contributor

    Fuyuan Bie (@biefy) are you able to add some context given your comment from January 2024.

  14. biefy commented on Jun 27, 2025

    @biefy
    Contributor

    In AKS Istio addon, starting from Istio 1.22, in gateway api injection template, service's health check path has been set to probing kube-proxy (:10256 /healthz).
    If you want the whole cluster's LoadBalancer typed services health check pointing to above path, change the cluster's loadBalancerProfile.clusterServiceLoadBalancerHealthProbeMode to Shared.

    az aks update --name my-cluster
        --resource-group rg-my-cluster
        --cluster-service-load-balancer-health-probe-mode Shared

    ARM API:
    https://learn.microsoft.com/en-us/azure/templates/microsoft.containerservice/managedclusters?pivots=deployment-language-arm-template#managedclusterloadbalancerprofile-1

  15. Stevenjin8 commented on Jul 1, 2025

    @Stevenjin8

    Fuyuan Bie (@biefy) Simon Waight (@sjwaight) can we remove the resolution/fixed-release label? I realize there is a workaround, but the main issue is that there needs to be a workaround at all. Trying to debug an issue with gateway connectivity is incredibly frustrating when there are no helpful error messages. Does the error lie in the Gateway CRDs? The k8s Service? The running pods? The installation of the GatewayClass? My own ISP? The Azure Load Balancer health check is probably the last place I would look.

    Its also not just Istio that has GWAPI getting started docs that do not have an HTTPRoute for /. NGINX Fabric has similar docs.

    I also wonder if its correct for a path (?) to be marked unhealthy just because / responds with a 404. 404 is a client side error

  16. microsoft-github-policy-service commented on Jul 9, 2025

    @microsoft-github-policy-service
    Contributor

    Thanks for reaching out. I'm closing this issue as it was marked with "resolution/fix-released" and it hasn't had activity for 7 days.

  17. keithmattix commented on Dec 18, 2025

    @keithmattix
    Contributor

    Reopening just to make sure we got closure here. Is the official supported solution to set loadBalancerProfile.clusterServiceLoadBalancerHealthProbeMode to Shared

  18. microsoft-github-policy-service commented on Dec 26, 2025

    @microsoft-github-policy-service
    Contributor

    Thanks for reaching out. I'm closing this issue as it was marked with "resolution/fix-released" and it hasn't had activity for 7 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions