You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] LoadBalancer services use incorrect readiness logic with appProtocol=httpΒ #3646
Per Kubernetes spec, endpoints of the service should be determined by the Endpoints/EndpointSlice health, which is determined by the pod's readinessProbe.
From a practical standpoint, this breaks usage of Istio Gateway-API, which create Services like this. However, we do not respond to GET / on port 80 (unless the user configures a route matching that, of course). If we do respond to it, we are likely forwarding it to some other service, which is fairly incorrect.
The end result is that users of Istio Gateway on Azure will see traffic not work due to failing health check. We can recommend users manually set a bunch of annotations to make things work, but this is tedious for users and leads to different usage for Azure vs other platforms. We could automatically detect + insert these for azure users, but we prefer to avoid vendor specific workarounds.
Thanks Pengfei Ni (@feiskyer). I am aware of how to fix it as a user. However, I am not a user but a maintainer of Istio (which, btw, now is shipped with Azure), and would like Istio to work out the box on Azure, as it does on other K8s platforms without relying on azure-specific annotations.
Just had this hit me as well and wasted about a day debugging specifically why.
I was experimenting with Kubernetes on Azure and tried my preferred ingress controller Ingress-Nginx and it would never work and there were no event logs or logs of any kind as to why this failure occurred just an inaccessibly not actually public Service.
Frustratingly, I eventually took a step back and tried a tutorial from Microsoft with Terraform here which seemed to work, eventually diff-ing and distilling it down to the fact that that controller and many modern controllers now specify this appProtocol=http by default. Thankfully, that controller has a flag to disable it so I can now use it without issue, but this is a fairly aggravating stain and blocker to adopt and utilize Azure AKS and alas has caused me wasted time and I'm sure will continue to cause others until this is fixed.
Please fix! At the very least just ignore this new option and don't completely block traffic, that should be a bare minimum.
Update: a new shared LB health probe would be introduced in cloud provider azure v1.29.0: feat: support shared load balancer health probe mode. By setting clusterServiceLoadBalancerHealthProbeMode to shared, all cluster services will share one health probe targeting the kube-proxy port 10256 and /healthz by default. The health check port and path can be configured by clusterServiceSharedLoadBalancerHealthProbePort and clusterServiceSharedLoadBalancerHealthProbePort.
Does this update to AKS k8s >= 1.28.5 imply the Gateway API should work with the Istio-based service mesh add-on for AKS as well? The docs currently mention the following limitation:
Gateway API for Istio ingress gateway or managing mesh traffic (GAMMA) are currently not yet supported with Istio addon.
In AKS Istio addon, starting from Istio 1.22, in gateway api injection template, service's health check path has been set to probing kube-proxy (:10256 /healthz).
If you want the whole cluster's LoadBalancer typed services health check pointing to above path, change the cluster's loadBalancerProfile.clusterServiceLoadBalancerHealthProbeMode to Shared.
az aks update --name my-cluster
--resource-group rg-my-cluster
--cluster-service-load-balancer-health-probe-mode Shared
Fuyuan Bie (@biefy)Simon Waight (@sjwaight) can we remove the resolution/fixed-release label? I realize there is a workaround, but the main issue is that there needs to be a workaround at all. Trying to debug an issue with gateway connectivity is incredibly frustrating when there are no helpful error messages. Does the error lie in the Gateway CRDs? The k8s Service? The running pods? The installation of the GatewayClass? My own ISP? The Azure Load Balancer health check is probably the last place I would look.
Its also not just Istio that has GWAPI getting started docs that do not have an HTTPRoute for /. NGINX Fabric has similar docs.
I also wonder if its correct for a path (?) to be marked unhealthy just because / responds with a 404. 404 is a client side error
Reacted by Keith Mattix II, Mike Morris and Aaron Lucas
microsoft-github-policy-service commented on Jul 9, 2025
Reopening just to make sure we got closure here. Is the official supported solution to set loadBalancerProfile.clusterServiceLoadBalancerHealthProbeMode to Shared
microsoft-github-policy-service commented on Dec 26, 2025
Activity
keithmattix commentedon May 9, 2023
/cc Shashank Barsin (@shashankbarsin)
shashankbarsin commentedon May 9, 2023
Pengfei Ni (@feiskyer), Paul Miller (@paulgmiller), Chase Wilson (@chasewilson)
feiskyer commentedon May 10, 2023
Please follow https://cloud-provider-azure.sigs.k8s.io/topics/loadbalancer/#custom-load-balancer-health-probe to set the correct request path with annotations if the default request path doesn't work.
howardjohn commentedon May 10, 2023
Thanks Pengfei Ni (@feiskyer). I am aware of how to fix it as a user. However, I am not a user but a maintainer of Istio (which, btw, now is shipped with Azure), and would like Istio to work out the box on Azure, as it does on other K8s platforms without relying on azure-specific annotations.
biefy commentedon May 10, 2023
We are taking a look. Thanks John Howard (@howardjohn) for reporting this issue.
AndrewFarley commentedon Aug 4, 2023
Just had this hit me as well and wasted about a day debugging specifically why.
I was experimenting with Kubernetes on Azure and tried my preferred ingress controller Ingress-Nginx and it would never work and there were no event logs or logs of any kind as to why this failure occurred just an inaccessibly not actually public Service.
Frustratingly, I eventually took a step back and tried a tutorial from Microsoft with Terraform here which seemed to work, eventually diff-ing and distilling it down to the fact that that controller and many modern controllers now specify this
appProtocol=httpby default. Thankfully, that controller has a flag to disable it so I can now use it without issue, but this is a fairly aggravating stain and blocker to adopt and utilize Azure AKS and alas has caused me wasted time and I'm sure will continue to cause others until this is fixed.Please fix! At the very least just ignore this new option and don't completely block traffic, that should be a bare minimum.
Azure Kubernetes version (if it matters): 1.25.11
jtv8 commentedon Aug 30, 2023
If it helps anyone, here's a signpost to the relevant source code in the provider.
Does this need a corresponding issue at kubernetes-sigs/cloud-provider-azure?
akshayjnambiar commentedon Jan 5, 2024
Do we have any update on this issue?
biefy commentedon Jan 12, 2024
We are testing an internal fix from AKS.
feiskyer commentedon Jan 16, 2024
Update: a new shared LB health probe would be introduced in cloud provider azure v1.29.0: feat: support shared load balancer health probe mode. By setting
clusterServiceLoadBalancerHealthProbeModetoshared, all cluster services will share one health probe targeting the kube-proxy port 10256 and /healthz by default. The health check port and path can be configured byclusterServiceSharedLoadBalancerHealthProbePortandclusterServiceSharedLoadBalancerHealthProbePort.Refer kubernetes-sigs/cloud-provider-azure#4891 for the details.
nathanweeks commentedon Apr 19, 2024
Does this update to AKS k8s >= 1.28.5 imply the Gateway API should work with the Istio-based service mesh add-on for AKS as well? The docs currently mention the following limitation:
7 remaining items
Stevenjin8 commentedon Jun 9, 2025
Not stale. Can this be stale proof please
sjwaight commentedon Jun 10, 2025
Fuyuan Bie (@biefy) are you able to add some context given your comment from January 2024.
biefy commentedon Jun 27, 2025
In AKS Istio addon, starting from Istio 1.22, in gateway api injection template, service's health check path has been set to probing kube-proxy (
:10256 /healthz).If you want the whole cluster's LoadBalancer typed services health check pointing to above path, change the cluster's
loadBalancerProfile.clusterServiceLoadBalancerHealthProbeModetoShared.az aks update --name my-cluster --resource-group rg-my-cluster --cluster-service-load-balancer-health-probe-mode SharedARM API:
https://learn.microsoft.com/en-us/azure/templates/microsoft.containerservice/managedclusters?pivots=deployment-language-arm-template#managedclusterloadbalancerprofile-1
Stevenjin8 commentedon Jul 1, 2025
Fuyuan Bie (@biefy) Simon Waight (@sjwaight) can we remove the resolution/fixed-release label? I realize there is a workaround, but the main issue is that there needs to be a workaround at all. Trying to debug an issue with gateway connectivity is incredibly frustrating when there are no helpful error messages. Does the error lie in the Gateway CRDs? The k8s Service? The running pods? The installation of the GatewayClass? My own ISP? The Azure Load Balancer health check is probably the last place I would look.
Its also not just Istio that has GWAPI getting started docs that do not have an HTTPRoute for
/. NGINX Fabric has similar docs.I also wonder if its correct for a path (?) to be marked unhealthy just because
/responds with a 404. 404 is a client side errormicrosoft-github-policy-service commentedon Jul 9, 2025
Thanks for reaching out. I'm closing this issue as it was marked with "resolution/fix-released" and it hasn't had activity for 7 days.
keithmattix commentedon Dec 18, 2025
Reopening just to make sure we got closure here. Is the official supported solution to set
loadBalancerProfile.clusterServiceLoadBalancerHealthProbeModeto Sharedmicrosoft-github-policy-service commentedon Dec 26, 2025
Thanks for reaching out. I'm closing this issue as it was marked with "resolution/fix-released" and it hasn't had activity for 7 days.