Skip to content

Increase TLSRoute hostnames limit from 16 to 1024 - #4332

Merged
k8s-ci-robot merged 1 commit into
kubernetes-sigs:mainfrom
sap-contributions:increase-tls-routes
May 8, 2026
Merged

k8s-ci-robot merged 1 commit into
kubernetes-sigs:mainfrom
sap-contributions:increase-tls-routes

Conversation

@alexanderstephan

@alexanderstephan alexanderstephan commented Dec 8, 2025 •

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind feature
/kind documentation

What this PR does / why we need it:
This PR updates the TLSRoute CRD validation rationale and proposes increasing the maxItem bound for hostnames from 16 to 4096. This change is proposed only to accommodate very large orgs. In large orgs, the current limit of 16, this can lead to hundred of thousands TLSRoute objects. These objects multiply storage, watch traffic, and controller memory/CPU, which drives up API-server latency and risks OOMs and instability.

In a similar PR the limit has been increased: #3205
To safely deploy the change the author employed a XValidation rule. For my case, such a rule would likely be rejected for being too complex. One idea would be to add the validation to a Validating Webhook. However, the webhook as been removed:

The validating webhook has been removed. CEL validation is now built-in to CRDs and replaces the webhook. (#2595, @robscott)

Do you have any ideas on how to solve this issue? I would be happy for further assistance on how to tackle this.

Rationale:

Which issue(s) this PR fixes:

No issue yet.

Does this PR introduce a user-facing change?:

The TLSRoute CRD validation has been adjusted to allow up to 4096 hostnames and rules per TLSRoute resource. Operators must validate kube-apiserver, etcd and Gateway controller behavior with representative manifests prior to enabling the new limit in production.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. kind/documentation Categorizes issue or PR as related to documentation. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 8, 2025
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Welcome @alexanderstephan!

It looks like this is your first PR to kubernetes-sigs/gateway-api πŸŽ‰. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gateway-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. πŸ˜ƒ

@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 8, 2025
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

Hi @alexanderstephan. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@alexanderstephan alexanderstephan changed the title Increase TLSRoute hostnames limit to 4096 Increase TLSRoute hostnames limit from 16 to 4096 Dec 8, 2025
@youngnick

youngnick commented Dec 9, 2025 •

Copy link
Copy Markdown
Contributor

Thanks for doing the calculations @alexanderstephan, that's helpful.

However, adding this would mean that there's no further headroom for adding any other constructs to TLSRoute. In other conversations, we've talked about adding ALPN matching, and a couple of other additional complexities that escape me right now, but if we use all the space available for storing hostnames, we'll have none left for expansion.

On top of that, I'm not sure how this would work. Because TLSRoute is about sending all traffic that matches the hostname list to a single backend, are you anticipating having that backend using a certificate that has up to 4096 SANs? That seems like a very large amount, that I'd be surprised if it's supported in most certificate handlers (it would certainly massively increase the size of the certificate).

I'd like to understand the use case you're aiming for here better. What sort of use cases involve serving 4000 hostnames from one backend service, presumably with a single certificate?

@youngnick

Copy link
Copy Markdown
Contributor

Also, we did stop shipping a validating webhook, because CEL does everything we want, generally, and the complexity cost calculations are also generally a good indication that we're keeping the API complexity under control.

@alexanderstephan

alexanderstephan commented Dec 12, 2025 •

Copy link
Copy Markdown
Contributor Author

Thanks for looking into this, @youngnick!

However, adding this would mean that there's no further headroom for adding any other constructs to TLSRoute. In other conversations, we've talked about adding ALPN matching, and a couple of other additional complexities that escape me right now, but if we use all the space available for storing hostnames, we'll have none left for expansion.

I see. So, you're suggesting we should lower the limit more, e.g., to 2048?

That seems like a very large amount, that I'd be surprised if it's supported in most certificate handlers (it would certainly massively increase the size of the certificate).

I think it actually possible to have 10k+ hostnames per certificate from what I have seen. However, this does not apply here for this case since we can also have multiple certificates as explained below.

I'd like to understand the use case you're aiming for here better. What sort of use cases involve serving 4000 hostnames from one backend service, presumably with a single certificate?

So, the umbrella topic here would be "multi-tenant SaaS with custom domains".
Here, a single backend shard can potentially serve thousands of tenant custom domains. Our deployments terminate TLS at the backend, selecting certs dynamically via SNI. So, TLSRoute’s role is to steer all those SNI matches to the right termination tier.
In this context, it makes sense to consolidate many low-traffic domains behind one backend as it is more efficient.

Also, we did stop shipping a validating webhook, because CEL does everything we want, generally, and the complexity cost calculations are also generally a good indication that we're keeping the API complexity under control.

Okay, that makes sense. I guess this can be looked into in the next step when the motivation for this change is more clear.

@alexanderstephan

Copy link
Copy Markdown
Contributor Author

Hi @youngnick, Do you have any insights for further steps on this issue? Thanks!

@k8s-triage-robot

Copy link
Copy Markdown

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 7, 2026
@youngnick

Copy link
Copy Markdown
Contributor

Hi @alexanderstephan, I'd prefer to keep the increase to a smaller one. 16 to 4096 is 8 orders of magnitude, that seems like too big a jump.

To rerun some of your calculations:

  • maximum hostname size is 256 characters
  • we need to allow at least 6, maybe 8 characters of overhead per hostname (indenting, - and maybe quotes). Let's go worst-case and call it 8.

4096 is, as you said : (256+8)*4096 = 1,081,344 bytes. That doesn't leave much overhead.

If we increase by 6 orders of magnitude, and go with 1024, then we have: (256+8)*1024 = 270,336 bytes.

I think that's a better place to start, and gives you a significant increase, while keeping plenty of space for further changes.

I do think you may find other limitations inside implementations when using large numbers of hostname entries like this, though.

@rikatz, ping for your thoughts here.

@youngnick

Copy link
Copy Markdown
Contributor

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 4, 2026
@alexanderstephan

Copy link
Copy Markdown
Contributor Author

Hi @youngnick, Thanks for looking into it again! Even 1024 would improve the situation a lot! Then, I would change it and also try to investigate for further side effects of this change.

@linux-foundation-easycla

linux-foundation-easycla Bot commented May 4, 2026 •

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 4, 2026
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels May 4, 2026
@rikatz

rikatz commented May 4, 2026

Copy link
Copy Markdown
Member

Can we make 128 first, and then move forward? I agree with Nick, making it 4096 is too much, and I think even 1024 is a lot and doesnΒ΄t give us much room to mistakes.

Thanks!

@Garfield96

Copy link
Copy Markdown

Multitenant environments can have 10000+ tenants, which would result in a lot of TLSRoute resources if only 128 entries are allowed. In my opinion, resource size is not big concern. If a user decides to have a lot of hostnames, it is the users responsibility to make sure it doesn't exceed the ETCD limits. In addition, the ETCD limit of 1.5 MiB is just the default. Users may decide to increase or decrease it, which would invalidate the calculations. Varying lengths of hostnames make the size impact even more unpredictable. I agree with @youngnick that other infrastructure components, like certificate handlers, might not be able to handle a very large amount of hostnames. However, this problem is completely unpredictable and I believe this also falls into the responsibility of the developer.

Hence, I support the proposal of @youngnick to start with 1024 entries.

@rikatz

rikatz commented May 6, 2026

Copy link
Copy Markdown
Member

sounds good, I think we can at least stick with it on 1024 then

Can you please rebase and fix the conflicts?

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 6, 2026
@alexanderstephan alexanderstephan changed the title Increase TLSRoute hostnames limit from 16 to 4096 Increase TLSRoute hostnames limit from 16 to 1024 May 7, 2026
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 7, 2026
@alexanderstephan

Copy link
Copy Markdown
Contributor Author

Hi @rikatz, I updated all occurrences I found and rebased onto the main branch. Please verify whether all changes are at right place as I lack the familiarity with API structure. Thanks!

@youngnick

Copy link
Copy Markdown
Contributor

Note that, even once it's merged, this will not take effect until the release of v1.6, due end of June. We don't backport changes to earlier versions.

@rikatz

rikatz commented May 8, 2026

Copy link
Copy Markdown
Member

/lgtm
/approve
thanks. As Nick pointed, this will be part of 1.6 release

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 8, 2026
@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexanderstephan, rikatz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 8, 2026
@alexanderstephan

Copy link
Copy Markdown
Contributor Author

@youngnick, @rikatz
No issue, we're not in a hurry. But very happy to see this coming, thanks guys! πŸŽ‰

@k8s-ci-robot
k8s-ci-robot merged commit a0b01b6 into kubernetes-sigs:main May 8, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/documentation Categorizes issue or PR as related to documentation. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants