Repository navigation
Increase TLSRoute hostnames limit from 16 to 1024 - #4332
Conversation
|
Welcome @alexanderstephan! |
|
Hi @alexanderstephan. Thanks for your PR. I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Thanks for doing the calculations @alexanderstephan, that's helpful. However, adding this would mean that there's no further headroom for adding any other constructs to TLSRoute. In other conversations, we've talked about adding ALPN matching, and a couple of other additional complexities that escape me right now, but if we use all the space available for storing hostnames, we'll have none left for expansion. On top of that, I'm not sure how this would work. Because TLSRoute is about sending all traffic that matches the I'd like to understand the use case you're aiming for here better. What sort of use cases involve serving 4000 hostnames from one backend service, presumably with a single certificate? |
|
Also, we did stop shipping a validating webhook, because CEL does everything we want, generally, and the complexity cost calculations are also generally a good indication that we're keeping the API complexity under control. |
|
Thanks for looking into this, @youngnick!
I see. So, you're suggesting we should lower the limit more, e.g., to 2048?
I think it actually possible to have 10k+ hostnames per certificate from what I have seen. However, this does not apply here for this case since we can also have multiple certificates as explained below.
So, the umbrella topic here would be "multi-tenant SaaS with custom domains".
Okay, that makes sense. I guess this can be looked into in the next step when the motivation for this change is more clear. |
|
Hi @youngnick, Do you have any insights for further steps on this issue? Thanks! |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
|
Hi @alexanderstephan, I'd prefer to keep the increase to a smaller one. 16 to 4096 is 8 orders of magnitude, that seems like too big a jump. To rerun some of your calculations:
4096 is, as you said : (256+8)*4096 = 1,081,344 bytes. That doesn't leave much overhead. If we increase by 6 orders of magnitude, and go with 1024, then we have: (256+8)*1024 = 270,336 bytes. I think that's a better place to start, and gives you a significant increase, while keeping plenty of space for further changes. I do think you may find other limitations inside implementations when using large numbers of @rikatz, ping for your thoughts here. |
|
/remove-lifecycle stale |
|
Hi @youngnick, Thanks for looking into it again! Even 1024 would improve the situation a lot! Then, I would change it and also try to investigate for further side effects of this change. |
987a6af to
3a6c41b
Compare
|
Can we make 128 first, and then move forward? I agree with Nick, making it 4096 is too much, and I think even 1024 is a lot and doesnΒ΄t give us much room to mistakes. Thanks! |
|
Multitenant environments can have 10000+ tenants, which would result in a lot of TLSRoute resources if only 128 entries are allowed. In my opinion, resource size is not big concern. If a user decides to have a lot of hostnames, it is the users responsibility to make sure it doesn't exceed the ETCD limits. In addition, the ETCD limit of 1.5 MiB is just the default. Users may decide to increase or decrease it, which would invalidate the calculations. Varying lengths of hostnames make the size impact even more unpredictable. I agree with @youngnick that other infrastructure components, like certificate handlers, might not be able to handle a very large amount of hostnames. However, this problem is completely unpredictable and I believe this also falls into the responsibility of the developer. Hence, I support the proposal of @youngnick to start with 1024 entries. |
|
sounds good, I think we can at least stick with it on 1024 then Can you please rebase and fix the conflicts? /ok-to-test |
3a6c41b to
b5762f4
Compare
|
Hi @rikatz, I updated all occurrences I found and rebased onto the |
|
Note that, even once it's merged, this will not take effect until the release of v1.6, due end of June. We don't backport changes to earlier versions. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alexanderstephan, rikatz The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@youngnick, @rikatz |
What type of PR is this?
/kind feature
/kind documentation
What this PR does / why we need it:
This PR updates the TLSRoute CRD validation rationale and proposes increasing the
maxItembound for hostnames from 16 to 4096. This change is proposed only to accommodate very large orgs. In large orgs, the current limit of 16, this can lead to hundred of thousands TLSRoute objects. These objects multiply storage, watch traffic, and controller memory/CPU, which drives up API-server latency and risks OOMs and instability.In a similar PR the limit has been increased: #3205
To safely deploy the change the author employed a XValidation rule. For my case, such a rule would likely be rejected for being too complex. One idea would be to add the validation to a Validating Webhook. However, the webhook as been removed:
Do you have any ideas on how to solve this issue? I would be happy for further assistance on how to tackle this.
Rationale:
Which issue(s) this PR fixes:
No issue yet.
Does this PR introduce a user-facing change?: