Repository navigation
GEP-5224: Pre-routing (Implementable) - #5271
kubernetes-prow[bot] merged 7 commits into
Conversation
Signed-off-by: Jackie Elliott <jaellio@microsoft.com>
|
This issue is currently awaiting triage. If Gateway API community members determine this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Hi @jaellio. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
| // +optional | ||
| // +listType=atomic | ||
| // +kubebuilder:validation:MaxItems=16 | ||
| HTTPPreRoutingFilters []HTTPPreRoutingFilter `json:"httpPreRoutingFilters,omitempty"` |
There was a problem hiding this comment.
To make this more future proof, why not just call this filters? Then we can note which filter types can be combined with which protocols, allowing us to eventually add filters for other protocols without a large number of additional fields.
There was a problem hiding this comment.
Attempting to merge this with @howardjohn's comment below which closely aligned with @youngnick's preference.
Both Nick and John are suggesting a split based roughly on protocols or network layers. I've been very nervous that protocol specific filters would be hard to scale and could unnecessarily lead to confusion of where a filter should be placed if it could theoretically fit in either layer or protocol. If we went with network layers, the concept of an "application" filter would likely be confusing to users.
John's comment at the end of today's community meeting stuck with me though - he really just wants to distinguish between filters that should apply to a connection vs a request. So with all that said, why not just split this into filters.connection and filters.request?
There was a problem hiding this comment.
I really don't have strong opinions here but that works for me
Signed-off-by: Jackie Elliott <jaellio@microsoft.com>
nirrozenbaum
left a comment
There was a problem hiding this comment.
@jaellio thanks!
I left some comments, most of them are minor/nits.
the only one worth discussing is the RequestHeaderModifier filter part.
I strongly think we should extend that filter to support dynamic values rather than recommending ExtensionRef for body based routing use cases.
extending that filter would also be very useful in MultiStageRoute GEP.
Ideally, I'd like to see AI use cases represented as first class citizens, e.g., body based routing should work similar in any gateway. I think recommending ExtensionRef for the mechanism to do that could cause fragmentation to the ecosystem instead of converging on a single way to do this basic capability.
/ok-to-test
| * **`RequestMirror`.** Mirroring clones the request to a separate backend | ||
| and discards the mirror's response. It does **not** mutate the primary | ||
| request in any way, so by construction it cannot influence which route | ||
| the primary is matched to. The pre-routing phase is defined as the phase | ||
| whose output *can* influence route selection. A variant that provably | ||
| cannot doesn't belong. |
There was a problem hiding this comment.
I could actually see some utility in mirroring a request before mutations are applied. I don't think this is a priority, but i wouldn't permanently rule it out if we see users request it.
There was a problem hiding this comment.
Could you share the use cases you are thinking of? My initial thoughts were that specifying mirroring backendRefs pre-route selection felt unintuitive. Still going to exclude RequestMirroring for now, but removed absolute exclusion comments.
| * **`GRPCRoute`.** GRPCRoute is HTTP/2 in practice. If community consensus | ||
| extends pre-routing to GRPCRoute, no new field is required: the same | ||
| `Listener.httpFilters` (on an HTTP/HTTPS listener) applies. A | ||
| future revision would loosen the "HTTPRoute only" restriction in godoc and | ||
| extend conformance to cover GRPCRoute matching. |
There was a problem hiding this comment.
Maybe worth considering that https://gateway-api.sigs.k8s.io/reference/api-spec/main/spec/#grpcroutefilter currently only includes a subset of the filters allowed on HTTPRoute? I think mostly that has been to ensure implementations and conformance test coverage exist before promoting, but some functionality like URLRewrite could feel awkward even if technically implementable.
There was a problem hiding this comment.
I remove the HTTPRoute only restriction
to implementable Signed-off-by: Jackie Elliott <jaellio@microsoft.com>
Signed-off-by: Jackie Elliott <jaellio@microsoft.com>
youngnick
left a comment
There was a problem hiding this comment.
Damn, I thought I hit submit yesterday. Sorry!
| // +optional | ||
| // +listType=atomic | ||
| // +kubebuilder:validation:MaxItems=16 | ||
| HTTPFilters []HTTPListenerFilter `json:"httpFilters,omitempty"` |
There was a problem hiding this comment.
I think we should have a general Filters struct (filters in YAML), that then contains separate lists based on the metadata that is passed to those filters.
We start with filters.http, but that leaves it open for us to add filters.tcp, filters.udp or similar later. And it's way simpler than trying to do everything in one list/slice, where we would have to have a lot of CEL to check, for example, that a hypothetical TCP filter (like an ACL or something) came before a HTTP one.
So, my suggestion in tl;dr format:
- Create a new
PreRoutingFilters(filtersin YAML) struct - That struct contains a HTTP (
httpin YAML) entry, that is a[]HTTPListenerFilter
that gives us extensibility without adding a lot of definitional complexity to the actual list of filters.
| HTTPListenerFilterExternalAuth HTTPListenerFilterType = "ExternalAuth" | ||
| HTTPListenerFilterExtensionRef HTTPListenerFilterType = "ExtensionRef" |
| // +optional | ||
| // +listType=atomic | ||
| // +kubebuilder:validation:MaxItems=16 | ||
| HTTPFilters []HTTPListenerFilter `json:"httpFilters,omitempty"` |
There was a problem hiding this comment.
The HTTP fields also needs documentation about the circumstances under which it's valid (that is, only when protocol is HTTP or HTTPS, maybe GRPC?), and what filters are allowed to use when it is (any HTTP data, including metadata like headers and the request body if required).
This will be more relevant once we eventually add other filter types, so we could just add the protocol part if people think that's better.
Signed-off-by: Jackie Elliott <jaellio@microsoft.com>
| * **Defining an L4 pre-routing surface.** SNI/TLS-based selection | ||
| ([GEP-2643](../gep-2643/index.md)) and TCP/UDP-layer pre-routing filters | ||
| are lower-layer forms of pre-routing and are not in scope here. The API | ||
| is deliberately structured so that connection-oriented pre-routing (see | ||
| [Extensibility to Other Route Kinds](#extensibility-to-other-route-kinds)) | ||
| can be added as a `filters.connection` sibling to `filters.requests` | ||
| by a follow-up GEP without renaming or moving the outer `filters` | ||
| field. |
There was a problem hiding this comment.
I think this is okay, but I don't know where we would put UDP filters in this case? Since that doesn't fit into connection either, really?
There was a problem hiding this comment.
Added a section on UDP filters to note it is out of scope but could be captured in a requests.datagram field.
youngnick
left a comment
There was a problem hiding this comment.
I think that filters.requests is fine as a name, as long as we're clear that the requests requires a HTTP-like transport, whether that is HTTP1, HTTP/2 or HTTP/3.
I don't quite agree with filters.connections as a name, but I do agree with the concept of using a sibling field to requests, so I think this LGTM.
Signed-off-by: Jackie Elliott <jaellio@microsoft.com>
| // Today only the request phase (Requests) is defined. A future | ||
| // connection-oriented phase (Connection) is anticipated for L4 | ||
| // pre-routing (for example, source-IP allow/deny, per-connection rate | ||
| // limits, SNI-derived decisions) but is deliberately not introduced by | ||
| // this GEP. When Connection is added, existing configurations that only | ||
| // populate Requests remain valid. |
There was a problem hiding this comment.
Non-blocking, can handle this in PR that bumps to experimental and adds API types.
| // Today only the request phase (Requests) is defined. A future | |
| // connection-oriented phase (Connection) is anticipated for L4 | |
| // pre-routing (for example, source-IP allow/deny, per-connection rate | |
| // limits, SNI-derived decisions) but is deliberately not introduced by | |
| // this GEP. When Connection is added, existing configurations that only | |
| // populate Requests remain valid. | |
| // Today only the request phase (Requests) is defined. In the future | |
| // a connection-oriented phase (Connection) may be added. |
|
|
||
| **New reasons introduced by this GEP:** | ||
|
|
||
| * **`Accepted=False`, `Reason=InvalidListenerFilter`.** Used when a filter in |
There was a problem hiding this comment.
Recommend adding InvalidListenerFilterOrder as a valid reason below this point.
| // every request accepted on this Listener, before route matching | ||
| // is performed. The list order is load-bearing: | ||
| // implementations MUST execute the filters in the exact order they | ||
| // appear here and MUST NOT reorder them. |
There was a problem hiding this comment.
| // appear here and MUST NOT reorder them. | |
| // implementations MUST execute the filters in the exact order they | |
| // appear here and MUST NOT reorder them. If an implementation can | |
| // not implement the filters in the order they are specified, the | |
| // implementation MUST set the "Accepted" Listener condition to "false" | |
| // with the "InvalidListenerFilterOrder" reason. |
| The shape of `ListenerFilter` follows the established | ||
| `GRPCRouteFilter` pattern (see | ||
| [grpcroute_types.go](https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1/grpcroute_types.go#L507)): | ||
| a discriminated union of a `Type` enum and one payload field per variant. |
There was a problem hiding this comment.
The section above connects this to HTTPRouteFilter, and then this one compares it with GRPCRouteFilter. It may be simpler to just keep all references relative to HTTPRouteFilter.
|
|
||
| The shape of `ListenerFilter` follows the established | ||
| `GRPCRouteFilter` pattern (see | ||
| [grpcroute_types.go](https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1/grpcroute_types.go#L507)): |
There was a problem hiding this comment.
For a link like this, always prefer a link to a commit hash so it's future proof. (Using "t" as a shortcut on any GH page like this will change the link from a branch to the latest commit hash on that branch).
| The intent of this MUST is stated explicitly in godoc on the field, and is | ||
| verified by conformance tests (see [Conformance](#conformance)). This is a | ||
| deliberate contrast to | ||
| [`HTTPRouteRule.Filters`](../../reference/api-types/httproute.md#filters-optional), |
There was a problem hiding this comment.
Similar comment throughout, but recommend absolute GitHub links here to a git commit hash.
There was a problem hiding this comment.
Nit: This is inconsistent with the feature name written in the GEP now.
|
LGTM once Rob's comments are sorted. Nice work on the UDP section. |
nirrozenbaum
left a comment
There was a problem hiding this comment.
lgtm from me as well after Rob’s comments
Signed-off-by: Jackie Elliott <jaellio@microsoft.com>
|
Thanks @jaellio! /lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jaellio, nirrozenbaum, robscott The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| ### Top-level `ListenerFilterPolicy` CRD via Policy Attachment (GEP-713) | ||
|
|
||
| **Considered.** Introduce a new CRD (analogous to | ||
| [`BackendTrafficPolicy`](../gep-3388/index.md)) with `targetRefs` pointing at | ||
| `Gateway`, `Listener` (via `sectionName`), or `ListenerSet`. Filters live in | ||
| the policy `spec.filters`. |
There was a problem hiding this comment.
How to avoid config bloating and management burden?
Did we consider making ListenerFilters a top-level CRD? This would keep the ordering while allowing the same filter list to be reused across multiple listeners.
Currently, Listener.filters is hardcoded. Adding a distinct listener based on hostname (likely) requires copy-pasting the entire filter list. Now imagine having 100+ listeners on the same port.
| ### Attach at Gateway (`GatewaySpec.filters`) | ||
|
|
||
| **Considered.** A single shared list at Gateway scope, applied uniformly to | ||
| every listener on the Gateway. | ||
|
|
||
| **Rejected** because: | ||
|
|
||
| * Filter payloads depend on protocol (an HTTP filter on a TCP listener is | ||
| meaningless). Listener scope makes protocol-appropriateness expressible | ||
| directly, without additional per-protocol validation on a Gateway-wide list. |
There was a problem hiding this comment.
Security concern.
Did we consider per-protocol per-port ListenerFilters (similar to per-port frontend mTLS configuration)?
With multiple listeners distinct based on hostname, it's dangerously easy to miss a vulnerability where all BUT ONE have ExternalAuth configured.
Or we'd like to put it on users and recommend setting up ValidatingAdmissionPolicies / OPA Gatekeeper guards?
What type of PR is this?
/kind gep
What this PR does / why we need it:
Defines the experimental GEP for pre-routing
Which issue(s) this PR fixes:
Fixes #5224
Does this PR introduce a user-facing change?:
Was AI used in preparing this PR?
AI was used to help write this GEP but the process was interactive (populated with debate results and manual probing) and the GEP was reviewed/edited by me.
/label ail/3