Repository navigation
gep-4768: add TelemetryPolicy API proposal - #4872
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: gkhom The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @gkhom. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
|
|
||
| # 1. Tracing Configuration | ||
| tracing: | ||
| mode: "On" |
There was a problem hiding this comment.
the mode here is kind of a weird value. Usually we avoid to have bools or "wana be bools" (On/Off, True/False) and instead make them more explicit on Behavior (Disabled, PartiallyEnabled, FullyEnabled)
If we can only have On/Off, wouldn't it make more sense to say "if tracing is null, tracing should be disabled" instead?
There was a problem hiding this comment.
(btw this applies to every "Mode" defined here)
There was a problem hiding this comment.
The reason I added this explicit field is to allow implementations to define their own default behaviors. If we interpret a null/omitted block as "disabled", it would force "off-by-default" behavior for all implementations. For example, if a TelemetryPolicy is defined that only has explicit config for tracing, then it would imply that metrics and logs are disabled instead of the implementation's default. With the proposed schema we end up with the follow three distinct states:
- Omitted (Null): No preference expressed, the implementation uses its native default.
- Explicitly On: Ensure the feature is active.
- Explicitly Off: Ensure the feature is inactive.
As an added bonus, the explicit field allows a feature to be disabled without having to delete complex configuration (e.g., field mappings or CEL expressions). The example demonstrates this in the accessLogs block.
Is it reasonable to allow implementations to maintain their own telemetry defaults? If so, I'm happy to rename the enum values to "enabled/disabled" if it helps.
There was a problem hiding this comment.
I think this is still arguable. Mode can imply lots of things, can be "Partial", can imply operating modes. I think the field name here should probably be reworked into something more explicit of the intent of it (tho tbh I don't have a better name now and I don't like enabled/disabled much as well). I need to think more on it
There was a problem hiding this comment.
I also find it challenging to come up with a better name for the field. Alternatively, we could indeed consider removing this field entirely. We would need to define the default behaviors in the absence of these blocks. For example, the defaults could be:
tracing: default disabled
tracing/parentBasedSampling: default disabled
metrics: default enabled
access logs: default enabled
The downside of this is that the defaults aren't consistent across the different blocks. Wdyt?
There was a problem hiding this comment.
let me try to put on a different way and you tell me if it makes sense:
- a null configuration means "admin has no opinion, use the implementation defaults". We do not test conformance on it
- a tracing configuration, with numerator=0 means "disabled sampling rate" so the admin is explicitly saying "I DO NOT WANT" telemetry.
- same applies for parentBasedSampling
- On the other hand, any of these options with a number bigger than 0 means "the admin has an opinion and wants to have this configuration"
Probably a similar approach can be applied to metrics, where if you set a configuration with an override that drop every metric, you are disabling the metrics.
There was a problem hiding this comment.
Agreed on mode: On/Off being not ideal.
a tracing configuration, with numerator=0 means "disabled sampling rate" so the admin is explicitly saying "I DO NOT WANT" telemetry.
On this, I dont really like having a value as an indication of "disable it" this feels very indirect.
Whats the current state of the art WRT to defaults? do most implementation default to have tracing configs on or off by default?
The enum based approach is IMO better.
Why should not we have something simple like: "if TelemetryPolicy CRD exists and have tracing configs -- tracing configs are taken from this crd. If not, implementation does whatever their default is (whether its specific configs or OFF by default). I dont think we need to attempt standardizing what happens when this CRD is not applied in the cluster.
We can (and maybe should) support a modeDisabled enum though to standardize "how do I disable telemetry".
There was a problem hiding this comment.
It's better to not have "unspecified" have a meaning that could vary between implementations. We should be explicit.
So, if we need the default to be "whatever the implementation wants" (which I think is probably still arguable), then this should be an +optional Enum, with an ImplementationDefault value, that is the default if the field is unspecified.
I'd really prefer to have it disabled unless enabled, but I acknowledge that it's a breaking change and would almost certainly not be in user's best interests, so I think this is a good compromise.
Notably, having a default value supplied like that means that the field may get serialized with that value into etcd; this is very preferable in my mind, as it makes this ImplementationDefault mode extremely explicit.
| # 3. Access Logs Configuration | ||
| accessLogs: | ||
| mode: "Off" # Explicitly disabled while keeping the configuration intact | ||
| matches: "response.code >= 500" # Conditional logging, CEL filtering for errors |
There was a problem hiding this comment.
honest question here: is CEL matching for logs something usually supported by OTEL exporters, or is this something specific from Istio that we are carrying?
There was a problem hiding this comment.
I don't think OTel exporters support this, but OTel collectors support the filterprocessor which is based on a "CEL-like" language: OTTL. It could be argued that if the collectors filter logs, the gateway doesn't need to support this. However, I believe that filtering at the source saves resources and not all deployments utilize a collector.
There was a problem hiding this comment.
Oh I am not arguing against saving resource, but we need to think what happens for collectors that support a different type of filter:
maybe we need to say that filter support is by its own extended, but try to have a more common filter language (unless we can have a major agreement on every implementation on converting the CEL filtering to its own filtering, which I think will be harder)
There was a problem hiding this comment.
I agree that translating CEL into proxy-specific filter languages will be challenging for some implementations.
The Matches field is currently designated with Support: Implementation-specific. Implementations unable to natively evaluate or translate CEL would reject the policy with an UnsupportedField condition. Or would Extended support align better with Gateway API standards?
Do you think keeping CEL but strictly treating it as Extended/Implementation-specific is an acceptable compromise?
There was a problem hiding this comment.
yes it is, keeping as implementation specific, and then we can promote later to extended. But if we have a field that is implementation specific only, does it make sense to cover on this GEP now?
|
If it helps with some of this API design, here is the ObservabilityPolicy API used by NGINX Gateway Fabric to configure the nginx-otel module. It attaches to a Route. NGINX supports setting span names and trace context as well, which I don't see in the proposed API. In addition, we configure the exporter settings at the Gateway level in our NginxProxy API, which includes things like Since we already support all of these fields, the new TelemetryPolicy would need to support them as well in some form before we would be able to migrate to it. |
|
|
||
| const ( | ||
| // CustomAttributeTypeHeader extracts the value from an HTTP header. | ||
| CustomAttributeTypeHeader CustomAttributeType = "Header" |
There was a problem hiding this comment.
can we be explicit of what can be added or not? I would expect that we are careful with things like "Authorization" headers and other sensitive headers.
There was a problem hiding this comment.
I think that the API should remain non-opinionated on this. There are use-cases in which even sensitive headers/values are required in telemetry.
There was a problem hiding this comment.
I think we should be clear: Implementations SHOULD NOT allow passing sensitive values without a very good reason, as logging or storing sensitive values is more often than not a security risk.
I'd prefer to start with MUST NOT, and loosen this to SHOULD NOT if we find specific use cases that need it. But I'd prefer to start with secure-by-default, and only loosen with big warnings and maybe an Extended feature for it, so that we don't automatically build CVEs into every implementation.
| const ( | ||
| // CustomAttributeTypeHeader extracts the value from an HTTP header. | ||
| CustomAttributeTypeHeader CustomAttributeType = "Header" | ||
| // CustomAttributeTypeMetadata extracts the value from proxy metadata or context. |
There was a problem hiding this comment.
so, Metadata is a bit of a confusing concept in this context. I am worried we are making it very "Envoy" centric (unless @sjberman tells me he gets the concept of "metadata" here out of Nginx)
As we are leaning towards OTEL, I think it would be good to be more explicit on what each attribute means (eg.: https://opentelemetry.io/docs/specs/semconv/registry/attributes/http/ ) and use some OTEL references here
There was a problem hiding this comment.
No concept of metadata in nginx.
There was a problem hiding this comment.
Yeah, I'm pretty strongly opposed to "fetch a key from metadata"; it's completely unstructured and completely nonportable.
There was a problem hiding this comment.
I agree, relying on metadata is likely not a good idea. I will update the proposal to use references to OTel attributes.
| } | ||
|
|
||
| type TelemetryPolicySpec struct { | ||
| // Identifies the target gateways to which this policy attaches (GEP-713). |
There was a problem hiding this comment.
we must be specific here that at least one target is required
There was a problem hiding this comment.
Made it explicit (with comment and validation) that at least one target must be present.
| // Identifies the target gateways to which this policy attaches (GEP-713). | ||
| TargetRefs []NamespacedPolicyTargetReference `json:"targetRefs"` | ||
|
|
||
| // Configuration for distributed tracing options. |
There was a problem hiding this comment.
The comment must be more complete. Think on users using this API. We must write here what is tracing, what Gateway users will get from it.
This applies for every comment on every struct, I would recommend taking a look into https://gateway-api.sigs.k8s.io/reference/api-spec/main/spec/#backendtlspolicy
Each API field must contain:
- What is the field
- What happens when a user configure that field (or in the absence of that field)
- Are there specific conflicting options?
- Anything else that may help users to use
Also, given this is a field documentation for implementations, you must consider:
- What an implementation must know when considering that field? (does setting it has any caveat? Does the implementation needs to consider some condition status?)
- What the permutation of possible values can cause that the implementation must care? (think on conformance tests)
Think on https://gateway-api.sigs.k8s.io/guides/api-design/. The documentation must be separated between user facing and implementation facing (when writing implementation/developers facing, use the tag <gateway:util:excludeFromCRD></gateway:util:excludeFromCRD>)
Also, when defining fields you must define what kind of support you expect for it. Eg.:
- core - means that the field is required to be recognized by the implementation for this feature to work. eg.: targetRefs
- extended - means that the feature/field should be implemented by implementations that claim its support, but are not mandatory for the whole feature. eg.: a telemetryPolicy may support tracing, or metrics, or accessLogs, but doesn't require all of them to work
- implementationSpecific - We don't do conformance test, and a user using it knows they will not have portability between implementations. We should avoid this kind of support, and use only if strictly required on new apis. An example is my question about the cel matching for logs, as I don't think every implementation supports it
There was a problem hiding this comment.
+1 defining support levels at this stage is very important. Also worth keeping in mind, everything in core and extended should be covered by conformance tests, otherwise it risks becoming implementation-specific anyway.
There was a problem hiding this comment.
I've added a first round of documentation to the API definition, including support levels. The documentation can be more elaborate in defining semantics but I'll do that as a follow up once we roughly agree on the API structure.
| tracing: | ||
| mode: "On" | ||
| provider: | ||
| endpoint: "otel-collector.monitoring.svc:4317" |
There was a problem hiding this comment.
should probably use backendRef style API to align with other APIs
There was a problem hiding this comment.
+1 otherwise we have to discuss again how to specify TLS, ... (as also mentioned in https://github.com/kubernetes-sigs/gateway-api/pull/4872/changes/BASE..084523a5d6aa69293ca1b9577fabdab8487ba0da#r3260666205)
There was a problem hiding this comment.
Updated the example and API struct to use backendRef.
| // --- Metrics Types --- | ||
|
|
||
| type MetricsConfig struct { | ||
| // Mode explicitly controls if metric generation is enabled. Valid values are "On" or "Off". |
There was a problem hiding this comment.
What does it mean to turn off metrics generation? It does not seem reasonable to entirely disable all metrics for an application
There was a problem hiding this comment.
It seems plausible to me that a user can gain sufficient insights from traces and/or access logs alone. Out of curiosity, why do you think metrics must always be enabled?
There was a problem hiding this comment.
Metrics are pull based which generally means they are quite a bit cheaper than logs/traces to enable without using. Obviously some cost to maintain.
For Envoy specifically there is no way to dynamically disable metrics so this API would not be implementable on ~50% of implementations.
There was a problem hiding this comment.
Fair point about Envoy's core stats engine being static, but I think that we can achieve dynamic disablement at the filter/logger level using the Runtime Discovery Service (RTDS).
I'm not an expert on this but I think it's possible to link the telemetry collection filters (or StatsAccessLogger instances) to a standard RuntimeFilter, the Gateway controller can dynamically enable or bypass metric recording via RTDS config updates. This way a restart should not be required just to toggle metric generation.
There was a problem hiding this comment.
Do we have any usecase that requires metric disablement? I think that right now 100% of the proxies generate metrics, if so - I would argue no need to define metric disablement here
There was a problem hiding this comment.
Users often turn (some of) them off in istio/envoy for scale reasons. see https://github.com/howardjohn/gateway-api-bench#route-scale
| Mode TelemetryMode `json:"mode,omitempty"` | ||
|
|
||
| // List of configurations to customize specific metric families. | ||
| Overrides []MetricOverride `json:"overrides,omitempty"` |
There was a problem hiding this comment.
What does this actually do? Is it customizing an existing metric? adding a new one?
There was a problem hiding this comment.
Is this maybe trying to formalize relabeling? 🤔
There was a problem hiding this comment.
The intent is to customize an existing metric. It's quite common for users to require additional attributes/labels for metrics.
There was a problem hiding this comment.
Is there any expectation for how this should be done? Realistically there are two (and a half) options:
-
envoy/agentgateway/nginx or any other proxies support dynamic metric overrides (whether it is names, attributes or something else). I dont think thats the case (but could be wrong)
-
Something augments those metrics on top of the proxy and does these overrides. And this something is the thing that prometheus should scrape.
-
Some kind of config in a metric collector/scraper that does that after it scrapes the original metrics and before it ships it to the metric store
For #3 - we dont need to standardize in gateway-api cause this is a job of a metric collector
For #1 - I dont think its supported right now. But this would be the most logical expectation I would expect the CRD fields would be implemented if we were to add them.
For #2 I am not sure if everyone runs their proxy in some kind of an envelope that wraps it and is capable of doing that.
I do agree that adding attributes or standardizing metrics and/or schemas are useful, but maybe we should start by guiding what metrics proxies should export (is it OTEL? is it working with OTEL community to define proxy metrics, is it something like gateway.k8s.io/?) I'm not quite sure -- just throwing it out there
| MetricAttributeTypeLiteral MetricAttributeType = "Literal" | ||
| ) | ||
|
|
||
| type MetricAttribute struct { |
There was a problem hiding this comment.
So we can add fields only right, not remove them?
There was a problem hiding this comment.
Indeed, the current API spec only allows additions. I do think that removal is useful as well. Do you think we should accommodate this in the first iteration? It could be as simple as adding a field: action: remove.
| // This is required if Type is "Literal". | ||
| LiteralValue *string `json:"literalValue,omitempty"` | ||
|
|
||
| // StandardValue specifies a standard log property (e.g., "RequestStartTime", "Duration"). |
There was a problem hiding this comment.
Are these defined? or implementation specific?
There was a problem hiding this comment.
I think that these should indeed be defined in this proposal. I'll add an explicit list of supported values, mapping them to the relevant OTel SemConv attributes. Hopefully, this can substitute relying on metadata as well.
| } | ||
|
|
||
| type TracingProvider struct { | ||
| Endpoint string `json:"endpoint,omitempty"` |
There was a problem hiding this comment.
How about TLS options and custom headers, per #4775 (comment)
There was a problem hiding this comment.
Updated the API to use backendRef so that should cover the TLS options. Do we still need something separate for custom headers?
There was a problem hiding this comment.
I'm thinking from the perspective of NGINX, which supports setting custom headers in the requests to the exporter backend.
There was a problem hiding this comment.
I've added support for custom headers in the TracingProvider.
| LogFieldTypeStandard LogFieldType = "Standard" | ||
| ) | ||
|
|
||
| type LogField struct { |
There was a problem hiding this comment.
It's not clear to me what the fields LogField in the log represents, is it a JSON field for specifying additional key-value pairs?
How do you plan to support customization of nested JSON structures? These are commonly used in access logs, e.g., in the Airlock Microgateway we use the Elastic Common Schema (ECS).
Since access logs tend to be highly implementation-specific, IMO a format string or an extension point is probably the best approach here.
There was a problem hiding this comment.
The LogField struct is indeed intended to specify key-value pairs for the access log output. For nested JSON structures like ECS, we can support dot-notation (e.g., user.id) in the name field, which implementations can then expand into structured objects during serialization.
There was a problem hiding this comment.
I believe dot-notation in a flat field ({"my.key": "bar"}) is both common and strictly required in OTEL semconv. So uplifting that into structured objects doesn't seem great
There was a problem hiding this comment.
Thanks, I overlooked that. To clean this up, I split the config into two separate structs:
- Metrics & Traces (
Attribute): Uses a single stringnamefor standard flat key-value attributes. - Access logs (
LogField): Uses apathlist (containing path segments) to build nested JSON logs. Individual segments can contain dots that will be preserved verbatim. For example:path: ["mcp.method"]maps to a flat key containing dots:{"mcp.method": "<value>"}.path: ["http", "request", "start_time"]maps to a standard nested structure:{"http": {"request": {"start_time": "<value>"}}}.
| // Identifies the target gateways to which this policy attaches (GEP-713). | ||
| TargetRefs []NamespacedPolicyTargetReference `json:"targetRefs"` | ||
|
|
||
| // Configuration for distributed tracing options. |
There was a problem hiding this comment.
+1 defining support levels at this stage is very important. Also worth keeping in mind, everything in core and extended should be covered by conformance tests, otherwise it risks becoming implementation-specific anyway.
Co-authored-by: Rob Scott <rob.scott87@gmail.com>
| // (i.e., as a span tag). | ||
| // | ||
| // +required | ||
| Name string `json:"name"` |
There was a problem hiding this comment.
v1.HTTPHeaderName would be appropriate for the HeaderName field below. Recommend defining a net new type for this, here's what AI recommends (which also seems reasonable to me):
// AttributeName defines the key of a span attribute or tag.
//
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=256
// +kubebuilder:validation:Pattern=`^[a-zA-Z0-9_.:/-]+$`
type AttributeName string
| 2. **Attribute Type Handling and Failure Modes**: | ||
| How should attribute data types (such as boolean, integer, or string) be handled and preserved when emitting custom span attributes or mapping dynamic header values? | ||
| Should an optional field (e.g., `valueType`) be added to the `Attribute` struct to allow explicitly specifying the output type for custom attribute keys, while relying on OpenTelemetry Semantic Conventions for standard keys? |
There was a problem hiding this comment.
That makes sense. We can add this in the future if needed. In the meantime it may make sense to move this to "potential future additions" instead of "open questions" so it's not blocking the proposal from graduating to experimental.
|
/label tide/merge-method-squash |
Co-authored-by: Rob Scott <rob.scott87@gmail.com>
|
/ok-to-test |
|
Thanks @gkhom! /lgtm |
snorwin
left a comment
There was a problem hiding this comment.
I reviewed all the changes again and overall it looks good to me to proceed forward. I'm especially looking forward to the conformance tests, I think we should really make sure that if this feature ends up in 1.7 as experimental, the conformance tests are already implemented, or at least some of them.
@gkhom thanks for the effort you put into this, really appreciated.
/lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gkhom, rikatz, robscott, snorwin The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/ok-to-test |
What type of PR is this?
/kind gep
What this PR does / why we need it:
Added an initial proposal for the TelemetryPolicy API (gep-4768). This includes an example, the Go structs, and comparison with prior art. (See: #4768)
Does this PR introduce a user-facing change?: