🔴 Live TV: aws observability/helm charts/pull/

louisall · 2026-03-24T11:17:14Z

Summary

Adds OTEL-based Container Insights support to the helm chart, enabled by default. Deploys node-exporter, kube-state-metrics, and a cluster-scraper (as an AmazonCloudWatchAgent CR in the agents array) alongside the existing CloudWatch agent to collect Kubernetes metrics via OTLP and export them to the CloudWatch Monitoring endpoint. Includes feature-targeted agent routing where each feature (containerInsights, applicationSignals, otelContainerInsights) declares a targetAgent field controlling which agent receives its config.

Changes

Chart infrastructure:

New OTEL collector configs for per-node scraping (cadvisor, kubeletstats, node-exporter, EFA, EBS CSI, DCGM, Neuron) and cluster-level scraping (kube-state-metrics, API server)
Node-exporter DaemonSet with TLS, kube-state-metrics Deployment with TLS, cluster-scraper as an agents array entry (mode: deployment) managed by the operator
OTEL config merge logic that preserves user-supplied otelConfig alongside generated pipelines (generated config takes precedence on collision), with fromYaml error validation
containerInsights.enabled flag to independently control legacy Container Insights pipeline
Region-aware image resolution via repositoryDomainMap (public, China, GovCloud) with restrictedRepository/restrictedTag for restricted regions
KSM/node-exporter receivers and pipelines gated by their respective enabled flags — disabled components don't produce scrape configs
scrapeTimeout helper using min(metricResolution, 10s) to prevent Prometheus rejection when metricResolution < 10s
Fixed monitorAllServices logic to use applicationSignals.enabled + targetAgent routing instead of inspecting $agent.config (which is nil for default installs)
KSM naming aligned with sidecar component pattern (kube-state-metrics.name/kube-state-metrics.serviceAccountName helpers), RBAC split into separate clusterrole/clusterrolebinding files
Removed health probes and health_check OTEL extensions from agent CRs — operator manages agent health; avoids port 13133 conflict with hostNetwork: true
otelConfig field omitted from CRs when no OTEL features target the agent (instead of emitting empty config)

Integration tests:

Updated EKS resource counts and name patterns for new OTEL resources
Extended minikube default scenario to validate OTEL resources exist
New otlp-disabled and otlp-custom-otel-config scenarios
New feature-targeted scenarios: default, multi-agent, otlp-disabled, split-features, appsignals-disabled, ci-disabled, both-disabled, custom-otel-config, user-config-override
Added ValidateDaemonSetExists, ValidateServiceExists, ValidateServiceAccountExists helpers to K8sClient

Documentation:

Hierarchical AGENTS.md files at each directory boundary for AI-assisted development context
.helmignore to exclude .md files from chart packaging

Configuration

New values under otelContainerInsights:

enabled (default: true) — gates all OTEL CI resources
targetAgent (default: "cloudwatch-agent") — agent receiving node-level OTEL config
clusterScraperAgent (default: "cloudwatch-agent-cluster-scraper") — agent receiving cluster-level OTEL config
metricResolution (default: "30s") — scrape interval
cloudwatchMetricsEndpoint — optional endpoint override

kubeStateMetrics and nodeExporter sections at root level with enabled, name, image, resources, serviceAccount. Resources only created when both the component's enabled flag and otelContainerInsights.enabled are true.

containerInsights.enabled (default: true) independently controls the legacy CI pipeline. Both can run simultaneously.

Testing

make all passes (helm lint, secret scan, go fmt/lint)
All 25 minikube scenarios pass (0 failures)
Deployed and verified on EKS test cluster

Add otelContainerInsights feature to the existing chart, gated behind otelContainerInsights.enabled (defaults to true). Deploys node-exporter, kube-state-metrics, and a cluster-scraper alongside the existing agent to collect metrics via OTLP and export to CloudWatch Monitoring endpoint. Key changes: - New OTEL collector configs for per-node and cluster-level scraping - Node-exporter DaemonSet, kube-state-metrics Deployment, cluster-scraper Deployment - OTEL config merge logic preserving user-supplied otelConfig alongside generated pipelines - containerInsights.enabled flag to independently control legacy CI pipeline - Public image fallback via repositoryDomainMap for non-EKS clusters - Integration tests for enabled, disabled, and config merge scenarios

Add Minikube scenarios and update EKS tests to cover otelContainerInsights: - Update EKS resource counts, name patterns, and RBAC assertions for OTLP resources - Extend Minikube default scenario to validate OTLP resources exist - New otlp-disabled scenario verifying resources are absent when flag is off - New otlp-custom-otel-config scenario validating otelConfig merge and precedence - Add ValidateDaemonSetExists and ValidateServiceExists to K8sClient

Hierarchical AGENTS.md files providing context at each directory boundary: root, chart, templates (linux/windows/rosa/webhooks), CRDs, and integration tests. Also adds .helmignore to exclude .md files from Helm packaging and updates .gitignore for Terraform artifacts.

louisall · 2026-03-24T12:53:37Z

Known CI failure: Minikube integration tests timeout

All Minikube scenarios fail with context deadline exceeded on the helm install step. This is expected and will resolve once the updated CloudWatch Agent image ships with OTEL component support.

Root cause: This PR sets otelContainerInsights.enabled: true by default, which deploys a cloudwatch-agent-cluster-scraper Deployment. This deployment uses the public CW agent image (public.ecr.aws/cloudwatch-agent/cloudwatch-agent), which does not yet include the OTEL components referenced in the generated config (otlphttp exporter, kubeletstats receiver, metricstarttime processor, etc.). The cluster-scraper pod starts, fails config validation, and enters CrashLoopBackOff. Since the Terraform helm provider uses wait=true by default, it blocks until all helm-managed pods are ready, hits the 300s timeout, and fails.

The Go test assertions themselves pass — they validate resource existence and config content, not pod health. Verified locally by running all 16 scenarios with --wait=false: 16/16 pass.

What will fix it: When CW agent image released with the required OTEL collector components. Once that image is published to public ECR, the cluster-scraper will start successfully and wait=true will no longer timeout.

Manual validation performed:

Merge precedence logic verified on EKS cluster with 3 scenarios: OTLP enabled + colliding user config, OTLP disabled + user config, OTLP enabled without user config
All 16 minikube Go tests pass when deployed with --wait=false
helm template dry-runs confirm correct config rendering for all scenarios

…container attr in KSM - Add transform/set_cloud_resource_id processor to all daemonset and cluster-scraper pipelines, building the EKS cluster ARN from cloud.region, cloud.account.id, and k8s.cluster.name (placed after resourcedetection) - Add container key to groupbyattrs/otelci_ksm and promote to k8s.container.name in the KSM pipeline

mitali-salvi · 2026-03-25T20:52:05Z

+## | false                    | false                         | No CI metrics             |
+otelContainerInsights:
+  enabled: true
+  metricResolution: "15s"


Will this change the metric resolution in all components of the otelContainerInsights pipeline ?

Yes. Weve decided to change it back to 30s though

…tion Add targetAgent fields to containerInsights, applicationSignals, and otelContainerInsights to route feature configs to specific agents. - Add build-default-config and build-default-otel-config helpers that dynamically construct per-agent CW Agent JSON and OTEL YAML configs based on targetAgent matching - Promote cluster-scraper from standalone Deployment to CR entry in agents array (mode: deployment), managed by operator - Add universal health check: all agents get health_check OTEL extension and liveness/readiness probes regardless of otelContainerInsights.enabled - Separate KSM RBAC into dedicated file, add TLS with web-config-file, remove tokenreviews/subjectaccessreviews permissions - Rename otelci prefix to otel_container_insights in cluster-scraper config - Fix kubeletstats endpoint: K8S_NODE_NAME -> HOST_IP - Add boolean validation for otelContainerInsights.enabled - Apply metricResolution to cluster-scraper config (was hardcoded 30s) - Add 9 feature-targeted integration test scenarios - Update AGENTS.md steering files

…ealth probes/extensions, fix monitorAllServices logic, add fromYaml error validation, gate KSM/node-exporter receivers by enabled flags, add scrapeTimeout helper, simplify double-negative condition, split KSM RBAC files, remove unnecessary prometheus annotations, omit empty otelConfig from CRs

…ip_verify

) * scope configmap permission to namesapce * integ tests * updates

Add scope.name and scope.version to OTEL transform processors for: - node-exporter: scope name from github repo, version from image tag - kube-state-metrics: scope name from github repo, version from image tag - cadvisor: update scope name from prometheus receiver to cadvisor - dcgm-exporter: scope name and version from image tag - neuron-monitor: scope name and version from image tag - apiserver: extract version from kubernetes_build_info git_version attribute Add helper templates for region-aware scope version resolution for node-exporter and kube-state-metrics (restrictedTag vs public tag).

…er and cadvisor

…config - Add nodemetadatacache extension and nodemetadataenricher processor to cluster-scraper OTel config; insert enricher in KSM pipeline after resourcedetection to set per-node host.* and cloud.availability_zone from Lease cache - Remove host.* delete_key statements from ksm_clean_resource (enricher now sets correct values) - Add unconditional_removal_prefixes/keys to cluster-scraper awsattributelimit (parity with DaemonSet config for node label cleanup) - Add namespace-scoped Lease Role/RoleBinding for cluster-scraper (get/list/watch) and Lease permissions for DaemonSet agent (get/create/update/delete)

Reverts commits 7422f65 and 9324321 to remove the lease-based node metadata enrichment changes from the Helm chart

… only (#287)" This reverts commit b12a6c5.

…daemonset mode

…d release-* to CI triggers

…-service-acct)

… and regex

* Add OTEL Container Insights infrastructure (#282) * Add v6.0.0 release notes * Cluster-scraper deployment now uses cloudwatch-agent SA (cluster-scraper SA retained as dead resource) * Bump chart version to 6.0.1 and add release notes * Scope config-map permission for cloudwatch-agent to namespace only (#287) * scope configmap permission to namesapce * integ tests * updates * Retain configmap get permission at cluster scope (#291) * Fix template whitespace control and hardcoded test names - Use {{- if instead of {{ if in cloudwatch-agent-role.yaml and cloudwatch-agent-rolebinding.yaml to prevent leading blank lines - Extract agentName constant in configmap_permission_scoping_test.go to replace hardcoded strings * Keep configmaps get at cluster scope, scope create/update to namespace - Add configmaps get-only rule to ClusterRole - Narrow namespace-scoped Role to create and update only - Update test assertions to match * Run integration tests on PRs targeting release-* branches * Set hostNetwork: true for default $.agent * Feature/ksm enrichment and label preservation (#296) * Fix agent.config bleed-through into cluster-scraper deployment (#298) * Fix scrapeTimeout, add missing scrape_timeout, remove dead RBAC, cleanup docs and tests (#297) * Add ClusterIP Service for node-exporter and use service DNS for scraping (#304) * Refactor fluent-bit region-specific (isolated) config using tpl conditionals (#271) * Pin GitHub Actions to commit SHAs (#306) * minor fixes (#307) * release v6.1.0 * Add LIS CSI metrics support to OTEL container insights config (#308) Add prometheus receiver, scope transform, attribute promote transform, and pipeline for local instance store CSI driver metrics collection, mirroring the existing EBS CSI pattern. Scrapes pods with label app=ec2-instance-store-plugin in kube-system namespace and promotes instance_id and volume_id to resource attributes. * feat(otel): Add k8sattributes processor node filter to daemonset (#311) * fix(integration-test): include node-exporter-service in EKS resource counts (#312) * Add OTEL-native Container Insights log pipelines (app, host) (#310) * release v6.2.0 * Address PR review: use v0.18 CRD schema, add version bumps to release notes --------- Co-authored-by: Louise Allen <147041218+louisall@users.noreply.github.com> Co-authored-by: Kaushik Surya <kausyas@amazon.com> Co-authored-by: Kaushik Surya <108111936+sky333999@users.noreply.github.com> Co-authored-by: Petru Anica-Popa <66786772+petruanica@users.noreply.github.com> Co-authored-by: Trey Richbourg <63218430+treyrichbourg@users.noreply.github.com> Co-authored-by: POOJA REDDY NATHALA <poojardy@amazon.com> Co-authored-by: Michael O'Neill <miconeil@amazon.com> Co-authored-by: Mitali Salvi <mitsalvi@amazon.com>

* Add OTEL Container Insights infrastructure (aws-observability#282) * Add v6.0.0 release notes * Cluster-scraper deployment now uses cloudwatch-agent SA (cluster-scraper SA retained as dead resource) * Bump chart version to 6.0.1 and add release notes * Scope config-map permission for cloudwatch-agent to namespace only (aws-observability#287) * scope configmap permission to namesapce * integ tests * updates * Retain configmap get permission at cluster scope (aws-observability#291) * Fix template whitespace control and hardcoded test names - Use {{- if instead of {{ if in cloudwatch-agent-role.yaml and cloudwatch-agent-rolebinding.yaml to prevent leading blank lines - Extract agentName constant in configmap_permission_scoping_test.go to replace hardcoded strings * Keep configmaps get at cluster scope, scope create/update to namespace - Add configmaps get-only rule to ClusterRole - Narrow namespace-scoped Role to create and update only - Update test assertions to match * Run integration tests on PRs targeting release-* branches * Set hostNetwork: true for default $.agent * Feature/ksm enrichment and label preservation (aws-observability#296) * Fix agent.config bleed-through into cluster-scraper deployment (aws-observability#298) * Fix scrapeTimeout, add missing scrape_timeout, remove dead RBAC, cleanup docs and tests (aws-observability#297) * Add ClusterIP Service for node-exporter and use service DNS for scraping (aws-observability#304) * Refactor fluent-bit region-specific (isolated) config using tpl conditionals (aws-observability#271) * Pin GitHub Actions to commit SHAs (aws-observability#306) * minor fixes (aws-observability#307) * release v6.1.0 * Add LIS CSI metrics support to OTEL container insights config (aws-observability#308) Add prometheus receiver, scope transform, attribute promote transform, and pipeline for local instance store CSI driver metrics collection, mirroring the existing EBS CSI pattern. Scrapes pods with label app=ec2-instance-store-plugin in kube-system namespace and promotes instance_id and volume_id to resource attributes. * feat(otel): Add k8sattributes processor node filter to daemonset (aws-observability#311) * fix(integration-test): include node-exporter-service in EKS resource counts (aws-observability#312) * Add OTEL-native Container Insights log pipelines (app, host) (aws-observability#310) * release v6.2.0 * Address PR review: use v0.18 CRD schema, add version bumps to release notes --------- Co-authored-by: Louise Allen <147041218+louisall@users.noreply.github.com> Co-authored-by: Kaushik Surya <kausyas@amazon.com> Co-authored-by: Kaushik Surya <108111936+sky333999@users.noreply.github.com> Co-authored-by: Petru Anica-Popa <66786772+petruanica@users.noreply.github.com> Co-authored-by: Trey Richbourg <63218430+treyrichbourg@users.noreply.github.com> Co-authored-by: POOJA REDDY NATHALA <poojardy@amazon.com> Co-authored-by: Michael O'Neill <miconeil@amazon.com> Co-authored-by: Mitali Salvi <mitsalvi@amazon.com>

louisall added 3 commits March 24, 2026 11:00

louisall requested review from mitali-salvi and sky333999 March 24, 2026 12:54

mitali-salvi reviewed Mar 25, 2026

View reviewed changes

JayPolanco reviewed Mar 25, 2026

View reviewed changes

Comment thread charts/amazon-cloudwatch-observability/templates/linux/_otel-container-insights-config.tpl Resolved

sky333999 reviewed Mar 26, 2026

View reviewed changes

mitali-salvi reviewed Mar 26, 2026

View reviewed changes

louisall added 3 commits March 27, 2026 13:30

release v6.0.0

4c6d130

Use CA cert for node-exporter TLS verification instead of insecure_sk…

c97c7ae

…ip_verify

mitali-salvi reviewed Mar 27, 2026

View reviewed changes

Comment thread ...cloudwatch-observability/templates/linux/_otel-container-insights-cluster-scraper-config.tpl Resolved

Comment thread charts/amazon-cloudwatch-observability/templates/linux/kube-state-metrics.yaml Resolved

Scope config-map permission for cloudwatch-agent to namespace only (#287

b12a6c5

) * scope configmap permission to namesapce * integ tests * updates

mitali-salvi changed the base branch from main to release-6.x March 27, 2026 20:11

louisall and others added 11 commits March 30, 2026 15:09

PR feedback

7e7160f

Fixing scope name

793dd42

Added instrumentation.cloudwatch.pipeline label

94e2006

Merge branch 'release-6.0.0' into container-insights-otlp

ae78830

Added node enrichment for KSM node metrics

0d472ef

Fixing hardcoded port. Adding scrape metatdata transform for api-serv…

48955bb

…er and cadvisor

Adding missing cw_k8s_ci_v0_scrape_metadata transform to ebc pipeline

886baaa

AGENTS.md file updates

7422f65

revert: undo lease-based KSM enrichment and AGENTS.md updates

c8d79bc

Reverts commits 7422f65 and 9324321 to remove the lease-based node metadata enrichment changes from the Helm chart

sky333999 force-pushed the release-6.0.0 branch from b12a6c5 to 0c281c0 Compare April 10, 2026 16:25

sky333999 added 5 commits April 10, 2026 16:31

Revert "Scope config-map permission for cloudwatch-agent to namespace…

c56b16b

… only (#287)" This reverts commit b12a6c5.

Support configurable hostNetwork per agent; default to true only for …

72bc7ae

…daemonset mode

Add hostNetwork integration tests, fix k8sclient.go missing brace, ad…

25a3b72

…d release-* to CI triggers

Fix EKS test: use correct KSM ServiceAccount name (kube-state-metrics…

5fd16ce

…-service-acct)

Fix EKS test: account for cluster-scraper-monitoring service in count…

4b25e1d

… and regex

sky333999 approved these changes Apr 13, 2026

View reviewed changes

sky333999 merged commit af10440 into release-6.0.0 Apr 13, 2026
16 checks passed

sky333999 deleted the container-insights-otlp branch April 13, 2026 05:27

Conversation

louisall commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Configuration

Testing

Uh oh!

louisall commented Mar 24, 2026

Known CI failure: Minikube integration tests timeout

Manual validation performed:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mitali-salvi Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

louisall Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

louisall commented Mar 24, 2026 •

edited

Loading