[nathalapooja]

Summary

Add Local Instance Store (LIS) CSI driver metrics support to the OTEL container insights pipeline, mirroring the existing EBS CSI pattern. This enables the OTEL collector to scrape LIS CSI metrics and export them with proper OTEL attributes via the OTLP endpoint.

Changes

Added 4 components to _otel-container-insights-config.tpl:

1. Receiver (prometheus/cw_k8s_ci_v0_lis_csi_node) — Prometheus scrape config targeting pods with label app=ec2-instance-store-plugin in kube-system namespace, filtered to the current node and the metrics port (8101).

2. Scope Transform (transform/cw_k8s_ci_v0_set_scope_lis_csi) — Sets instrumentation scope attributes:

   • cloudwatch.pipeline = "lis-csi"
   • cloudwatch.solution = "k8s-otel-container-insights"
   • cloudwatch.source = "cloudwatch-agent"

3. Attribute Promote Transform (transform/cw_k8s_ci_v0_lis_csi_promote) — Promotes instance_id and volume_id from datapoint attributes to resource attributes (same as EBS CSI, since LIS CSI uses the same dimension sets).

4. Pipeline (metrics/cw_k8s_ci_v0_lis_csi_node) — Wires the receiver through the standard processor chain: scrape metadata filter → set unit → metric start time → cluster name → lis_csi promote → node name → resource detection → cloud resource ID → k8s node attributes → lis_csi scope → clear schema → workload → attribute limit → batch → OTLP export.

Metrics Collected (13 metrics)

The LIS CSI driver exposes these raw Prometheus metrics (scraped as-is by the OTEL pipeline):

Metric	Type
aws_ec2_instance_store_csi_read_bytes_total	Counter
aws_ec2_instance_store_csi_write_bytes_total	Counter
aws_ec2_instance_store_csi_read_ops_total	Counter
aws_ec2_instance_store_csi_write_ops_total	Counter
aws_ec2_instance_store_csi_read_seconds_total	Counter
aws_ec2_instance_store_csi_write_seconds_total	Counter
aws_ec2_instance_store_csi_ec2_exceeded_iops_seconds_total	Counter
aws_ec2_instance_store_csi_ec2_exceeded_tp_seconds_total	Counter
aws_ec2_instance_store_csi_volume_queue_length	Gauge
aws_ec2_instance_store_csi_read_io_latency_seconds	Histogram
aws_ec2_instance_store_csi_write_io_latency_seconds	Histogram
aws_ec2_instance_store_csi_nvme_collector_errors_total	Counter (collector-internal)
aws_ec2_instance_store_csi_nvme_collector_scrapes_total	Counter (collector-internal)

Note on metric names: The OTEL pipeline passes through the raw Prometheus metric names (aws_ec2_instance_store_csi_*) as-is. This differs from the enhanced container insights (EMF) pipeline which renames them to node_diskio_instance_store_* via a metricstransform processor.
Note on collector-internal metrics: nvme_collector_errors_total and nvme_collector_scrapes_total are included for exporter health observability. They are low-cardinality and follow the same _total → unit "1" convention.

Testing

End-to-end validated on EKS cluster cwa-liscsi-test-arm64 (arm64 nodes).

Setup

• Installed aws-ec2-local-instance-store-csi-driver EKS addon with {"metrics":{"enabled":true}}
• Installed eks-pod-identity-agent EKS addon
• Created Pod Identity association for CloudWatch agent service account
• Deployed test PVC (ec2-instance-store-sc) and pod generating continuous disk I/O
• Installed helm chart from this branch with agent.image.tag=latest

Validation

Queried the OTLP PromQL API at https://monitoring.us-west-2.amazonaws.com/api/v1/query with SigV4 signing:

All 13 metrics returned results ✅

Attribute validations ✅

• @resource.instance_id present, starts with i- (promoted from datapoint to resource scope)
• @resource.volume_id present (promoted from datapoint to resource scope)
• instance_id absent from datapoint scope (confirms promotion)
• volume_id absent from datapoint scope (confirms promotion)
• @instrumentation.cloudwatch.pipeline = lis-csi
• @instrumentation.cloudwatch.solution = k8s-otel-container-insights
• @resource.k8s.cluster.name = cwa-liscsi-test-arm64
• @resource.service.name = lis-csi-node

[miconeilaws]

Can we check with @louisall or @spanaik about the E2E testing? We must ensure these metrics conform to the same semantic conventions, resource attributes, K8s labels etc as the exsiting OtelCI metrics