🔥 Jadwal & Hasil: awslabs/amazon kinesis client/pull/

wisemanIV · 2026-04-06T20:07:05Z

Add alternative metrics publishing path that sends KCL metrics to the CloudWatch OTEL endpoint in OTLP format. Users can choose between the existing CloudWatch PutMetricData API and the new OTEL endpoint via MetricsConfig.metricsBackend().

New classes:

MetricsBackend enum (CLOUDWATCH, CLOUDWATCH_OTEL)
OtelMetricsFactory, OtelMetricsScope, OtelPublisherRunnable
OtelMetricsPublisher with OTLP Histogram conversion
OtelMetricKey for metric accumulation

Key behaviors:

OTEL path defaults to 30s publish interval (vs 10s for CloudWatch)
Maps KCL dimensions to OTEL semantic conventions
Auto-detects resource attributes from KCL configuration
Custom resource attributes override auto-detected ones
Backward compatible: default remains CloudWatch endpoint

Tests:

OtelMetricsPublisherTest (batching, conversion, dimension mapping)
OtelPublisherRunnableTest (flush, timeout, drain, shutdown)
OtelMetricsConfigTest (backend selection, buffer time defaults)

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Add alternative metrics publishing path that sends KCL metrics to the CloudWatch OTEL endpoint in OTLP format. Users can choose between the existing CloudWatch PutMetricData API and the new OTEL endpoint via MetricsConfig.metricsBackend(). New classes: - MetricsBackend enum (CLOUDWATCH, CLOUDWATCH_OTEL) - OtelMetricsFactory, OtelMetricsScope, OtelPublisherRunnable - OtelMetricsPublisher with OTLP Histogram conversion - OtelMetricKey for metric accumulation Key behaviors: - OTEL path defaults to 30s publish interval (vs 10s for CloudWatch) - Maps KCL dimensions to OTEL semantic conventions - Auto-detects resource attributes from KCL configuration - Custom resource attributes override auto-detected ones - Backward compatible: default remains CloudWatch endpoint Tests: - OtelMetricsPublisherTest (batching, conversion, dimension mapping) - OtelPublisherRunnableTest (flush, timeout, drain, shutdown) - OtelMetricsConfigTest (backend selection, buffer time defaults)

wisemanIV · 2026-04-09T09:05:38Z

@chenylee-aws @lucienlu-aws @ehasah-aws would appreciate your review.

Declare opentelemetry-api, opentelemetry-sdk-common, and opentelemetry-sdk-metrics as explicit compile-scope dependencies to satisfy maven-dependency-plugin analyze-only which flags used but undeclared transitive dependencies. Versions managed by the OTel BOM.

wisemanIV · 2026-04-20T09:10:19Z

please can you review / approve? @chenylee-aws @lucienlu-aws @ehasah-aws

john-karp

As a user of both Otel and Kinesis, this looks pretty interesting. Leaving some comments.

Remove auto-detected service.name and cloud.provider resource attributes from MetricsConfig to let the host application own service-level resource configuration. KCL identifies itself via instrumentation scope. Use aws.kinesis.stream_name per OTel semantic conventions for Kinesis instead of the generic messaging.destination.name attribute. Mark all OpenTelemetry dependencies as optional so they are not pulled transitively into consumer projects that do not use OTel metrics.

Remove jqwik test dependency and version property since no property-based tests currently exist in the codebase.

wisemanIV · 2026-04-26T18:52:55Z

As a user of both Otel and Kinesis, this looks pretty interesting. Leaving some comments.

Thanks for the feedback @john-karp. I've implemented most of the suggestions with the exception of the unit types. Please see explanation and let me know if you disagree.

john-karp

I'm not sure this design is interacting with otel at the right level of abstraction. It is hard-coding the OtlpHttpMetricExporter, and as far as I can tell it is not pulling in standard environment configuration of attributes and exporters.

Looking at:

It would seem the standard way for a library to natively instrument itself with OTEL is:

"Libraries should only use the OpenTelemetry API" (that is, not the SDK packages)
"Native instrumentation should use GlobalOpenTelemetry.getOrNoop() as the default {@link OpenTelemetry} instance, and expose APIs for setting a custom instance."
"The Instrumentation scope represents a logical unit within the application code with which the emitted telemetry can be associated. [...] In the case of a library or framework, a common approach is to use an identifier as scope that is unique to the library or framework, such as a fully qualified name and version of the library or framework."

So then I think the pattern would be to use meterBuilder/getMeter instantiated by the OpenTelemetry instance that was returned from GlobalOpenTelemetry.getOrNoop(), with the appropriate instrumentation scope name set.

That is difficult though if you are trying to process pre-aggregated MetricDatum objects. Based on the pattern that's used in my application, you could change OtelMetricsScope to override addData, which handles individual data points. And then each OtelMetricsScope could have an internal cache for looking up OTEL meter instances by name, instantiated with the proper meter type.

Then you'd get auto-configuration of exporter and attributes, and you could skip all the buffering logic.

…trumentation Replace the custom publisher model (OtlpHttpMetricExporter, custom batching, SDK dependencies) with native OTel library instrumentation following OpenTelemetry best practices for libraries. Key changes: - OtelMetricsScope now implements MetricsScope directly and records raw observations on OTel Meter instruments (DoubleHistogram, LongCounter) - OtelMetricsFactory accepts an OpenTelemetry instance, defaults to GlobalOpenTelemetry.get() for standard SDK autoconfiguration - Only opentelemetry-api is a compile dependency (optional); SDK deps moved to test scope - Removed OtelMetricsPublisher, OtelPublisherRunnable, OtelMetricKey - New MetricsBackend.OTEL enum value; CLOUDWATCH_OTEL deprecated - MetricsConfig exposes optional OpenTelemetry field; otelEndpoint and otelResourceAttributes deprecated

wisemanIV · 2026-04-27T21:43:37Z

I'm not sure this design is interacting with otel at the right level of abstraction. It is hard-coding the OtlpHttpMetricExporter, and as far as I can tell it is not pulling in standard environment configuration of attributes and exporters.

Looking at:

https://opentelemetry.io/docs/concepts/instrumentation/libraries/#opentelemetry-api

https://opentelemetry.io/docs/languages/java/instrumentation/#native-instrumentation

https://opentelemetry.io/docs/languages/java/api/#globalopentelemetry

https://opentelemetry.io/docs/concepts/instrumentation-scope/

It would seem the standard way for a library to natively instrument itself with OTEL is:

"Libraries should only use the OpenTelemetry API" (that is, not the SDK packages)

"Native instrumentation should use GlobalOpenTelemetry.getOrNoop() as the default {@link OpenTelemetry} instance, and expose APIs for setting a custom instance."

"The Instrumentation scope represents a logical unit within the application code with which the emitted telemetry can be associated. [...] In the case of a library or framework, a common approach is to use an identifier as scope that is unique to the library or framework, such as a fully qualified name and version of the library or framework."

So then I think the pattern would be to use meterBuilder/getMeter instantiated by the OpenTelemetry instance that was returned from GlobalOpenTelemetry.getOrNoop(), with the appropriate instrumentation scope name set.

That is difficult though if you are trying to process pre-aggregated MetricDatum objects. Based on the pattern that's used in my application, you could change OtelMetricsScope to override addData, which handles individual data points. And then each OtelMetricsScope could have an internal cache for looking up OTEL meter instances by name, instantiated with the proper meter type.

Then you'd get auto-configuration of exporter and attributes, and you could skip all the buffering logic.

I agree with the feedback. I was trying to add an exported to the existing metrics pipeline but needed to go higher in the architecture.

wisemanIV · 2026-04-27T21:45:22Z

@john-karp thanks for the feedback. I've made some significant changes to be more OTel native.

john-karp

Thanks for making the changes. I hope the maintainers accept it. 😅

john-karp · 2026-04-28T20:59:08Z

+        if (obs.unit == StandardUnit.COUNT) {
+            LongCounter counter = meter.counterBuilder(obs.name)
+                    .setUnit(convertUnit(obs.unit))
+                    .build();
+            counter.add((long) obs.value, attrs);
+        } else {
+            DoubleHistogram histogram = meter.histogramBuilder(obs.name)
+                    .setUnit(convertUnit(obs.unit))
+                    .build();
+            histogram.record(obs.value, attrs);


I think this mapping works pretty well for the majority of the metrics in KCL. There are some StandardUnit.COUNT metrics that might be better represented as Gauge though:

CurrentLeases, TotalLeases, NumWorkers (DynamoDBLeaseRenewer, DynamoDBLeaseTaker)

ExpiredLeases, VeryOldLeases (the snapshot-style emission sites in DynamoDBLeaseTaker, LeaseAssignmentManager)

ActiveStreams.Count, StreamsPendingDeletion.Count (Scheduler)

QueueSize (StreamIdCacheManager)

NumStreamsToSync (PeriodicShardSyncManager)

NumNewAssignmentsForWorker., LeaderLeasesByWorker., etc. (LeaseAssignmentManager)

Hard-coding that list of exceptions doesn't seem great though; it makes it harder for people to know to update it here when they add or change a metric of KCL.

Ideally, all of the callers of MetricsScope.addData could be more explicit about what meter type semantics they are using. Then OtelMetricsScope could use that info to choose the type to use (and CloudWatchMetricsScope could ignore it). But that would require changes to the MetricsScope interface, and changes to every caller of MetricsScope.addData. And I'm not sure how open the KCL maintainers are to that.

Yeah, great call on the gauge vs counter distinction, those snapshot-style metrics (CurrentLeases, TotalLeases, QueueSize, etc.) really are gauges semantically. Agreed that hard-coding a list of exceptions here would be brittle and easy to forget when new metrics get added.

I think the right long-term fix is what you described - having callers of addData declare the instrument semantics explicitly, so OtelMetricsScope can pick the right instrument type. That would mean extending the MetricsScope interface (maybe an addGauge method or an instrument type hint parameter) and updating the call sites. CloudWatch side could just ignore the hint since PutMetricData doesn't distinguish.

That's a bigger change though, and probably better as a follow-up once we see how the maintainers feel about the current PR. For now the counter/histogram split by unit is a reasonable default that works for the majority of KCL metrics. I'll file an issue to track the gauge support as a future improvement.

Record OTel observations immediately in addData() instead of buffering until end(). OTel API calls are designed to be fast and low-allocation, so buffering adds unnecessary overhead and delays timestamps. Remove MetricObservation class which is no longer needed. Make opentelemetry-api a regular (non-optional) dependency to avoid ClassLoader issues when MetricsConfig references OpenTelemetry classes. The API jar is small (~200KB) with zero transitive deps and noop runtime cost when no SDK is configured. Remove unused test-scope dependencies (opentelemetry-sdk, opentelemetry-sdk-testing, jqwik) since tests only use OpenTelemetry.noop() from the API package.

john-karp

I thought of a few more things to consider. Also, I think overall the unit test coverage might be worth taking a look at.

Cache built Attributes on first addData() call and reuse for subsequent calls since the MetricsScope contract guarantees dimensions are added before data. Remove CLOUDWATCH_OTEL enum value entirely since it was never in a GA release. Remove deprecated otelEndpoint and otelResourceAttributes fields from MetricsConfig. Use DoubleCounter instead of LongCounter to avoid precision loss from the long cast. Use EnumMap for UNIT_MAP for better enum key performance. Use GlobalOpenTelemetry.getOrNoop() instead of .get() to avoid the side effect of permanently locking in the noop instance. Add hard-coded GAUGE_METRIC_NAMES set for snapshot-style metrics (CurrentLeases, TotalLeases, NumWorkers, etc.) that should be recorded as DoubleGauge rather than counters. Add OtelMetricNameTransformer with 8-step pipeline to normalize KCL metric names to OTel conventions: special-case rename map, colon substitution, .Time to .duration, strip .Count, PascalCase splitting, lowercase, aws.kinesis.client. namespace prefix, and BNF validation.

Add OtelMetricNameTransformerTest (30 tests) covering all 11 special-case map entries, colon substitution, .Time to .duration, .Count stripping, PascalCase splitting with uppercase runs, namespace prefix, and edge cases. Add OtelMetricsScopeTest (37 tests) covering gauge/counter/histogram instrument selection, dimension-to-attribute mapping, MetricsLevel filtering, dimension filtering, end-state enforcement, unit conversion, and noop lifecycle tests. Add OtelMetricsFactoryTest (10 tests) covering scope creation, shutdown behavior, noop OpenTelemetry integration, and constructor validation.

wisemanIV · 2026-05-01T21:09:15Z

I thought of a few more things to consider. Also, I think overall the unit test coverage might be worth taking a look at.

Thanks for the feedback @john-karp. Suggested changes implemented and unit tests extended.

lucienlu-aws

Change generally looks good to me. Have you tested this in a running KCL app with OTEL configured and verified it works on OTEL end already?

lucienlu-aws · 2026-05-15T23:49:17Z

+     * KCL metric names (original, pre-transformation) that should be recorded as gauges
+     * rather than counters or histograms.
+     */
+    static final Set<String> GAUGE_METRIC_NAMES = Collections.unmodifiableSet(new HashSet<>(Arrays.asList(


What is the behavior if we add a metric to KCL and don't include it here? Would it be recorded as a counter in OTEL?

If a new metric is added and NOT in GAUGE_METRIC_NAMES:

If its unit is StandardUnit.COUNT → it's recorded as a DoubleCounter (monotonically increasing)
If its unit is anything else (MILLISECONDS, BYTES, etc.) → it's recorded as a DoubleHistogram
So yes, a metric with StandardUnit.COUNT that isn't in the gauge set would be a counter. This is correct for most count metrics (RecordsProcessed, Success, etc.) but would be wrong for point-in-time counts like "CurrentLeases" — which is why those are explicitly listed in GAUGE_METRIC_NAMES.

lucienlu-aws · 2026-05-16T00:11:56Z

+ * </ol>
+ */
+@Slf4j
+class OtelMetricNameTransformer {


Since this is implicitly protected I'm assuming this can have backwards incompatible changes internally? Can you mark this as @KinesisClientInternalAPI?

yes agreed. pushing

lucienlu-aws · 2026-05-16T00:16:43Z

+        map.put("NumWorkers", "workers");
+        map.put("NumWorkersWithInvalidEntry", "workers.invalid_entry");
+        map.put("NumWorkersWithFailingWorkerMetric", "workers.failing_worker_metric");
+        SPECIAL_CASE_MAP = Collections.unmodifiableMap(map);


What happens if we add a metric that is not in the map and does not comply with OTEL names? Considering the case where we add a metric and forget to add it here and the transformName method transforms it to something OTEL doesn't like. Does it not send the metric at all?

The metric is still sent — it's never dropped. The validate() method sanitizes illegal characters by replacing them with _ and logs a warning. So the worst case is a slightly mangled metric name with a log warning, not a lost metric. The OTel SDK itself is also lenient — it will accept most names and only reject truly malformed ones at the exporter level (which would log an error but not crash).

wisemanIV · 2026-05-18T22:52:19Z

Change generally looks good to me. Have you tested this in a running KCL app with OTEL configured and verified it works on OTEL end already?

Yes. We deployed a sample KCL app on ECS Fargate (account 322962300869, us-east-1) using MetricsBackend.OTEL with the 3.4.3-SNAPSHOT from this branch. The app configures an OTel SDK SdkMeterProvider with a custom MetricExporter that sends OTLP/JSON to CloudWatch's monitoring.us-east-1.amazonaws.com/v1/metrics endpoint with SigV4 signing. We confirmed:

34 KCL metrics are being exported successfully (HTTP 200 from CloudWatch)
Metrics are queryable via CloudWatch's PromQL endpoint at /api/v1/query
Metric names follow the aws.kinesis.client.* convention (e.g., aws.kinesis.client.records_processed, aws.kinesis.client.record_processor.process_records.duration, aws.kinesis.client.current_leases)
Labels include aws.kinesis.operation, aws.kinesis.shard.id, @resource.service.name
A Grafana dashboard visualizes the metrics end-to-end

…nsformer Mark OtelMetricNameTransformer as internal API since it is package-private, used only by OtelMetricsScope, and its transformation logic is an implementation detail that may change without notice.

Move addWorkerIdentifier before addData in DynamoDBLeaseRenewer to ensure the OTel scope records observations with all dimensions attached. Add ordering contract documentation to MetricsScope javadoc and a TODO for future AutoCloseable improvement. Add unit tests verifying the dimension ordering behavior.

wisemanIV · 2026-05-19T08:17:22Z

Retested with latest changes:

Complete the dimension ordering fix across all affected files: - DynamoDBLeaseDiscoverer - DynamoDBLeaseCoordinator - DynamoDBLeaseTaker - WorkerMetricStatsReporter - DynamoDBLeaseRenewer (second usage in renewLease method) This ensures the WorkerIdentifier dimension is attached to all OTel metric observations, not just those recorded after the finally block.

The sample-app directory contains a demo deployment with binary JARs and CDK build artifacts. It should not be part of the library PR.

…ting The splitSegment method now inserts '_' at digit→uppercase transitions (e.g., Retry3Count → retry3_count), matching the test expectation.

wisemanIV temporarily deployed to manual-approval April 6, 2026 20:07 — with GitHub Actions Inactive

wisemanIV had a problem deploying to manual-approval April 14, 2026 14:05 — with GitHub Actions Failure

john-karp reviewed Apr 23, 2026

View reviewed changes

wisemanIV added 2 commits April 26, 2026 19:44

chore(deps): Remove unused jqwik dependency

d46c281

Remove jqwik test dependency and version property since no property-based tests currently exist in the codebase.

wisemanIV had a problem deploying to manual-approval April 26, 2026 18:50 — with GitHub Actions Failure

john-karp reviewed Apr 27, 2026

View reviewed changes

Comment thread amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/OtelMetricsPublisher.java Resolved

Comment thread amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/OtelMetricKey.java Resolved

Comment thread amazon-kinesis-client/.jqwik-database Resolved

wisemanIV had a problem deploying to manual-approval April 27, 2026 21:36 — with GitHub Actions Failure

chore: Remove stray .jqwik-database file

b0ad1a0

wisemanIV had a problem deploying to manual-approval April 27, 2026 21:40 — with GitHub Actions Failure

john-karp reviewed Apr 28, 2026

View reviewed changes

wisemanIV had a problem deploying to manual-approval April 29, 2026 10:54 — with GitHub Actions Failure

john-karp reviewed May 1, 2026

View reviewed changes

wisemanIV added 2 commits May 1, 2026 21:56

wisemanIV had a problem deploying to manual-approval May 1, 2026 21:08 — with GitHub Actions Failure

lucienlu-aws reviewed May 16, 2026

View reviewed changes

refactor(metrics): Add @KinesisClientInternalApi to OtelMetricNameTra…

0c10303

…nsformer Mark OtelMetricNameTransformer as internal API since it is package-private, used only by OtelMetricsScope, and its transformation logic is an implementation detail that may change without notice.

wisemanIV had a problem deploying to manual-approval May 18, 2026 22:56 — with GitHub Actions Failure

john-karp reviewed May 18, 2026

View reviewed changes

Comment thread amazon-kinesis-client/src/main/java/software/amazon/kinesis/metrics/OtelMetricsScope.java Resolved

wisemanIV had a problem deploying to manual-approval May 19, 2026 08:15 — with GitHub Actions Failure

wisemanIV had a problem deploying to manual-approval May 21, 2026 16:44 — with GitHub Actions Failure

wisemanIV had a problem deploying to manual-approval May 21, 2026 18:14 — with GitHub Actions Failure

chore: Remove sample-app from PR

79da52a

The sample-app directory contains a demo deployment with binary JARs and CDK build artifacts. It should not be part of the library PR.

wisemanIV force-pushed the feat/cloudwatch-otel-metrics-support branch from c81e6f3 to 79da52a Compare May 21, 2026 18:17

wisemanIV temporarily deployed to manual-approval May 21, 2026 18:17 — with GitHub Actions Inactive

style: Apply spotless formatting

408ccbf

wisemanIV had a problem deploying to manual-approval May 22, 2026 13:48 — with GitHub Actions Failure

fix(metrics): Handle digit-to-uppercase transition in camelCase split…

e90c913

…ting The splitSegment method now inserts '_' at digit→uppercase transitions (e.g., Retry3Count → retry3_count), matching the test expectation.

wisemanIV temporarily deployed to manual-approval May 22, 2026 13:52 — with GitHub Actions Inactive

lucienlu-aws approved these changes May 22, 2026

View reviewed changes

lucienlu-aws merged commit 4faf650 into awslabs:master May 22, 2026
13 of 16 checks passed

Conversation

wisemanIV commented Apr 6, 2026

Uh oh!

wisemanIV commented Apr 9, 2026

Uh oh!

wisemanIV commented Apr 20, 2026

Uh oh!

john-karp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wisemanIV commented Apr 26, 2026

Uh oh!

john-karp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wisemanIV commented Apr 27, 2026

Uh oh!

wisemanIV commented Apr 27, 2026

Uh oh!

john-karp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

john-karp Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

wisemanIV Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

john-karp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wisemanIV commented May 1, 2026

Uh oh!

lucienlu-aws left a comment

Choose a reason for hiding this comment

Uh oh!

lucienlu-aws May 15, 2026

Choose a reason for hiding this comment

Uh oh!

wisemanIV May 18, 2026

Choose a reason for hiding this comment

Uh oh!

lucienlu-aws May 16, 2026

Choose a reason for hiding this comment

Uh oh!

wisemanIV May 18, 2026

Choose a reason for hiding this comment

Uh oh!

lucienlu-aws May 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wisemanIV May 18, 2026

Choose a reason for hiding this comment

Uh oh!

wisemanIV commented May 18, 2026

Uh oh!

Uh oh!

wisemanIV commented May 19, 2026

john-karp left a comment •

edited

Loading

lucienlu-aws May 16, 2026 •

edited

Loading