Skip to content

Resolve shard hash key range from the lease via ShardInfo instead of ListShards - #1828

Merged
chenylee-aws merged 3 commits into
awslabs:masterfrom
chenylee-aws:hashkey-shardinfo
Oct 19, 2026
Merged

chenylee-aws merged 3 commits into
awslabs:masterfrom
chenylee-aws:hashkey-shardinfo

Conversation

@chenylee-aws

@chenylee-aws chenylee-aws commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

With the default skipShardSyncAtWorkerInitializationIfLeasesExist=false, ProcessTask called shardDetector.shard() on every batch to get the shard's hash key range for KPL de-duplication. On a cold or stale shard cache (worker start, reshard, lease takeover) that calls ListShards on the data path, where a failure used to drop the batch.

KCL stores the hash key range on every lease since 2.3, so this change takes it from the lease instead:

  • DynamoDBLeaseCoordinator.convertLeaseToAssignment copies the lease's range into a new optional ShardInfo field (new constructor overload; existing constructors unchanged; excluded from equals/hashCode/toString).
  • ShardConsumerArgument.hashKeyRange() reads it from ShardInfo and caches it per consumer, falling back to the ShardDetector only for legacy leases or custom coordinators that don't set it. A ShardDetector failure propagates, so the batch is retried by the dispatch retry from Retry records delivery in shardConsumer #1822 instead of being delivered without range filtering.
  • ProcessTask takes the resolved range instead of a ShardDetector.

No behavior change for skip=true (still no range filtering). Additive API only.

Testing:

  • Unit: ConsumerStatesTest, ShardInfoTest, DynamoDBLeaseCoordinatorTest, ProcessTaskTest; mvn clean install passes.
  • End-to-end on a 2-shard stream with KPL aggregates whose user records hash into their own shard (100 "in") or into the other shard (100 "cross", the post-reshard duplicate case). A TaskExecutionListener recorded ShardInfo.hashKeyRange() per process task and an SDK interceptor classified every ListShards caller:
Run ShardInfo range Delivered in / cross Data-path ListShards
This PR, skip=false matches the lease and shard ranges 100 / 0 (filtered) 0
This PR, skip=true set, unused 100 / 100 (unchanged: no filtering) 0
This PR, legacy leases (range removed) null 100 / 0 1 (fallback, then backfilled)
master, skip=false n/a 100 / 0 1 (ProcessTask.<init>)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@chenylee-aws chenylee-aws changed the title Carry the lease hash key range in ShardInfo Resolve shard hash key range from the lease via ShardInfo instead of ListShards Oct 15, 2026
@chenylee-aws
chenylee-aws merged commit a43767a into awslabs:master Oct 19, 2026
5 of 6 checks passed
@chenylee-aws
chenylee-aws deleted the hashkey-shardinfo branch October 19, 2026 17:33
vincentvilo-aws pushed a commit to vincentvilo-aws/amazon-kinesis-client that referenced this pull request Oct 26, 2026
chenylee-aws added a commit to chenylee-aws/amazon-kinesis-client that referenced this pull request Oct 26, 2026

This branch is waiting to be deployed

1 waiting deployment
manual-approval — 204fe3f8 Waiting Oct 19, 2026 by chenylee-aws via wait-for-approval #352
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants