Skip to content

Steal 1 lease from workers at target in lease count-based balancing - #1837

Merged
chenylee-aws merged 1 commit into
awslabs:masterfrom
jahyunk:fix/lease-count-min-one-lease
Oct 28, 2026
Merged

chenylee-aws merged 1 commit into
awslabs:masterfrom
jahyunk:fix/lease-count-min-one-lease

Conversation

@jahyunk

@jahyunk jahyunk commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

With LEASE_COUNT_BASED assignment, LeaseCountBasedLeaseAssignmentDecider.balanceWorkerVariance() only moves a lease off a worker that holds more than the target. When every loaded worker holds exactly the target, a new worker gets no leases and stays at 0.

Example from a production fleet: 21520 leases, 1215 active workers, target ceil(21520 / 1215) = 18. 1193 workers hold 18, and 17 workers (including a newly deployed host) hold 0. The leader logs No new lease assignment performed in this iteration on every run, so the new host never processes traffic.

KCL 2.x did not have this gap. DynamoDBLeaseTaker.chooseLeasesToSteal steals 1 lease when the worker needs more than 1 and the most loaded worker has exactly the target:

// steal 1 if we need > 1 and max loaded worker has target leases.
if ((needed > 1) && (numLeasesToSteal == 0)) {
    numLeasesToSteal = 1;
}

This change applies the same rule in the two-pointer loop of balanceWorkerVariance():

  • A worker that holds exactly the target (excess == 0) gives 1 lease to a worker that needs more than 1 (needed > 1).
  • A worker that needs only 1 lease and has no over-target worker to take from moves the left pointer forward, so the remaining under-target workers are served in the same run.

Unchanged:

  • Workers above the target still give first.
  • Leases pending a graceful handoff are still skipped.
  • When there are at least as many workers as leases (target 1), needed is never above 1, so nothing changes.
  • A worker at the target only drops to target - 1, so it does not take a lease back.

Testing:

  • Added balanceWorkerVariance_allLoadedWorkersAtTarget_emptyWorkerStealsUpToTargetMinusOne and balanceWorkerVariance_workerNeedsOnlyOneLease_noStealFromWorkersAtTarget.
  • mvn -pl amazon-kinesis-client spotless:check passes.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Match the KCL 2.x DynamoDBLeaseTaker rule: steal 1 lease when the worker needs more than 1 and the most loaded worker has exactly the target. Without this, a new worker gets no leases when all other workers are at the target.
@chenylee-aws
chenylee-aws merged commit 79b0cbe into awslabs:master Oct 28, 2026
1 of 2 checks passed
vincentvilo-aws pushed a commit to vincentvilo-aws/amazon-kinesis-client that referenced this pull request Nov 3, 2026
…wslabs#1837)

Match the KCL 2.x DynamoDBLeaseTaker rule: steal 1 lease when the worker needs more than 1 and the most loaded worker has exactly the target. Without this, a new worker gets no leases when all other workers are at the target.
vincentvilo-aws added a commit to vincentvilo-aws/amazon-kinesis-client that referenced this pull request Nov 5, 2026
balanceWorkerVariance() returns early when getAssignableWorkers() is empty. The
stubStorageView() helper, used by two tests added in awslabs#1837, didn't stub the
assignable workers, so one test failed and the other passed without balancing.
Stub them from the worker-to-leases map.

This branch is waiting to be deployed

1 waiting deployment
manual-approval — da22a569 Waiting Oct 28, 2026 by jahyunk via wait-for-approval #362
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants