Skip to content

Add fixed-width threshold option for variance-based lease rebalancing - #1814

Merged
ehasah-aws merged 10 commits into
awslabs:masterfrom
ehasah-aws:low-throughput-rebalancing
Sep 24, 2026
Merged

ehasah-aws merged 10 commits into
awslabs:masterfrom
ehasah-aws:low-throughput-rebalancing

Conversation

@ehasah-aws

@ehasah-aws ehasah-aws commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

  • New boolean config useAbsoluteReBalanceThreshold (default false) is added.
  • When enabled, the rebalancing threshold range is calculated as fixed percentage points (avg Β± threshold/2) instead of relative to the average (avg Β± avgΓ—threshold/100). This prevents the range from collapsing at low CPU.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@ehasah-aws
ehasah-aws deployed to manual-approval September 22, 2026 16:47 — with GitHub Actions Active
@ehasah-aws ehasah-aws changed the title Fix frequent lease rebalancing on low/zero throughput shards Add fixed-width threshold option for variance-based lease rebalancing Sep 22, 2026
@ehasah-aws
ehasah-aws marked this pull request as ready for review September 22, 2026 20:54
private boolean allowThroughputOvershoot = true;

/**
* When set to true, the reBalanceThresholdPercentage is applied as fixed percentage points.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use {@link #reBalanceThresholdPercentage}?

WorkerMetricStats mostLoadedWorker = null;

log.info("Range for re-balance upper threshold {} and lower threshold {}", upperLimit, lowerLimit);
log.info("Range for re-balance upper threshold {} and lower threshold {}", upperLimit, Math.max(0, lowerLimit));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we clamp down lower and upper to be within 0-100 ?

/**
* When set to true, the reBalanceThresholdPercentage is applied as fixed percentage points.
* The rebalancing range is calculated as
* [fleetAvgCPU - reBalanceThresholdPercentage/2, fleetAvgCPU + reBalanceThresholdPercentage/2].

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid using cpu sicne this is across all metrics? Maybe use fleetAvgMetricValue ?

upperLimit = Math.min(100, workerMetricsValueAvg + halfBand);
lowerLimit = Math.max(0, workerMetricsValueAvg - halfBand);
} else {
upperLimit = workerMetricsValueAvg * (1.0D + (double) reBalanceThreshold / 100);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets apply the clamping here too?

@ehasah-aws
ehasah-aws merged commit 902fb6e into awslabs:master Sep 24, 2026
4 of 6 checks passed

This branch had an error being deployed

1 failed deployment
manual-approval β€” 64bd5f8b Deployed Sep 24, 2026 by ehasah-aws via wait-for-approval #327
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants