Skip to content

Table Migration metrics fixes - #1811

Merged
akidambisrinivasan merged 1 commit into
awslabs:masterfrom
akidambisrinivasan:master
Sep 10, 2026
Merged

akidambisrinivasan merged 1 commit into
awslabs:masterfrom
akidambisrinivasan:master

Conversation

@akidambisrinivasan

Copy link
Copy Markdown
Contributor

Fix the flaky rollback detection from phase2 to phase1. This is because anytime leader changes, it doesnt have the latest LAM state yet until LAM runs again, therefore it ends up using stale data to determine the phase2 vs phase1 worker and triggers a rollback.

Put in 2 fixes for this:

  1. Reset state (LAM state) when leadership is lost and dont perform rollback check without latest LAM state
  2. Monitor consistent rollback signals and not a one off, worker restart triggering the rollback. So a rollback signal must exist consecutively 20 iterations before rollback will be performed

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Fix the flaky rollback detection from phase2 to phase1. This is because
anytime leader changes, it doesnt have the latest LAM state yet until LAM
runs again, therefore it ends up using stale data to determine the phase2
vs phase1 worker and triggers a rollback.

Put in 2 fixes for this:
1. Reset state (LAM state) when leadership is lost and dont perform rollback check
   without latest LAM state
2. Monitor consistent rollback signals and not a one off, worker restart triggering
   the rollback. So a rollback signal must exist consecutively 20 iterations before
   rollback will be performed
@akidambisrinivasan
akidambisrinivasan merged commit a2d4be5 into awslabs:master Sep 10, 2026
6 of 7 checks passed

This branch was successfully deployed

1 active deployment
manual-approval β€” 49fcfa48 Deployed Sep 8, 2026 by akidambisrinivasan via wait-for-approval #304
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants