Skip to content

Add Metrics for table migration - #1779

Merged
akidambisrinivasan merged 2 commits into
awslabs:masterfrom
akidambisrinivasan:master
Jul 13, 2026
Merged

akidambisrinivasan merged 2 commits into
awslabs:masterfrom
akidambisrinivasan:master

Conversation

@akidambisrinivasan

@akidambisrinivasan akidambisrinivasan commented Jun 20, 2026 •

Copy link
Copy Markdown
Contributor

Metrics Summary:

Leader emitted always:
TableMigration:
- StatusOrdinal (NONE): Ordinal of DDB status (0=UNKNOWN, 1=INIT, 2=DEPLOYED, 3=PENDING, 4=COMPLETE) WorkerMetrics:
- FleetMinSupportCode (NONE): Minimum support code across all lease-owning workers

Leader emitted during migration:
TableMigration:
- Phase1Worker (COUNT): Workers with SINGLE_TABLE_MIGRATION support code minus Phase2Workers
- Phase2Worker (COUNT): Workers writing WorkerMetricStats to lease table
- PrePhase1Worker (COUNT): Workers on older versions without migration support code
- WriteFault (COUNT): 1 on DDB write failure, 0 on success
- DeleteFault (COUNT): 1 on DDB write failure, 0 on success
- CompletionFault (COUNT): 1 when transactional write of COMPLETE fails, 0 on success TableMigrationAsyncMove:
- Success (COUNT): 1 on successful move of coordinator state entries, 0 on failure
- Time (MILLISECONDS): Duration of the async move operation
- BatchCount (COUNT): Number of successfully moved batches

All workers emitted during migration:
TableMigration:
- ReadFault (COUNT): 1 on DDB read failure in readStateFromLegacy, 0 on success
- Time (MILLISECONDS): Duration of state machine run TableMigrationInitialize:
- Success (COUNT): 1 on successful initialize, 0 on failure
- Time (MILLISECONDS): Duration of initialize

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Metrics Summary:

Leader emitted always:
  TableMigration:
    - StatusOrdinal (NONE): Ordinal of DDB status (0=UNKNOWN, 1=INIT, 2=DEPLOYED, 3=PENDING, 4=COMPLETE)
  WorkerMetrics:
    - FleetMinSupportCode (NONE): Minimum support code across all lease-owning workers

Leader emitted during migration:
  TableMigration:
    - Phase1Worker (COUNT): Workers with SINGLE_TABLE_MIGRATION support code minus Phase2Workers
    - Phase2Worker (COUNT): Workers writing WorkerMetricStats to lease table
    - PrePhase1Worker (COUNT): Workers on older versions without migration support code
    - WriteFault (COUNT): 1 on DDB write failure, 0 on success
    - CompletionFault (COUNT): 1 when transactional write of COMPLETE fails, 0 on success
  TableMigrationAsyncMove:
    - Success (COUNT): 1 on successful move of coordinator state entries, 0 on failure
    - Time (MILLISECONDS): Duration of the async move operation
    - BatchCount (COUNT): Number of successfully moved batches

All workers emitted during migration:
  TableMigration:
    - ReadFault (COUNT): 1 on DDB read failure in readStateFromLegacy, 0 on success
    - Success (COUNT): 1 on successful state machine run, 0 on failure
    - Time (MILLISECONDS): Duration of state machine run
  TableMigrationInitialize:
    - Success (COUNT): 1 on successful initialize, 0 on failure
    - Time (MILLISECONDS): Duration of initialize
} catch (final InvalidStateException | ProvisionedThroughputException e) {
throw new DependencyException("Failed to write table migration status " + status + " to legacy", e);
} finally {
scope.addData(WRITE_FAULT_METRIC, success ? 0 : 1, StandardUnit.COUNT, MetricsLevel.SUMMARY);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This emits WriteFault=1 whenever success is false, but success stays false on two benign branches: the lost conditional write at line 1209 ("Another worker advanced state") and the "already exists in legacy, another worker created it" case at 1199. So WriteFault will spike during normal operation, and the doc defines it as "1 on DDB write failure." Can we set success=true on those branches so WriteFault only counts real DDB dependency failures?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those two are still write failures, the worker decided to create state and failed, and worker decided to update state and failed, I wanted to capture those race, eventually it should resolve, but its useful to see those race especially when it leads to inconsistent state due to a bug and doesn't resolve. Let me know what you think.

state,
e);
} finally {
scope.addData(WRITE_FAULT_METRIC, success ? 0 : 1, StandardUnit.COUNT, MetricsLevel.SUMMARY);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleteStateSafe emits WriteFault here, and writeStatusToLegacy emits WriteFault too (line 1215). A single handleLeaderLockResult cycle can do both (the Phase 1 rollback reset deletes and then writes), so two WriteFault datapoints land on the same scope. Could the delete path use its own metric name, e.g. DeleteFault, so each fault points at a specific op?


success = true;
} finally {
MetricsUtil.addSuccessAndLatency(scope, success, startTime, MetricsLevel.SUMMARY);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When another leader completes the migration we throw InvalidStateException at line 324 as a deliberate signal to release the lock. That throw happens before success = true, so this finally records Success=0 for the TableMigration operation on what's actually the normal completion handoff. The DependencyException we propagate on a transient DDB read blip hits the same path. So Success dips to 0 right when a cutover succeeds and on ordinary throttles. Is that intended?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the method itself doesnt or shouldnt fail the only exceptions are intended to be thrown to let the caller release the lock. Perhaps returning a boolean would have been a better way to achieve that. My main intention of the metric here is to capture the latency of the method to make sure we can monitor that it doesnt take too long since it happens during isLeader checks which should be fast. Ill just keep latency and remove the success since individual failures like ddb read/write failures etc are already covered by their own metrics.

…ss metric

- Introduce DELETE_FAULT_METRIC for deleteStateSafe to disambiguate from
  WriteFault
- Remove Success metric and retain latency (Time) tracking for
@nikhil-m-amazon

Copy link
Copy Markdown

LGTM after the latest commit, TY.

@akidambisrinivasan
akidambisrinivasan merged commit be6e6a8 into awslabs:master Jul 13, 2026
7 checks passed

This branch was previously deployed

1 inactive deployment
manual-approval β€” 5cd4e299 Deployed Jul 10, 2026 by akidambisrinivasan via wait-for-approval #275
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants