Repository navigation
Add Metrics for table migration - #1779
Conversation
Metrics Summary:
Leader emitted always:
TableMigration:
- StatusOrdinal (NONE): Ordinal of DDB status (0=UNKNOWN, 1=INIT, 2=DEPLOYED, 3=PENDING, 4=COMPLETE)
WorkerMetrics:
- FleetMinSupportCode (NONE): Minimum support code across all lease-owning workers
Leader emitted during migration:
TableMigration:
- Phase1Worker (COUNT): Workers with SINGLE_TABLE_MIGRATION support code minus Phase2Workers
- Phase2Worker (COUNT): Workers writing WorkerMetricStats to lease table
- PrePhase1Worker (COUNT): Workers on older versions without migration support code
- WriteFault (COUNT): 1 on DDB write failure, 0 on success
- CompletionFault (COUNT): 1 when transactional write of COMPLETE fails, 0 on success
TableMigrationAsyncMove:
- Success (COUNT): 1 on successful move of coordinator state entries, 0 on failure
- Time (MILLISECONDS): Duration of the async move operation
- BatchCount (COUNT): Number of successfully moved batches
All workers emitted during migration:
TableMigration:
- ReadFault (COUNT): 1 on DDB read failure in readStateFromLegacy, 0 on success
- Success (COUNT): 1 on successful state machine run, 0 on failure
- Time (MILLISECONDS): Duration of state machine run
TableMigrationInitialize:
- Success (COUNT): 1 on successful initialize, 0 on failure
- Time (MILLISECONDS): Duration of initialize
| } catch (final InvalidStateException | ProvisionedThroughputException e) { | ||
| throw new DependencyException("Failed to write table migration status " + status + " to legacy", e); | ||
| } finally { | ||
| scope.addData(WRITE_FAULT_METRIC, success ? 0 : 1, StandardUnit.COUNT, MetricsLevel.SUMMARY); |
There was a problem hiding this comment.
This emits WriteFault=1 whenever success is false, but success stays false on two benign branches: the lost conditional write at line 1209 ("Another worker advanced state") and the "already exists in legacy, another worker created it" case at 1199. So WriteFault will spike during normal operation, and the doc defines it as "1 on DDB write failure." Can we set success=true on those branches so WriteFault only counts real DDB dependency failures?
There was a problem hiding this comment.
Those two are still write failures, the worker decided to create state and failed, and worker decided to update state and failed, I wanted to capture those race, eventually it should resolve, but its useful to see those race especially when it leads to inconsistent state due to a bug and doesn't resolve. Let me know what you think.
| state, | ||
| e); | ||
| } finally { | ||
| scope.addData(WRITE_FAULT_METRIC, success ? 0 : 1, StandardUnit.COUNT, MetricsLevel.SUMMARY); |
There was a problem hiding this comment.
deleteStateSafe emits WriteFault here, and writeStatusToLegacy emits WriteFault too (line 1215). A single handleLeaderLockResult cycle can do both (the Phase 1 rollback reset deletes and then writes), so two WriteFault datapoints land on the same scope. Could the delete path use its own metric name, e.g. DeleteFault, so each fault points at a specific op?
|
|
||
| success = true; | ||
| } finally { | ||
| MetricsUtil.addSuccessAndLatency(scope, success, startTime, MetricsLevel.SUMMARY); |
There was a problem hiding this comment.
When another leader completes the migration we throw InvalidStateException at line 324 as a deliberate signal to release the lock. That throw happens before success = true, so this finally records Success=0 for the TableMigration operation on what's actually the normal completion handoff. The DependencyException we propagate on a transient DDB read blip hits the same path. So Success dips to 0 right when a cutover succeeds and on ordinary throttles. Is that intended?
There was a problem hiding this comment.
You are right, the method itself doesnt or shouldnt fail the only exceptions are intended to be thrown to let the caller release the lock. Perhaps returning a boolean would have been a better way to achieve that. My main intention of the metric here is to capture the latency of the method to make sure we can monitor that it doesnt take too long since it happens during isLeader checks which should be fast. Ill just keep latency and remove the success since individual failures like ddb read/write failures etc are already covered by their own metrics.
β¦ss metric - Introduce DELETE_FAULT_METRIC for deleteStateSafe to disambiguate from WriteFault - Remove Success metric and retain latency (Time) tracking for
|
LGTM after the latest commit, TY. |
Metrics Summary:
Leader emitted always:
TableMigration:
- StatusOrdinal (NONE): Ordinal of DDB status (0=UNKNOWN, 1=INIT, 2=DEPLOYED, 3=PENDING, 4=COMPLETE) WorkerMetrics:
- FleetMinSupportCode (NONE): Minimum support code across all lease-owning workers
Leader emitted during migration:
TableMigration:
- Phase1Worker (COUNT): Workers with SINGLE_TABLE_MIGRATION support code minus Phase2Workers
- Phase2Worker (COUNT): Workers writing WorkerMetricStats to lease table
- PrePhase1Worker (COUNT): Workers on older versions without migration support code
- WriteFault (COUNT): 1 on DDB write failure, 0 on success
- DeleteFault (COUNT): 1 on DDB write failure, 0 on success
- CompletionFault (COUNT): 1 when transactional write of COMPLETE fails, 0 on success TableMigrationAsyncMove:
- Success (COUNT): 1 on successful move of coordinator state entries, 0 on failure
- Time (MILLISECONDS): Duration of the async move operation
- BatchCount (COUNT): Number of successfully moved batches
All workers emitted during migration:
TableMigration:
- ReadFault (COUNT): 1 on DDB read failure in readStateFromLegacy, 0 on success
- Time (MILLISECONDS): Duration of state machine run TableMigrationInitialize:
- Success (COUNT): 1 on successful initialize, 0 on failure
- Time (MILLISECONDS): Duration of initialize
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.