Repository navigation
A sole voter whose snapshot marker runs ahead of its applied entries (#6111 stale-snapshot floor) has no operator override either #9498
Description
Follow-up to #9449.
Gap
#9449 adds POST /api/v1/cluster/accept-diverged/{database} (and the gRPC AcceptDivergedDatabase twin), which lifts a per-database quarantine and its per-database read floor on a node that is the sole voter. It deliberately does not touch the node-wide stale-snapshot floor (ArcadeStateMachine.staleSnapshotAppliedFloor, issue #6111), which reinitialize() publishes when the Ratis snapshot marker runs ahead of the entries this node actually applied.
That floor is also part of LocalResyncState.inProgress(), so it keeps /api/v1/ready at 503 and clamps LINEARIZABLE reads. It is lifted only by a successful snapshot download, and triggerSnapshotDownload() / retryUnfilledSnapshotGap() refuse on a leader (it cannot resync from itself). On a one-voter cluster no peer can take leadership, so, as ha-raft/CLAUDE.md says, the wedge is terminal until an operator acts, and there is no operator action short of rebuilding the node.
Why it was out of scope for #9449
#9449 is about the per-database quarantine named by a DivergenceCause. The global floor is a different state with a different owner (the replay position, not a skipped entry), it is not per database, and lifting it means accepting that the whole node may be missing committed entries. That needs its own decision.
Suggested fix
Either extend the #9449 override with a node-wide variant (root-only, sole voter only, audited the same way), or document the terminal state with its recovery procedure. Per the repo rule, a new API needs the HTTP handler, OpenAPI, gRPC/proto parity and the Remote clients together.
Activity
address review: package-private test hook instead of reflection for t…
address review: roll back on any failure, restore only the moved per-…
feat(#9498): operator override to lift the stale-snapshot read floor …
robfrank commentedon Oct 9, 2026
Implemented by #9593. A sole voter held behind the node-wide stale-snapshot read floor (#6111) can now be unstuck by an operator with the new root-only
POST /api/v1/cluster/accept-stale-snapshot(also gRPCAcceptStaleSnapshot, andRemoteServer/RemoteGrpcServerclients). It records the node's own position as applied, so the next restart does not raise the floor again, then clears the floor, wakes waiting readers and logs a WARNING audit line. If the write fails, it puts back exactly what it changed and the floor stays. It is refused on a node with peers (409 /FAILED_PRECONDITION), while a download or install is running (409), and when no floor stands (404 /NOT_FOUND). The cluster alert and the sole-voter log line now point to this route.Still open:
❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.
feat(#9498): operator override to lift the stale-snapshot read floor …