Repository navigation
HA: a node serves a snapshot of a database it has quarantined, and its reported applied index overstates what the copy holds #8468
Description
Follow-up of #8454 (residual of the fix in the PR that closes it).
Gap
The #8454 fix has the node serving a snapshot (SnapshotHttpHandler) report its Raft applied index in X-ArcadeDB-Snapshot-Applied-Index, read before the copy is captured, and the installing follower refuses a copy whose index is below what it already applied. The premise is that every entry up to the serving node's lastAppliedIndex is in the copy it serves.
That premise does not hold for a database the serving node has quarantined (ArcadeStateMachine.divergedDatabases). A quarantine deliberately leaves the skipped entry unapplied while every later entry advances lastAppliedIndex past it (see the comment on divergedDatabases and ha-raft/CLAUDE.md). A leader that has quarantined database D therefore reports an applied index that covers an entry its copy of D does not hold, the follower's check passes, and the follower installs a copy that is short of that entry.
SnapshotHttpHandler does not consult the quarantine at all (grep -n 'isDatabaseDiverged\|quarantin' ha-raft/src/main/java/com/arcadedb/server/ha/raft/SnapshotHttpHandler.java returns nothing), so this is also a pre-existing problem independent of #8454: a quarantined node serves its diverged copy of D as authoritative.
Why it was left out of #8454
#8454 is about the follower getting ahead of a healthy leader's apply thread. A diverged leader copy is a different defect with a different fix, most likely one of:
- have the handler refuse (503) to serve a database it has quarantined, so the follower retries against a healthy leader; or
- report, for a quarantined database, the index just below the entry the quarantine skipped instead of
lastAppliedIndex.
Either one changes how a leader behaves while one of its own databases is quarantined, and needs its own review and test.
Entry point
SnapshotHttpHandler.handleSnapshot -> servedAppliedIndex(server) on a node where isDatabaseDiverged(databaseName) is true.
Activity
fix(#8468): refuse to serve a snapshot of a database this node has qu…
robfrank commentedon Sep 27, 2026
Fixed in #8484. A node that had quarantined a database still served snapshots of it. Its reported applied index covered the entry it had skipped, so a follower could install a copy missing a committed entry without any error. The snapshot endpoint now answers 503 for a quarantined database, whatever the cause, including a quarantine restored after a restart. The follower's install then fails and keeps its own copy. The check runs again after the snapshot image is captured, so a quarantine recorded mid-request is also refused.
Behavior change: while the leader has a database quarantined, followers cannot install that database. The leader logs a warning saying to transfer leadership.
Still open:
❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.