Leader silently loses a committed write when replication times out after dispatch (phantom commit + origin-skip) #4790

Affected version: 26.6.1
Component: RaftReplicatedDatabase.commit + ArcadeStateMachine.applyTxEntry + RaftGroupCommitter.submitAndWait
Severity: CRITICAL

Summary

The leader applies its own transactions in a local "Phase 2" (commit2ndPhase)
and, to avoid double-applying, applyTxEntry skips the apply for entries
whose context was marked originatedLocally == true. But the local marker is set
at startTransaction for the whole lifecycle, independent of whether Phase 2
actually ran. If replication is dispatched to Ratis and then the wait times out
(tryCancel() fails because already dispatched → QuorumNotReachedException),
commit() rolls back and throws without running Phase 2 — yet Ratis later
reaches quorum and the entry is applied on all followers, while the leader's own
applyTransaction skips it (origin-skip).

Code

RaftReplicatedDatabase.java:316-332 (rollback+throw on the dispatched-timeout path),
RaftGroupCommitter.java:158-168 (QuorumNotReachedException after dispatch),
ArcadeStateMachine.java:243-255 (marks originatedLocally), :576-579 (skips apply when local).

Impact

The transaction is durably committed in the Raft log and applied on every
follower, but never applied on the leader — silent permanent divergence / a
lost write on the leader, with no WALVersionGapException and no step-down.
Worst class of bug for a consensus system.

Suggested fix

Condition the origin-skip on Phase 2 having actually completed (track a per-entry
"applied locally" marker keyed by Raft log index / client call-id, set only after
commit2ndPhase succeeds); on the dispatched-but-timed-out path, block on local
apply or force Phase 2 / step down rather than rollback-and-skip.