Skip to content

Replica stuck in STALLED #4728

New issue
New issue
Closed
Bug
Closed
Replica stuck in STALLED#4728
Bug
Assignees
lvca
Milestone
26.7.1

Description

@danieljuhl
danieljuhl
opened on Jun 24, 2026

ArcadeDB Server v26.7.1-SNAPSHOT (build 661e52fce3b84f452f02cc131b9c5608674deb67/1782283285125/main)

I just upgraded our ArcadeDB cluster, afterwards I saw a lot of the following errors in the log. The follower did not have any load, or high I/O.

Jun 24 09:24:28 arcadedb-01 java[113566]: 2026-06-24 09:23:58.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 8 (current lag=1757). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:24:58 arcadedb-01 java[113566]: 2026-06-24 09:24:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=1789). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:25:28 arcadedb-01 java[113566]: 2026-06-24 09:24:58.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=1821). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:25:58 arcadedb-01 java[113566]: 2026-06-24 09:25:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=1855). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:26:28 arcadedb-01 java[113566]: 2026-06-24 09:25:58.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=1887). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:26:58 arcadedb-01 java[113566]: 2026-06-24 09:26:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=1917). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:27:28 arcadedb-01 java[113566]: 2026-06-24 09:26:58.168 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=1947). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:27:58 arcadedb-01 java[113566]: 2026-06-24 09:27:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=1977). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:28:28 arcadedb-01 java[113566]: 2026-06-24 09:27:58.168 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=2005). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:28:58 arcadedb-01 java[113566]: 2026-06-24 09:28:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=2035). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:29:28 arcadedb-01 java[113566]: 2026-06-24 09:28:58.168 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=2061). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:29:58 arcadedb-01 java[113566]: 2026-06-24 09:29:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=2089). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:30:28 arcadedb-01 java[113566]: 2026-06-24 09:29:58.168 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=2115). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:30:58 arcadedb-01 java[113566]: 2026-06-24 09:30:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 2 (current lag=2139). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:31:28 arcadedb-01 java[113566]: 2026-06-24 09:30:58.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=2165). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:32:03 arcadedb-01 java[113566]: 2026-06-24 09:31:28.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 2 (current lag=2189). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:32:33 arcadedb-01 java[113566]: 2026-06-24 09:32:03.168 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=2219). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:33:03 arcadedb-01 java[113566]: 2026-06-24 09:32:33.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=2243). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:33:33 arcadedb-01 java[113566]: 2026-06-24 09:33:03.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=2267). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:34:03 arcadedb-01 java[113566]: 2026-06-24 09:33:33.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 2 (current lag=2289). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:34:33 arcadedb-01 java[113566]: 2026-06-24 09:34:03.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 4 (current lag=2313). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:35:03 arcadedb-01 java[113566]: 2026-06-24 09:34:33.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 2 (current lag=2335). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.
Jun 24 09:35:12 arcadedb-01 java[113566]: 2026-06-24 09:35:03.169 SEVER [ClusterMonitor] Replica '10.0.0.2_2434' STALLED: matchIndex stuck at -1 while leader advanced by 6 (current lag=2361). This will trigger a leader election if it continues. Likely cause: replica disk saturation or network stall. Check replica I/O, GC, and heartbeat connectivity.

Activity

  1. robfrank
    added this to the 26.7.1 milestone on Jun 24, 2026
  2. lvca
    self-assigned this
    on Jun 24, 2026
  3. lvca
    added the
    Bug
    issue type on Jun 24, 2026
  4. lvca
    added 4 commits that reference this issue on Jun 25, 2026

    fix: [#4728] address PR review - SSL-aware resync + close output stream

    d337a57

    fix: [#4728] address 2nd PR review round

    e626642

    fix: [#4728] address 3rd PR review round

    0c85200

    fix: [#4728] auto-recover a replica stuck STALLED at matchIndex=-1 (#…

    c127114

  5. lvca commented on Jun 25, 2026

    @lvca
    lvca
    on Jun 25, 2026
    Member

    Fixed

  6. lvca
    closed this as completedon Jun 25, 2026
  7. lvca
    added 4 commits that reference this issue on Jun 25, 2026

    fix: [#4728] address PR review - shutdown hang + batchRAM race

    294e4b2

    fix: [#4728] follow-ups - wait for leader on forward + bound snapshot… (

    1fe612b

    fix(#4841): reset ClusterMonitor per-replica lag state on each leader…

    d38a9fc

    review(#5074): interrupt-leak regression test + uninterruptible-wait …

    e5c294c
  8. hoogenbj
    added a commit that references this issue on Jul 7, 2026

    fix(ArcadeData#5068): refcount page-flush suspension so overlapping s…

    e3fd04a
  9. lvca
    added a commit that references this issue on Jul 15, 2026

    fix(#5295): flag a never-appended follower (matchIndex=-1) as STALLED…

    ff20829
  10. robfrank
    added 6 commits that reference this issue on Aug 14, 2026

    fix: [#4728] auto-recover a replica stuck STALLED at matchIndex=-1 (#…

    04461db

    fix: [#4728] follow-ups - wait for leader on forward + bound snapshot… (

    33b3b54

    fix(#4841): reset ClusterMonitor per-replica lag state on each leader…

    2ad64af

    fix(#5068): refcount page-flush suspension so overlapping suspenders …

    cca7ef2

    review(#5074): interrupt-leak regression test + uninterruptible-wait …

    54a4f37

    fix(#5295): flag a never-appended follower (matchIndex=-1) as STALLED…

    53d8400
  11. lvca
    added 3 commits that reference this issue on Aug 16, 2026

    fix(#6200): the deferred backlog is released per database, never zero…

    b7bd3eb

    fix(#6200): a release moves off the total only what the per-database …

    21da075

    fix(#6199, #6200): the flush drains wake on the page, and one suspend…

    a2c243d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions