Had another crash. Also in Containment, also to do with EntityGrid (but in ::AddEntity(), not ::GetNeighboursByType(). Both end up in spatial::UniformGrid<WorldObjectId, glm::vec<2, float, (glm::qualifier)3>, 128, spatial::BasicCell<WorldObjectId, glm::vec<2, float, (glm::qualifier)3> > >::GetOverlappingCells(glm::vec<2, float, (glm::qualifier)3> const&, float) const (darwinia + 0x2d1bc5), though.

Thanks for the report -- and for including the crash info. I found both the crashes you reported in our Sentry instance. After digging through both I'm fairly convinced this crash is impossible with consistent memory.

The two events:
Same sort call both times, and both fault addresses end in ...ffff8 -- i.e. each is an 8-byte read starting 8 bytes below a page boundary. The sorted elements are 8-byte pairs of int32 cell coordinates, and the buffer is page-aligned (we use mimalloc, whose randomized arena placement also explains the odd ~4 TiB addresses). So in both crashes, the sort read data[-1]: libstdc++'s __unguarded_linear_insert walked backwards off the front of the array and dereferenced the element below it.

Why that shouldn't be possible: the unguarded insertion path is only entered after a guarded comparison against *first returns false -- that earlier result is the promise that the backward walk terminates before reaching the front. Blowing through element zero requires the comparator to contradict itself between two evaluations of the same data. I went through our comparator looking for a way to violate strict weak ordering:

• The elements are integer cell coordinates, freshly written by the same function a few lines earlier, bounds-clamped to the grid.
  
• The comparator computes squared float distances from cell centers (always finite -- derived from clamped ints and fixed grid geometry) to the query center. A NaN can only enter via the query center, and in that case every distance becomes NaN, so all comparisons uniformly return false -- degenerate but still a self-consistent ordering. There's no way to get a mix of NaN and finite keys. Overflow to +inf is likewise consistent.
  
• The scratch buffer, the query center, and everything captured by the comparator are only ever touched by the game's main thread -- the simulation loop is single-threaded, and both crashing call paths are in it.

So: deterministic comparator, immutable inputs for the duration of the sort, single thread. The only way __unguarded_linear_insert walks off the front is if a value in that buffer (or the captured query center) changed between two reads within the same sort call. That's not a bug our code can express -- something outside the process modified memory, or a load returned inconsistent data.

Normally I'd stop at "flaky RAM, please memtest" -- and I'd still gently suggest ruling that out -- but there's a correlation in our crash data that made me want to flag this to you rather than just closing it: your kernel string is 7.2.0-rc1-sulix+, and the only other user machine producing "impossible" crashes is also on a 7.2-rc kernel (7.2.0-rc2-1-mainline, an AMD Strix Point laptop with RADV -- completely different CPU/GPU vendors from your Haswell + Arc A380). That machine produced a spray of similarly nonsensical crashes in one week: wild-pointer SEGVs across unrelated subsystems, a corrupted vector-upload size of exactly 2^30 bytes, audio-mixer graph pointer corruption. Every other Linux user we see is on a stable kernel (7.0.x, 6.18.x, 6.12.x...), and none of them has ever hit any of these signatures. Two unrelated machines, both on 7.2-rc, both exhibiting transient user-memory inconsistency that a hot sort comparator is very good at converting into a SEGV.

The cross-vendor spread makes me suspect something core rather than a GPU driver -- mm, or FP/SIMD state save-restore on context switch, that sort of thing. If you do get a chance to run on a stable kernel for a while, I'd love to know whether the crashes simply stop.

Hi Steven, i'm the user with arch linux "7.2.0-rc2-1-mainline, an AMD Strix Point laptop with RADV".
I will try with stable kernel asap

I also have the issue on linux 7.1.3-arch1-3. please look in your sentry, you'll find my crash
thanks!

Also have the same issue on linux-lts 6.18.38-3. and FYI "normal production" release don't have any issue. i can play during hours without any crashes

Hey @grs_93 -- I haven't seen any Sentry events come from you in the past 24 hours. I've seen this happen when the crash is too early, or Sentry decides to defer posting its reports until a subsequent launch. Can you explain the circumstances for your crashes? Does it happen immediately on game startup? Do you manage to play a game level for a while before it crashes?

Oh, I think I found your crashes -- they're not on the Steam ID you're posting here under. So here's what I found. The crashes on the stable kernels are *realistic* and likely a bug in MiniAudio that I need to investigate. All the crashes on 7.2.0-rc are all weird, nonsensical ones. I'll try to get a new beta out soon, once I hunt down the MiniAudio behavior.

Hi, glad you found it. About my issues, it's really very random, sometime it happens just after i start playing Containment level. like 5 seconds after. and sometime I can play Containment for 10 minutes or so. and then crash. I'm ready to try a new build. Thanks for your support!

Hi Steven, FYI I tried with "SoundBackend = null" (that should disable miniaudio maybe?). and I also get the crash

Originally posted by Steven:

Oh, I think I found your crashes -- they're not on the Steam ID you're posting here under. So here's what I found. The crashes on the stable kernels are *realistic* and likely a bug in MiniAudio that I need to investigate. All the crashes on 7.2.0-rc are all weird, nonsensical ones. I'll try to get a new beta out soon, once I hunt down the MiniAudio behavior.

Hello, any news about the issue ? thanks a lot!

Thanks very much for looking into this.

A few updates from my side:
- I managed to complete 'Containment', and haven't had a similar crash in EntityGrid since on other levels. (Though I did also upgrade from -rc1 → -rc2 at around that time.)
- Nor have I been able to reproduce this by re-entering Containment, but since it's a random crash, that could just be bad (good?) luck.
- I have had a few crashes in miniaudio (all in descendants of ma_node_read_pcm_frames: one in the stb_vorbis decoder). Not sure if these are the same 'impossible' crashes or the separate miniaudio bug. I seem to have a pretty reliable instance of this in Yard at the moment.
- I ran a couple of full passes of memtest86+, and no errors were found. (Equally, there hasn't been anything else on this machine which would point towards memory issues. It's not overclocked, regularly does much more memory-heavy things than Darwinia without issue.)
- Playing on a different machine (Ryzen 7 7850U), also with a 7.2.0-rc1 kernel, hasn't resulted in any crashes so far — not even the miniaudio ones.

Let me know if there's anything else I can provide which would help.

I have the miniaudio crashdump

Message: Process 184284 (Main thread) of user 60177 dumped core.

Stack trace of thread 184304:
#0 0x0000557aa9556127 ma_node_input_bus_read_pcm_frames (darwinia + 0x612127)
#1 0x0000557aa953fd73 ma_node_read_pcm_frames (darwinia + 0x5fbd73)
#2 0x0000557aa953f9f4 ma_node_graph_read_pcm_frames (darwinia + 0x5fb9f4)
#3 0x0000557aa93e5f82 _ZL22MiniAudio_DataCallbackP9ma_devicePvPKvj (darwinia + 0x4a1f82)
#4 0x0000557aa95510be ma_device__on_data (darwinia + 0x60d0be)
#5 0x0000557aa9550c96 ma_device__handle_data_callback (darwinia + 0x60cc96)
#6 0x0000557aa954b73a ma_device_write_to_stream__pulse (darwinia + 0x60773a)
#7 0x0000557aa954b649 ma_device_on_write__pulse (darwinia + 0x607649)
#8 0x00007f5e7171ba36 n/a (libpulse.so + 0x2da36)
#9 0x00007f5e716b1cb0 pa_pdispatch_run (libpulsecommon-17.0.so + 0x43cb0)
#10 0x00007f5e7170050d n/a (libpulse.so + 0x1250d)
#11 0x00007f5e716b6c33 n/a (libpulsecommon-17.0.so + 0x48c33)
#12 0x00007f5e716b7ad7 n/a (libpulsecommon-17.0.so + 0x49ad7)
#13 0x00007f5e71714050 pa_mainloop_dispatch (libpulse.so + 0x26050)
#14 0x00007f5e717147a2 pa_mainloop_iterate (libpulse.so + 0x267a2)
#15 0x0000557aa954ad1f ma_device_data_loop__pulse (darwinia + 0x606d1f)
#16 0x0000557aa952e567 ma_worker_thread (darwinia + 0x5ea567)
#17 0x00007f5e82697f42 n/a (libc.so.6 + 0x97f42)
#18 0x00007f5e8271efac n/a (libc.so.6 + 0x11efac)

Stack trace of thread 184284:
#0 0x0000557aa9216578 _ZSt16__insertion_sortIN9__gnu_cxx17__normal_iteratorIPSt4pairIiiESt6vectorIS3_SaIS3_EEEENS0_5__ops15_Iter_comp_iterIZNK7spatial11UniformGridI13WorldObjectIdN3glm3vecILi2EfLNSE_9qualifierE3EEELi128ENSB_9BasicCellISD_SH_EEE19GetOverlappingCellsERKSH_fEUlRKT_RKT0_E_EEEvSN_SN_SQ_ (darwinia + 0x2d2578)
#1 0x0000557aa9215bc5 _ZNK7spatial11UniformGridI13WorldObjectIdN3glm3vecILi2EfLNS2_9qualifierE3EEELi128ENS_9BasicCellIS1_S5_EEE19GetOverlappingCellsERKS5_f (darwinia + 0x2d1bc5)
#2 0x0000557aa9217ac5 _ZN10EntityGrid9AddEntityEPK6Entity (darwinia + 0x2d3ac5)
#3 0x0000557aa9217836 _ZN10EntityGrid7RebuildEv (darwinia + 0x2d3836)
#4 0x0000557aa939cb07 _ZN8Location7AdvanceEi (darwinia + 0x458b07)
#5 0x0000557aa93b219f _ZL10RunTheGamev.llvm.5821829136952856098 (darwinia + 0x46e19f)
#6 0x0000557aa93b00af _Z7AppMainv (darwinia + 0x46c0af)
#7 0x0000557aa9371999 main (darwinia + 0x42d999)
#8 0x00007f5e82627781 n/a (libc.so.6 + 0x27781)
#9 0x00007f5e826278b9 __libc_start_main (libc.so.6 + 0x278b9)
#10 0x0000557aa91057a5 _start (darwinia + 0x1c17a5)

crashes with SIGSEGV/SEGV_MAPERR on the miniaudio worker thread, in ma_node_input_bus_read_pcm_frames, called through ma_node_graph_read_pcm_frames; MiniAudio_DataCallback to PulseAudio backend. Fault offset is darwinia+0x612127

I got another crash in the native Linux build of Darwinia. This one appears to be on the main game thread, inside obstruction/fence checking while advancing a Centipede entity.

The relevant stack trace is:

```text
#0 ObstructionGrid::GetBuildings(float, float, float, int)
#1 Entity::CheckForFences()
#2 Centipede::Advance(Unit*)
#3 Unit::AdvanceEntities(int)
#4 Team::Advance(int)
#5 Location::Advance(int)
#6 RunTheGame()
#7 AppMain()
#8 main()
```

The faulting frame is:

```text
ObstructionGrid::GetBuildings(...)
darwinia + 0x47741c
```

The full relevant portion was:

```text
#0 0x000055db43f7341c ObstructionGrid::GetBuildings(...) (darwinia + 0x47741c)
#1 0x000055db440021bf Entity::CheckForFences() (darwinia + 0x5061bf)
#2 0x000055db43feb024 Centipede::Advance(Unit*) (darwinia + 0x4ef024)
#3 0x000055db43fc99d4 Unit::AdvanceEntities(int) (darwinia + 0x4cd9d4)
#4 0x000055db43fc4b16 Team::Advance(int) (darwinia + 0x4c8b16)
#5 0x000055db43f54b3d Location::Advance(int) (darwinia + 0x458b3d)
#6 0x000055db43f6a19f RunTheGame() (darwinia + 0x46e19f)
#7 0x000055db43f680af AppMain() (darwinia + 0x46c0af)
#8 0x000055db43f29999 main (darwinia + 0x42d999)
```

At the time of the crash, the MiniAudio worker thread was active but did not appear to be the faulting thread. It was processing the audio node graph:

```text
ma_node_input_bus_read_pcm_frames
ma_node_read_pcm_frames
ma_node_input_bus_read_pcm_frames
ma_node_read_pcm_frames
ma_node_graph_read_pcm_frames
MiniAudio_DataCallback
ma_device__on_data
ma_device__handle_data_callback
ma_worker_thread
```

I analyzed the latest crash further in IDA.

So the crash occurs in:

ObstructionGrid::GetBuildings(float, float, float, int)
darwinia + 0x47741c

The exact faulting instruction is:

mov r12, [rcx+rax*8]

The surrounding code calculates a spatial-grid cell address:

movsxd rcx, dword ptr [r13+4] ; y
movsxd rax, dword ptr [r13] ; x
imul rcx, [rbx+18h] ; y * grid width
lea rax, [rax+rax*2] ; x * 3
lea rcx, [rcx+rcx*2]
shl rcx, 3 ; y * width * 24
add rcx, [rbx] ; grid storage base
mov r12, [rcx+rax*8] ; crash here
mov r15, [rcx+rax*8+8]

So it appears the crash happens while accessing a cell in the `ObstructionGrid`/`UniformGrid`, before `Location::GetBuilding()` is called.

The effective cell address appears to be equivalent to:

gridStorage + 24 * (y * gridWidth + x)


From what I understand, `GetOverlappingCells()` generates the `(x, y)` pairs used here and appears to clamp them to the grid dimensions, so this may indicate corrupted/stale cell coordinates, invalid grid storage, or some other memory/lifetime issue rather than a simple unchecked coordinate.

The stack for this crash was:

ObstructionGrid::GetBuildings
Entity::CheckForFences
Centipede::Advance
Unit::AdvanceEntities
Team::Advance
Location::Advance
RunTheGame
...

So the failure seems to happened while a Centipede was being advanced and checking fences/buildings.

I hope that can help!