Repository navigation
benchmarks: memory, disk and IO per phase behind BENCH_INSTRUMENT=2026-11 - #422
Merged
Merged
Conversation
โฆ6-11 (#421) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #421. Part of #352 (items 1, 2 and 3).
Under BENCH_INSTRUMENT=2026-11 the runner reads memory, disk and IO at the end of each phase. With the variable unset nothing runs and a row is byte-identical to before: the lane hook is guarded by an environment variable only the 2026-11 runner sets, and a test checks it.
How it works: the lane prints its PHASE marker, and at one that ends a phase (build, post-build, warm-up, close) it writes a request file into a directory the runner mounted into the container and waits. A thread in the runner (memory_probe.py, about 400 lines, three hook lines in runner.py and one in bench_common.py) reads the cgroup files of the cell, forces a full GC where the engine is a JVM (jcmd for a server, JPype for an embedded engine), reads disk, then restarts memory.peak. Nothing runs inside a timed section. The time spent is on the row.
Headline: mem_owned_mib, cgroup anon plus shmem of the cell's containers at phase end. Every engine has it. Per runtime and context only: jvm_used_after_gc_mib, _committed_mib, direct_mib, and proc_anon_mib and rss_mib for embedded engines. Also mempeak_mib (kernel-reset when the runner may write memory.peak, otherwise the sampler's maximum, named in mem_probe_peak_kind), diskmib and data_mib (less the empty start), diskwal_mib, disk_close*, and io_read_mib and _write_mib. The design, what compares across engines and what does not, and the pinned-heap effect are in PROTOCOL.md and FAIRNESS.md. export_web adds one column, owned memory after queries, for lanes whose rows all carry 2026-11.
Tested: unit tests on synthetic cgroup, smaps, io.stat and jcmd input, the lane and host handshake, and that 2026-10 has no new field or column (test_memory_probe.py); the full experiments suite passes (784 passed, 15 skipped). Laptop smoke on cores 12 and 13: embedded ArcadeDB and served PostgreSQL on the document lane fill every field; jcmd probing was run against a real ArcadeDB server container (heap used after GC 20 MiB, committed 256 MiB). A served JVM cell through the runner was not completed (Neo4j did not become ready at the 4g smoke cap). On this laptop the runner cannot write memory.peak, so peaks are the sampler's. The 2026-11 export_web path covers the new column only; the other 2026-11 table changes are not part of this PR.
This stacks on main after #411 and is independent of the change that stops pinning the embedded ArcadeDB heap.
๐ค Generated with Claude Code