[shashb27]

Description

Fixes #1951. Full credit for the diagnosis to @asottile's report and @zsol's 2021 profiling: the tight await asyncio.wait(pending, return_when=FIRST_COMPLETED) loop in schedule_formatting is the culprit — every asyncio.wait call re-registers completion callbacks on all still-pending tasks, so each completion costs O(pending) bookkeeping, quadratic overall.

This PR replaces the re-wait loop with a single done callback per task that pushes the finished task onto an asyncio.Queue, which the loop then consumes. Each completion now costs O(1), results are still processed in completion order, and cancelled/failed task handling, the SIGINT/SIGTERM handlers, and cache/report behavior are unchanged.

I didn't use asyncio.as_completed (zsol's original suggestion) because on Python 3.10/3.11 it yields wrapper futures rather than the original tasks, and this loop needs the original task to map back to its source path and to inspect .cancelled()/.exception() — Black still supports those versions.

Benchmarks (Windows 11, Python 3.13.14, editable install, fresh BLACK_CACHE_DIR per run, directories of N trivial x = 1 files, matching the issue's repro shape):

Files	main	this PR
1,000	3.45s	3.13s
10,000	67.09s	18.34s (18.13s second run)

10x the input now costs 5.9x wall time instead of 19.4x; the remaining sub-linearity is process-pool startup and fixed per-file cost.

Happy to close this in favor of a maintainer's own fix if one is in flight — I searched open/closed PRs and the issue thread and found no competing attempt, but I may have missed something.

Checklist - did you ...

• Implement any code style changes under the --preview style, following the
  stability policy? — N/A, no style change (performance only)
• Add an entry in CHANGES.md if necessary?
• Add / update tests if necessary? — new tests/test_schedule_formatting.py
  covering the scheduling loop end to end (success counts + failure drain);
  pytest tests/ -n auto green locally on Windows (502 passed, 4 skipped)
• Add new / update outdated documentation? — N/A

[github-actions]

diff-shades results comparing this PR (6ed1008) to main (d078e8e):

--preview style: no changes

--stable style: no changes

---

What is this? | Workflow run | diff-shades documentation

[cobaltt7]

Are these benchmarks run on compiled Black? mypyc significantly improves formatting as well.

[shashb27]

Honest answer: no - the benchmarks were run on a pure-Python dev install, not a mypyc-compiled build. The quadratic part lives in asyncio's wait-loop bookkeeping though (every asyncio.wait(FIRST_COMPLETED) call re-registers callbacks on all pending tasks), and stdlib asyncio isn't compiled by mypyc - so I'd expect the relative overhead to be as bad or worse in compiled builds, since the formatting work gets faster while the event-loop bookkeeping stays Python-speed. Happy to re-run the comparison on a mypyc build if that's useful.

[cobaltt7]

Yes, please do run the benchmarks again on mypyc

[shashb27]

Ran them on mypyc-compiled builds of both sides (HATCH_BUILD_HOOKS_ENABLE=1 wheels, Windows, Python 3.13): main at bdcad23 vs this branch at 63a819b. Same corpus of generated ~30-line files each run, black --check, cache pointed at a throwaway dir, same thread count.

files	main	this PR
500	3.5 s	3.6 s
2 000	10.5 s	10.3 s
8 000	59.7 s	34.5 s
16 000	208.1 s	70.6 s

So the finding holds in compiled builds, and it looks like the mypyc speedup actually makes the scheduling overhead relatively worse, as expected — the quadratic term is stdlib asyncio bookkeeping that doesn't get compiled, while the formatting work around it gets faster. The scaling ratio makes it plain: from 8 000 to 16 000 files, main takes 3.5x longer for 2x the files, this branch 2.05x (linear). At small repo sizes they're identical, so there's no regression for the common case.

[cobaltt7]

Thanks!