Skip to content

Add process pool initializer to manage-imports script - #12900

Merged
mekarpeles merged 3 commits into
internetarchive:masterfrom
jimchamp:fix-importbot
Jun 11, 2026
Merged

Add process pool initializer to manage-imports script#12900
mekarpeles merged 3 commits into
internetarchive:masterfrom
jimchamp:fix-importbot

Conversation

@jimchamp

Copy link
Copy Markdown
Collaborator

Fixes bug causing the importbot container to restart endlessly.

Prior to Python 14, the default multiprocessing start method in Linux environments was fork. Now, spawn is the default.

spawn creates a new Python interpreter process, which does not inherit the parent process's web.config. The web.config in spawned processes did not contain the necessary credentials for connecting to our DB, causing an unhandled error which stopped the container.

This code creates an initializer method which calls load_config when a new process is spawned. Now, new processes contain the necessary credentials, and the container no longer restarts ceaselessly.

Technical

Testing

Screenshot

Stakeholders

Copilot AI review requested due to automatic review settings June 11, 2026 20:12
@jimchamp jimchamp added the Patch Deployed This PR has been deployed to production independently, outside of the regular deploy cycle. label Jun 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes importbot restart loop under Python 3.14+ by ensuring child processes started via multiprocessing (now defaulting to spawn on Linux per PR description) load the Open Library config before running import work.

Changes:

  • Add a multiprocessing.Pool worker initializer that calls load_config(configfile) in each spawned process.
  • Thread the resolved configfile path through CLI flags into import_all() so the pool initializer can use it.

Comment thread scripts/manage-imports.py

@cdrini cdrini left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm! Change makes sense, and @jimchamp tested that it works correctly on prod and fixes the restart issue. One optional non-blocking suggestion.

Comment thread scripts/manage-imports.py Outdated
@jimchamp jimchamp changed the title Add thread pool initializer to manage-imports script Add process pool initializer to manage-imports script Jun 11, 2026
Comment thread scripts/manage-imports.py Outdated
@mekarpeles mekarpeles self-assigned this Jun 11, 2026
@mekarpeles

Copy link
Copy Markdown
Member

Thank you!!!

@mekarpeles
mekarpeles merged commit f186b2f into internetarchive:master Jun 11, 2026
4 checks passed
@jimchamp
jimchamp deleted the fix-importbot branch June 11, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Patch Deployed This PR has been deployed to production independently, outside of the regular deploy cycle.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants