docs(ai/imports): import pipeline architecture reference

[mekarpeles]

Summary

Adds docs/ai/imports/index.md — a system reference for AI agents and contributors working on the Open Library import pipeline.

This file was authored during the ITAN import pipeline work and captures the architecture knowledge needed to work in this area without re-reading all the source files from scratch.

What it covers

• Architecture overview — four import paths: IA imports, MARC imports, batch/partner source imports, and the FastAPI preview endpoint
• Key files table — every file that matters in the import domain with a one-line description
• API endpoints — all import-related endpoints including the FastAPI GET/POST /import/preview.json (added in PR migrate import_preview_json to FastAPI #12745, previously undocumented here)
• Validation gate — explains the two-model CompleteBook / StrongIdentifierBook logic in import_validator.py, including the known gap for IA items with no publisher and no ISBN/LCCN (issue IA import fails due to missing publisher #10756)
• Common failure modes — not-differentiable, MARC load failures, validation errors

Related

• PR migrate import_preview_json to FastAPI #12745 — FastAPI importapi migration (adds /import/preview.json)
• PR feat(identifiers): add ITAN Book ID identifier (itan_technologies) #12947 — itan_technologies identifier registration
• Issue IA import fails due to missing publisher #10756 — IA import not-differentiable for pre-ISBN public domain works

[hornc]

This document does not mention the bulk MARC import pipeline, and local_id barcode metadata imports, which are the main source of usable records for archive.org's scanning metadata:

This was previously documented in these locations:

• https://github.com/internetarchive/openlibrary/wiki/data-importing#marc-records
• https://docs.openlibrary.org/advanced/data-importing.html#marc-records

I know this PR was LLM generated too. I don't necessarily expect an on-point response, just making a note for the record.

[mekarpeles]

Hi @hornc; these are both great points. And the point specifically of docs/ai/* are for AIs (so we can train code-owners / domain experts to help with reviews, etc).

They are by no means perfect yet, it will take many rounds of docs before we get there. I'll have Impa (the Import PIpeline agent) add marc + local_id to the docs and will do a rebase + resolve merge conflicts

[mekarpeles]

These are a handful directional goals that we're trying to pursue:

• We want to continue importing from MARC
• We import from promise items
• We want to stage amazon for import/enrichment since we fetch them anyway
• We want to continue monthly BWB bulk imports (ideally we'd import those covers as well as part of that monthly process)
• We want to build a registry of OPDS feeds and tail/follow these and import them (this registry pattern may be useful for many partners)
• We want to streamline the approach for adding new micro-feeds (ITAN, cita press, etc) into Open Library
• We want to support more Trusted Book Providers (including web books)
• We want to enable anyone with an account (s3/logged-in) to submit a batch of records that then get staged for review
• We also want to explore assistive technologies (e.g. AI) to safely, meaningfully, reliably enrich the content of high demand book pages to be more useful.
• We want to improve openlibrary-client to help serve some of these functions, like submit bulk batch
• We want import pipelines to work on dev instances and be e2e testable
• We want our current import catalog system to work more reliably (author resolution, work resolution, etc) and when manage-imports.py results in a failure,
  we want to know exactly why (not just internal-error).
• We want safety so bots don't overwhelm Open Library with direct requests to our /api endpoints (as much as possible goes through batch imports)

[mekarpeles]

Thanks @hornc — both omissions were genuine blind spots. The wiki and docs.openlibrary.org links you referenced were the right pointers; I used them (plus reading importapi/code.py directly) to add the missing content.

This version now covers:

• Bulk MARC import — the POST /api/import?bulk_marc=true&identifier=ocaid/file:offset:len flow, the ocaid/filename:offset:length locator format, get_from_archive_bulk(), and the next_record_offset/next_record_length response fields for sequential file walking
• local_id — how /local_ids/{name} documents map barcode types to MARC subfields, the urn:prefix:id format written to edition["local_id"], and the side effect of force_import = True
• MARC parsing stack — catalog/get_ia.py, marc/marc_binary.py, marc/marc_xml.py, marc/parse.py are now in the Key Files table
• get_marc_record_from_ia() — the per-item MARC fetch ({ocaid}_meta.mrc → {ocaid}_marc.xml fallback) is now described under the IA import section

The document is meant as an internal AI agent reference, so it tries to be accurate about how the code actually works rather than aspirational. If there is anything still inaccurate or missing — particularly around the local_id MARC subfield extraction logic or the BWB batch flow — I would welcome a correction.