[Scott McKay (skottmckay)]

Description

Changes required to enable writing external initializers to a buffer when using OrtCompileApi

Motivation and Context

Enable WebNN usage in sandboxed scenario so models >2GB can be created.

[Scott McKay (skottmckay)]

mingmingtasd (@mingmingtasd)
Ningxin Hu (@huningxin)

Please take a look to see if this will satisfy the WebNN requirements for compiling models > 2GB in the sandbox.

[mingmingtasd (mingmingtasd)]

Thanks for the plan! I think it satisfies the non-compiling EP case like WebGPU, but not the compiling EP case like OV EP. We need both in the sandbox.

For the compiling EPs like OV EP, the main payload is the compiled blob stored in the EPContext node's ep_cache_context attribute rather than the initializers, just as the plan said "does not support graph metadata that independently exceeds protobuf's limit". But I see that there are some experimental APIs OrtCompileApi_ModelCompilationOptions_SetEpContextDataWriteFunc and OrtApi_SessionOptions_SetEpContextDataReadFunc which support writing/reading the compiled blob to/from a caller-owned buffer instead of a real file (with filesystem access). The buffer can be retrieved via a fake/logical file name (no filesystem access). These APIs were introduced by #28624. And I guess that's the second solution you mentioned here.

[Scott McKay (skottmckay)]

But I see that there are some experimental APIs OrtCompileApi_ModelCompilationOptions_SetEpContextDataWriteFunc and OrtApi_SessionOptions_SetEpContextDataReadFunc which support writing/reading the compiled blob to/from a caller-owned buffer instead of a real file (with filesystem access). The buffer can be retrieved via a fake/logical file name (no filesystem access). These APIs were introduced by #28624. And I guess that's the second solution you mentioned here.

The new experimental API was initially intended to enable encryption/decryption but could also be used to work around the sandbox limitations. You'd need to manage the buffers and track the associated filename but in theory it would work as a mechanism.

Problem is that the EP implementation owns creating the EPContext node including whether it uses the write function pointer or not. ORT doesn't control how the write is done. We could try and invert that control (e.g. add a function to create an EPContext node that takes the bytes and other node properties/attributes to create the node and does the write to file or uses the provided function pointer), but that's a new API and also requires EP implementation changes for it to be used.

Could also be that the EP implementation prefers a pointer to do the write via so it can be passed into their compilation implementation, and us adding a function for creating an EPContext node where they provide the buffer to write might not fit with that. Gopalakrishnan Nallasamy (@GopalakrishnanN) do you know if that's the case?

[Scott McKay (skottmckay)]

Personally I would lean towards adding a CreateEPContextNode function and enforcing usage (reject call to CreateNode for EPContext node). That ensures any write function is consistently used. Needs feedback from all IHVs though to confirm that approach is valid.

[Copilot]

Copilot review overview

🟡 Changes recommended

External-buffer outputs are committed before later model serialization can fail, violating the documented failure contract.

Review effort: Balanced
Findings: 3 · 2

Open (5)

• Publish external buffer only after model save succeeds · New
• Test external initializers in control-flow subgraphs · New
• Test misaligned external buffer fallback · New
• Correct design document type name · New
• Document default external initializer alignment policy · New

What changed in this PR

Adds in-memory external initializer serialization and optional zero-copy loading, enabling compiled models whose weights exceed protobuf’s 2 GB limit.

Changes:

• Adds C/C++ Compile API controls for external initializer buffers and alignment.
• Implements two-pass buffer serialization and recursive direct-buffer loading.
• Adds API documentation and integration tests.

File	Description
onnxruntime/​test/​framework/​inference_session_test.cc	Tests buffer serialization, loading, and file output.
onnxruntime/​core/​session/​plugin_ep/​ep_api.cc	Uses the shared EP context configuration.
onnxruntime/​core/​session/​model_compilation_options.h	Declares new compilation options.
onnxruntime/​core/​session/​model_compilation_options.cc	Stores and validates buffer settings.
onnxruntime/​core/​session/​inference_session.cc	Enables optional direct-buffer loading.
onnxruntime/​core/​session/​ep_context_config.h	Centralizes EP context callback state.
onnxruntime/​core/​session/​compile_api.h	Declares new C API implementations.
onnxruntime/​core/​session/​compile_api.cc	Implements and registers the APIs.
onnxruntime/​core/​session/​abi_session_options.cc	Strengthens external-buffer argument validation.
onnxruntime/​core/​graph/​model.h	Declares stream-based model serialization.
onnxruntime/​core/​graph/​model.cc	Implements stream-based model serialization.
onnxruntime/​core/​graph/​graph.cc	Adds aligned serialization and direct recursive loading.
onnxruntime/​core/​framework/​tensor_external_data_info.cc	Aligns threshold-boundary prepacked data.
onnxruntime/​core/​framework/​ep_context_utils.cc	Implements two-pass buffer generation.
onnxruntime/​core/​framework/​ep_context_options.h	Adds buffer destination and alignment state.
onnxruntime/​core/​framework/​ep_context_options.cc	Adds buffer-option lookup.
include/​onnxruntime/​core/​session/​onnxruntime_session_options_config_keys.h	Exposes direct-buffer loading configuration.
include/​onnxruntime/​core/​session/​onnxruntime_cxx_inline.h	Implements C++ wrappers.
include/​onnxruntime/​core/​session/​onnxruntime_cxx_api.h	Declares C++ wrappers.
include/​onnxruntime/​core/​session/​onnxruntime_c_api.h	Extends and documents the Compile API.
include/​onnxruntime/​core/​graph/​graph.h	Declares serialization and loading interfaces.
docs/​design/​Compile_API_In_Memory_External_Initializers.md	Documents the design and validation plan.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.