Repository navigation
Expose CoreML EP via OrtEpFactory - #31975
Conversation
Register an internal CoreMLEpFactory with the plugin-EP infrastructure so that, on supported Apple systems with discovered accelerator hardware, the CoreML EP appears in GetEpDevices(), can be added to a session with SessionOptionsAppendExecutionProvider_V2, and is eligible for EP selection policies, notably PREFER_NPU, which can now select the Apple Neural Engine on supported systems instead of silently falling back to the CPU EP. Fixes microsoft#29066 The factory registers with vendor ID OrtDevice::VendorIds::APPLE, matching the vendor ID reported by Apple device discovery, so device ordering during EP selection prefers CoreML for Apple hardware via the vendor-affinity rule. In builds that also register the internal WebGPU EP, including the official macOS packages, PREFER_GPU and MAX_PERFORMANCE now select CoreML instead of WebGPU when both EPs claim the same Apple GPU, because CoreML has matching vendor affinity. The factory advertises at most one NPU and one GPU from device discovery. It advertises no devices below Core ML 5, the EP's minimum supported version. The NPU additionally requires Core ML 6 (iOS 16 / macOS 13), because earlier Core ML versions have no compute-units mode that enables the Neural Engine without also enabling the GPU. MLComputeUnitsCPUAndNeuralEngine requires Core ML 6, so an NPU selection could not otherwise be honored. The CPU device is deliberately left to the CPU EP. CPU-selecting policies choose a single preferred EP for the CPU device, so advertising it would displace the ORT CPU EP and other CPU-based EPs. CoreML-on-CPU remains available through an explicit MLComputeUnits=CPUOnly option, and Core ML can use its internal CPU path for operations unsupported by the selected compute units. When MLComputeUnits is not specified, the factory derives it from the selected devices: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU+GPU. User options may narrow this set, but may not enable an accelerator that was not selected. Tests in onnxruntime/test/autoep/test_selection.cc cover factory registration, explicit GPU and NPU+GPU selection, inference, PREFER_NPU/PREFER_GPU and their policy aliases, compute-unit defaulting and conflict validation, version checks, device validation and selection limits, CPU-only narrowing, and transactional cleanup after partial device-creation failures.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
@microsoft-github-policy-service agree company="Parallels" |
|
Hi Edward Chen (@edgchen1), would you be the right person to review this PR, or could you suggest an appropriate reviewer? It exposes the CoreML EP via OrtEpFactory and enables device-based EP selection. Thanks! |
The CoreML EP is implemented as part of ONNX Runtime, not provided by Apple, so its factory now reports "Microsoft" and VendorIds::MICROSOFT, like the other internal EP factories (CPU, DML, and WebGPU). Discovered Apple hardware devices keep their "Apple" / VendorIds::APPLE identifiers. VendorIds::APPLE remains the named constant for the vendor ID reported by Apple platform device discovery. Because the factory vendor no longer matches the Apple hardware vendor, CoreML is no longer preferred based on vendor matching during device ordering. When another EP claims the same hardware device, ordering falls back to the EP name tie-breaker. The CoreMLVendorAffinityPrecedesEpNameOrdering test was deleted because it tested the vendor affinity behavior removed by this change. The remaining vendor assertions now expect Microsoft identifiers for the factory and Apple identifiers for the hardware device.
Rename kCoremlComputeUnits_* to kCoremlProviderOption_MLComputeUnits_*.
Call the native binding through the _pybind_state module.
There was a problem hiding this comment.
Pull request overview
Registers CoreML with device-based EP selection on supported Apple systems.
Changes:
- Adds an internal CoreML EP factory for NPU/GPU discovery and selection.
- Derives and validates
MLComputeUnitsfrom selected devices. - Adds Apple vendor IDs and comprehensive AutoEP tests.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
onnxruntime/test/python/onnxruntime_test_python.py |
Tests Apple vendor IDs. |
onnxruntime/test/autoep/test_selection.cc |
Tests CoreML discovery, selection, validation, and cleanup. |
onnxruntime/python/onnxruntime_inference_collection.py |
Exposes Apple vendor ID to Python. |
onnxruntime/core/session/plugin_ep/ep_library_internal.h |
Declares CoreML factory creation. |
onnxruntime/core/session/plugin_ep/ep_library_internal.cc |
Registers the CoreML factory. |
onnxruntime/core/session/plugin_ep/ep_factory_coreml.h |
Defines the internal factory interface. |
onnxruntime/core/session/plugin_ep/ep_factory_coreml.cc |
Implements discovery, validation, and provider creation. |
onnxruntime/core/providers/coreml/model/host_utils.h |
Defines the NPU-selection version requirement. |
onnxruntime/core/providers/coreml/coreml_options.cc |
Uses shared compute-unit constants. |
onnxruntime/core/platform/apple/device_discovery.cc |
Uses the shared Apple vendor ID. |
include/onnxruntime/core/providers/coreml/coreml_provider_factory.h |
Publishes compute-unit option constants. |
include/onnxruntime/core/framework/ortdevice.h |
Adds the Apple vendor ID. |
π‘ Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
After the CoreML factory switched to the Microsoft vendor ID, OrtDevice::VendorIds::APPLE was no longer used. Apple device discovery keeps its local kApplePciVendorId constant (0x106B) as before.
The CoreML EP factory is an internal EP: there is no library boundary between it and ORT, so its entry points do not need to convert exceptions into OrtStatus.
The device claiming and option validation helpers of CoreMLEpFactory were public static members only so that unit tests could call them directly. Move them into an anonymous namespace in ep_factory_coreml.cc, fold OrtEpDevice creation and publication into GetSupportedDevices, and remove the tests that called the helpers directly.
Move the CoreML EP selection tests from test_selection.cc to a dedicated CoreML test file.
There was a problem hiding this comment.
π‘ Changes recommended
Core ML version-boundary behavior lacks deterministic test coverage.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Balanced
|
Edward Chen (@edgchen1), I'd like to discuss one more topic. It does not expand the scope of this PR. I tried to add a CPU device to the CoreML factory, but two existing tests then failed: 1. OrtEpLibrary.PluginEp_Custom_Op_Inference_With_Prefer_Cpu This test registers example_ep, which implements Custom_Mul, and uses PREFER_CPU. In this configuration, the CPU candidates were ordered as: CoreMLExecutionProvider -> example_ep -> CPUExecutionProvider The policy selected CoreML and the ORT CPU EP. Consequently, example_ep was not added to the session. CoreML could not execute Custom_Mul, and the ORT CPU EP could not execute it either. Session creation failed with βCould not find an implementation for Custom_Mul(1) nodeβ. 2. ModelPackageTest.LoadModelPackageAndRunInference_PreferCpu This test creates a package containing CPU and NPU variants targeting example_ep, then uses PREFER_CPU. The policy again selected CoreML and the ORT CPU EP. GetVariantSelectionEpInfo captures only provider_list[0], and VariantSelector::SelectVariant considers only ep_infos[0]. Since the first EP was CoreML and the package had no CoreML variant, the test failed with βNo suitable model variant found for the configured execution providersβ. This is a package-variant selection failure and keeping more EPs in the policy result would not fix it on its own: the resolver must also consider later candidates. There is already a TODO for this in model_package_variant_selector.cc. ================================== As a possible solution, I would propose supporting a ranked chain of CPU EPs, which would give another CPU EP the opportunity to handle the nodes the first one does not support. For ordinary ONNX sessions, explicit registration already provides the relevant behavior: applications can append different EPs through separate AppendExecutionProvider_V2 calls, and graph partitioning considers them in registration order. Nodes not assigned to one EP remain available to later EPs, with the ORT CPU EP last. A policy could keep multiple eligible CPU EPs in ranked order instead of reducing them to one primary EP plus the ORT CPU EP. This would reuse existing graph-partitioning behavior. Model packages would need a corresponding extension: consider EPs in priority order and choose the first one with a compatible variant. I also measured CoreML CPU against the ORT CPU EP on an Apple M3 Pro using static, batch-1 models. These are latencies in milliseconds, each pair is two separate runs:
CoreML on the CPU is up to 4Γ faster than the ORT CPU EP here. I should also note that these numbers are for MLProgram, not the current NeuralNetwork default, which has known gaps in FP16 support. With NeuralNetwork and CPUOnly, both FP16 models ran entirely on the ORT CPU EP. What do you think? |
## Description `tests/magika_idle_threads.rs` (added in #4189) fails on macOS with ONNX Runtime 1.31. Building the magika session leaves 8 extra threads against a fixed allowance of 2. The cause is not a thread pool. ORT 1.31 registers Core ML as a built-in EP that claims the Apple NPU/GPU (microsoft/onnxruntime#31975). The `ort` 2.0.0-rc.12 crate sets the `MAX_EFFICIENCY` (= `PREFER_NPU`) EP selection policy on every `SessionBuilder::new()`. So magika's session now runs on Core ML / Apple Neural Engine (ANE), and building it starts 5β7 ANE service threads that are idle, not spinning. The fixed allowance of 2 can't account for that. The test now builds a pool-free control session (`magika::Session::builder().with_intra_threads(1)`) first and uses the threads it adds as the allowance. The control goes through the same EP selection, so whatever the EP starts cancels out, while a reintroduced intra-op pool still shows up on top. As a precaution, Rust CI now also runs `cargo test` on `macos-26`. It only ran on Ubuntu, which is why this slipped through. Closes #4191 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `crates/headroom-core/tests/magika_idle_threads.rs`: measure the allowance against a pool-free control session instead of a fixed `2`, with comments explaining the ORT 1.31 Core ML/ANE cause. Test-only; no production code changes. - `.github/workflows/rust.yml`: turn the `test` job into a matrix with `ubuntu-latest` and `macos-26`, using the same ONNX Runtime setup step. Ubuntu keeps its `test (ubuntu)` check name and runs the same steps as before. macOS runs only `cargo test`; `cargo fmt` and `cargo clippy` stay on Ubuntu. `fail-fast: false`, so one OS failing doesn't cancel the other. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Rust: `cargo fmt --all -- --check`, `cargo clippy --workspace -- -D warnings`, `cargo test --workspace` with `ORT_DYLIB_PATH` set so the magika tests actually run - [ ] Unit tests pass (`pytest`): only the pre-push `ci-precheck-python` subset was run (177 passed, 4 skipped); the full suite was not - [x] Linting passes (`ruff check .`): no Python files changed; pre-commit hooks passed - [ ] Type checking passes (`mypy headroom`): N/A, no Python changes - [x] New tests added for new functionality (updated regression test) - [x] Manual testing performed (see Real Behavior Proof) ### Test Output `cargo test -p headroom-core --test magika_idle_threads`, with the #4187 fix (`with_intra_threads(1)`) in place and with it temporarily reverted: ```text # macOS arm64, ORT 1.31.0, before this PR (fixed allowance of 2) magika session left 8 extra OS thread(s) running ... FAILED # macOS arm64, ORT 1.31.0, this branch test result: ok. 1 passed; 0 failed (6/6 repeat runs) # macOS arm64, ORT 1.29.0, this branch test result: ok. 1 passed; 0 failed # #4187 fix reverted, this branch: the test still catches the pool macOS, ORT 1.31: 12 extra vs 9 allowed -> FAILED macOS, ORT 1.29: 7 extra vs 2 allowed -> FAILED Linux (Docker aarch64), ORT 1.31: 7 extra vs 1 allowed -> FAILED ``` Full Rust suite on macOS with ORT 1.31.0 (the same command the new `macos-26` job runs): ```text ORT_DYLIB_PATH=.../libonnxruntime.1.31.0.dylib cargo test --workspace 68 suites, 1658 passed, 0 failed ``` Pre-push `make ci-precheck` on this branch: ```text ci-precheck-python: 177 passed, 4 skipped β ci-precheck PASSED β safe to push. ``` ## Real Behavior Proof - Environment: macOS 26 arm64 (16 cores), Rust 1.95, `pip install onnxruntime==1.31.0` and 1.29.0 dylibs via `ORT_DYLIB_PATH`; Linux via Docker `rust:1.95` (aarch64) with pip onnxruntime 1.31.0. - Exact command / steps: the reporter's repro from #4191 (`pip install 'onnxruntime==1.31.0'`, `export ORT_DYLIB_PATH=<capi/libonnxruntime.*.dylib>`, `cargo test -p headroom-core --test magika_idle_threads`), on `main` and on this branch; then the same test with the #4187 `with_intra_threads(1)` fix temporarily reverted, on macOS 1.31, macOS 1.29 and Linux 1.31. To find the cause, a raw `ort` probe built the magika model with `intra_threads=1` under the default policy and under `with_auto_device(PreferCPU)`, counting threads with `ps -M` and listing the EP devices. - Observed result: fails on `main` with ORT 1.31 (8 extra vs 2) and passes on this branch, including the reporter's exact steps. With the #4187 fix reverted it fails on all three setups (12 vs 9, 7 vs 2, 7 vs 1), so it still guards #4187. Probe: EP devices are CPU, CoreML/NPU and CoreML/GPU. The default policy adds 6β7 threads and takes ~420 ms to build; `PreferCPU` adds 0 and takes ~12 ms. - Not tested: the new `macos-26` CI job on GitHub runners (it runs on this PR), ORT 1.30.0 at runtime, x86_64 Linux, Intel macOS, and low-core macOS hosts. Details: - ORT 1.30.0: the 1.30 vs 1.31 difference comes from the ORT source diff and release notes, plus the reporter's data. - Low-core macOS hosts: with only 2β4 cores the control allowance can absorb a small pool, so the guard is weaker there. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A - Stable/default behavior changed: no. The PR changes only the test and CI; runtime code is untouched. - Kill switch / disable path: N/A - Unsafe override required: no. - Qualification impact: adds one macOS CI job (only when Rust paths change). - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (none needed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` β it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Not fixed here: on macOS with ORT β₯ 1.31, headroom's magika detection runs through Core ML on the ANE instead of the CPU EP. That comes from `ort`'s default policy; nobody chose it. Visible cost: a one-off ~400 ms session build plus 5β7 idle threads. Per-call latency and output parity were not measured. Pinning it to CPU needs `with_auto_device(PreferCPU)`, which magika 1.1.0's `Builder` doesn't expose. An environment-level CPU EP does not override the policy. This is raised on #4191 as a possible follow-up issue on the `magika` crate. - Option 1 from the issue (count CPU-burning threads instead) was rejected: the #4187 pool only spun briefly after each inference, so idle CPU was ~0% with or without it and the check wouldn't catch a regression. - This change was developed with GitHub Copilot CLI assistance; all measurements above were run locally by me. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4194" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Romin Parekh <rominparekh@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
Description
This PR registers an internal CoreMLEpFactory with the plugin-EP infrastructure so that, on
supported Apple systems, the CoreML EP participates in device-based EP selection.
The CoreML EP now:
of silently falling back to the CPU EP.
The implementation follows the existing internal-factory pattern used by the CPU, DML, and
WebGPU EPs.
The factory reports "Microsoft" as its vendor name and OrtDevice::VendorIds::MICROSOFT as its
vendor ID, matching the other internal factories: the factory vendor identifies who provides the
EP implementation, not the hardware vendor. Discovered Apple hardware devices keep their existing
"Apple" vendor name and PCI vendor ID (pci_vendor_ids::kApple, 0x106B).
Fixes #29066.
The factory advertises at most one NPU and one GPU from Apple device discovery. Core ML selects
compute-unit classes rather than individual physical devices, so additional devices of the same
type would not be individually addressable and are intentionally ignored.
No devices are advertised below Core ML 5 (macOS 12 / iOS 15), the minimum version accepted by
the current CoreML EP.
The NPU additionally requires Core ML 6 (iOS 16 / macOS 13). Earlier versions do not provide
MLComputeUnitsCPUAndNeuralEngine, the only mode that enables the Neural Engine without also
enabling the GPU. Core ML 5 can still use the ANE through "ALL", but it cannot honor an NPU-only
device selection.
The CPU device is deliberately not advertised. CPU-selection policies select one primary CPU EP
and add the ORT CPU EP as a fallback when needed. If CoreML advertised the CPU, it would compete
with the ORT CPU EP and other CPU-based plugin EPs to become the primary CPU EP. Models that
depend on operators provided by an EP that was not selected could then fail. CoreML-on-CPU
remains available through MLComputeUnits=CPUOnly, and Core ML may use its internal CPU path for
operations unsupported by the selected compute units.
When the caller does not provide MLComputeUnits, the factory derives one value from the complete
device selection: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU + GPU. A
user-provided MLComputeUnits value may narrow which selected accelerators CoreML uses, or disable
them all with CPUOnly, but it may not enable an accelerator that was not selected.
The CoreML EP now implements IExecutionProvider::GetProviderOptions() so its effective options can be
checked. It calls CoreMLOptions::ToProviderOptions(), which reports these options using the public option
names defined in coreml_provider_factory.h.
PREFER_NPU, MAX_EFFICIENCY, and MIN_OVERALL_POWER can now select CoreML on systems where the
factory advertises the Neural Engine.
In builds that register both CoreML and WebGPU, including the official macOS packages, both EPs
advertise the same Apple GPU. Both factories report "Microsoft" as their vendor, which does not
match the Apple hardware vendor. Device ordering therefore falls back to the EP name tie-breaker,
where "CoreMLExecutionProvider" sorts before "WebGpuExecutionProvider". As a result, PREFER_GPU
and MAX_PERFORMANCE select CoreML instead of WebGPU. This ordering comes from the EP names rather
than an explicit preference for CoreML. CoreML uses MLComputeUnits=CPUAndGPU for that selection,
which excludes the Neural Engine. WebGPU remains available through explicit device selection.
Tests
This patch adds 12 CoreML AutoEP tests covering: