Skip to content

Expose CoreML EP via OrtEpFactory - #31975

Merged
Scott McKay (skottmckay) merged 13 commits into
microsoft:mainfrom
maxim-davgalev:coreml-ep-factory
Sep 29, 2026
Merged

Scott McKay (skottmckay) merged 13 commits into
microsoft:mainfrom
maxim-davgalev:coreml-ep-factory

Conversation

@maxim-davgalev

@maxim-davgalev maxim-davgalev commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR registers an internal CoreMLEpFactory with the plugin-EP infrastructure so that, on
supported Apple systems, the CoreML EP participates in device-based EP selection.

The CoreML EP now:

  • appears in GetEpDevices() as NPU and GPU OrtEpDevice entries.
  • can be added with SessionOptionsAppendExecutionProvider_V2 by selecting either or both devices.
  • participates in SetEpSelectionPolicy, so PREFER_NPU can select the Apple Neural Engine instead
    of silently falling back to the CPU EP.

The implementation follows the existing internal-factory pattern used by the CPU, DML, and
WebGPU EPs.

The factory reports "Microsoft" as its vendor name and OrtDevice::VendorIds::MICROSOFT as its
vendor ID, matching the other internal factories: the factory vendor identifies who provides the
EP implementation, not the hardware vendor. Discovered Apple hardware devices keep their existing
"Apple" vendor name and PCI vendor ID (pci_vendor_ids::kApple, 0x106B).

Fixes #29066.

The factory advertises at most one NPU and one GPU from Apple device discovery. Core ML selects
compute-unit classes rather than individual physical devices, so additional devices of the same
type would not be individually addressable and are intentionally ignored.

No devices are advertised below Core ML 5 (macOS 12 / iOS 15), the minimum version accepted by
the current CoreML EP.

The NPU additionally requires Core ML 6 (iOS 16 / macOS 13). Earlier versions do not provide
MLComputeUnitsCPUAndNeuralEngine, the only mode that enables the Neural Engine without also
enabling the GPU. Core ML 5 can still use the ANE through "ALL", but it cannot honor an NPU-only
device selection.

The CPU device is deliberately not advertised. CPU-selection policies select one primary CPU EP
and add the ORT CPU EP as a fallback when needed. If CoreML advertised the CPU, it would compete
with the ORT CPU EP and other CPU-based plugin EPs to become the primary CPU EP. Models that
depend on operators provided by an EP that was not selected could then fail. CoreML-on-CPU
remains available through MLComputeUnits=CPUOnly, and Core ML may use its internal CPU path for
operations unsupported by the selected compute units.

When the caller does not provide MLComputeUnits, the factory derives one value from the complete
device selection: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and ALL for NPU + GPU. A
user-provided MLComputeUnits value may narrow which selected accelerators CoreML uses, or disable
them all with CPUOnly, but it may not enable an accelerator that was not selected.

The CoreML EP now implements IExecutionProvider::GetProviderOptions() so its effective options can be
checked. It calls CoreMLOptions::ToProviderOptions(), which reports these options using the public option
names defined in coreml_provider_factory.h.

PREFER_NPU, MAX_EFFICIENCY, and MIN_OVERALL_POWER can now select CoreML on systems where the
factory advertises the Neural Engine.

In builds that register both CoreML and WebGPU, including the official macOS packages, both EPs
advertise the same Apple GPU. Both factories report "Microsoft" as their vendor, which does not
match the Apple hardware vendor. Device ordering therefore falls back to the EP name tie-breaker,
where "CoreMLExecutionProvider" sorts before "WebGpuExecutionProvider". As a result, PREFER_GPU
and MAX_PERFORMANCE select CoreML instead of WebGPU. This ordering comes from the EP names rather
than an explicit preference for CoreML. CoreML uses MLComputeUnits=CPUAndGPU for that selection,
which excludes the Neural Engine. WebGPU remains available through explicit device selection.

Tests

This patch adds 12 CoreML AutoEP tests covering:

  • factory registration and NPU/GPU advertisement.
  • explicit single-device and NPU+GPU V2 selection.
  • PREFER_NPU, PREFER_GPU, and their policy aliases.
  • incompatible option validation.
  • duplicate device selections.
  • device enumeration limits.
  • cleanup of previously created devices after a subsequent creation failure.
  • the MLComputeUnits value derived from the selected devices (NPU, GPU, and NPU + GPU) and caller-provided values that narrow the selection.

Register an internal CoreMLEpFactory with the plugin-EP
infrastructure so that, on supported Apple systems with discovered
accelerator hardware, the CoreML EP appears in
GetEpDevices(), can be added to a session with
SessionOptionsAppendExecutionProvider_V2, and is eligible for EP
selection policies, notably PREFER_NPU, which can now select the Apple
Neural Engine on supported systems instead of silently falling back
to the CPU EP.

Fixes microsoft#29066

The factory registers with vendor ID OrtDevice::VendorIds::APPLE,
matching the vendor ID reported by Apple device discovery, so device
ordering during EP selection prefers CoreML for Apple hardware via
the vendor-affinity rule.

In builds that also register the internal WebGPU EP, including the
official macOS packages, PREFER_GPU and MAX_PERFORMANCE now select
CoreML instead of WebGPU when both EPs claim the same Apple GPU,
because CoreML has matching vendor affinity.

The factory advertises at most one NPU and one GPU from device
discovery. It advertises no devices below Core ML 5, the EP's minimum
supported version. The NPU additionally requires Core ML 6
(iOS 16 / macOS 13), because earlier Core ML versions have no
compute-units mode that enables the Neural Engine without also
enabling the GPU. MLComputeUnitsCPUAndNeuralEngine requires Core ML 6,
so an NPU selection could not otherwise be honored.

The CPU device is deliberately left to the CPU EP. CPU-selecting
policies choose a single preferred EP for the CPU device, so
advertising it would displace the ORT CPU EP and other CPU-based EPs.
CoreML-on-CPU remains available through an explicit
MLComputeUnits=CPUOnly option, and Core ML can use its internal CPU
path for operations unsupported by the selected compute units.

When MLComputeUnits is not specified, the factory derives it from the
selected devices: CPUAndNeuralEngine for NPU, CPUAndGPU for GPU, and
ALL for NPU+GPU. User options may narrow this set, but may not enable
an accelerator that was not selected.

Tests in onnxruntime/test/autoep/test_selection.cc cover factory
registration, explicit GPU and NPU+GPU selection, inference,
PREFER_NPU/PREFER_GPU and their policy aliases, compute-unit
defaulting and conflict validation, version checks, device validation
and selection limits, CPU-only narrowing, and transactional cleanup
after partial device-creation failures.
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@maxim-davgalev

Copy link
Copy Markdown
Contributor Author

maxim-davgalev please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

@microsoft-github-policy-service agree company="Parallels"

@maxim-davgalev

Copy link
Copy Markdown
Contributor Author

Hi Edward Chen (@edgchen1), would you be the right person to review this PR, or could you suggest an appropriate reviewer? It exposes the CoreML EP via OrtEpFactory and enables device-based EP selection. Thanks!

The CoreML EP is implemented as part of ONNX Runtime, not provided by
Apple, so its factory now reports "Microsoft" and
VendorIds::MICROSOFT, like the other internal EP factories (CPU, DML,
and WebGPU). Discovered Apple hardware devices keep their "Apple" /
VendorIds::APPLE identifiers. VendorIds::APPLE remains the named
constant for the vendor ID reported by Apple platform device
discovery.

Because the factory vendor no longer matches the Apple hardware
vendor, CoreML is no longer preferred based on vendor matching during
device ordering. When another EP claims the same hardware device,
ordering falls back to the EP name tie-breaker. The
CoreMLVendorAffinityPrecedesEpNameOrdering test was deleted because it
tested the vendor affinity behavior removed by this change. The
remaining vendor assertions now expect Microsoft identifiers for the
factory and Apple identifiers for the hardware device.
Rename kCoremlComputeUnits_* to kCoremlProviderOption_MLComputeUnits_*.
Call the native binding through the _pybind_state module.
Copilot AI balanced review requested due to automatic review settings August 28, 2026 05:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Registers CoreML with device-based EP selection on supported Apple systems.

Changes:

  • Adds an internal CoreML EP factory for NPU/GPU discovery and selection.
  • Derives and validates MLComputeUnits from selected devices.
  • Adds Apple vendor IDs and comprehensive AutoEP tests.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
onnxruntime/test/python/onnxruntime_test_python.py Tests Apple vendor IDs.
onnxruntime/test/autoep/test_selection.cc Tests CoreML discovery, selection, validation, and cleanup.
onnxruntime/python/onnxruntime_inference_collection.py Exposes Apple vendor ID to Python.
onnxruntime/core/session/plugin_ep/ep_library_internal.h Declares CoreML factory creation.
onnxruntime/core/session/plugin_ep/ep_library_internal.cc Registers the CoreML factory.
onnxruntime/core/session/plugin_ep/ep_factory_coreml.h Defines the internal factory interface.
onnxruntime/core/session/plugin_ep/ep_factory_coreml.cc Implements discovery, validation, and provider creation.
onnxruntime/core/providers/coreml/model/host_utils.h Defines the NPU-selection version requirement.
onnxruntime/core/providers/coreml/coreml_options.cc Uses shared compute-unit constants.
onnxruntime/core/platform/apple/device_discovery.cc Uses the shared Apple vendor ID.
include/onnxruntime/core/providers/coreml/coreml_provider_factory.h Publishes compute-unit option constants.
include/onnxruntime/core/framework/ortdevice.h Adds the Apple vendor ID.

πŸ’‘ Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

After the CoreML factory switched to the Microsoft vendor ID,
OrtDevice::VendorIds::APPLE was no longer used. Apple device
discovery keeps its local kApplePciVendorId constant (0x106B)
as before.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

The CoreML EP factory is an internal EP: there is no library boundary
between it and ORT, so its entry points do not need to convert
exceptions into OrtStatus.
The device claiming and option validation helpers of CoreMLEpFactory
were public static members only so that unit tests could call them
directly. Move them into an anonymous namespace in ep_factory_coreml.cc,
fold OrtEpDevice creation and publication into GetSupportedDevices,
and remove the tests that called the helpers directly.
Move the CoreML EP selection tests from test_selection.cc to a dedicated CoreML test file.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟑 Changes recommended

Core ML version-boundary behavior lacks deterministic test coverage.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 1
  • Review effort level: Balanced

@maxim-davgalev

Copy link
Copy Markdown
Contributor Author

Edward Chen (@edgchen1), I'd like to discuss one more topic. It does not expand the scope of this PR.
As you may have noticed, the factory does not advertise a CPU device. Here is why.

I tried to add a CPU device to the CoreML factory, but two existing tests then failed:

1. OrtEpLibrary.PluginEp_Custom_Op_Inference_With_Prefer_Cpu

This test registers example_ep, which implements Custom_Mul, and uses PREFER_CPU. In this configuration, the CPU candidates were ordered as:

CoreMLExecutionProvider -> example_ep -> CPUExecutionProvider

The policy selected CoreML and the ORT CPU EP. Consequently, example_ep was not added to the session. CoreML could not execute Custom_Mul, and the ORT CPU EP could not execute it either. Session creation failed with β€œCould not find an implementation for Custom_Mul(1) node”.

2. ModelPackageTest.LoadModelPackageAndRunInference_PreferCpu

This test creates a package containing CPU and NPU variants targeting example_ep, then uses PREFER_CPU. The policy again selected CoreML and the ORT CPU EP. GetVariantSelectionEpInfo captures only provider_list[0], and VariantSelector::SelectVariant considers only ep_infos[0]. Since the first EP was CoreML and the package had no CoreML variant, the test failed with β€œNo suitable model variant found for the configured execution providers”.

This is a package-variant selection failure and keeping more EPs in the policy result would not fix it on its own: the resolver must also consider later candidates. There is already a TODO for this in model_package_variant_selector.cc.

==================================

As a possible solution, I would propose supporting a ranked chain of CPU EPs, which would give another CPU EP the opportunity to handle the nodes the first one does not support.

For ordinary ONNX sessions, explicit registration already provides the relevant behavior: applications can append different EPs through separate AppendExecutionProvider_V2 calls, and graph partitioning considers them in registration order. Nodes not assigned to one EP remain available to later EPs, with the ORT CPU EP last.

A policy could keep multiple eligible CPU EPs in ranked order instead of reducing them to one primary EP plus the ORT CPU EP. This would reuse existing graph-partitioning behavior. Model packages would need a corresponding extension: consider EPs in priority order and choose the first one with a compatible variant.

I also measured CoreML CPU against the ORT CPU EP on an Apple M3 Pro using static, batch-1 models. These are latencies in milliseconds, each pair is two separate runs:

Model ORT CPU EP CoreML MLProgram, CPUOnly CoreML speedup
ResNet-50 FP16 20.64 / 20.85 5.35 / 5.20 3.86Γ— / 4.01Γ—
ResNet-50 FP32 20.89 / 20.95 9.98 / 10.17 2.09Γ— / 2.06Γ—
BERT-SQuAD FP16 87.03 / 87.28 25.02 / 25.02 3.48Γ— / 3.49Γ—
BERT-SQuAD FP32 85.39 / 85.39 54.75 / 55.13 1.56Γ— / 1.55Γ—

CoreML on the CPU is up to 4Γ— faster than the ORT CPU EP here.
I think these results show that it makes sense to discuss letting CPU selection use more than one CPU EP: a specialized CPU EP can be much faster than the default one, but it does not cover every operator, and the tests above show that a single-EP selection is not enough.

I should also note that these numbers are for MLProgram, not the current NeuralNetwork default, which has known gaps in FP16 support. With NeuralNetwork and CPUOnly, both FP16 models ran entirely on the ORT CPU EP.

What do you think?

JD Davis (JerrettDavis) added a commit to headroomlabs-ai/headroom that referenced this pull request Nov 4, 2026
## Description

`tests/magika_idle_threads.rs` (added in #4189) fails on macOS with ONNX
Runtime 1.31. Building the magika session leaves 8 extra threads against
a fixed allowance of 2.

The cause is not a thread pool. ORT 1.31 registers Core ML as a built-in
EP that claims the Apple NPU/GPU (microsoft/onnxruntime#31975). The
`ort` 2.0.0-rc.12 crate sets the `MAX_EFFICIENCY` (= `PREFER_NPU`) EP
selection policy on every `SessionBuilder::new()`. So magika's session
now runs on Core ML / Apple Neural Engine (ANE), and building it starts
5–7 ANE service threads that are idle, not spinning. The fixed allowance
of 2 can't account for that.

The test now builds a pool-free control session
(`magika::Session::builder().with_intra_threads(1)`) first and uses the
threads it adds as the allowance. The control goes through the same EP
selection, so whatever the EP starts cancels out, while a reintroduced
intra-op pool still shows up on top.

As a precaution, Rust CI now also runs `cargo test` on `macos-26`. It
only ran on Ubuntu, which is why this slipped through.

Closes #4191

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `crates/headroom-core/tests/magika_idle_threads.rs`: measure the
allowance against a pool-free control session instead of a fixed `2`,
with comments explaining the ORT 1.31 Core ML/ANE cause. Test-only; no
production code changes.
- `.github/workflows/rust.yml`: turn the `test` job into a matrix with
`ubuntu-latest` and `macos-26`, using the same ONNX Runtime setup step.
Ubuntu keeps its `test (ubuntu)` check name and runs the same steps as
before. macOS runs only `cargo test`; `cargo fmt` and `cargo clippy`
stay on Ubuntu. `fail-fast: false`, so one OS failing doesn't cancel the
other.

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [x] Rust: `cargo fmt --all -- --check`, `cargo clippy --workspace --
-D warnings`, `cargo test --workspace` with `ORT_DYLIB_PATH` set so the
magika tests actually run
- [ ] Unit tests pass (`pytest`): only the pre-push `ci-precheck-python`
subset was run (177 passed, 4 skipped); the full suite was not
- [x] Linting passes (`ruff check .`): no Python files changed;
pre-commit hooks passed
- [ ] Type checking passes (`mypy headroom`): N/A, no Python changes
- [x] New tests added for new functionality (updated regression test)
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

`cargo test -p headroom-core --test magika_idle_threads`, with the #4187
fix (`with_intra_threads(1)`) in place and with it temporarily reverted:

```text
# macOS arm64, ORT 1.31.0, before this PR (fixed allowance of 2)
magika session left 8 extra OS thread(s) running ... FAILED
# macOS arm64, ORT 1.31.0, this branch
test result: ok. 1 passed; 0 failed      (6/6 repeat runs)
# macOS arm64, ORT 1.29.0, this branch
test result: ok. 1 passed; 0 failed

# #4187 fix reverted, this branch: the test still catches the pool
macOS, ORT 1.31:                      12 extra vs 9 allowed -> FAILED
macOS, ORT 1.29:                       7 extra vs 2 allowed -> FAILED
Linux (Docker aarch64), ORT 1.31:      7 extra vs 1 allowed -> FAILED
```

Full Rust suite on macOS with ORT 1.31.0 (the same command the new
`macos-26` job runs):

```text
ORT_DYLIB_PATH=.../libonnxruntime.1.31.0.dylib cargo test --workspace
68 suites, 1658 passed, 0 failed
```

Pre-push `make ci-precheck` on this branch:

```text
ci-precheck-python: 177 passed, 4 skipped
βœ… ci-precheck PASSED β€” safe to push.
```

## Real Behavior Proof

- Environment: macOS 26 arm64 (16 cores), Rust 1.95, `pip install
onnxruntime==1.31.0` and 1.29.0 dylibs via `ORT_DYLIB_PATH`; Linux via
Docker `rust:1.95` (aarch64) with pip onnxruntime 1.31.0.
- Exact command / steps: the reporter's repro from #4191 (`pip install
'onnxruntime==1.31.0'`, `export
ORT_DYLIB_PATH=<capi/libonnxruntime.*.dylib>`, `cargo test -p
headroom-core --test magika_idle_threads`), on `main` and on this
branch; then the same test with the #4187 `with_intra_threads(1)` fix
temporarily reverted, on macOS 1.31, macOS 1.29 and Linux 1.31. To find
the cause, a raw `ort` probe built the magika model with
`intra_threads=1` under the default policy and under
`with_auto_device(PreferCPU)`, counting threads with `ps -M` and listing
the EP devices.
- Observed result: fails on `main` with ORT 1.31 (8 extra vs 2) and
passes on this branch, including the reporter's exact steps. With the
#4187 fix reverted it fails on all three setups (12 vs 9, 7 vs 2, 7 vs
1), so it still guards #4187. Probe: EP devices are CPU, CoreML/NPU and
CoreML/GPU. The default policy adds 6–7 threads and takes ~420 ms to
build; `PreferCPU` adds 0 and takes ~12 ms.
- Not tested: the new `macos-26` CI job on GitHub runners (it runs on
this PR), ORT 1.30.0 at runtime, x86_64 Linux, Intel macOS, and low-core
macOS hosts. Details:
- ORT 1.30.0: the 1.30 vs 1.31 difference comes from the ORT source diff
and release notes, plus the reporter's data.
- Low-core macOS hosts: with only 2–4 cores the control allowance can
absorb a small pool, so the guard is weaker there.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A
- Stable/default behavior changed: no. The PR changes only the test and
CI; runtime code is untouched.
- Kill switch / disable path: N/A
- Unsafe override required: no.
- Qualification impact: adds one macOS CI job (only when Rust paths
change).
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` β€” it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- Not fixed here: on macOS with ORT β‰₯ 1.31, headroom's magika detection
runs through Core ML on the ANE instead of the CPU EP. That comes from
`ort`'s default policy; nobody chose it. Visible cost: a one-off ~400 ms
session build plus 5–7 idle threads. Per-call latency and output parity
were not measured. Pinning it to CPU needs
`with_auto_device(PreferCPU)`, which magika 1.1.0's `Builder` doesn't
expose. An environment-level CPU EP does not override the policy. This
is raised on #4191 as a possible follow-up issue on the `magika` crate.
- Option 1 from the issue (count CPU-burning threads instead) was
rejected: the #4187 pool only spun briefly after each inference, so idle
CPU was ~0% with or without it and the check wouldn't catch a
regression.
- This change was developed with GitHub Copilot CLI assistance; all
measurements above were run locally by me.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4194"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Romin Parekh <rominparekh@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Expose the CoreML EP via OrtEpFactory so it appears in GetEpDevices() and participates in automatic EP selection

4 participants