Repository navigation
Pin C# RunAsync arguments until completion - #32015
Merged
Akshay Sonawane (apsonawane) merged 2 commits intoAug 13, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the C# InferenceSession.RunAsync pathway by ensuring managed arrays used for native interop remain pinned until the async execution completes, and by tightening cleanup behavior so pinned handles are freed on both success and failure. This reduces the risk of use-after-move GC issues when native code retains array addresses beyond the P/Invoke call.
Changes:
- Update
OrtRunAsyncinterop to pass raw pointers (IntPtr) and pin managedIntPtr[]arrays until the native callback completes. - Make the async callback host disposable and ensure
GCHandles / pinned handles are freed in all code paths. - Strengthen the async inference test to stress object lifetimes via forced GC while native work is in flight.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| csharp/test/Microsoft.ML.OnnxRuntime.Tests.Common/InferenceTest.cs | Adds forced GC/finalizer pressure during an outstanding RunAsync to exercise lifetime correctness. |
| csharp/src/Microsoft.ML.OnnxRuntime/NativeMethods.shared.cs | Adjusts OrtRunAsync delegate signature to accept pointer arguments (IntPtr) instead of managed arrays. |
| csharp/src/Microsoft.ML.OnnxRuntime/InferenceSession.shared.cs | Pins RunAsync argument arrays via GCHandle until callback completion; adds CallbackHost.Dispose() usage and improves failure-path cleanup. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Ti-Tai Wang (titaiwangms)
approved these changes
Aug 13, 2026
Akshay Sonawane (apsonawane)
deleted the
fix/csharp-runasync-pinned-arguments
branch
August 13, 2026 22:09
Eli Belash (Nucs)
added a commit
to SciSharp/NumSharp
that referenced
this pull request
Oct 13, 2026
…oAsync and RunAsync on ONNX Runtime's own native asynchronous run (no .NET thread waits), for ORT 1.30.0 and newer; the task completes off ORT's intra-op thread, where a continuation that disposes the session never returns, and a thread that blocks on it under a synchronization context is released, where ORT's own task waited forever
Work package 7 of docs/plans/onnxruntime-rework.md (item C3a; gap G8).
SIGNATURES (InferenceSessionExtensions.Async.cs; the class is partial now).
static bool IsRunAsyncSupported { get; }
Task RunIntoAsync(session, inputs, outputs, RunOptions runOptions = null, string casting = "safe", OnnxCodecMode mode = Auto)
Task<IReadOnlyDictionary<string, NDArray>> RunAsync(session, inputs, IReadOnlyCollection<string> outputNames = null,
RunOptions runOptions = null, string casting = "safe", OnnxCodecMode mode = Auto)
RunIntoAsync is the asynchronous RunInto: the same inputs, output rules, overlap rule (V9) and pass-through rule
(V12), because it uses the same code. RunAsync is the asynchronous Run with one limit that ONNX Runtime sets.
WHAT IT BUILDS ON. InferenceSession.RunAsync schedules the run on the session's intra-op thread pool and
completes a task from a native callback. It reads the handle of every output value, so the outputs must exist
before the run. Four facts, each measured with a standalone probe on ORT 1.31.0, decide the design:
1. THE VERSION. Before 1.30.0 the managed code handed its name and value arrays to the native run as plain
P/Invoke arguments, which are pinned only for the call, while the native side keeps the pointers until the
run ends (microsoft/onnxruntime#32015; 1.31.0 pins them with GC handles in its CallbackHost). So both forms
throw NotSupportedException on an older assembly, from the call itself and before every other check but the
null checks. The loaded version is read once from the managed assembly: 1.16.0 reports 0.0.0.0, 1.29.0
1.29.0.0, 1.30.0 1.30.0.0, 1.31.0 1.31.0.0. An assembly that reports none counts as older. The message
names the version found, the native library's version, the issue and the synchronous verb to call.
IsRunAsyncSupported lets a caller choose.
2. THE COMPLETION THREAD. ORT completes its task on an intra-op thread of the session (its completion source
runs continuations inline: the probe's continuation ran on a non-pool thread). Code that continues there
runs inside the session's thread pool, and session.Dispose() called there DID NOT RETURN (8 s, then the
probe's watchdog). That is ordinary code: `using var session = ...; await session.RunAsync(...);` ends with
that dispose. So the caller's task comes from a TaskCompletionSource with RunContinuationsAsynchronously: a
continuation runs on the thread pool (or the caller's context), never on ORT's thread. The bridge's own end
of the run (PendingRun.Finish) runs where ORT completes, with no hop of its own: it completes the
pass-through outputs, releases every handle, and only then completes the task. A thread that BLOCKS on the
task is released at once (a blocking wait is not a continuation).
3. THE CALLER'S CONTEXT. ORT awaits its completion source without ConfigureAwait(false), so it captures the
caller's SynchronizationContext, or else the caller's TaskScheduler when it is not the default. With a
context that nobody pumps, ORT's own task never completed: Wait(2 s) returned false, then true once the
probe pumped the context by hand. A UI thread that blocks on the task would wait forever. StartNative hides
the context for the length of ORT's call and restores it; from inside a task of another scheduler (which
cannot be hidden) it makes ORT's call from the default scheduler, at the cost of one thread-pool hop.
4. THE THREAD POOL. ORT refuses an asynchronous run on a session with 1 intra-op thread: its task is faulted
with "intra op thread pool must have at least one thread for RunAsync". The documentation states the rule
(2 or more); the bridge cannot read a session's options, so the failure arrives through the task.
ERRORS. Everything the method can check is checked before the run and throws from the call: null arguments,
the version, the casting rule, the mode, an empty output list, unknown names, dtypes that do not cast, shapes
that contradict the model, a View input that needs a copy, an overlap under View (the message names
RunIntoAsync: SeparateInputsFromOutputs takes the verb now). A failure inside ORT faults the task with ORT's
OnnxRuntimeException: a wrong output shape, RunOptions.Terminate = true (the way to cancel), 1 intra-op
thread. In both cases every handle and temporary is released; a refused call releases what it had prepared
(PendingRun.Abort, which does nothing once ORT has the run, because ORT may be reading the memory).
IN FLIGHT. ORT reads the shared inputs and writes the outputs on its own threads while the caller has its
thread back. The run holds a counted reference on every buffer (the OrtTensor handles), so DISPOSING an input
or an output array before the task completes is safe. WRITING an input before then is a race on values, not
on memory; OnnxCodecMode.Copy feeds a copy of every input taken during the call, so the caller's arrays are
free at once. The PendingRun object is what keeps the handles alive: ORT roots its callback state with a GC
handle, that state references ORT's task, whose continuation is Finish on the PendingRun. A dropped task still
releases its run.
RunAsync ALLOCATES ITS OUTPUTS, so it needs their shape before the run. It accepts a numeric tensor whose
declared shape is fixed, and an output that is a fed input passed through (allocated from the fed tensor,
whatever the declaration says). It refuses, with NotSupportedException and before it reads an input:
- a dimension the model fixes only at run time ("('M', 'N')"): allocate the array and call RunIntoAsync;
- a declaration without dimensions. The metadata reports a declared scalar (shape []) and a tensor of unknown
rank identically, as dims=[] (measured on a SequenceLength output and a rank-agnostic Identity), so the
method does not guess. RunIntoAsync takes a 0-d array;
- a string tensor, a sequence, a map: ORT creates these itself; the synchronous forms return them;
- an element type NumSharp lacks.
A name asked more than once is fetched once; a null name, no name and an unknown name are usage errors. When
the task is faulted the arrays the method allocated are disposed.
TESTS. New SessionRunAsyncTests.cs (20 tests; sessions with 2 intra-op threads; every wait has a deadline and
every test a timeout, because the defects guarded against are deadlocks). On the floor version the first test
checks the refusal and the 19 others report Inconclusive with the reason; the ORT 1.31.0 leg runs them all.
- The version rule: refusal text, thrown from the call, before the other argument checks, nothing prepared.
- RunIntoAsync writes the caller's arrays like RunInto (two inputs); right after the await no export is live
and the arrays are uniquely referenced. Every zero-copy dtype, with 0-d and empty tensors.
- The continuation runs on the thread pool (or inline when the run had ended), never on ORT's thread, and
disposes the session there (a 600x600 product keeps the run in flight at the await).
- A thread that owns a never-pumped synchronization context blocks on the task and is released; nothing is
posted to the context and it is current again after the call.
- The same from inside a task of a single-thread scheduler.
- Inputs and the output disposed while the run is in flight: the whole result is in the buffer (read through
a second view of it).
- A dropped task still releases its run.
- Argument errors throw from the call (11 cases) and release what was prepared.
- An output that overlaps an input under the default mode: computed from the values before the run.
- ORT's failures fault the task and release the run: a wrong output shape, a terminated run, 1 intra-op thread.
- 64 runs in flight on one session.
- Copy mode: the inputs are uniquely referenced right after the call and are overwritten in flight; the
result is the product of the values at the call.
- An output that is a fed input, through both forms.
- RunAsync: an output of each of the 12 element types (new model cast_fanout.onnx: X [2,3] float32 -> Cast ->
as_<dtype> [2,3]); the names asked, each once; the four refusals, with no buffer taken from the pool and a
released input not even looked at; an input error releases the inputs already prepared; a failed run gives
every allocated buffer back to the pool.
ApiSurfaceTests: the approved list gains the three members.
VERIFIED. ONNX suite in Release: 286 / 286 on net10.0 and net8.0 with ORT 1.31.0; 267 passed + 19 Inconclusive
on net10.0 and net8.0 with ORT 1.16.0. Mutation sample with ORT 1.31.0: 18 of 18 killed (continuations inline
on ORT's thread, the context visible, the scheduler visible, a wrong version threshold, handles never
released, pass-through outputs not completed, allocated arrays kept after a failure, a refusal that releases
nothing, each of the four RunAsync refusals removed, names not de-duplicated, a pass-through output allocated
from its declaration, the overlap rule skipped, an empty output list passed on, ORT's failure swallowed, the
input mode ignored). The first mutant showed the hazard directly: with continuations on ORT's thread, the
tests that dispose a session after an await ran into their 120 s timeouts.
Coverage of the bridge (coverlet, Debug, net10.0, ORT 1.31.0): 90.4 % lines (1,670 / 1,847), 87.9 % branches
(824 / 937); it was 89.2 % and 83.3 % at the review, over 1,151 lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request improves the memory management and safety of async inference execution in the ONNX Runtime C# API, particularly around the
RunAsyncmethod. The main changes ensure that arrays passed to native code are properly pinned and unpinned, reducing the risk of memory errors, and that resources are disposed of correctly in both success and failure scenarios. The test code is also updated to more rigorously exercise object lifetimes.Memory management and resource safety improvements
CallbackHostclass now implementsIDisposableand manages the pinning and unpinning of input and output arrays usingGCHandle, ensuring that native code receives stable pointers and that handles are freed even on exceptions. [1] [2] [3]OrtCallbackandRunAsyncInternalmethods are updated to explicitly dispose of theCallbackHostand free handles in all code paths, further strengthening resource safety. [1] [2]Native interop changes
OrtRunAsyncis updated to pass pointers (IntPtr) instead of managed arrays, matching the new pinned memory approach.OrtRunAsync) are updated to use the new pointer properties fromCallbackHostinstead of directly passing arrays.Test improvements
RunOptionsinstance and forces garbage collection, testing that managed resources are not prematurely collected while native work is outstanding.These changes collectively make async inference safer and more robust, especially in scenarios with concurrent or long-running operations.