Skip to content

Associate UHID devices to displays on Android 15 - #6009

Closed
yume-chan wants to merge 2 commits into
Genymobile:devfrom
yume-chan:feat/associate-input-port
Closed

yume-chan wants to merge 2 commits into
Genymobile:devfrom
yume-chan:feat/associate-input-port

Conversation

@yume-chan

@yume-chan yume-chan commented Apr 16, 2025 •

Copy link
Copy Markdown
Contributor

Fixes #4829
Fixes #5547
Fixes #5557

Only works on Android 15 and newer because it requires ASSOCIATE_INPUT_DEVICE_TO_DISPLAY permission.

When running multiple instances of scrcpy --display-id or scrcpy --new-display, each display will have its own mouse pointer.

Has no effect on keyboard, the whole system still only has one input focus.

I used FakeContext.get().getSystemService(Context.INPUT_SERVICE) to get InputManager, because initially I was testing addPortAssociation, which doesn't exist on InputManagerGlobal. This simplifies the code, especially for ClipboardManager, where we can use the stable public API (https://github.com/yume-chan/leap-scrcpy/blob/f9aaf1b05118261d75b82ba88b462f08e37eecdc/server/app/src/main/java/leap/scrcpy/server/FakeContext.kt#L62-L72)

@rom1v

rom1v commented Apr 16, 2025

Copy link
Copy Markdown
Collaborator

Thank you ๐Ÿ‘ I will review later.

Just a quick question:

I used FakeContext.get().getSystemService(Context.INPUT_SERVICE) to get InputManager, because initially I was testing addPortAssociation, which doesn't exist on InputManagerGlobal.

Doesn't it cause a regression for an issue fixed by 5bd7514 on some Android versions?

@yume-chan

Copy link
Copy Markdown
Contributor Author

Doesn't it cause a regression for an issue fixed by 5bd7514 on some Android versions?

I see #4074 is caused by ActivityThread.currentApplication() being null, but now there is a polyfilled one from

private static void fillAppContext() {
try {
Application app = new Application();
Field baseField = ContextWrapper.class.getDeclaredField("mBase");
baseField.setAccessible(true);
baseField.set(app, FakeContext.get());
// activityThread.mInitialApplication = app;
Field mInitialApplicationField = ACTIVITY_THREAD_CLASS.getDeclaredField("mInitialApplication");
mInitialApplicationField.setAccessible(true);
mInitialApplicationField.set(ACTIVITY_THREAD, app);
} catch (Throwable throwable) {
// this is a workaround, so failing is not an error
Ln.d("Could not fill app context: " + throwable.getMessage());
}
}

So If I change com.genymobile.scrcpy.wrappers.InputManager#create back to

    static InputManager create() {
        try {
            Object im = android.hardware.input.InputManager.class.getDeclaredMethod("getInstance").invoke(null);
            return new InputManager(im);
        } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
            throw new RuntimeException(e);
        }
    }

It still works (and calls FakeContext.get().getSystemService(Context.INPUT_SERVICE) through several layers of indirections)

@rom1v

rom1v commented Apr 17, 2025

Copy link
Copy Markdown
Collaborator

This simplifies the code, especially for ClipboardManager, where we can use the stable public API

I just tested to simplify ClipboardManager by using FakeContext.get().getSystemService(FakeContext.CLIPBOARD_SERVICE) as follow: 363c9a5

But then, when I paste:

java.lang.SecurityException: Package android does not belong to 2000
	at android.os.Parcel.createExceptionOrNull(Parcel.java:3261)
	at android.os.Parcel.createException(Parcel.java:3245)
	at android.os.Parcel.readException(Parcel.java:3228)
	at android.os.Parcel.readException(Parcel.java:3170)
	at android.content.IClipboard$Stub$Proxy.getPrimaryClip(IClipboard.java:459)
	at android.content.ClipboardManager.getPrimaryClip(ClipboardManager.java:253)
	at com.genymobile.scrcpy.wrappers.ClipboardManager.getText(ClipboardManager.java:27)
	at com.genymobile.scrcpy.device.Device.getClipboardText(Device.java:106)
	at com.genymobile.scrcpy.device.Device.setClipboardText(Device.java:119)
	at com.genymobile.scrcpy.control.Controller.setClipboard(Controller.java:568)
	at com.genymobile.scrcpy.control.Controller.handleEvent(Controller.java:279)
	at com.genymobile.scrcpy.control.Controller.control(Controller.java:192)
	at com.genymobile.scrcpy.control.Controller.lambda$start$1$com-genymobile-scrcpy-control-Controller(Controller.java:200)
	at com.genymobile.scrcpy.control.Controller$$ExternalSyntheticLambda1.run(D8$$SyntheticClass:0)
	at java.lang.Thread.run(Thread.java:1012)
Caused by: android.os.RemoteException: Remote stack trace:
	at android.app.AppOpsManager.checkPackage(AppOpsManager.java:9636)
	at com.android.server.clipboard.ClipboardService.clipboardAccessAllowed(ClipboardService.java:1329)
	at com.android.server.clipboard.ClipboardService.clipboardAccessAllowed(ClipboardService.java:1313)
	at com.android.server.clipboard.ClipboardService.-$$Nest$mclipboardAccessAllowed(ClipboardService.java:0)
	at com.android.server.clipboard.ClipboardService$ClipboardImpl.getPrimaryClip(ClipboardService.java:663)

@yume-chan

Copy link
Copy Markdown
Contributor Author

For ClipboardManager, the mContext field needs to be replaced with FakeContext, see my link in OP: https://github.com/yume-chan/leap-scrcpy/blob/f9aaf1b05118261d75b82ba88b462f08e37eecdc/server/app/src/main/java/leap/scrcpy/server/FakeContext.kt#L62-L72

@rom1v

rom1v commented Apr 17, 2025

Copy link
Copy Markdown
Collaborator

Indeed, thank you, it works: a899752

try {
FileDescriptor old = fds.put(id, fd);
// Must be unique across the system
String inputPort = "scrcpy:" + Os.getpid() + ":" + id;

@rom1v rom1v Apr 24, 2025 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is only one virtual display per scrcpy session, do we need one input port per UHID device?

I think we could just use String inputPort = "scrcpy:" + Os.getpid(); and call addUniqueIdAssociationByPort() only once (and all UHID devices would use the same input port associated to the single display id). That way, storing an inputPort per device would be unnecessary.

I'm not asking you to make any change, I would just like to know what you think about it.

(Currently, I cannot test virtual displays because of #5523 on my current device.)

@yume-chan yume-chan Apr 25, 2025 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I never thought about that. Initially I saw Android Studio is using different phys values for each device: https://cs.android.com/android-studio/platform/tools/adt/idea/+/mirror-goog-studio-main:streaming/screen-sharing-agent/app/src/main/cpp/virtual_input_device.cc;l=66-68;drc=fc06f05ca98ff7d3c8558976fd7eaf1b5480e2e8

But I believe multiple devices can have the same phys value, it means they are in one physical device, but with multiple input features. And addUniqueIdAssociationByPort still works.

The original commit has been backed up at https://github.com/yume-chan/scrcpy/tree/feat/associate-input-port-2

@yume-chan
yume-chan force-pushed the feat/associate-input-port branch from 0ef8f64 to 6d17ee6 Compare April 25, 2025 04:32
@rom1v

rom1v commented Apr 25, 2025

Copy link
Copy Markdown
Collaborator

(Currently, I cannot test virtual displays because of #5523 on my current device.)

I tested on two devices:

  • Motorola Edge 30 Neo with Android 14
  • Redmi Note 13 with Android 14

On both of them, running either your initial branch or your new one, the UHID mouse pointer does not appear on the virtual display (but it correctly disappears from the main display).

I run with:

scrcpy --new-display -M
scrcpy --new-display -M --start-app=org.videolan.vlc  # to get an app running

@yume-chan

yume-chan commented Apr 25, 2025 •

Copy link
Copy Markdown
Contributor Author

I see. So Android 14 still only has one mouse pointer across all displays. We need not only associate the UHID device to the display, but also move the mouse cursor to the display, which is impossible because the API is not available outside system server (same as my original comment #4829 (comment)).

I guess adding association alone is enough for virtual touchscreen devices

Only Android 15 added seperated mouse cursors for each display (https://cs.android.com/android/_/android/platform/frameworks/native/+/da10dd34cb41f0f117fe87f568f16f6025ab3d72)

@rom1v

rom1v commented Apr 25, 2025

Copy link
Copy Markdown
Collaborator

Thank you.

Here is a branch: pr6009.3 (not tested, because I currently have no device with Android 15 not impacted by #5523), with the following changes:

  • extract the change for using the public InputManager API to a separate commit
  • pass the display unique id in the "synchronized" DisplayData instance (DisplayManager can only be called from the "display" thread currently, as there is no synchronization, so pass the display unique id along with the display id and the position mapper)
  • wait for the virtual display to be available before creating the UhidManager (with a timeout of 1 second), using the same mechanism as for --start-app
  • handle the case where the unique id is null (in that case, do not associate an input port)
  • (as a side effect) also associate in case displayId == 0 (should we make a special case?)

TODO:

  • retrieve the display unique id also with the fallback mechanism in DisplayManager.parseDisplayInfo()

@rom1v

rom1v commented Apr 26, 2025

Copy link
Copy Markdown
Collaborator

TODO:

  • retrieve the display unique id also with the fallback mechanism in DisplayManager.parseDisplayInfo()

It would require to rework the parsing, as a single regex is not sufficient (depending on the device, the values are not always in the same order). Since it's just a fallback for some old devices, it is probably useless, so I will not do it until necessary.

So I think the branch can be considered ready.

Could you please review/test? I managed to test it successfully on a Pixel 6a.

@yume-chan

Copy link
Copy Markdown
Contributor Author

The timing between adding port association and creating UHID devices doesn't matter, so I think it could be simplified a little (just adding the association when initializing UHID manager and remove it on exit).

The association will NOT be removed by Android OS when Scrcpy process exits, although it won't cause any harm, if we want to make sure it's 100% clean, maybe we need to add it to the CleanUp process.

Otherwise looks good to me and tests OK on my Redmi K70 Pro running Android 15.

@rom1v

rom1v commented Apr 30, 2025 •

Copy link
Copy Markdown
Collaborator

Thank you for your review.

The timing between adding port association and creating UHID devices doesn't matter, so I think it could be simplified a little (just adding the association when initializing UHID manager and remove it on exit).

When scrcpy runs without UHID device, I want to avoid adding a useless association. So it must be done only when at least one UHID device is created (and only once the virtual display id is known, whichever happens first).

Also, I think that my branch only adds the association for "new virtual displays" (not when --display-id=XX is passed).

What makes it more complicated is that currently, DisplayManager may not be called from the controller thread (and I don't want to add unnecessary synchronization, but maybe I'll have to so that I can get the displayUniqueId from a displayId in the controller thread, I will think about it).

@rom1v

rom1v commented May 3, 2025

Copy link
Copy Markdown
Collaborator

Also, I think that my branch only adds the association for "new virtual displays" (not when --display-id=XX is passed).

Fixed, now it should be called both for virtual displays and non-main display (I disabled the association for the main display, since it works by default).

What makes it more complicated is that currently, DisplayManager may not be called from the controller thread (and I don't want to add unnecessary synchronization, but maybe I'll have to so that I can get the displayUniqueId from a displayId in the controller thread, I will think about it).

I added synchronization and now call getDisplayInfo() from the controller thread.

The association will NOT be removed by Android OS when Scrcpy process exits, although it won't cause any harm, if we want to make sure it's 100% clean, maybe we need to add it to the CleanUp process.

OK. Since it is harmless, for simplicity I just remove the association when it closes normally, but not if the process is killed.

New version: pr6009.7

rom1v added a commit that referenced this pull request May 22, 2025
Use the public ClipboardManager API, with the FakeContext as context.

PR #6009 <#6009>

Suggested by: Simon Chan <1330321+yume-chan@users.noreply.github.com>
rom1v pushed a commit that referenced this pull request May 22, 2025
Use the public InputManager API.

PR #6009 <#6009>

Signed-off-by: Romain Vimont <rom@rom1v.com>
rom1v added a commit that referenced this pull request May 22, 2025
Do not use reflection to retrieve the method for every call.

PR #6009 <#6009>
rom1v added a commit that referenced this pull request May 22, 2025
The DisplayManager and its method getDisplayInfo() may be used from both
the Controller thread and the video (main) thread.

PR #6009 <#6009>
rom1v pushed a commit that referenced this pull request May 22, 2025
This allows the mouse pointer to appear on the correct display (only for
devices running Android 15+).

Fixes #5547 <#5547>
PR #6009 <#6009>

Signed-off-by: Romain Vimont <rom@rom1v.com>
@rom1v

rom1v commented May 22, 2025 •

Copy link
Copy Markdown
Collaborator

I rebased: pr6009.8.

@yume-chan Could you please test and confirm that it works correctly before I merge? I was able to test with an older version of the branch, but the device I used has been updated and is now impacted by #5523).

@yume-chan

Copy link
Copy Markdown
Contributor Author

I tested pr6009.8 works on my Redmi K70 Pro running Android 15.

@rom1v

rom1v commented Jun 2, 2025

Copy link
Copy Markdown
Collaborator

@yume-chan Thank you for your tests ๐Ÿ‘

I just noticed that 3b36f29 breaks copy-paste from the Android device to the computer: the callback passed to clipboardManager.addPrimaryClipChangedListener() is never called.

@anotheruserofgithub

anotheruserofgithub commented Jun 3, 2025 •

Copy link
Copy Markdown

@rom1v Just a side remark in Controller.waitDisplayData(long) called from Controller.getUhidManager():

long timeout = deadline - System.currentTimeMillis();
if (timeout < 0) {
return null;
}
displayDataAvailable.wait(timeout);
It should test if (timeout <= 0) because if timeout is zero thenObject.wait(long) waits for a notification (or a spurious wakeup).

Other classes in the java.util.concurrent package behave differently when timeout is zero or negative, though you often have to delve into the implementation to be sure how they work. For instance, ReentrantLock.tryLock(long, TimeUnit) (source, called from here) will not synchronize and not wait at all, while if you create a Condition from a ReentrantLock then Condition.await(long, TimeUnit) (source, ConditionObject created here) will still yield the lock. It was more clearly documented e.g. in JDK 11 but comments disappeared in more recent versions.

@rom1v

rom1v commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator

@anotheruserofgithub Indeed, we should not call wait(0):

If timeout is zero, however, then real time is not taken into consideration and the thread simply waits until notified.

Since this is on dev (not related to this PR), could you open a PR (so that the commit has your authorship)?

Even if it does not change anything in practice, I would not test timeout <= 0 because it feels wrong to not read the value if timeout is 0. I'd prefer something like:

diff --git server/src/main/java/com/genymobile/scrcpy/control/Controller.java server/src/main/java/com/genymobile/scrcpy/control/Controller.java
index 5e64a4c5b..24d827fda 100644
--- server/src/main/java/com/genymobile/scrcpy/control/Controller.java
+++ server/src/main/java/com/genymobile/scrcpy/control/Controller.java
@@ -699,7 +699,9 @@ public class Controller implements AsyncProcessor, VirtualDisplayListener {
                 if (timeout < 0) {
                     return null;
                 }
-                displayDataAvailable.wait(timeout);
+                if (timeout > 0) {
+                    displayDataAvailable.wait(timeout);
+                }
                 data = displayData.get();
             }
 

@anotheruserofgithub

Copy link
Copy Markdown

could you open a PR (so that the commit has your authorship)?

I don't wish to have authorship, this is such a minor thing.

I would not test timeout <= 0 because it feels wrong to not read the value if timeout is 0.

You're right! So it's definitely your authorship. ;)

rom1v added a commit that referenced this pull request Jun 3, 2025
Calling wait(0) results in waiting without a timeout, which is
unintended.

Refs #6009 comment <#6009 (comment)>
@rom1v

rom1v commented Jun 3, 2025

Copy link
Copy Markdown
Collaborator

@anotheruserofgithub OK, thanks. Merged: 52f5d08

@yume-chan yume-chan changed the title Associate UHID devices to virtual displays on Android 14+ Associate UHID devices to virtual displays on Android 15 Jun 4, 2025
@yume-chan yume-chan changed the title Associate UHID devices to virtual displays on Android 15 Associate UHID devices to displays on Android 15 Jun 4, 2025
@yume-chan

Copy link
Copy Markdown
Contributor Author

I just noticed that 3b36f29 breaks copy-paste from the Android device to the computer: the callback passed to clipboardManager.addPrimaryClipChangedListener() is never called.

I see. ClipboardManager uses main looper to post the callback (https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/content/ClipboardManager.java;l=102;drc=61197364367c9e404c7da6900658f1b16c42d0da), so the main looper needs to be running. That's pretty annoying.

I can either

  1. Manually run the main looper on a new thread like yume-chan@1044101 (looks little bit hacky)
  2. Let the main thread run the main looper (as normal Android app would do), and run the main scrcpy method on another thread
  3. While doing the first half of 2, make the main method event-based using the main looper, so it doesn't block on completion.await()

Still simpler than old ClipboardService wrapper I would say.

(While the main looper is running, it can also be used elsewhere so we don't need to create so many HandlerThreads, for example with camera: yume-chan@08de0b6 (although camera might be a bad example because it shouldn't be blocked by other tasks))

@rom1v

rom1v commented Jun 4, 2025

Copy link
Copy Markdown
Collaborator

Let the main thread run the main looper [โ€ฆ] make the main method event-based using the main looper

Yes, I think it would be a good solution (even for other Android features that might use the main looper).

Since this is a separate issue, I will probably merge this PR without the clipboard refactor, and this can be done afterwards.

@rom1v

rom1v commented Jun 5, 2025

Copy link
Copy Markdown
Collaborator

Merged on dev (without the ClipboardManager refactor).

I will open a new PR with a looper.

@rom1v rom1v closed this Jun 5, 2025
rom1v pushed a commit that referenced this pull request Jun 5, 2025
Use the public InputManager API.

PR #6009 <#6009>

Signed-off-by: Romain Vimont <rom@rom1v.com>
rom1v added a commit that referenced this pull request Jun 5, 2025
Do not use reflection to retrieve the method for every call.

PR #6009 <#6009>
rom1v added a commit that referenced this pull request Jun 5, 2025
The DisplayManager and its method getDisplayInfo() may be used from both
the Controller thread and the video (main) thread.

PR #6009 <#6009>
rom1v pushed a commit that referenced this pull request Jun 5, 2025
This allows the mouse pointer to appear on the correct display (only for
devices running Android 15+).

Fixes #5547 <#5547>
PR #6009 <#6009>

Signed-off-by: Romain Vimont <rom@rom1v.com>
rom1v added a commit that referenced this pull request Jun 5, 2025
This will allow the main thread to process any event posted to the main
looper.

Refs #6009 comment <#6009 (comment)>
rom1v added a commit that referenced this pull request Jun 5, 2025
Use the public ClipboardManager API, with the FakeContext as context.

This requires an active main looper, otherwise clipboard changes are not
processed.

PR #6009 <#6009>

Suggested by: Simon Chan <1330321+yume-chan@users.noreply.github.com>
rom1v added a commit that referenced this pull request Jun 5, 2025
Use the public ClipboardManager API, with the FakeContext as context.

This requires a running main looper, otherwise clipboard changes are not
processed.

PR #6009 <#6009>

Suggested by: Simon Chan <1330321+yume-chan@users.noreply.github.com>
@rom1v rom1v mentioned this pull request Jun 5, 2025
rom1v added a commit that referenced this pull request Jun 6, 2025
Instead of blocking the main thread until completion, run a looper.

This will allow the main thread to process any event posted to the main
looper.

Refs #6009 comment <#6009 (comment)>
rom1v added a commit that referenced this pull request Jun 6, 2025
Use the public ClipboardManager API, with the FakeContext as context.

This requires a running main looper, otherwise clipboard changes are not
processed.

PR #6009 <#6009>

Suggested by: Simon Chan <1330321+yume-chan@users.noreply.github.com>
rom1v added a commit that referenced this pull request Jun 11, 2025
Instead of blocking the main thread until completion, run a looper.

This will allow the main thread to process any event posted to the main
looper.

Refs #6009 comment <#6009 (comment)>
PR #6129 <#6129>
rom1v added a commit that referenced this pull request Jun 11, 2025
Use the public ClipboardManager API, with the FakeContext as context.

This requires a running main looper, otherwise clipboard changes are not
processed.

Refs #6009 <#6009>
PR #6129 <#6129>

Suggested by: Simon Chan <1330321+yume-chan@users.noreply.github.com>
rom1v added a commit that referenced this pull request Nov 4, 2025
On some devices, DisplayInfo does not have a "uniqueId" field. This
field is only used for correct UHID behavior on virtual displays, so its
absence should not prevent scrcpy from working.

Refs #6009 <#6009>
Fixes #6461 <#6461>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants