Skip to content

VA-API: accept DRM_FORMAT_RG88 for NV12 chroma plane - #7070

Closed
niawag wants to merge 1 commit into
Genymobile:masterfrom
niawag:patch-1
Closed

niawag wants to merge 1 commit into
Genymobile:masterfrom
niawag:patch-1

Conversation

@niawag

@niawag niawag commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Disclaimer: AI was used to debug scrcpy and find this fix

Description

On NVIDIA GPUs using nvidia-vaapi-driver, VA-API hardware decoding may export NV12 DRM PRIME frames using DRM_FORMAT_RG88 for the chroma plane.

Currently, scrcpy only accepts DRM_FORMAT_GR88:

return desc->nb_layers == 2
    && desc->layers[0].format == DRM_FORMAT_R8
    && desc->layers[0].nb_planes == 1
    && desc->layers[1].format == DRM_FORMAT_GR88
    && desc->layers[1].nb_planes == 1;

With nvidia-vaapi-driver, the exported DRM PRIME descriptor is:

objects=2 layers=2
layer 0: format=DRM_FORMAT_R8   planes=1
layer 1: format=DRM_FORMAT_RG88 planes=1

As a result, scrcpy rejects every decoded frame with:

VA-API: unsupported DRM PRIME layout (layers=2, format=0x20203852)
Frame update failed

and the video is not displayed correctly.

FFmpeg's VA-API DRM format mapping supports both NV12 representations:

DRM_MAP(NV12, 2, DRM_FORMAT_R8, DRM_FORMAT_RG88),
DRM_MAP(NV12, 2, DRM_FORMAT_R8, DRM_FORMAT_GR88),

This patch therefore accepts both formats for the NV12 chroma plane.

Change

diff --git a/app/src/interop/interop_vaapi.c b/app/src/interop/interop_vaapi.c
--- a/app/src/interop/interop_vaapi.c
+++ b/app/src/interop/interop_vaapi.c
@@
 static bool
 sc_interop_vaapi_check_nv12_layout(const AVDRMFrameDescriptor *desc) {
     return desc->nb_layers == 2
         && desc->layers[0].format == DRM_FORMAT_R8
         && desc->layers[0].nb_planes == 1
-        && desc->layers[1].format == DRM_FORMAT_GR88
+        && (desc->layers[1].format == DRM_FORMAT_GR88
+            || desc->layers[1].format == DRM_FORMAT_RG88)
         && desc->layers[1].nb_planes == 1;
 }

Tested on

  • Linux Mint 22.3
  • NVIDIA GeForce GTX 1080
  • NVIDIA driver 580.178.04
  • nvidia-vaapi-driver 0.0.18
  • scrcpy 5.0
  • FFmpeg/libavcodec 6.1.1
  • H.264 video stream
  • VA-API hardware decoding

Before the patch:

INFO: Interop: vaapi
INFO: Video decoding: vaapi
INFO: Texture (VA-API): 1072x2400
ERROR: VA-API: unsupported DRM PRIME layout (layers=2, format=0x20203852)
ERROR: Frame update failed

The DRM PRIME descriptor reported by the NVIDIA VA-API driver was:

DRM PRIME: objects=2 layers=2
DRM PRIME layer 0: format=0x20203852 planes=1
  plane 0: object=0 offset=0 pitch=1088
DRM PRIME layer 1: format=0x38384752 planes=1
  plane 0: object=1 offset=0 pitch=1088

where:

0x20203852 = DRM_FORMAT_R8
0x38384752 = DRM_FORMAT_RG88

After accepting DRM_FORMAT_RG88, VA-API hardware decoding and VA-API/OpenGL interop work correctly:

INFO: Renderer: opengl
INFO: OpenGL version: 4.6.0 NVIDIA 580.178.04
INFO: Interop: vaapi
INFO: Video decoding: vaapi
INFO: Texture (VA-API): 1072x2400

The video is displayed correctly and the colors are correct.

Disclaimer: AI was used to debug scrcpy and find this fix

## Description

On NVIDIA GPUs using `nvidia-vaapi-driver`, VA-API hardware decoding may export NV12 DRM PRIME frames using `DRM_FORMAT_RG88` for the chroma plane.

Currently, scrcpy only accepts `DRM_FORMAT_GR88`:

```c
return desc->nb_layers == 2
    && desc->layers[0].format == DRM_FORMAT_R8
    && desc->layers[0].nb_planes == 1
    && desc->layers[1].format == DRM_FORMAT_GR88
    && desc->layers[1].nb_planes == 1;
```

With `nvidia-vaapi-driver`, the exported DRM PRIME descriptor is:

```text
objects=2 layers=2
layer 0: format=DRM_FORMAT_R8   planes=1
layer 1: format=DRM_FORMAT_RG88 planes=1
```

As a result, scrcpy rejects every decoded frame with:

```text
VA-API: unsupported DRM PRIME layout (layers=2, format=0x20203852)
Frame update failed
```

and the video is not displayed correctly.

FFmpeg's VA-API DRM format mapping supports both NV12 representations:

```c
DRM_MAP(NV12, 2, DRM_FORMAT_R8, DRM_FORMAT_RG88),
DRM_MAP(NV12, 2, DRM_FORMAT_R8, DRM_FORMAT_GR88),
```

This patch therefore accepts both formats for the NV12 chroma plane.

## Change

```diff
diff --git a/app/src/interop/interop_vaapi.c b/app/src/interop/interop_vaapi.c
--- a/app/src/interop/interop_vaapi.c
+++ b/app/src/interop/interop_vaapi.c
@@
 static bool
 sc_interop_vaapi_check_nv12_layout(const AVDRMFrameDescriptor *desc) {
     return desc->nb_layers == 2
         && desc->layers[0].format == DRM_FORMAT_R8
         && desc->layers[0].nb_planes == 1
-        && desc->layers[1].format == DRM_FORMAT_GR88
+        && (desc->layers[1].format == DRM_FORMAT_GR88
+            || desc->layers[1].format == DRM_FORMAT_RG88)
         && desc->layers[1].nb_planes == 1;
 }
```

## Tested on

- Linux Mint 22.3
- NVIDIA GeForce GTX 1080
- NVIDIA driver 580.178.04
- `nvidia-vaapi-driver` 0.0.18
- scrcpy 5.0
- FFmpeg/libavcodec 6.1.1
- H.264 video stream
- VA-API hardware decoding

Before the patch:

```text
INFO: Interop: vaapi
INFO: Video decoding: vaapi
INFO: Texture (VA-API): 1072x2400
ERROR: VA-API: unsupported DRM PRIME layout (layers=2, format=0x20203852)
ERROR: Frame update failed
```

The DRM PRIME descriptor reported by the NVIDIA VA-API driver was:

```text
DRM PRIME: objects=2 layers=2
DRM PRIME layer 0: format=0x20203852 planes=1
  plane 0: object=0 offset=0 pitch=1088
DRM PRIME layer 1: format=0x38384752 planes=1
  plane 0: object=1 offset=0 pitch=1088
```

where:

```text
0x20203852 = DRM_FORMAT_R8
0x38384752 = DRM_FORMAT_RG88
```

After accepting `DRM_FORMAT_RG88`, VA-API hardware decoding and VA-API/OpenGL interop work correctly:

```text
INFO: Renderer: opengl
INFO: OpenGL version: 4.6.0 NVIDIA 580.178.04
INFO: Interop: vaapi
INFO: Video decoding: vaapi
INFO: Texture (VA-API): 1072x2400
```

The video is displayed correctly and the colors are correct.
@rom1v

rom1v commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Thank you.

Can you test the binary generated by the CI, please, and confirm that the issue is fixed? https://github.com/rom1v/scrcpy/actions/runs/37375162897

EDIT: fixed in scrcpy 5.0.1

rom1v pushed a commit that referenced this pull request Oct 6, 2026
The chroma plane of exported NV12 surfaces was expected to be
`DRM_FORMAT_GR88`, as exported by the Mesa and Intel drivers. But
`nvidia-vaapi-driver` labels it `DRM_FORMAT_RG88`, so every frame was
rejected:

    VA-API: unsupported DRM PRIME layout (layers=2, format=0x20203852)

Accept both formats.

Fixes #7073 <#7073>
PR #7070 <#7070>

Signed-off-by: Romain Vimont <rom@rom1v.com>
@niawag

niawag commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Binary tested, it's working great, thanks !

@rom1v

rom1v commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Merged into dev: 2e8363a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants