Wayland Screen Capture for a Remote-Desktop Agent: XDG Portals, PipeWire, and the x11vnc Fallback

This post documents how the ET Ducky Linux agent captures the screen on Wayland desktops: the xdg-desktop-portal ScreenCast/RemoteDesktop flow over D-Bus, the PipeWire-to-GStreamer pipeline that encodes the stream, how the helper process gets into the user's session from a system service, why the portal deliberately refuses to persist input permission, and where the old x11vnc path still earns its keep as a fallback. Everything here is the shipped implementation — a single self-contained Python helper plus the agent code that spawns it.

Why x11vnc stopped being enough

The first-generation Linux capture path shelled out to x11vnc against the logged-in user's X display. It works against real Xorg — and fails on XWayland's default visuals with BadMatch on XGetImage, which matters because Ubuntu 22.04+, Fedora, and RHEL 9+ desktops default to GNOME on Wayland. Rather than accumulate per-compositor capture code, the replacement targets the freedesktop portal API: GNOME, KDE, and wlroots compositors all ship an xdg-desktop-portal backend, the portal owns the permission UX, and Wayland and X11 sessions produce identical output through one code path.

Getting into the session

The structural problem for any agent is that xdg-desktop-portal is a per-user D-Bus service, and PipeWire screen capture requires permission from the active session's compositor. A system service cannot call it. The agent (running as an unprivileged etducky system user) therefore spawns the helper into the logged-in user's session with sudo -n -u <user>, backed by a narrow sudoers entry, and reads frames back over a Unix socket.

Two details cost real debugging time. First, the socket is an abstract socket (leading NUL byte, no filesystem presence) specifically because systemd sandboxing directives on the agent unit — PrivateTmp, BindPaths — apply to filesystem paths; an abstract name sidesteps an entire class of "works in dev, times out under systemd" failures. Second, sudo is invoked without -i: a login shell resets the environment and can leave the helper without a usable D-Bus session address, whereas direct exec plus the helper's own fallback to /run/user/<uid>/bus (the systemd-conventional path libdbus also probes) connects reliably.

The agent's selection logic is try-first, not detect-first: if a user is logged in and the helper binary exists, spawn it and give it fifteen seconds to bind the socket; on failure, fall back to x11vnc. Detecting "is the portal available" up front would duplicate the helper's own startup logic — the helper is the probe, and its exit codes (1 portal denied, 2 D-Bus failed, 3 PipeWire failed, 4 GStreamer failed, 5 socket bind failed) make each failure mode legible in the journal.

The portal flow, step by step

The helper drives two portal interfaces: org.freedesktop.portal.ScreenCast and org.freedesktop.portal.RemoteDesktop. The ordering rules are strict and mostly undocumented outside the spec:

1. CreateSession — on the RemoteDesktop interface if you ever want input. Input cannot be added to an existing ScreenCast-only session; combining capture and input means creating the session on RemoteDesktop and layering ScreenCast onto it. On compositors without a RemoteDesktop backend, CreateSession returns NotSupported and the helper falls back to a ScreenCast-only session — capture works, input calls become silent no-ops, and the dashboard shows a view-only session.
2. SelectDevices (RemoteDesktop only) — request keyboard and pointer, and it must happen between CreateSession and SelectSources or input is not granted at Start.
3. SelectSources — monitor capture with the cursor embedded in the stream. This is where persistence lives, covered below.
4. Start — returns the PipeWire stream node IDs and the granted input-device bitmask. A RemoteDesktop session that comes back with zero granted devices means the operator declined input in the dialog; the helper degrades itself to view-only rather than failing.
5. OpenPipeWireRemote — returns a file descriptor for the PipeWire connection over D-Bus fd-passing. Capturing by portal-issued fd, rather than connecting to the daemon by name, makes the portal the authority on which PipeWire instance to use.

One mechanical footnote: portal request/response correlation works by constructing an expected request-object path from your bus name and a caller-chosen token, subscribing to its Response signal, then issuing the call with a 30-second timeout — a user staring at a permission dialog takes as long as they take. Some portal versions return a different request handle than the constructed one; the helper re-subscribes to the actual handle and re-awaits, which is the kind of version-drift accommodation this API quietly requires.

Persistence, and why unattended input is impossible by design

The portal's persistence knob (persist_mode) lets a user's sharing choice be remembered across sessions. The shipped configuration is asymmetric on purpose:

• ScreenCast-only sessions request permanent persistence (persist_mode=2) — view-only capture can be remembered.
• RemoteDesktop sessions request none (persist_mode=0) — because the portal rejects persistence on SelectSources when input devices were requested, with "Remote desktop sessions cannot persist." Unattended once-and-forever input access is considered too dangerous to grant; per-session prompting is the design, not a limitation of this implementation.

This is the honest answer to "can a Wayland agent do fully unattended remote control the way X11 tools can": through the portal, no — a human in the session approves input each time. When that guarantee is unacceptable (headless boxes, lights-out maintenance), the answer is not to fight the portal but to use a different layer: the x11vnc path on Xorg systems, or out-of-band management below the OS entirely.

The encoding pipeline

Frames flow through a GStreamer pipeline the helper builds from the portal-issued fd:

pipewiresrc fd=<fd> path=<node> do-timestamp=true
  ! videoconvert
  ! videorate ! video/x-raw,framerate=<max-fps>/1
  <encoder chain>
  ! appsink emit-signals=true sync=false max-buffers=2 drop=true

A negotiation lesson is embedded in that ordering: putting a framerate caps filter directly after the source breaks negotiation, because the compositor decides the actual capture rate. videorate downstream is the correct rate-capping mechanism. The appsink's max-buffers=2 drop=true keeps the pipeline real-time — a slow consumer drops frames instead of building latency.

Encoder selection probes GStreamer element factories in priority order: vah264enc (VA-API hardware — roughly 5% CPU and under 500 KB/s for 1080p), then x264enc (software, ~30% CPU, ubiquitous), then jpegenc as the legacy fallback (~20% CPU and 5+ MB/s — always available, an order of magnitude more bandwidth). Both H.264 encoders are constrained to baseline profile (avc1.42E01E) because VA-API's default is main and browser WebCodecs decoders — Safari and older Firefox in particular — are stricter than Chrome. Latency tuning is zerolatency/ultrafast on x264 and low-power plus CBR on VA-API, targeting under one frame of encoder buffering. The H.264 path emits a one-shot SPS+PPS config frame before the first NALU; decoding on the browser side is covered in the WebCodecs post.

Input injection

When the RemoteDesktop grant succeeded, input goes back through the same portal session: NotifyPointerMotionAbsolute, NotifyPointerButton, NotifyPointerAxis, and NotifyKeyboardKeysym. Absolute pointer coordinates are stream-relative, which is why the pipeline reports the negotiated stream dimensions back to the portal layer — without the real width and height, absolute motion lands in the wrong place. Each injection call is fire-and-log; a failed injection should degrade the session, not kill it.

Deployment shape

The helper ships as one dependency-light Python file under /opt/etducky/agent/bin/, using only PyGObject and GStreamer packages present in Ubuntu and Fedora default repositories — no virtualenv, no pip, no compiled extensions, because an RMM agent's install story has to be "install the package and it works." The frame contract on the Unix socket is length-prefixed binary (JPEG frames, H.264 NALUs with a keyframe flag, and the H.264 config record), so the agent side stays codec-agnostic and the same relay handles every encoder the helper might have chosen.