Overview: Spatial Computing Platform for AR Glasses

Portal OS is composed of a Wayland compositor stack and a suite of system services that render and stream a full desktop environment to AR glasses over a dedicated wireless link. All visual output flows to the glasses through hardware-encoded H.265 video streamed over RTP/UDP, while the tactile surface serves as both keyboard and trackpad. The platform adds 3DoF head tracking, on-device voice AI powered by the Hexagon DSP, an emulation layer for x86/Windows applications, and a spatial domain model that lets windows be organized across virtual zones projected into the wearer’s field of view.

What Portal OS Does

At its core, Portal OS solves a deceptively simple problem: run a complete Linux desktop and make it usable through AR glasses instead of a monitor. This requires rethinking every layer of the traditional stack, from how frames are rendered and captured, to how input flows when there is no physical keyboard or trackpad in the conventional sense, to how AI workloads share a single NPU between voice processing and language model inference.

Portal OS runs Wayfire (a Wayland compositor) with two logical outputs: the tactile surface used for the on-screen keyboard, and a virtual display output whose composited frames are captured via DMA-BUF, hardware-encoded to H.265, packetized as RTP, and broadcast over UDP to the glasses. The glasses run an Android/Unity receiver that decodes the stream, renders it to a virtual quad in 3D space, and sends 3DoF head orientation data back over UDP, closing the loop with homography-based spatial warp correction on the compositor side.

System Architecture at a Glance

The following diagram shows the major data flows across the platform’s subsystems:

flowchart TB
    subgraph the hardware["the hardware (X Elite)"]
        WF["Wayfire Compositor<br/>DRM backend · the tactile surface + the display output"]
        SP["Spatial Plugins<br/>Zone assignment · Dimension switching"]
        WP["Spatial Warp<br/>Homography head-tracking correction"]
        WM["Window Manager Daemon<br/>Toplevel tracking · Zone policy"]
        
        KB["Keyboard Daemon<br/>DRM/KMS scanout on the tactile surface<br/>Fusion engine · evdev touch"]
        
        STR["Streaming Pipeline<br/>DMA-BUF → IRIS H.265 → RTP/UDP"]
        
        VAD["Voice Pipeline<br/>VAD → STT → NLU → TTS"]
        NPU["Hexagon DSP<br/>QNN HTP v73 · 45 TOPS"]
        LLM["LLM Daemon<br/>Qwen3-0.6B on DSP"]
        
        NET["Network AP<br/>5 GHz the wireless AP · the DHCP server DHCP"]
        PCP["PCP Daemon<br/>App registry · Capability routing"]
        
        WF --> SP
        SP --> WP
        WM <-->|"Unix socket IPC"| SP
        KB -->|"uinput → virtual devices"| WF
        WF -->|"DMA-BUF capture"| STR
        VAD --> NPU
        LLM --> NPU
    end
    
    STR -->|"RTP/UDP H.265"| GLASSES
    NET -.->|"5 GHz Wi-Fi AP"| GLASSES
    
    subgraph GLASSES["AR glasses Glasses"]
        RX["Android/Unity Receiver<br/>H.265 decode · 3DoF tracking"]
    end
    
    GLASSES -->|"3DoF head orientation UDP"| WP
    GLASSES -->|"Mic audio UDP"| VAD

Major Subsystems

Portal OS is organized as a Rust workspace of 30+ crates plus a C streaming daemon and C++ Wayfire plugin shims. Each subsystem is independently buildable, testable, and deployable as a systemd service. The table below maps each subsystem to its role:

Subsystem	Crate(s)	Role	Service
Spatial Domain Model	`portal-spatial`	Zone types, dimension configs, assignment policies, IPC protocol, homography math	(library)
Spatial Plugin	`portal-spatial-plugin`	Wayfire cdylib: Wayland global, zone registry, dimension state	`portal.service`
Spatial Warp	`portal-warp`	Homography-based GLSL shaders for head-tracking correction	`portal.service`
Window Manager	`portal-wm`	Foreign toplevel tracking, zone assignment policy, IPC client	`portal-wm.service`
Keyboard/Input	`portal-input`	DRM/KMS direct scanout, fusion engine, gesture recognition, evdev	`portal-input.service`
Streaming (C)	`portal/streaming/`	DMA-BUF capture → IRIS H.265 encode → RTP/UDP broadcast	`portal-stream.service`
Streaming (Rust)	`portal-stream`	RTP packetization, HMAC-SHA256 authentication, protocol types	(library/daemon)
Voice	`portal-voice`	VAD (QNN DSP), STT (sherpa-onnx), NLU (MiniLM), TTS (Elara VITS)	`portal-voice.service`
LLM	`portal-llm`	Qwen3-0.6B inference via GenieX llama.cpp on Hexagon DSP	`portal-llm.service`
NPU Runtime	`portal-npu-runtime`	Safe Rust wrappers for Qualcomm Hexagon FastRPC	(library)
PCP	`portal-pcp/*` (15 crates)	Portal Capability Protocol: app registry, AT-SPI2, IPC, daemon	`portal-pcp.service`
Context Engine	`portal-context`	Event ingestion, decision making, template synthesis	(library)
Launcher	`portal-launcher`	Action registry, fuzzy search, plugin system	`portal-launcher.service`
Shell	`portal-shell`	Status bar, notifications, window switcher	(library)
Style	`portal-style`	Design token system: colors, typography, spacing, themes	(library)
Common	`portal-common`	Shared error types, IPC auth, test harness, mock hardware	(library)

The Spatial Domain Model

Portal’s defining innovation is the spatial domain model, which extends the flat 2D desktop metaphor into a structured spatial layout that maps naturally to how AR glasses render content. The model has three core abstractions:

Zones divide the 2560×1080 glasses canvas into logical regions: Center (primary focal area), LeftPeripheral and RightPeripheral (side areas for secondary windows), and Overlay (floating UI elements). The zone assignment engine uses view position to automatically route windows, for example, a code editor might land in Center while a chat window drifts to RightPeripheral.

Dimensions represent complete spatial configurations, Desktop, VR, AR, and Tablet modes, each defining which zones are active and how they are arranged. Users switch dimensions with keyboard shortcuts (Ctrl+Alt+Right / Ctrl+Alt+Left), instantly reconfiguring the workspace layout.

Homography-based warp correction applies a perspective transform to the compositor output based on real-time 3DoF head tracking data from the glasses. As the wearer turns their head, the spatial warp plugin recomputes a homography matrix and adjusts the GLSL shader uniforms so the rendered content appears stable and anchored in physical space.

The Streaming Pipeline

The streaming pipeline is the critical path that determines perceived system responsiveness. It operates as a zero-copy DMA-BUF pipeline: Wayfire composites the the display output to a GPU buffer, the streaming daemon imports that buffer via EGL (handling Qualcomm’s UBWC tiled layout by rendering through a scratch FBO to linearize it), feeds the linearized frames into GStreamer’s appsrc, which passes them to the IRIS V4L2 H.265 hardware encoder (v4l2h265enc), then through rtph265pay for RTP packetization, and finally out via udpsink to the glasses.

The end-to-end frame-to-glasses latency is measured at approximately 6 ms (5 ms IRIS encode + 1 ms Wi-Fi transit), well within the 20 ms budget. An optional HMAC-SHA256 stream authentication layer (RFC 5285 two-byte header extension) can sign each RTP packet to prevent tampering or injection on the wireless link.

On-Device AI: Voice and LLM

Portal OS runs two AI workloads entirely on-device, sharing the Hexagon v73 DSP (45 TOPS):

The voice pipeline flows from the glasses’ microphone (UDP audio at ~2 ms transit) through RNNoise denoising (~1 ms), QNN-based Voice Activity Detection on the DSP (~50 ms), sherpa-onnx Speech-to-Text on CPU (~200 ms), MiniLM-based Natural Language Understanding (~15 ms), and finally Elara VITS Text-to-Speech with encoder on DSP (~800 ms) and decoder via onnxruntime-qnn plugin (~1.6 s). The full voice round-trip completes in approximately 2.7 seconds, under the 5-second budget.

The LLM daemon runs Qwen3-0.6B (Q4_0 GGUF quantization) through GenieX llama.cpp on the Hexagon DSP, achieving 0.3-0.5 second response times with thinking disabled. An NPU scheduler arbitrates DSP access between VAD, TTS, and LLM workloads to prevent resource contention.

Hardware Platform

The the hardware (the Portal OS hardware) is a purpose-built spatial computing host built around the X Elite SoC:

Component	Specification	Portal OS Role
SoC	X Elite (X1E80100)	Primary compute
GPU	Adreno X1-85 (Mesa 26.1.2, Freedreno/Turnip)	wgpu rendering, Wayland EGL, GLSL shaders
NPU	Hexagon v73 (45 TOPS, QNN SDK v2.44)	VAD, TTS, LLM inference
Video Encoder	Qualcomm IRIS (`/dev/video1`)	H.265 hardware encode for glasses stream
Wi-Fi	the wireless interface (the wireless driver, 5 GHz AP)	Dedicated streaming link to glasses
Touch	capacitive touch sensor	Tactile surface input (keyboard + trackpad)
Display	the tactile surface, 2240×1400 @ 60 Hz	Tactile keyboard surface (owned by keyboard daemon)
Glasses Output	Virtual the display output, 2560×1080 @ 60 Hz	Composited desktop → streamed to AR glasses
OS	Debian testing (trixie), kernel 7.0.13-iris	Custom kernel with IRIS encoder support

The AR glasses serve as the display endpoint: they receive the H.265 RTP stream on port 5600, decode via Android MediaCodec hardware decoder, render to a virtual quad in Unity with 3DoF head tracking, and feed orientation data back to the compositor for warp correction.

Service Topology

Portal OS runs as a constellation of 13 systemd services (plus 2 watchdogs), all communicating through Unix sockets and Wayland protocols under the shared runtime directory /run/portal:

flowchart LR
    subgraph Core["Core Services"]
        PORTAL["portal.service<br/>(Wayfire)"]
        INPUT["portal-input.service<br/>(Keyboard + Fusion)"]
        STREAM["portal-stream.service<br/>(H.265 → RTP)"]
    end
    
    subgraph Management["Window & App Management"]
        WM["portal-wm.service"]
        PCP["portal-pcp.service"]
        LAUNCHER["portal-launcher.service"]
    end
    
    subgraph AI["AI Services"]
        VOICE["portal-voice.service"]
        LLM["portal-llm.service"]
    end
    
    subgraph Infra["Infrastructure"]
        NET["portal-network.service"]
        DBUS["portal-dbus.service"]
        HEALTH["portal-health-monitor"]
        WLO1["wlo1-watchdog"]
    end
    
    PORTAL -->|"PartOf"| STREAM
    PORTAL -->|"BindsTo"| INPUT
    WM -->|"Wayland"| PORTAL
    WM -->|"Unix socket"| PCP
    LAUNCHER --> PCP
    VOICE --> LLM

Repository Structure

The repository is a single Cargo workspace rooted at the project top level, with all Portal OS crates under portal/, Unity receiver projects under unity/, cross-component documentation under docs/, and deployment automation under scripts/:

portal/
├── spatial/              # Spatial domain model (zones, dimensions, IPC, homography)
├── spatial-plugin/       # Wayfire spatial plugin (Rust cdylib → C++ shim)
├── warp/                 # Spatial warp rendering (homography GLSL shaders)
├── wm/                   # Window manager daemon (toplevel tracking, zone policy)
├── input/                # Keyboard/input (DRM/KMS, fusion engine, gestures)
├── stream/               # RTP protocol types (Rust library)
├── streaming/            # Main streaming pipeline (C/GStreamer/IRIS)
├── voice/                # Voice pipeline (VAD, STT, NLU, TTS, DSP arbitration)
├── llm/                  # On-device LLM daemon (GenieX llama.cpp)
├── npu-runtime/          # Hexagon FastRPC safe Rust wrappers
├── pcp/                  # Portal Capability Protocol (15 sub-crates)
├── context/              # Context engine (event ingestion, template synthesis)
├── launcher/             # Universal launcher (fuzzy search, action registry)
├── shell/                # Shell layer (status bar, notifications)
├── style/                # Design token system
├── common/               # Shared utilities (errors, IPC auth, test harness)
├── compositor/           # Wayfire configs (wayfire.ini, spatial-warp.toml)
├── protocols/            # Wayland protocol XML definitions
├── wayfire-plugins/      # C++ shim plugins for Wayfire (CMake)
├── streaming/            # C streaming daemon source
├── systemd/              # Service unit files
├── network/              # the wireless AP, the DHCP server configs
└── android/              # Android receiver apps

unity/
├── SpatialPortal_3DoF/   # Unity 3DoF head tracking receiver

Emulation Layer

Portal OS includes transparent x86/Windows application support via Box64 (v0.4.3, x86/x64 dynamic recompiler on ARM64) and Wine 10.0. The portal-binfmt-fix service ensures Box64, not QEMU, handles x86 ELF binaries through binfmt_misc, which is critical for Wine’s internal architecture. File watchers (portal-file-watcherd) and a Wine monitor (portal-wine-monitor) automatically detect and register new .exe binaries. The PCP AT-SPI2 integration provides accessibility capability derivation for both native and emulated applications.

Where to Go Next

This overview establishes the mental model for the entire Portal OS platform. The documentation catalog is organized as a progressive deep dive, each section builds on the concepts introduced here:

Getting Started:

Repository Structure and Workspace Layout, Detailed crate-by-crate breakdown

Deep Dives, Explore any subsystem that sparks your curiosity, from the Dual-Display Wayland Architecture through the streaming pipeline’s DMA-BUF to H.265 encode path, the voice AI pipeline, and the PCP application framework.

Next
Repository Structure and Workspace Layout

Last updated: Aug 10, 2026