VR Overlay Protocol (VROP) Specification

Version: 0.1.0-draft
Status: Draft
Authors: catnet Team
Date: 2025-07-17

Abstract

The VR Overlay Protocol (VROP) defines a standardized communication protocol between VR overlay hosts and plugins, enabling cross-platform development of VR overlay applications. VROP provides a message-based interface for overlay lifecycle management, rendering, input handling, and system integration while maintaining security through capability-based permissions.

1. Introduction

1.1 Motivation

Current VR overlay development requires deep integration with platform-specific APIs (OpenVR, OpenXR), limiting portability and increasing development complexity. VROP abstracts these platform details behind a simple, message-based protocol that can be implemented over various transport mechanisms.

1.2 Goals

Platform Independence: Plugins work across OpenVR, OpenXR, and future VR platforms
Language Agnostic: Support for any language that can serialize/deserialize messages
Security: Sandboxed execution with fine-grained permissions
Performance: Minimal overhead for real-time VR applications
Extensibility: Forward-compatible protocol design

1.3 Non-Goals

Direct GPU access (use platform-specific extensions)
Low-level VR runtime control
Real-time audio processing (separate protocol)

2. Protocol Overview

2.1 Architecture

direction: down

# Components
Plugin: "Plugin\n\(WASM/Native\)" {
  style: {
    fill: "#F18F01"
    stroke: "#F18F01"
    font-color: "#ffffff"
    stroke-width: 2
  }
}

Client: "VROPClient\n\(Plugin Host\)" {
  style: {
    fill: "#2E86AB"
    stroke: "#2E86AB"
    font-color: "#ffffff"
    stroke-width: 2
  }
}

Dashboard: "Dashboard Plugin" {
  style: {
    fill: "#A23B72"
    stroke: "#A23B72"
    font-color: "#ffffff"
    stroke-width: 2
  }
}

Host: "VROS\n\(System Host\)" {
  style: {
    fill: "#2E86AB"
    stroke: "#2E86AB"
    font-color: "#ffffff"
    stroke-width: 2
  }
}

Runtime: "VR Runtime\n\(OpenVR/XR\)" {
  style: {
    fill: "#2D6E3B"
    stroke: "#2D6E3B"
    font-color: "#ffffff"
    stroke-width: 2
  }
}

# Connections
Plugin <-> Client: "VROP Messages\n\(Serialized\)" {
  style: {
    stroke: "#666666"
    stroke-width: 2
  }
}

Client -> Runtime {
  style: {
    stroke: "#666666"
    stroke-width: 2
  }
}

Dashboard <-> Host: "VROP Messages" {
  style: {
    stroke: "#666666"
    stroke-width: 2
  }
}

Host -> Runtime {
  style: {
    stroke: "#666666"
    stroke-width: 2
  }
}

2.2 Transport Mechanisms

VROP supports multiple transport layers:

In-Process (Native Plugins): Direct function calls
IPC (Separate Process): Shared memory + lock-free queues
WASM (Sandboxed): WASI-based message passing
Network (Remote): WebSocket/gRPC (future)

2.3 Message Format

VROP defines a message structure that is independent of the serialization format:

struct VropMessage {
    /// Message type identifier
    msg_type: u32,
    
    /// Request ID for request/response correlation
    request_id: Option<u64>,
    
    /// Timestamp (nanoseconds since epoch)
    timestamp: u64,
    
    /// Message payload (serialized data)
    payload: Vec<u8>,
}

2.4 Serialization Formats

VROP is serialization-agnostic. Implementations may choose formats based on their requirements:

Format	Use Case	Pros	Cons
Bincode	High-performance native	Extremely fast, compact	Rust-specific
MessagePack	Cross-language	Good performance, wide support	Larger than Bincode
JSON	Debugging, web	Human-readable, universal	Slower, larger
Protobuf	Enterprise, gRPC	Schema validation, evolution	Complexity
rkyv	Zero-copy	Fastest deserialization	Rust-only

The transport layer negotiates the serialization format during initialization.

3. Message Types

3.1 Lifecycle Messages

Initialize Request (0x0001)

struct InitializeRequest {
    /// Protocol version
    protocol_version: String,
    
    /// Plugin capabilities request
    requested_capabilities: Vec<String>,
    
    /// Plugin metadata
    plugin_info: PluginInfo,
}

struct PluginInfo {
    name: String,
    version: String,
    author: String,
    description: String,
}

Initialize Response (0x0002)

struct InitializeResponse {
    /// Granted capabilities
    granted_capabilities: Vec<String>,
    
    /// Host information
    host_info: HostInfo,
    
    /// Session ID
    session_id: String,
}

struct HostInfo {
    name: String,
    version: String,
    vr_runtime: String, // "openvr", "openxr"
    platform: String,   // "windows", "linux", "macos"
}

Shutdown Request (0x0003)

struct ShutdownRequest {
    reason: String,
    save_state: bool,
}

3.2 Overlay Management

Create Overlay Request (0x0101)

struct CreateOverlayRequest {
    /// Unique overlay identifier
    overlay_id: String,
    
    /// Overlay type
    overlay_type: OverlayType,
    
    /// Initial properties
    properties: OverlayProperties,
}

enum OverlayType {
    World3D,
    Dashboard,
    System,
    Notification,
}

struct OverlayProperties {
    name: String,
    width: u32,
    height: u32,
    position: Transform3D,
    visible: bool,
    alpha: f32,
    sort_order: i32,
}

Update Overlay Request (0x0102)

struct UpdateOverlayRequest {
    overlay_id: String,
    updates: OverlayPropertyUpdates,
}

struct OverlayPropertyUpdates {
    position: Option<Transform3D>,
    visible: Option<bool>,
    alpha: Option<f32>,
    sort_order: Option<i32>,
}

3.3 Rendering

Submit Frame Request (0x0201)

struct SubmitFrameRequest {
    overlay_id: String,
    frame_data: FrameData,
}

enum FrameData {
    /// Shared memory buffer
    SharedMemory {
        buffer_id: String,
        offset: u64,
        size: u64,
        format: PixelFormat,
    },
    
    /// Direct texture handle (native plugins only)
    TextureHandle {
        handle: u64,
        format: PixelFormat,
    },
    
    /// Compressed image data
    CompressedImage {
        data: Vec<u8>,
        format: ImageFormat,
    },
}

enum PixelFormat {
    RGBA8,
    BGRA8,
    RGB10A2,
    // ...
}

Frame Timing Notification (0x0202)

struct FrameTimingNotification {
    frame_id: u64,
    predicted_display_time: u64,
    frame_interval: u64,
}

3.4 Input Handling

Input Event (0x0301)

struct InputEvent {
    overlay_id: String,
    event_type: InputEventType,
    timestamp: u64,
    device_id: String,
}

enum InputEventType {
    ControllerHover {
        position: Vec2,
        distance: f32,
    },
    ControllerClick {
        position: Vec2,
        button: u32,
    },
    KeyPress {
        key_code: u32,
        modifiers: u32,
    },
    // ...
}

3.5 System Integration

Notification Request (0x0401)

struct NotificationRequest {
    title: String,
    message: String,
    icon: Option<Vec<u8>>,
    duration_ms: u32,
    priority: NotificationPriority,
}

Haptic Feedback Request (0x0402)

struct HapticFeedbackRequest {
    device_id: String,
    duration_ms: u32,
    frequency: f32,
    amplitude: f32,
}

4. Capability System

4.1 Core Capabilities

overlay.create: Create and manage overlays
overlay.world3d: Create world-space overlays
overlay.dashboard: Create dashboard overlays
input.receive: Receive input events
input.haptic: Trigger haptic feedback
system.notification: Show system notifications
ipc.shared_memory: Use shared memory buffers

4.2 Extended Capabilities

gpu.direct_texture: Submit GPU texture handles
tracking.read: Access tracking data
tracking.spaces: Create tracking spaces
audio.spatial: Spatial audio playback
network.external: External network access

5. Security Model

5.1 Sandboxing

WASM plugins run in a sandboxed environment with:

No direct file system access
No direct network access (without capability)
Memory isolation
CPU time limits
Resource quotas

5.2 Permission Model

struct PluginManifest {
    // ... plugin info ...
    
    permissions: PermissionSet,
}

struct PermissionSet {
    required: Vec<String>,
    optional: Vec<String>,
    
    /// Resource limits
    max_memory_mb: u32,
    max_cpu_percent: u8,
    max_overlays: u8,
}

6. Error Handling

6.1 Error Response (0xFFFF)

struct ErrorResponse {
    request_id: u64,
    error_code: u32,
    error_message: String,
    details: Option<HashMap<String, Value>>,
}

6.2 Error Codes

0x1000: Protocol error
0x2000: Permission denied
0x3000: Resource limit exceeded
0x4000: Invalid parameter
0x5000: Overlay not found
0x6000: VR runtime error

7. Performance Considerations

7.1 Message Batching

Multiple messages can be batched:

struct BatchedMessage {
    messages: Vec<VropMessage>,
}

7.2 Shared Memory

For high-frequency data (frames, tracking), use shared memory:

struct SharedMemorySetup {
    buffer_id: String,
    size: u64,
    permissions: MemoryPermissions,
}

8. Examples

8.1 Basic Overlay Plugin (Pseudo-code)

// Initialize connection
send(InitializeRequest {
    protocol_version: "1.0.0",
    requested_capabilities: vec!["overlay.create", "input.receive"],
    plugin_info: PluginInfo { ... },
});

let init_response = receive::<InitializeResponse>();

// Create overlay
send(CreateOverlayRequest {
    overlay_id: "my_overlay",
    overlay_type: OverlayType::World3D,
    properties: OverlayProperties {
        width: 1920,
        height: 1080,
        ...
    },
});

// Main loop
loop {
    // Render frame
    let frame_data = render_frame();
    send(SubmitFrameRequest {
        overlay_id: "my_overlay",
        frame_data: FrameData::SharedMemory { ... },
    });
    
    // Handle input
    if let Some(input) = try_receive::<InputEvent>() {
        handle_input(input);
    }
}

// Request dashboard capability
send(InitializeRequest {
    requested_capabilities: vec!["overlay.dashboard"],
    ...
});

// Create dashboard overlay
send(CreateOverlayRequest {
    overlay_type: OverlayType::Dashboard,
    properties: OverlayProperties {
        name: "My Widget",
        width: 400,
        height: 300,
        ...
    },
});

9. Protocol Extensions

9.1 Vendor Extensions

Vendors can add custom messages using reserved ranges:

0x8000-0x8FFF: OpenVR extensions
0x9000-0x9FFF: OpenXR extensions
0xA000-0xFFFF: Reserved for future use

9.2 Version Negotiation

struct VersionNegotiation {
    supported_versions: Vec<String>,
    preferred_version: String,
    required_features: Vec<String>,
}

10. Implementation Notes

10.1 Host Implementation (catnet)

VROP defines two types of hosts:

VROPClient

The generic plugin host for overlay plugins:

Loads and executes overlay plugins
Manages overlay rendering and input
Enforces plugin sandboxing
Handles resource limits
Communicates with VROS

VROS

The system-level host for the dashboard:

Hosts the dashboard plugin
Provides system services (process management, configuration)
Orchestrates VROPClient instances
Manages inter-process communication
Enforces security policies

Both hosts must:

Validate all capability requests
Enforce resource limits
Handle plugin crashes gracefully
Provide backwards compatibility
Implement efficient message routing

10.2 Plugin SDK

Provide SDKs for:

Rust (native bindings)
C/C++ (via bindgen)
JavaScript/TypeScript (WASM)
Python (WASM)
C# (WASM)

10.3 Testing

Protocol conformance test suite
Performance benchmarks
Security fuzzing
Integration tests with real VR runtimes

11. Future Considerations

11.1 Planned Features

Mesh submission for 3D overlays
Advanced input (hand tracking, eye tracking)
Multi-user synchronization
Cloud rendering support

11.2 Protocol Evolution

Minor version bumps for additions
Major version bumps for breaking changes
Deprecation warnings for removed features
Feature flags for experimental features

Appendix A: Message Type Registry

Range	Category	Description
0x0000-0x00FF	Lifecycle	Initialization, shutdown
0x0100-0x01FF	Overlay	Creation, updates, deletion
0x0200-0x02FF	Rendering	Frame submission, timing
0x0300-0x03FF	Input	Events, haptics
0x0400-0x04FF	System	Notifications, settings
0x0500-0x05FF	Tracking	Pose data, spaces
0x0600-0x06FF	Audio	Spatial audio
0x0700-0x07FF	Network	External communication
0x0800-0x7FFF	Reserved	Future use
0x8000-0xFFFE	Extensions	Vendor specific
0xFFFF	Error	Error responses

Appendix B: Capability Registry

Capability	Description	Risk Level
overlay.create	Create basic overlays	Low
overlay.world3d	World-space overlays	Medium
overlay.dashboard	Dashboard integration	Low
input.receive	Receive input events	Low
input.haptic	Haptic feedback	Low
system.notification	System notifications	Medium
gpu.direct_texture	GPU texture access	High
tracking.read	Read tracking data	Low
network.external	Internet access	High

References

OpenVR IVROverlay Interface
OpenXR Overlay Extension Proposals
WebAssembly System Interface (WASI)
Bincode Specification (https://github.com/bincode-org/bincode)
MessagePack Specification (https://msgpack.org/)
JSON-RPC 2.0 Specification
catnet Plugin Architecture (internal)

Abstract​

1. Introduction​

1.1 Motivation​

1.2 Goals​

1.3 Non-Goals​

2. Protocol Overview​

2.1 Architecture​

2.2 Transport Mechanisms​

2.3 Message Format​

2.4 Serialization Formats​

3. Message Types​

3.1 Lifecycle Messages​

Initialize Request (0x0001)​

Initialize Response (0x0002)​

Shutdown Request (0x0003)​

3.2 Overlay Management​

Create Overlay Request (0x0101)​

Update Overlay Request (0x0102)​

3.3 Rendering​

Submit Frame Request (0x0201)​

Frame Timing Notification (0x0202)​

3.4 Input Handling​

Input Event (0x0301)​

3.5 System Integration​

Notification Request (0x0401)​

Haptic Feedback Request (0x0402)​

4. Capability System​

4.1 Core Capabilities​

4.2 Extended Capabilities​

5. Security Model​

5.1 Sandboxing​

5.2 Permission Model​

6. Error Handling​

6.1 Error Response (0xFFFF)​

6.2 Error Codes​

7. Performance Considerations​

7.1 Message Batching​

7.2 Shared Memory​

8. Examples​

8.1 Basic Overlay Plugin (Pseudo-code)​

8.2 Dashboard Widget​

9. Protocol Extensions​

9.1 Vendor Extensions​

9.2 Version Negotiation​

10. Implementation Notes​

10.1 Host Implementation (catnet)​

VROPClient​

VROS​

10.2 Plugin SDK​

10.3 Testing​

11. Future Considerations​

11.1 Planned Features​

11.2 Protocol Evolution​

Appendix A: Message Type Registry​

Appendix B: Capability Registry​

References​