The Streaming Protocol¶

The wire protocol between the Selkies server and the web client, for anyone implementing a client, embedding the engine, or debugging with a network inspector. Everything rides one WebSocket (proxied at <base>/websocket), carrying a mix of binary frames and terse text messages. There is no negotiation dance: connect, receive settings, start receiving media.

Connection and roles¶

The client connects to wss://host/<subfolder>websocket. Role assignment happens one of two ways:

• Fragment mode (default): the URL fragment the page was opened with decides the role, #shared (view only), #collab (full control), #player2 through #player4 (gamepad slot only), #display2-right and friends (second monitor surface). No fragment means primary.
• Token mode (when the server was started with SELKIES_MASTER_TOKEN): the client must present ?token=<token> in the WebSocket URL. Tokens and their roles are registered by the orchestrator via POST /tokens on the internal control port with the master token as a bearer credential. Close codes: 4001 invalid token, 4002 revoked, 4029 reconnecting too fast.

On success the server sends MODE websockets, an auth confirmation with the assigned role, and a server_settings JSON message containing every tunable setting with its value, allowed range or enum, and locked flag, this single message is what renders the sidebar UI, which is why locking a setting server side removes the control everywhere.

Binary messages, server to client¶

The first byte of every binary frame is a type tag:

These headers are produced by pixelflux itself (see the wire format details); the Python server broadcasts them untouched. A full frame is just a stripe at Y offset 0 with full height. The client feeds H.264 to a WebCodecs VideoDecoder per stream, decodes JPEG stripes with createImageBitmap, and composites stripes onto the canvas at their Y offsets. The frame type byte reflects what the encoder actually emitted, clients use it to recover decoder state after drops.

Binary messages, client to server¶

Text messages, client to server¶

Compact comma separated commands. The important families:

Every input bearing message is filtered by role: viewers get settings and video control only, players get their js slot, mk_control grants of mouse and keyboard can be toggled live in token mode.

Text and JSON messages, server to client¶

• cursor,{...}: PNG cursor images with hotspots, delivered out of band so the canvas cursor is pixel perfect without burning it into the video.
• clipboard,<b64> and the chunked variants: server clipboard changes.
• system_stats, gpu_stats, network_stats JSON blobs every few seconds for the Stats UI.
• stream_resolution and display_config_update for resize and multi monitor layout changes.
• VIDEO_STARTED, AUDIO_STOPPED, PIPELINE_RESETTING <display> lifecycle notices, and system,{"action":"reload"} when the client should reconnect fresh.

Backpressure¶

The server stamps every video frame with a wrapping uint16 id; the client periodically acknowledges the last id it presented. The server computes each client's desync, subtracts an RTT allowance (smoothed over recent samples), and if a client falls more than the allowed window behind, frames are withheld for that client before encode where possible, keeping H.264 reference chains valid. A client stalled beyond a timeout is disconnected. The intended behavior: one person on hotel wifi does not blur the session for four people on a LAN.

Settings flow¶

Client sends SETTINGS,{"framerate":60,"h264_crf":20,...}; the server validates each key against its schema (range clamp, enum membership, locked flag) and applies what survives, live where pixelflux supports it (quality, framerate, paint over) or with a pipeline restart where it does not (encoder, color mode). The authoritative state then flows back in server_settings so all connected clients converge.

Implementing a client: a minimal path¶

1. Open the WebSocket with binaryType = 'arraybuffer', send START_VIDEO and START_AUDIO after receiving server_settings.
2. Demux on byte zero. Feed 0x00 and 0x04 frames (strip the header, respect the keyframe flag) into a WebCodecs H.264 decoder configured from the stripe dimensions; paint 0x03 JPEGs at their offsets; queue 0x01 Opus into an audio decoder.
3. Send CLIENT_FRAME_ACK with the latest presented frame id a few times per second.
4. Map your input events to the kd, ku, m or m2 grammar.

That is a functioning viewer; everything else (clipboard, files, gamepads, stats) is additive. Start from the example client in the pixelflux repository: example/screen_to_browser.py plus example/index.html are a complete working server and client pair, with the frame parsing in about a page of code.