Warper is a WebRTC based ultra-low latency video streaming prototype.
Warper uses the following architecture:
flowchart TD
videoSource["warp-drive<br>(streamer, C++)"]
gateway["wormhole<br>(web server, C#)"]
videoSink["warper<br>(web front-end, TypeScript)"]
videoSink--"input stream <br/><i>WebRTC</i>"-->videoSource--"video stream<br><i>WebRTC</i>"-->videoSink
gateway<-."static content / signaling<br/><i>HTTP / WebSocket</i>".->videoSink
videoSource<-."signaling<br><i>WebSocket<i>".->gateway
ice["STUN / TURN server(s)"]
ice<-..->videoSource
ice<-.->videoSink
Warp drive is the video stream source, written in C++ (currently only targets Windows, but it could be updated for multi-platform operation). It supports multiple frame sources, such as screen / window capture. The provided frames are converted to a supported format then they are encoded on the GPU. The encoded video frames are sent to the web front-end using a WebRTC data channel.
The following libraries are used:
- Frame capture is done by the Graphics Capture API
- Image preprocessing (e.g. color space conversion, resolution scaling) is done using Direct3D
- Video encoding is done by the NVEnc
- Video streaming is done using libdatachannel
Wormhole is an ASP.Net Core web-server, it connects streaming sources with streaming sinks. It also provides WebRTC signaling, which allows the stream source and sink to establish a peer-to peer-connection.
Warper is a web front-end, which displays the decoded video, provides user inputs to the stream source. It uses the following technologies:
- Signaling uses WebSockets.
- Video is received using WebRTC
- Video decoding uses WebCodecs
- The UI is implemented using Angular and TypeScript.
To establish a peer-to-peer connection between endpoints which might be behind NAT, STUN / TURN servers are used to connect the endpoints, a process which is called Interactive Connectivity Establishment. See this article for a detailed explanation.