Low-latency video over wireless
One thing I’ve been doing whilst waiting for my new quadcopter bits to come is playing around with the Raspberry Pi and its camera.
It is a wonderful beast. Photo above is a frame from a video I recorded on my laptop. The laptop was receiving the video stream from the Pi in my hand over an ad-hoc wireless link. It’s a cool red tint because of the lack of Infra-Red filter.
The Pi NoIR camera I have attached has some impressive specifications. It is now able to stream H.264 video at up to 45 frames per second at something crazy like 2592 x 1944 resolution. At 800x600 (VGA) or lower you can get 120 frames per second! And it can do this whilst barely touching the Raspberry Pi’s CPU, keeping it free for other stuff.
There’s lots of interest regarding streaming video from Pi’s. My use-case is slightly different to others; I’m not so much concerned about quality. I just want to get video frames from the Pi as fast as possible. I’m interested in low latency.
Here’s my tips for getting low latency video out of the Pi + Pi camera:
Use raspivid — getting video from other programs like gstreamer/ffmpeg/mplayer/the V4L2 wrapper for the camera, all add extra layers of extraction. raspivid can directly give you an unmodified H.264 video stream.
Use udp — if you’re gonna stream your video over the network, don’t bother trying to wrap it in any fancy protocols like RTSP. you don’t actually need the timing headers or anything. You definitely don’t need all the fancy stuff in TCP.
Keep a high frame-rate — the higher the frame rate, the less time between frames. It helps the human operators see motion more clearly [citation needed]. It also pushes more data over the network, the same as the following…
Don’t be scared of high bit-rates or high-resolutions — if you have a dedicated wireless link between your computer(s) and your Pi, you can probably get tens of megabytes per second. That’s plenty for streaming H.264 video. Higher bitrates and resolutions and framerates mean better quality video, but they also mean the video frames are produced more slowly and consumed more quickly (when played back). To keep latency down, you want to always be in a slight deficit of video frames. One trick I use is to set the framerate of the receiving player one FPS faster than the Pi camera’s framerate.
It’s pretty easy* to get H.264 video from a Pi. Install the program socat. Now you can stream video from the camera:
raspivid -t 0 -fps 30 -w 1280 -h 1024 -b 100000000 -o - |\ socat stdin udp-sendto::9001
Then on your receiving computer, get yourself socat and mplayer and be running linux:
socat udp-recv:9001 stdout | mplayer -framedrop -nocache -fps 31 -demuxer h264es -
Start the receiving video player first, then start streaming the video on your Pi secondly.
If you have another raspberry pi to receive video on, you could even do this:
socat udp-recv:9001 stdout | /opt/vc/bin/hello_video -
If you wanted to REALLY drop latency; you could go further and write your own C program to read data from the camera and send directly over UDP with as little buffering as possible, and display it on another computer with as little buffering as possible too. This means more stuttery error prone video, but you can get video with under a hundred milliseconds lag. Which I would argue is fast enough to remotely control a quadcopter.
BOOM! I have my quadcopter’s cheap hacky FPV sorted. :)
*I say this is easy, obviously it took me hours of trial and error to get this method working.