Add RTMP stream function with FFMPEG

[edwardxliu]

What is the purpose of the change

Provide darknet the ability to do RTMP video streaming via FFMPEG. Now user can stream the detector result video stream to a RTMP stream url like showed below.

./darknet detector stream cfg/coco.data cfg/yolov3-tiny.cfg yolov3-tiny.weights -stream_bitrate 500000 -stream_gop 12 -stream_width 1920 -stream_height 1080 -stream_fps 20 -stream_address rtmp://10.12.18.130:1935/rtmplive/rtmp rtsp://admin:[email protected]:554/cam/realmonitor

./darknet detector stream cfg/coco.data cfg/yolov3-tiny.cfg yolov3-tiny.weights -stream_bitrate 500000 -stream_gop 12 -stream_width 1920 -stream_height 1080 -stream_fps 20 -stream_address rtmp://10.12.18.130:1935/rtmplive/rtmp test.mp4

Brief change log

Create stream.cpp, stream.h, streamer.cpp, streamer.hpp 4 files where "streamer.cpp" handles the actual streaming function and streaming parameters tuning and "stream.cpp" is basically a copy of the demo.c with some changes to stream frames instead of being displayed in a window.

Small changes to detector.c for the interface of user input and streaming function. Other files except Makefile are not touched.

Tested on Ubuntu 16.04/18.04 and MacOS Catalina with FFMPEG (ver 3.4.8 and 4.1.3). Windows version is under development currently.

[AlexeyAB]

@edwardxliu Hi,
Thanks for PR!
Why is it better than: https://github.com/AlexeyAB/darknet#how-to-use-on-the-command-line

JSON and MJPEG server that allows multiple connections from your soft or Web-browser ip-address:8070 and 8090:

./darknet detector demo ./cfg/coco.data ./cfg/yolov3.cfg ./yolov3.weights test50.mp4 -json_port 8070 -mjpeg_port 8090 -ext_output

[edwardxliu]

@edwardxliu Hi,
Thanks for PR!
Why does it better than: https://github.com/AlexeyAB/darknet#how-to-use-on-the-command-line

JSON and MJPEG server that allows multiple connections from your soft or Web-browser ip-address:8070 and 8090:

./darknet detector demo ./cfg/coco.data ./cfg/yolov3.cfg ./yolov3.weights test50.mp4 -json_port 8070 -mjpeg_port 8090 -ext_output

@AlexeyAB Hi,
Ty for your reply.

The thing is that we are trying to build a live video streaming website with a large number of viewers at frontend. Like many other live streaming services, our streaming server receives live stream uploads in RTMP from a lot of web cameras and then scale the live stream content to countless viewers. Since the original streaming function of darknet dose not support many users accessing the video stream at the same time with good quality such as low latency, we add the RTMP streaming function so that it can send RTMP stream to our RTMP server.

So in terms of latency and concurrency, I think RTMP performs better than the original streaming protocol. But I'm not an expert in live streaming area, so maybe there are some other good solutions.

[edwardxliu]

@AlexeyAB Hi,

Recently I made some tests related to IP cameras and found that there is a certain minimum level of delay (which is around 1 sec) as below screenshot shows when using "darknet detector demo". The reason of the latency is that the input video stream is extracted by OPENCV and it's really difficult to reduce the input latency related to IP cameras.

So what I did was just replacing the input video stream extraction function with FFMPEG and the rest of the functions remain with OPENCV. The result shows that the latency reduces to around 200ms, which is almost the same comparing with usb cameras.

[migvel]

Hello,

is this pull request going to be merged into the main branch?

Regards.

[edwardxliu]

Hello,

is this pull request going to be merged into the main branch?

Regards.

Personally, in terms of the latency issue related to IP cameras, I think we need to determine if it is necessary to reduce the 1 sec latency by importing FFMPEG to this project along with OPENCV. If yes, then I think there should be a lot of doc works to do in terms of the download, installation, version and etc. related to the FFMPEG. Also, I can initiate another pull request only include the optimization part of the latency issue and exclude the streaming part if necessary.

In terms of the RTMP streaming part, as I mentioned before, due to the reason that the current Darknet can not meet the streaming requirement of my project, I did some changes to provide such a function and it works very well. However, without such a function, Darknet can still meet the needs of normal streaming cases. So considering the minimalist design style, it is not necessary to include this function to the current Darkent.

That's my personal opinions. What do you think @AlexeyAB ?

[migvel]

From my experience using a RTSP input stream into darknet, and getting the output as a JPEG stream is good enough for now,
since I transcode such output into RTSP and send it into a video server, would be handy to have the output stream directly in RTSP from Darknet.

For me latency is important, did your pull request improved such aspect by using it's own compiled FFMPEG intead OpenCV version?

[edwardxliu]

From my experience using a RTPS input stream into darknet, and getting the output as a JPEG stream is good enough for now,
since I transcode such output into RTPS and send it into a video server, would be handy to have the output stream directly in RTPS from Darknet.

For me latency is important, did your pull request improved such aspect by using it's own compiled FFMPEG intead OpenCV version?

Yes. I downloaded and installed the latest version of FFMPEG. Then I included FFMPEG related libraries (libswresample libswscale libavutil libavcodec libavformat) in Makefile. After that, I created image_ffmpeg.cpp and image_ffmpeg.h which only include the functions for handling the input video stream. Finally, I added a if statement in demo.c to switch between the original get_image_from_stream_resize function from image_opencv.cpp and get_image_from_ffmpeg_stream_resize function from image_ffmpeg.cpp created by me. What's more, in order to ensure the rest functions (including other OPENCV functions) work correctly, the get_image_from_ffmpeg_stream_resize function needs to return the same object with get_image_from_stream_resize. So I copied the mat_to_image function from image_opencv.cpp to image_ffmpeg.cpp.

In conclusion, the amendment for resolving the latency issue includes two new files creation (image_ffmpeg.cpp and image_ffmpeg.h) and some small changes in demo.c and Makefile. While the new files of stream.cpp, stream.h, streamer.cpp, streamer.hpp and some modification in detector.c are all for the RTMP streaming requirement mentioned, which can be excluded if unnecessary.

[migvel]

Thanks for the explanation @edwardxliu

[cenit]

what do you think about integrating it also in cmake toolchain?