GOP Cache

[seydx]

GOP Cache for Reduced Stream Latency

What's this about?

This PR adds an optional GOP (Group of Pictures) cache to go2rtc that significantly reduces stream startup latency by immediately delivering the last complete GOP to new clients instead of waiting for the next keyframe.

Why is this useful?

• Faster stream start: New clients get video immediately instead of waiting up to several seconds for the next I-frame
• Better user experience: Especially important for cameras with long GOP intervals (5-10 seconds)
• Mid-stream joins: Clients joining an active stream get the last complete GOP instantly

Best used with Preload

GOP cache requires at least one active viewer to populate the cache. For instant stream access even for the first viewer, combine this with the Preload Streams feature (#1762):

streams:
  camera: rtsp://192.168.1.100/stream#gop=1

preload:
  camera:  # keeps stream active and GOP cache populated

This combination ensures the GOP cache is always ready, providing instant playback for all clients including the very first one.

Demo

The video below shows the GOP cache in action. Notice how new viewers start with the cached GOP (slightly in the past) and immediately catch up to the live stream:

gop_demo_2.mp4

How it works

Configuration

Enable GOP cache per stream using the #gop=1 fragment:

streams:
  camera_fast: rtsp://192.168.1.123:554/stream#gop=1
  camera_normal: rtsp://192.168.1.123:554/stream  # no cache

RTSP Client Control

RTSP clients can disable GOP cache per-request even if enabled in config:

rtsp://go2rtc-server/camera_fast?gop=0

Testing

Tested with:

• H.264 and H.265 streams
• RTSP/Onvif sources with various GOP intervals (1-10 seconds)
• RTMP Sources
• HTTP(S) Sources (FLV)
• FFMPEG Sources (Files)
• WebRTC Sources (Ring etc)
• Streams with multiple sources (like ffmpeg for transcoding etc)
• Multiple simultaneous clients joining mid-stream
• Reconnection scenarios

Breaking Changes

None. Feature is opt-in and disabled by default.

Resolves: #1634

[NickM-27]

What is the impact of having this enabled? For example, if in frigate we made this be enabled for every stream would what would the impacts be for a large camera setup?

[AlexxIT]

It depends on how go2rtc is used in Frigate, as its use is not mandatory for the user.

If the same stream is used in go2rtc and Frigate (for example, for detection or recording), then this stream is constantly active in go2rtc. This PR will allow for GOP storage and faster access (start) to active streams.

I haven't looked into this PR yet. I was actually thinking about storing not just a single frame, but also some time in the past. For example, 15 seconds. So user could request a short recent video at any time.

[NickM-27]

Right, I'm mostly thinking about the disabled by default behavior, and if that is because there is some penalty for having it enabled by default for all streams.

For example if we had to consider memory usage or something.

[AlexxIT]

When I planned this feature, I wanted to enable it by default with no option to disable it.
Perhaps after the review this behavior and setting will change.

There's a small issue. For example, when proxying an RTSP -> RTSP stream via go2rtc, no stream processing occurs. Packets are transmitted as is.

When proxying RTSP -> WebRTC or MSE, packet processing already occurs inside go2rtc.
With GOP caching, we always have to process packets, even when proxying RTSP -> RTSP.

[felipecrs]

I think a setting to enable/disable GOP by default on all streams would be cool.

And I think it makes total sense that such setting were enabled for Frigate given Frigate keeps the streams loaded already.

The only concern I have with GOP is that the initial frame may be too old if the stream is not constantly loaded, and keeping that frame on screen may be misleading if the stream takes too long to load.

BTW, maybe go2rtc should drop the GOP cache when the cache is too old, like older than a minute. Not sure.

[AlexxIT]

If the GOP is very old, then what difference for the user whether he sees a black picture or this GOP? It's better to see something than nothing for a long time.

The only problem is with the screenshot. If the user expects to receive a current screenshot and receives something outdated, that's not good.

For the screenshot function will have to add some option like force_new or something like that.

[AlexxIT]

PS. I didn't notice you mentioned a minute. In this case, resetting the GOP makes sense.

[seydx]

I’ve been testing the GOP cache with camera.ui for a while now, and calling up the stream is instant. Much faster than just with active preload (because it starts always with a keyframe)

For streaming it’s not a problem, because as you can see in the video, the stream is immediately fast-forwarded and goes straight into live mode.

@felipecrs
The cache cannot be older than a minute. That’s not how it works. The GOP cache ALWAYS keeps the last 2 GOPs in cache, which means if one GOP = 2s, then we have 4s in the cache. And during streaming this time is fast-forwarded at 100fps

[felipecrs]

The cache cannot be older than a minute. That’s not how it works. The GOP cache ALWAYS keeps the last 2 GOPs in cache, which means if one GOP = 2s, then we have 4s in the cache. And during streaming this time is fast-forwarded at 100fps

I may have misunderstood indeed, but I still don't get it. What if there is no client consuming the stream? Won't the GOP cache become outdated in that case?

I.e. if a client stopped consuming the stream one hour ago, the GOP cache used in a new client connection will be from one hour ago, no?

[seydx]

The cache cannot be older than a minute. That’s not how it works. The GOP cache ALWAYS keeps the last 2 GOPs in cache, which means if one GOP = 2s, then we have 4s in the cache. And during streaming this time is fast-forwarded at 100fps

I may have misunderstood indeed, but I still don't get it. What if there is no client consuming the stream? Won't the GOP cache become outdated in that case?

I.e. if a client stopped consuming the stream one hour ago, the GOP cache used in a new client connection will be from one hour ago, no?

When the stream is no longer active, meaning it has no viewers and is also not active via preload, then the GOP cache is empty.

When the 1st viewer watches the stream, they always get the current data (since the GOP cache was empty).
The GOP cache is always kept as long as the stream is active, and always only the last 2 GOPs.

And that’s why it fits perfectly with the preload option. This ensures that the stream always stays active and thus the GOP cache is always filled.

And as already mentioned, it has NO impact on the live stream. Because as you can also see in the video, the GOP is fast-forwarded very quickly. The result of this is that the user immediately has the stream available and doesn’t have to wait for it to load.​​​​​​​​​​​​​​​​

[felipecrs]

Thank you, I understand now: the situation I mentioned is not even possible with your implementation.

Sorry for the confusion, and I agree this is a no-brainer feature that could even be enabled by default.

[seydx]

Quick (simplified) info on what the difference is to the recently implemented preload option. With preload, a constant connection to the camera is always maintained. This already significantly reduces the stream start time, because we don't have to connect to the camera, etc. etc.

The problem, however, is that go2rtc sends the data to clients immediately when they request it, which means we get data that does NOT start with an I-frame, but with a P-frame.

The decoders can't do anything with this, though, because P-frames always need the previous frames.

Example of 2 GOPs:
I P P P P I P P P P

And the GOP is basically the part that starts with an I-frame up to the next I-frame. And this can vary depending on the camera. Many cameras offer options like "IDR Interval" etc. And that indicates at what intervals we basically see the I-frames.

A decoder cannot start with a P-frame because it needs the previous P-frames, all the way back to the I-frame.
And the GOP cache fixes exactly this problem. Because it constantly analyzes the stream and keeps the last complete GOP in cache and also the current one being processed in cache.

The client gets the old complete GOP + the GOP currently being filled, the timestamps are manipulated so that the client has a fast forward effect and gets a seamless transition to the live stream.

The decoder can instantly start displaying the I-frame (this is independent of the P-frames and can be rendered alone).

[AlexxIT]

In some formats, go2rtc begins transmission with an I-frame. This is mandatory for MSE/MP4.

[seydx]

In some formats, go2rtc begins transmission with an I-frame. This is mandatory for MSE/MP4.

True, but the main problem is still that we need to wait for the I-frame without cache to render the stream for these formats

[AlexxIT]

This is quite a complex PR. I'm not ready to accept it yet.
It significantly impacts the data transmission pipeline, which even now isn't very clearly functional in all cases.

[seydx]

Added a new function to automatically update the fmtp line with correct parameter sets... fixes: #1634