Possible fix for Raspberry Pi 4 hardware-encoded H.264 streams

[octfx]

To preface: This PR is very much vibe coded. I don’t have real knowledge of the project internals or Go. The changes are based on trial and error while testing Raspberry Pi 4 hardware encoding with a Logitech MJPEG webcam.
I have no hard feelings when this PR is closed, I know that reviewing code takes time.

This is by no means an ask for merging this, but more of a "hoping" that parts of this code may help the projects or others.

Problem

I am using a Raspberry Pi 4 (64bit) for streaming to a RTMP Provider using different USB Webcams by employing this project and ffmpeg (v7.1.1).
For efficiency reasons I'm trying to use hardware encoding using h264_v4l2m2m.
While the command works, the WebRTC Preview and the image received on the rtmp side is broken, rtmp receives an artifacted grey image, the preview says it's corrupted.

Go2Rtc Config

As an example this is the config I'm using:

streams:
  Test:
    - exec:ffmpeg -hide_banner -nostats -use_wallclock_as_timestamps 1 -thread_queue_size 64 -f v4l2 -input_format mjpeg -framerate 24 -video_size 1920x1080 -i /dev/video0 -f lavfi -i "anullsrc=cl=stereo:r=48000" -c:a aac -b:a 128k -ar 48000 -ac 2 -b:v 5000k -maxrate 6000k -bufsize 15000k -vf "scale=in_range=pc:out_range=tv,pad=iw:1088:0:0:color=black,setsar=1,format=yuv420p" -c:v h264_v4l2m2m -g 48 -bf 0 -bsf:v h264_metadata=crop_bottom=8:video_full_range_flag=0:aud=insert -fps_mode cfr -f rtsp -rtsp_transport tcp {output}#killsignal=2#killtimeout=5

Changes

Using codex I've described the problems mentioned above, arriving at the changes presented in this PR.

Transcript from codex:

FLV/RTMP

• pkg/flv/muxer.go
  • Add onMetaData width/height and framerate (parsed from H.264 SPS).
  • Don’t emit placeholder AVC config when fmtp lacks SPS/PPS.
  • Inject a real AVC sequence header once SPS/PPS appear in-band.
  • Propagate discovered SPS/PPS to codec.FmtpLine (sprop-parameter-sets) so later consumers pick them up.

H.264 pipeline

• pkg/h264/sps.go
  • Add (*SPS) FPS() to compute fps from VUI timing (used in FLV metadata).
• pkg/h264/rtp.go
  • Capture SPS/PPS from in-band AVCC and use as latest parameter set.
  • Prepend latest SPS/PPS to IDR if they’re missing (fixes first-frame decode).
• pkg/h264/payloader.go
  • Remember latest SPS/PPS and prepend as STAP-A before IDR frames when needed.
• pkg/webrtc/api.go
  • Advertise H.264 High 4.1 (profile-level-id=640029) for better compatibility.

Result

With the changes included, and a fresh compile for my pi, the ffmpeg command now outputs a working image both in the preview and on my rtmp provider.

[AlexxIT]

Please fix spaces in your code. All lines are marked as changed. Although this is not the case.

[AlexxIT]

1. Thanks! I haven't studied everything carefully yet. I'll study it later.
2. I don't know why we need width, height and fps in RTMP. Useless parameters that no one uses.
3. Not sending H264 config is a bad idea. I understand that it will be crooked, but without it some players will not open the stream.
4. The project already has a code to restore SPS and PPS inside the stream if they are not there:
   

   go2rtc/pkg/h264/avcc.go

   Lines 14 to 29 in cd7fa5d

   	func RepairAVCC(codec *core.Codec, handler core.HandlerFunc) core.HandlerFunc {
   	sps, pps := GetParameterSet(codec.FmtpLine)
   	ps := JoinNALU(sps, pps)
   	return func(packet *rtp.Packet) {
   	// this can happen for FLV from FFmpeg
   	if NALUType(packet.Payload) == NALUTypeSEI {
   	size := int(binary.BigEndian.Uint32(packet.Payload)) + 4
   	packet.Payload = packet.Payload[size:]
   	}
   	if NALUType(packet.Payload) == NALUTypeIFrame {
   	packet.Payload = Join(ps, packet.Payload)
   	}
   	handler(packet)
   	}
   	}

   
   Yesterday I added a fix to this code - cd7fa5d
5. I took profile-level-id from real browsers. I will study this idea.

[octfx]

Just for your FYI and to verify I've cloned and compiled fresh from master cd7fa5d

The SPS and PPS fix seems to not be enough, I'm still seeing a grey and artifacted image on my RTMP Provider

[AlexxIT]

Please describe in more detail how you test so that I can repeat it myself.
It looks like you take a USB camera as input and get RTMP as output. In what player do you watch the result?

[octfx]

Sure, my current stack looks like follows:

A Raspberry PI 4 2GB, various USB Cameras (for testing mainly a Logitech C920 Webcam).
My RTMP Provider currently is Api.Video

Using Go2RTC's API I build up a yml (such das in the example above) and start go2rtc with this.

I watch the results on two players:

1. The one provided by api.video directly in their dashboard
2. Using the webrtc endpoint from go2rtc I connect via a react native expo app

The deem the result as broken when: api.video only shows a grey / artifacted image and/or when my webrtc preview in the expo app shows nothing

I also open a preview in my browser using stream.html (which just shows "Stream is broken")

If you need any more information I'd be happy to provide this

[AlexxIT]

In what browser and on what OS do you open stream.html?

[octfx]

Firefox (Librewolf) on Mac
But also tried Chrome (Mac)