Good render latency, horrible controller latency on Linux (ubuntu 23.10) and Quest 2

[Mrinohk]

Basically title. At first I thought it was the quest hand tracking taking over but I disabled it and the issue remains. Head tracking and encode/decode are plenty fast. I can move around in VR just fine. The controllers are way behind and seem to have a weird amount of smoothing applied making Beatsaber and Ragnarock basically unplayable. Any ideas are welcome. I've fiddled around the settings for both steamvr and alvr a ton and haven't found much affect from those settings.

Ryzen 7 5700x, RX6600XT, wired PC connection and 5GHz connection to the headset with the router being in the same room as both the PC and the headset.