output: fix tearing with direct scanout and hardware cursor

[ikalco]

Describe your PR, what does it fix/add?

related #9858 and #9890

this pr tries to fix tearing when its combined with other stuff (ds, hw cursor, maybe more? idk)
also draft cause seems to me some other old mechanisms (pre aq) may still be interfering with new ones

Is there anything you want to mention? (unchecked code, possible bugs, found problems, breaking compatibility, etc.)

Is it ready for merging, or does it need work?

draft

[UjinT34]

DS doesn't work with SW cursors. Might work with invisible SW cursors if the code allows it.
HW cursor movement breaks vrr on nvidia-drm (probably driver issue with atomic commit and cursor plane properties).
Locked cursors are handled differently with no_break_fs_vrr = 1. Might be the source of issues since tearing is usually needed for FPS games and they use locked cursors.

[abihf]

is there still any blocking for this PR? I've been using this patch for weeks and it's okay.

[ikalco]

is there still any blocking for this PR? I've been using this patch for weeks and it's okay.

I don't think its finished, gotta look over it still

[Honkazel]

@ikalco rebase?

[fxzzi]

rebase possible? ty ikalco

[Cloudperry]

I tried this on top of Hyprland dd33128 and it didn't seem to work properly. (This is the first time I have tested this PR.)

When I started a fullscreen game hyprctl monitors showed the same hex value on solitary and directScanoutTo. I guess this means that direct scanout was active. There were some graphical glitches and stuttering while it was active and I moved the mouse. When I went in game and the cursor was hidden, directScanoutTo was 0 and the stuttering/glitches stopped. When unhiding the cursor by going into a menu, directScanoutTo stayed as 0.

[fxzzi]

Yeah, just had some time to test too. Results came like this:

• There still seems to be some sort of inconsistency with DS, tearing, and hw cursor triggering at the same time. It's more apparent with animations disabled for some reason, but I often catch activelyTearing true but directScanoutTo 0.
• Moving hw cursor (cpu buffer) just kills DS for the rest of the time the game is fullscreened. Can sometimes be randomly recovered when flicking between workspaces or going between tiled / floating and fullscreen a few times.

[ikalco]

@fxzzi
can you tell me what app/game you use see if there's tearing cause I can't seem to get it? although its possible my gpu is too op hehe
also what options do you use for, misc:vrr, render:direct_scanout, general:allow_tearing, cursor:no_hardware_cursors

btw if you test the new commits out, I changed the activelyTearing and directScanoutTo to be simpler for the hyprctl monitors command

edit:
I just realized the weird behavior probably has something to do with nvidia so I'll try my old 1660 card sometime soon

[fxzzi]

I just realized the weird behavior probably has something to do with nvidia

likely, tho i've also noticed some strange behaviour on my friends AMD based PC (not tested this PR there yet). On certain games for him (Minecraft 1.8.9, Paradox games like crusader kings 3, hearts of iron 4, etc), if he changes workspace and back to the game, his fps is locked to a low number (like 30 or so) for about 10-15 seconds. Also happens when notifications show on screen etc.

I tested a few combinations with him:

# broken, fps locked for about 10 seconds
hyprctl keyword render:direct_scanout 1
hyprctl keyword general:allow_tearing 1
hyprctl keyword misc:vrr 2

# also broken
hyprctl keyword render:direct_scanout 0
hyprctl keyword general:allow_tearing 1
hyprctl keyword misc:vrr 2

# works (and so does vrr 0)
hyprctl keyword render:direct_scanout 0
hyprctl keyword general:allow_tearing 0
hyprctl keyword misc:vrr 2

I use the exact same setup with my nvidia GPU and don't experience this bug

can you tell me what app/game you use see if there's tearing cause I can't seem to get it?

I'm checking for tearing in a few games.

• Geometry Dash, which I run at double my monitor refresh rate experiences tearing as expected (340fps).
• Counter-Strike: Source, which I run at 3 below my refresh rate (167fps), theoretically shouldn't tear with vrr, tearing and ds all enabled at once, but still does. (i assume vrr should avoid tearing with tearing enabled, but without the latency increase of having tearing disabled)

EDIT: also maybe an nvidia related issue. Sometimes games feel stuttery and it magically fixes by going to a different TTY and back. idk y

cursor:no_hardware_cursors is false, using defaults for cursor stuff (cpu buffer on nvidia system, regular hw cursor on the amd sys)

vrr seems to be broken tho even if the cursor is invisible. Noticed this on ghost of tsushima. Controller works fine with VRR, but using my mouse causes my monitor OSD to stay at 170

[ikalco]

ok so the main issue is that the kernel doesn't really like anything changing, including the hw cursor position, while doing a tearing page flip
so my workaround is to use a seperate commit which shouldn't trigger any page flip in order to update the hw cursor position
however, the kernel only seems to like updating the hw cursor at vsync framerate, so moving the cursor limits the frames and could cause temporary stuttering

I'll try asking around about tearing and hw cursor, hopefully there are some better workarounds or a kernel mechanism I missed, but this is the best we got for now

---

theoretically shouldn't tear with vrr, tearing and ds all enabled at once, but still does. (i assume vrr should avoid tearing with tearing enabled, but without the latency increase of having tearing disabled)

not sure if this is the expected the behavior but it's up to the kernel since all we have are switches for tearing and vrr

also maybe an nvidia related issue. Sometimes games feel stuttery and it magically fixes by going to a different TTY and back. idk y

its probably related to the workaround limiting frames

vrr seems to be broken tho even if the cursor is invisible.

this is probably related to cursor:no_break_fs_vrr as UjinT34 suspected, but either way fixing vrr is kinda out of scope for this pr

[fxzzi]

thanks for the explanation, seems the kernel has more to do with ds tearing and vrr than I expected.

fixing vrr is kinda out of scope for this pr

yeah, I know, just thought I'd share anyway since it seems to be an issue in relation to them. my apologies :)

[Cloudperry]

I'm using an AMD gpu with Mesa git and Kernel 6.15.3. I have these tearing related settings:

general:allow_tearing 1
render:direct_scanout 1
misc:vrr 2

With the latest commits this PR doesn't cause stuttering or glitches, but still behaves a bit weirdly. When starting a game, tearing gets activated and scanout as well. However direct scanout keeps flipping between on and off unless I constantly move the mouse around. Also moving the mouse out of the game's monitor causes direct scanout to get stuck as off and it won't turn on when the cursor is back on the same monitor. Also it turns permanently off when the game hides the cursor (I was testing an FPS game).