StorageAnalyzer slows system responsiveness after opening heavily populated media folders

[dgw]

Describe the bug
System becomes slow or unresponsive after StorageAnalyzer has been used to view a folder containing many (thousands) media files.

To Reproduce
Steps to reproduce the behavior:

1. Place 5-10 thousand (or more) media files into a single folder, e.g. /storage/emulated/0/Pictures/TestFolder
   • Optionally add a .nomedia file to this folder — I am not sure yet if this influences the behavior
2. Scan device with StorageAnalyzer
3. Browse the StorageAnalyzer results; system responsiveness should still be normal
4. Enter the test folder containing thousands of media files
5. Scrolling may or may not work; SD Maid SE should now be slow to react to Back button presses or other input
6. When I have been able to get StorageAnalyzer to back out of the problematic folder, system responsiveness has not recovered, as if some background task(s) spawned by opening the folder stay(s) running

Expected behavior
Regardless of how many files are in a given folder, browsing StorageAnalyzer results should remain responsive.

Smartphone (please complete the following information):

• Device: ASUS ROG Phone 3 and OnePlus Pad
• OS: Android 12 and OxygenOS 14 (yes, it's just skinned Android)
• Root status: No
• SD Maid 2/SE version: 0.24.0-beta0

Debug log
I will be happy to add this later, if needed; it takes quite a while to scan my devices

Screenshots/Video
As above, happy to add if needed.

Additional context
Generally I end up restarting the whole device to get responsiveness back. I can usually do so through the normal menus, with patience, but occasionally it has been bad enough that I considered just holding the power button to force shutdown.

Edit: I have not been able to reproduce this extreme lag. Let's ignore it unless/until I can get a clear way to repro.

[d4rken]

If you disable previews in SD Maids settings, does it perform better?

[dgw]

With file previews disabled, StorageAnalyzer behaves much better in my test folder! The high file count on its own clearly isn't the problem.

Do you need me to check some other behavior? Maybe I could try removing the .nomedia file and let MediaServer index the folder overnight, then check StorageAnalyzer again with previews enabled? AFAIK there is no thumbnail cache for nomedia folders, which could be forcing thumbnail generation from scratch on all 5000 files.

Improvement opportunity there would be only to help the UI stay responsive while thumbs are created, since it looks like an external library called coil deals with the image/video files themselves. I wasn't able to follow the Kotlin code deep enough to see if it takes advantage of the system thumbnail cache, but even loading thousands of pre-generated thumbs could be slow.

[d4rken]

The .nomedia file shouldn't make a difference, coil ignores that and in my implementation also does not use any system media cache (using that would cause unwanted cross-interactions if we are cleaning caches, it would be bad to use caches too).

but even loading thousands of pre-generated thumbs could be slow.

This should not be a concern as it (should) only load thumbnails for files that are visible in the list (+/- a few extra at the start and at the end).

I tested this on my phone with 5k media files in the WhatsApp folder and it did only load those items that were visible. Fast scrolling even cancels loading for items no longer in view.

Is this on an external sdcard and could it be possible that the sdcard is very slow and these operations hangup the whole OS because the I/O operations are piling up?

[dgw]

I haven't managed to reproduce the extreme system-wide GUI lag I saw before, so let me strikeout that part of the original issue until/unless I can make it happen again.

Is this on an external sdcard and could it be possible that the sdcard is very slow and these operations hangup the whole OS because the I/O operations are piling up?

I thought of storage speed as a possible factor. Neither device I tested before opening the issue even supports expandable storage (😡 modern Android designs), but the unresponsiveness is more noticeable on the cheaper OnePlus device. Confoundingly, though, both advertise UFS 3.1 storage in their hardware specifications—so I'm inclined to think the difference is attributable to the different SoCs instead (Dimensity 9000 vs. Snapdragon 865).

I tested this on my phone with 5k media files in the WhatsApp folder and it did only load those items that were visible. Fast scrolling even cancels loading for items no longer in view.

The faster device (ROG 3/SD865) is able to scroll around a 20k-file folder with no issues as you describe here, except at the beginning of the list where StorageAnalyzer sorts the several-hundred-MB video files my test folder contains.

If I fast-scroll away from the huge video files down to single-digit-MB images, as soon as I can after opening the huge folder, I can spot thumbs being generated in quick sequence after some waiting. The delay suggests that thumbs are generated sequentially; and that once a large file starts being thumbnailed, the operation perhaps can't be canceled?

Maybe my earlier tests actually made the issue worse by queueing additional large videos, since I was scrolling not very far down, only around the top of the list where all the biggest (and therefore likely most expensive) files are.

---

While testing the scroll behavior, I found and replaced a couple of files that hadn't downloaded correctly and were sitting at 0 bytes. Normally I'd have no reason to look at the small end of the analyzer's report for a folder, so thanks for the excuse to play around!

[d4rken]

If I fast-scroll away from the huge video files down to single-digit-MB images, as soon as I can after opening the huge folder, I can spot thumbs being generated in quick sequence after some waiting. The delay suggests that thumbs are generated sequentially; and that once a large file starts being thumbnailed, the operation perhaps can't be canceled?

Sounds like thumbnailing the big videos is causing the slow down 🤔

I thought they are being created in parallel, but I'll have to check, I think it's just using coil's default setting at the moment. I just wrote a module so that coil can load the images through SD Maid's access methods (LocalGateway/SAFGateway etc).