[Feature] Multithreaded Screenscraper Scanning

[AROAH1337]

Is your feature request related to a problem? Please describe.
Currently, as far as I can tell, while the scraping of each individual metadata provider is asynchronous per-ROM, there is no support for Screenscraper's additional scraping threads for supporters, as each scan job waits until the others are done before moving onto the next game.

Describe the solution you'd like
Optimally, I'd like to allow each metadata source to scan completely asynchronously, but I think with the current design this would cause metadata to be overwritten by scans after the first provider's.

In the short-term, a separate, configurable "Screenscraper (multithreaded)" source could be added which does not permit being used with any other sources in the same scan job (or at least warns against it). Since different donation levels offer differing numbers of threads, there would need to be something like an environment value to configure the number of threads.

Describe alternatives you've considered
One could use a dedicated scraping utility, as I have in the past, but I don't know of a straightforward way to migrate the assets and data from that other utility into Romm's storage structure without a bunch of bash scripting that I'm frankly not knowledgeable enough to do in a reasonable length of time.

Additional context
I don't know how rate limiting works with the threaded scraping feature, so that may be something else to configure, since I seem to recall seeing a rate-limiting-workaround in the scanner code that waits a varying length of time between requests. Presumably, that would be at least less strict when you're a donor.

[spcano01]

Perhaps also allow us to link previously scraped media, if it matches rom name. I've used their app, and it works very well, and would prefer not to have 2 copies of media. One for ES-DE and one for ROMM.

But I will bc ROMM rocks.

[zurdi15]

Perhaps also allow us to link previously scraped media, if it matches rom name. I've used their app, and it works very well, and would prefer not to have 2 copies of media. One for ES-DE and one for ROMM.

But I will bc ROMM rocks.

Appreciate your kind words!

About media, to mantain consistency we use internal IDs to associate games with its metadata, so even if we make a feature to import all the already scrapped media, sadly it will be duplicated in order to make the internal architecture work properly

[AROAH1337]

What about creating symlinks to the imported files? Although that will become complicated when it comes to Docker, since you'd need to mount your entire external media storage location. Not to mention I've personally run into issues with Docker and symlinks between mounts on different filesystems...

Plus, you'd have the problem of mapping the media output of the external scraper to the internal DB, meaning you'd have to have users go through and reorganize and/or rename all of their media. At that point, it might just be easier to scrape everything a second time and use a separate dedup program to symlink in the opposite direction.

I think I've talked myself out of that one.

Besides, even if this did require refactoring the way the scanner works, all data sources could benefit from the change, both current and future ones.

[zurdi15]

With importing i meant identify the media and import it to the romm architecture to avoid scrap them again, forget about symlinks. In any case if that happens is going to be in a distant future, but I'll keep the issue opened because the SS multithreading is something I want to implement into RomM somehow

[gantoine]

this is supported as of 4.3.0!