Reimplement Search subsystem

[glassez]

New Search subsystem uses external Torznab indexers.
Ref.: qbittorrent/search-plugins#121

[ngosang]

Thank you! I tested it in Windows and it's working well but a few things. You can merge it after solving these issues. About code review, I thing there are more capable contributors. Just search for "python" or "py" in the whole project to be sure you don't forget dead code.

Issues:

• Size parsing is wrong. I see "unknown" in most results. Try searching movies or something larger than 10 GB
• If some indexer is not working there is no way to known. Print some error traces at least.
• If I disable all indexers the search never stops. Maybe add a check for that.
• If I delete 1 indexer the list is not updated. If I delete the same indexer again qBittorrent crashes.
• If I delete all indexers, close the indexers dialog and search for something the search never stops.

Improvements:

• Empty search in available in Torznab. It returns newest torrents and it's useful. Just remove the check.
• Indexers can't be edited. This is really useful because the Jackett API key and IP can change. You can reuse the "add" form, it should be easy.
• I have more improvements in mind but they can wait.

[glassez]

@ngosang
Thank you for the review and testing. Some of the mentioned disadvantages are known to me (I just haven't finished this job yet). I'll deal with the rest later, too.
The main thing is that in general it is acceptable.

[glassez]

Size parsing is wrong. I see "unknown" in most results. Try searching movies o

Does Torznab provide some predefined size format to be returned by each indexer?

[glassez]

If some indexer is not working there is no way to known. Print some error traces at least.

I've been thinking about error handling, but I haven't come to a decision yet. If all the indexer requests are failed, then we can just show the failed status. But what if only some of the indexers failed, and the rest returned the expected result?

[ngosang]

Does Torznab provide some predefined size format to be returned by each indexer?

All indexers follow the Torznab specs. There is a field called size with the value in bytes. Bear in mind this number can be huge. For example => 22987506683 (~21GB)

But what if only some of the indexers failed, and the rest returned the expected result?

1. I think it's a good idea to show error or warning traces in all cases.
2. We can use the status bar (bottom) to show total results and failing indexers. The user can check the traces for error details.
3. In the proposal => qBittorrent.pdf I suggested a button for testing all the indexers but I think it can be implemented later.

[glassez]

I think it's a good idea to show error or warning traces in all cases.

How should we show them? Just add messages to a log?

We can use the status bar (bottom) to show total results and failing indexers.

I'm not sure I can imagine what it might look like (sorry, I'm weak in UI design). In addition, it should be presented in such a way that it does not confuse when the Search tab is not current.

[glassez]

• Size parsing is wrong. I see "unknown" in most results. Try searching movies or something larger than 10 GB
• If I disable all indexers the search never stops. Maybe add a check for that.
• If I delete 1 indexer the list is not updated. If I delete the same indexer again qBittorrent crashes.
• If I delete all indexers, close the indexers dialog and search for something the search never stops.
• Empty search in available in Torznab. It returns newest torrents and it's useful. Just remove the check.
• Indexers can't be edited. This is really useful because the Jackett API key and IP can change. You can reuse the "add" form, it should be easy.

Should be fixed now. Please test again.

TODO:

• If some indexer is not working there is no way to known. Print some error traces at least.

[thalieht]

First time jackett user here so excuse me if i say something stupid.

• Isn't the API key global? Why do we have to add it manually to every indexer we add?
• Can the name field be automatically populated by the "Tornzab URL" (if empty before that)? In which case i would swap these 2 fields positions i.e. "Tornzab URL" first. Btw i would never have figured out what to put in "Tornzab URL" if i hadn't read Discusion: Breaking changes in qBittorrent search-engine search-plugins#121 (comment)

Indexers can't be edited. This is really useful because the Jackett API key and IP can change.

Some people will have many indexers. We'll need a way to bulk update them.

[glassez]

@thalieht
It seems that all your comments come from the false assumption that this is an implementation of Jackett support. This is not true. I never wanted to get attached to Jackett only, so this can be considered as support for any Torznab-compliant search engine.

[ngosang]

Should be fixed now. Please test again.

Sorry, I can't in the next weeks but it should be fine.

If some indexer is not working there is no way to known. Print some error traces at least.

Print error traces in the log, the "advanced" user will be able to see them in the log tab and that is enough to close related issues. About more "visible" message I don't have better ideas. 1) Warning icon in the search tab with a tooltip to check the log. 2) Same message in the status bar. 3) Add a search result with the error in the "torrent name". Eg: The indexer XXX is not working because: XXX. That result is fake and can't be downloaded. For the me urgent is the log message. This could be improved later.

It seems that all your comments come from the false assumption that this is an implementation of Jackett support. This is not true. I never wanted to get attached to Jackett only, so this can be considered as support for any Torznab-compliant search engine.

True. The current implementation is more generic and it's not coupled to Jackett.

Some people will have many indexers. We'll need a way to bulk update them.

This will be a nice enhancement but it can be implemented later. I prefer to merge this as is and the search feature will be improve over the time with small PRs.

[glassez]

Print error traces in the log

It's done currently (within latest changes).

[Chocobo1]

Maybe we should announce deprecation (in UI) of the old search engine first before dropping it entirely? A notice in log when old search is enabled and renaming the old search to Search (obsolete) would suffice. The new search would take over the name of Search.

[ngosang]

I prefer to drop support completely in 4.4.0 release. It will affect a lot of users but, since the decision is made there is no point in delaying. I can do intensive QA this weekend to be sure it's good enough in the first release so users don't have to go back.

You should add a big breaking change notice in the changelog. The users can stay in version 4.3.x for a while until they update. I will add a notice in the search-engine plugin repository too.

[glassez]

However, the problem is that WebAPI/WebUI have not yet been adapted.
Maybe I will have time to provide WebAPI for a new Search engine, but it is unlikely that I will be able to adapt WebUI. To be honest, I wouldn't want to have anything to do with WebUI at all. But I'm afraid that this PR will be blocked until WebUI follows it (it really pisses me off).

[FranciscoPombal]

@glassez

But I'm afraid that this PR will be blocked until WebUI follows it (it really pisses me off).

This PR need not be blocked, just the 4.4.0 release, pending the WebUI implementation.

[glassez]

But I'm afraid that this PR will be blocked until WebUI follows it (it really pisses me off).

This PR need not be blocked, just the 4.4.0 release, pending the WebUI implementation.

IMO, it is unacceptable to block a release while waiting for changes that do not have an ETA.

[FranciscoPombal]

@glassez

IMO, it is unacceptable to block a release while waiting for changes that do not have an ETA.

There is no obligation to make a major feature release until it's 100% ready.

I'm just saying that while it is an option to release the new search subsystem without a WebUI implementation, it is also an option to delay a release that includes the new search subsystem until there is one. IMO this is completely up to you, I'm fine with the decision either way (although in a perfect world, my preference would be that the release already includes the WebUI implementation).

[glassez]

There is no obligation to make a major feature release until it's 100% ready.

My opinion about "WebUI feature parity requirements" has not changed for years. We should not require it until we have an active web developer in the team, otherwise it hinders the development of the application. WebUI is not an integral part of the application for me, but just an external client application (although officially distributed as built-in).

But if the opinion of the community is that absence of "WebUI feature parity" should block the release, I would prefer not to merge it at all until someone adds WebUI implementation on top of this PR. Please be content with unsupported "rotting" feature, if WebUI is more important to you.

[glassez]

@ngosang Would those indexers handle rss feeds as well? I mean would it be possible for the indexers to replace/substitute the rss feature in qbt?

But they are different kind of features with different purposes. Search - to find a specific torrent purposely. RSS - for tracking news/updates of some sources /trackers and (optionally) autoloading new torrents based on some filtering rules. IIRC, Torznab indexes (at least provided by Jackett) can be used as RSS feeds by qBittorrent RSS subsystem.
Have you any detailed idea of how they can be integrated?

[glassez]

To be honest, I can imagine a backlash from the users if it changed without fore communication.
I don't mind it being included in the next release (4.5.0 for now) but I think users would need time to be prepared for the change, that is, the wiki help page for setting up the new system must be ready and there should be a deprecation warning when users enabled the current search system (in qbt log and in plugin repo/wiki pages, with both giving links to the aforementioned wiki page).

Sounds reasonable.
@ngosang
Can you deal with it?

[ngosang]

To be honest, I can imagine a backlash from the users if it changed without fore communication.
I don't mind it being included in the next release (4.5.0 for now) but I think users would need time to be prepared for the change, that is, the wiki help page for setting up the new system must be ready and there should be a deprecation warning when users enabled the current search system (in qbt log and in plugin repo/wiki pages, with both giving links to the aforementioned wiki page).

Yes.

• I will write the Wiki article ASAP (probably this week). It will contain all the details about the deprecation. migration, configuration, recommenced Torznab apps... I will post the Wiki link here when it's complete so you can add the Link in the qBittorrent UI.
• I can manage deprecation notices and complains in https://github.com/qbittorrent/search-plugins Other places like this repository or the official website are your thing.
• I don't have access to create new pages or edit the Wiki. Could you give me temporal access?
• I also would like to know an approximate date for the release. To do final testing, update the deprecation notice and pay attention to the related issues.

Would those indexers handle rss feeds as well? I mean would it be possible for the indexers to replace/substitute the rss feature in qbt?

For now we should keep Search and RSS feature as different things. Doing more changes will delay this integration.

Long answer:
Torznab API is compatible with RSS. That means you can put the Torznab URL in any RSS client and it should work as expected. In qBittorrent RSS too.
The "text search parameter" is optional in Torznab specs. If that field is empty or not defined, the Torznab API must return the list of latest/newest torrents. So, you can add the Torznab URL to any RSS client to discover new torrents.
Well know software like Sonarr/Radarr/Lidarr use that "new torrents RSS feed" to find new releases.
Example: Imagine you have a list of 100 movies you want to download (but they are not available yet because they are in theatres). You have to query the Torznab API periodically to check if the movie is available to download.

• You can make 100 queries searching for the movie name.
• You can make 1 request with empty text search, you get the latest releases and you search the 100 missing movies in that list. Radarr uses that method (called RSS in Radarr UI). Jackett and other Torznab providers have caching methods to speed up empty search.

The Torznab feed used for the Search Engine (this PR) can be used as RSS with the same configuration. In future releases we can use that Torznab config to "auto configure" RSS feeds. Or some kind of "auto download" feature based on RSS feeds. For now, the user can copy the Torznab URL 2 times. In Jackett you have 2 different buttons for Torznab/RSS to make it clear but it copies the same URL :)

[Chocobo1]

I don't have access to create new pages or edit the Wiki. Could you give me temporal access?

I suppose you meant this repo wiki, only @sledgehammer999 can help with that.

I also would like to know an approximate date for the release. To do final testing, update the deprecation notice and pay attention to the related issues.

It is hard to estimate, I would say about this year summer but it might be sooner or even later..

For now we should keep Search and RSS feature as different things. Doing more changes will delay this integration.

Of course, I was just imagining deeper integration since I know little of the indexer.

[glassez]

Guys, what would a deprecation notice look like in GUI?

[Chocobo1]

Guys, what would a deprecation notice look like in GUI?

For me, an warning (nobody would notice info message level) in log would suffice...
Or you can add it in search plugin dialog, probably near the buttons.

[glassez]

For me, an warning (nobody would notice info message level) in log would suffice...

Seems many users have no idea about the existence of the Log.

Or you can add it in search plugin dialog, probably near the buttons.

Here it can also go unnoticed if someone rarely changes/adds plugins. Maybe add it to the main interface of the Search Engine?

[ngosang]

Documentation => https://github.com/qbittorrent/search-plugins/wiki/New-Torznab-search-engine
Screenshots => https://github.com/qbittorrent/search-plugins/tree/master/docs

You should move both to the Wiki of this repository. I don't have access.
I think this is all from my side. If you need anything else I will try to help. Please don't delay these changes too long.

[glassez]

Damn, I really don't have time to do this job. @qbittorrent/frequent-contributors, @qbittorrent/bug-handlers, @jagannatharjun, maybe someone can proceed with it?

[Obegg]

As an end-user, if I understand correctly - I have to add each one of the indexers to qbittorrent, manually, one by one.
But what if you have too many? Is there no option/solution to copy all added indexers and paste them into qbittorrent?
I'm sure I'm not the only one who will be bothered by doing them one by one.
If there's no option, at least provide us (the end-users) with a guide of how to copy all indexers, one way to get all indexers from Jackett's web-ui is to right-click and view the page source and somehow from there do something.

TL:DR:
Make it simple to transition from the current method of search (Python) to the next method of search (Torznab).
Or I should say - Make it simple to import all search indexers.

[ngosang]

But what if you have too many?

In Radarr, Sonarr and related software you have to add one by one too. Please, do not delay this PR. When it's merged you can suggest more enhancements in other issues.

[nbusseneau]

Hey friends, unofficial plugin author here, getting up to speed on this. I was aware of #121 but had not checked in in a long time, I see not only a decision has officially been made for replacement, but also that a lot of things have been done. Great work 👍🏻

I see nothing moved here in the last months, is there background work ongoing? Is this still shipping in 4.5.0 as stated above? I don't see it in the 4.5.0 milestone. Trying to gauge how I should prepare my plugin's users for this.

[glassez]

@nbusseneau
This work has been temporarily suspended.
I'm still not completely sure that Search Feature should become Torznab only. At the very least, I would like to make sure that it supports (at least at the configuration file level) the use of local executables as search engines.
It is also still unclear what to do with the related Web UI. I can't do it myself, and I haven't seen anyone else who would like to do this part of the work yet.

I don't see it in the 4.5.0 milestone.

It is in 5.0 milestone (but this is just a convention).

[Anutrix]

I wonder if this will make it to 5.0

[github-actions]

This PR is stale because it has been 60 days with no activity. This PR will be automatically closed within 7 days if there is no further activity.

[Anutrix]

Any updates?

[github-actions]

This PR is stale because it has been 60 days with no activity. This PR will be automatically closed within 7 days if there is no further activity.