[Feature Request] Reorganise the codebase to support mulitple backends

[kentfredric]

I've done an initial set of changes that would make this work.

kentfredric@3600f0f

Already, some of the benefits of this restructuring are starting to appear. One being the creation of a result class, which makes the validation of the result itself theoretically much simpler, as well as unifies result decoding.

And this bug is opened for feedback/ideas on the branch.

The "adaptor" style is pretty much inside-out roles, creating classes that work as wrappers for classes we can't control to make them have an API we can control.

[kentfredric]

kentfredric/metacpan-api@master...adapter

Most the bugs ironed out, tests now passing for both www::mech and http tiny, just there's a few warnings to nail down now from www::mech.

Still yet to completely eliminate the _decode_result method and supercede it with result accessor validation.

[xsawyerx]

In short:

• Plus 1: I like the idea of switchable backends, supporting different user-agents.
• Plus 2: I like the refactoring (adapters and result set) in MetaCPAN/API.pm.
• Minus 1: I do not yet see why or when you would want MetaCPAN::API to use a different user-agent.
• Minus 2: I do not like the amount of code you need to write in each adapter. I can think of at least two ways to make it significantly shorter and easier. First I would need to understand why we need it. That's the "Minus 1" clause.

Sorry to be stepping in so late on this. This entire thing somehow eluded me.

[kentfredric]

Increasing backend support increases the depth of error handling one can provide, as well as adding support for extensions/features provided by one backend and not another.

For instance, WWW::Mechanize::Cached support /can/ be done at present by providing a fake user agent that emulates HTTP::Tiny , but this means error reporting and soforth to the userland via MetaCPAN::API will be restricted by the interface HTTP::Tiny provides.

I'll concede that it probably is an overcomplicated implementation at present, but its a first pass.

Something I'd like to see implemented eventually is a client-side object cache that maps directly to upstream objects, so that the MetaCPAN::API works more like an ORM, and queries internally fetch merely lists of objects that match a criteria, and then those objects are fetched if they're not cached already. ( Allowing persistent caching that never has to expire instead of caching individual queries that might share objects , and having those queries expire due to the fact the query itself will eventually become outdated )

And the amount of code in the adaptor isn't /that/ much.If you look at what its doing the adaptors are mostly just maps between interface types.

Thanks for looking at it though =)

[SineSwiper]

This smells like something for a separate module that MetaCPAN::API supports, instead of HTTP::Tiny. What you are essentially doing it making MetaCPAN::API's UserAgent work on a more generic level -and- putting that code into MCA. The former is a great idea, but the latter needs to be outside of MCA.

In fact, isn't that exactly what Plack was built for? As a generic "middleware" for web IO? Aren't you building something like Plack::App::Proxy? Or am I confusing things? Some of the Catalyst folks might have some answers to these questions.

[xsawyerx]

@kentfredric thank you for the extensive explanation! Very helpful!
I'm now sold on the idea of it..

I agree with @SineSwiper on the two separate issues. We should probably first consider how to do this (second, maybe even third pass on it) and only then start implementing this externally.

I'm tempted by the idea of MetaCPAN::API returning MetaCPAN::API::Result objects (and with the help of Moo, this becomes even less of a problem). We'll need to define those results much better though than I've done so far.

It would be great to get input from @olafalders on this.

[SineSwiper]

Yeah, I'm going to push the work I've done on HTTP::Tiny::Mech, and some of that could turn into WWW::UserAgent::Adapter.

BTW, this isn't quite Plack, and Plack is too heavy for this, anyway. @mst thinks that we should just beef up LWP, but I don't think he quite understands the need. HTTP::Tiny is designed for a small footprint, and I think we can have that while still maintaining a generic UA layer that supports all UAs.

[SineSwiper]

Here's the branch: https://github.com/SineSwiper/HTTP-Tiny-Mech/tree/topic/expand

I'd like to also remove the Moose stuff, since neither LWP, HTTP::Tiny, or WWW::Mechanize use Moose, and it's an unnecessary dependency.