Interceptor: Protocolize Async Features

[martinklepsch]

It would be nice to be able to use this library without core.async or with a different "deferred values" library (to use the manifold term).

A cursory look at the code makes me think it shouldn't be too hard to do but we'll see. Would you be interested in this kind of change?

[ohpauleez]

Thanks for reaching out with the idea and request!

I just want to clarify a few things to better understand expectations. Would you like to use the core of Pedestal (the interceptor chain) with other async facilities, or would you like to use all of Pedestal (including all the HTTP portions of the library) with other async facilities?

[martinklepsch]

Hey Paul, thanks for the reply :)

I'm only interested in the actual interceptor chain, not in other things pedestal provides.

Hope that answers your question.

Ps. I've been building something like an interceptor chain (enter-only) for a work project and the data being interceptable is just gold for the kind of problems we had to deal with. Later I discovered the interceptor lib here (thanks for making that a separate module!) which will help generalize my current approach and answer some outstanding questions I had about my own "thing". What I want to say is: interceptors are awesome! 😊

[ohpauleez]

@martinklepsch Thanks for following up and I'm glad you're finding the pattern (and the core lib) useful! I could definitely see the core async abstraction in the chain being programmed against a protocol instead of the core.async, and that change is a relatively small one.

Also, Pedestal's interceptor chain can be run in any combination of directions, which you might find useful for your purposes. There's more documented in the release notes, but you can call execute-only, which takes a key/direction, like :enter. This allows interceptor chains to process streams of data, as well as request-reply interactions.

[ohpauleez]

To anyone who is interested in this issue, there are some outstanding design tasks with the current chain API. go-async is a private function that supports two different arities, but after a refactor only one is used. Async handling needs to be improved when executing chains in only one direction -- Currently one-way chain execution works universally for synchronous interceptors, but only the :enter direction behaves as expected for async chains. This is documented in the tests.

Once that design challenge is met, the chain API would be in a place where an async protocol could be adopted.

[martinklepsch]

Thanks for the pointers regarding execute-only, already using this one :) One thing I'm planning to do which is probably a bit more unusual is handling errors in one interceptor (A) (which can only be identified by a later interceptor (B)) by re-enqueueing the interceptors from A leading up to B. Not sure that description makes sense but either way it will be interesting to see how that can be solved. That said from the code I've read so far I'm optimistic that it won't be too hard.

So as for this issue next steps would roughly be

• improve the async handling for one-way chain execution
• implement protocol for async stuff.

Please correct me if I'm wrong :)

[ohpauleez]

@martinklepsch - Enqueueing/Re-enqueueing is pretty common when you get into more advance uses of the chain; definitely a common pattern (for example, this is how the router works -- it finds a route match and then enqueues all the interceptors for that single route path). Post on the mailing list if you hit issues or need some advice.

As for a PR, I really only need the first bullet point (I already have a plan for the protocol refactor, allowing for a fast-path with core.async). Thanks!

[martinklepsch]

Sounds good. One thing I'd like the protocol refactor to include is the ability to use most of the namespaces without core.async on the classpath. Is that something you're taking into consideration or is there not much interest in that?

[ohpauleez]

I was not considering that, but this is a good time to bring things like that up.

[hlship]

In the home page, we discuss how Pedestal is based on protocols so that you can always roll your own solutions; the use of core.async w/ interceptors is the exception and should be addressed.

[hlship]

My thought is for a more general form of async, where the root contract is based on a simple Promise-like protocol, where you can register callback(s) for when a value is delivered to the promise. That's very easy to adapt to a core.async channel.

To Martin's comment (oh, back in 2017), it would be nice to split things up such that there was pedestal.interceptor library, and a pedestal.core-async that would provide the missing pieces and the dependencies on core.async - but I'm not aware of a way to do that in a fully compatible way for existing applications; I'd prefer to avoid upgrade notes like "and add this new dependency, and add this code snippet to set things up".