Implement a sequential dispatch pattern in re-frame

[qleguennec]

I'll explain the pattern we have been implementing for now:

• subscribe to a data query describing the data we need, e.g: @(subscribe [:get-data "idA" [:adress [:street])]
• either the data is present in the app-db, if so, return that data
• if the data isn't present in the app-db, dispatch a "query in progress" marker in the app-db, along with the http fx query to the server
• whenever the requested data arrives, we clear the "query in progress" marker, and resolve the initial subscription

We want that pattern to be sequential, that is, that the "query in progress" marker is always written to the app-db and propagated to the signal graph before another :get-data subscription dispatches a new http fx query.

e.g:

 (do 
      @(subscribe [:get-data "idA" [:adress [:street])
      @(subscribe [:get-data "idB" [:adress [:city])]))

Here, the second suscription cannot execute before the first subscription dispatched the "query in progress" marker, and before the changes in the app-db have been propagated to the signal graph.

As of now, no combination of dispatch-sync and reagent/flush was enough to make that pattern sufficiently sequential. Although those helped, the pattern doesn't seem to be truly deterministic, as some queries still sometimes run in the same time.

[mainej]

Will you clarify a little? It sounds like you're trying to do a few things:

1. Avoid re-requesting data if it's already present.
2. Ensure that only one request is in flight at once. (Or is there one "query in progress" marker per request, e.g. to show a spinner for that piece of data?)

For the first part, it sounds like you're dispatching from within a reg-sub? Something like this:

(reg-sub
 :get-data
 (fn [db [_ path]]
   (or (get-in db path)
       (do (dispatch [:fetch-data path])
           nil)))

This approach feels declarative: return my data, or fetch it if you need to. But it means your subscription is impure, which doesn't fit with re-frame's model. You can't control when and how often subscriptions fire. So, depending on how you've structured your views it's possible that the same request will be dispatched several times.

For example, imagine if two components subscribe to the same :get-data. To ensure the whole UI is consistent, re-frame will deref the app-db once, and pass it through to all the subscriptions. For both components, (get-in db path) will be empty and both subscriptions will dispatch :fetch-data. It doesn't make sense to try to mutate the db in the first subscription to short-circuit the second, because all subscriptions get the same immutable snapshot of the db. (The full story is more complicated, involving subscription caches and multiple renderings, but the moral is: subscriptions have to be pure.)

Usually the answer is to separate the events and the subscriptions. If you want to avoid re-requesting data that may have been fetched before, do something like this:

(reg-event-fx
 :fetch-data
 (fn [_ [_ path]]
   {:http-xhrio (request-for-path path)}))

(reg-event-fx
 :ensure-data
 (fn [{:keys [db]} [_ path]]
   (when-not (get-in db path)
     {:dispatch [:fetch-data path]})))

(reg-sub
 :get-data
 (fn [db [_ path]]
   (get-in db path)))

Then, dispatch :ensure-data on your schedule, i.e. when you first need the data, on page load or whatever. By taking control of when the event is dispatched, you can ensure it happens only once.

If you want to bring the fetch closer to the subscription, you can try with-let:

(defn view [path]
  (with-let [_ (dispatch [:ensure-data path])
             data (subscribe [:get-data path])]
    [:div (pr-str @data)]))

This usually works, but I'm always wary of it. It's not easy to reason about when views will mount or unmount. I feel safer being explicit, dispatching in response to some intentional event (button click, page load) instead of an implied event (view mount).

I admit, it's less declarative, but re-frame is unlikely to change to permit db mutation within subscriptions.

The second need—ensuring that only one request is in flight at once—has always been tricky in re-frame.

It sounds like you want a "request manager": a state machine that contains a request queue. You ask it to enqueue requests and it decides what to do:

1. Check whether there is a request in flight
   a. If there is, add the new one to the end of the queue.
   b. If there isn't, dispatch the new one immediately.
2. When an in-flight request finishes, trigger its callbacks, then check if there are more in the queue.
   a. If there are, pop one off and dispatch it.

For modeling state machines in re-frame, there's https://github.com/Day8/re-frame-async-flow-fx, but I think that https://github.com/lucywang000/clj-statecharts has more potential. You may get some ideas from https://github.com/ingesolvoll/glimt, which uses clj-statecharts to manage http requests in re-frame, though in a slightly different way than you're describing.