Search requests allow navigating back and forth

[tmetzke]

Description

As a follow-up of #16947, the search response schema is extended to allow navigating back and forth in search results. Currently, navigating back is not possible with the proposed schema.

Proposal

The search response schema is extended to contain the sortValues of the last and first items of the result set.
Regarding the example from the guidelines with the following request:

POST /v2/user-tasks/search

{
  "filter": {
    "processInstanceKey": 22456786958
  },
  "sort": [
    { "state": "ASC" }
  ],
  "page": {
    "limit": 3
  }
}

You'd receive the following (shortened) response:

200 OK

{
  "items": [
    { 
      "state": "CREATED",
      "processInstanceKey": 22456786958,
      "userTaskKey": 22456786345,
      ...
    },
    { 
      "state": "CREATED",
      "processInstanceKey": 22456786958,
      "userTaskKey": 22456786456,
      ...
    },
    { 
      "state": "COMPLETED",
      "processInstanceKey": 22456786958,
      "userTaskKey": 22456786678,
      ...
    }
  ],
  "page": {
    "totalItems": 345,
    "firstItemSortValues": [ "CREATED", 22456786345 ], <--- the "sortValues" of the first item
    "lastItemSortValues": [ "COMPLETED", 22456786678 ] <--- originally the "sortValues"
  }
}

The schema replaces the current sortValues attribute in the page object with firstItemSortValues and lastItemSortValues. Those contain the sortValues of the first and last item of the returned result set. The sort values arrays contain the values of the attributes from the requests sort object for the respective result item (in the example above: the state attribute value of the first and last item respectively), plus our internal tiebreaker (the internal key of the record).

This allows navigating the result set as follows:

• Fetching the previous result set (i.e. navigating back to the previous page): searchBefore with the values from firstItemSortValues, i.e. fetching the values before the current first item.
• Fetching the next result set (i.e. navigating to the next page): searchAfter with the values from lastItemSortValues, i.e. fetching the values after the current last item.
• Jumping to the first page: search request without searchBefore and searchAfter

Decisions

Will be filled in by the project DRIs after a peer review phase.

[aleksander-dytko]

@tmetzke the proposal makes sense to me and seems to provide a user value to develop custom frontend applications. Thank you ⭐

Just one question: can I jump to the last page with this approach?

[tmetzke]

@aleksander-dytko, no, because you don't know the sort values of the last item. You don't even know how many items you have overall. This is not really meant for classic pagination (with a page counter at the bottom and navigation to the first, previous, next, and last pages). It's more about supporting moving back and forth properly in infinite scrolling. Allowing to get the first page is simply possible since you can leave out the searchXYZ parameter and the ES/OS search starts from the beginning (it's what the initial search request does).

[tmetzke]

If we'd want to support "classic" offset pagination, we could add this. But with ES/OS, this is limited to 10k items. Which should be fine for many cases. But we can add this later on top with new parameters for the page in the request.

[aleksander-dytko]

Thanks for clarifying that @tmetzke. I agree with the approach above to start with the first iteration as in the proposal 🚀

[nathansandi]

Great proposal @tmetzke
Just one point:
Could you make explicit on the proposa the types and formats of firstItemSortValues and lastItemSortValues (e.g., timestamps, numeric IDs) to avoid any ambiguity.

[tmetzke]

@nathansandi, I added an example. The sort values arrays can contain any set of types since they reflect the values of the attributes defined in the request's sort object. In the example, we sort by state, so that's the first value in the sort values arrays (for the last or first item respectively). In the end, we add our tiebreaker (the internal record's key) to the arrays.

Hope that clarifies things.