API data fetch limit

[CBroz1]

I'm working on a python tool for fetching data from our uwazi instance, but I'm hitting some unexpected limits.

https://{my-instance}.uwazi.io/api/search?limit={X} returns a maximum of 60 rows, regardless of the limit set. I've used other filters to verify that more data is available. I've tried using other properties (e.g., offset=X, page=Y) to get further results, but none have worked. The documentation states that there's no maximum limit

Does anyone have any suggestions for iterative searches to fetch all data?

[txau]

Hello @CBroz1

I guess the param you are looking for is "from", so e.g.:

/api/search?from=0&limit=30&order=desc&sort=creationDate&treatAs=number&unpublished=false

will request the first 31 records (including record number 0).

There is indeed a limitation in how many you can request per API call at 10.000 records, that is due to:

Elasticsearch has a default limit of 10,000 results per query. This limit, known as the "Result window is too large" error, applies to the total number of documents returned in a single search request. While you can technically change the maximum number of results using the index.max_result_window setting, it's crucial to understand the potential impact on memory usage and performance.

You can still just paginate in your requests to get all the records. You don't necessarily need to get 10k per request, you could do smaller batches of e.g. 1000 records.

Also, as a rule of thumb, URLs from your uwazi instance library, can be used directly in API search, e.g., this URL:

https://summa.cejil.org/pt/library/?q=(allAggregations:!t,filters:(pa_s:(values:!(fadar409221emi))),from:0,includeUnpublished:!t,limit:30,order:desc,sort:creationDate,treatAs:number,types:!(%2758b2f3a35d59f31e1345b49f%27),unpublished:!f)

translates to:

https://summa.cejil.org/api/search?allAggregations=true&filters=%7B%7D&from=330&includeUnpublished=true&limit=30&order=desc&sort=creationDate&treatAs=number&unpublished=false&include=%5B%22permissions%22%5D

So building the needed queries with the right params for the API call can be done with the library UI.