Expose API through CURL / Python

[BeritJanssen]

Is your feature request related to a problem? Please describe.
Some users may want to fire off complex requests which are fiddly through UI or even URL manipulation.

Describe the solution you'd like
If they are comfortable with constructing JSON queries and sending requests through CURL or using a Python or R library, they should get the option to query the API directly.

Steps for exposing search API

• Construct ES queries in backend #1349
• Design endpoints
• Token authentication (see Django-Rest documentation)
• Implement throttling?
• Documentation of API (Perhaps using Swagger ?
  - [ ] Hash / save query models on backend (see also URL can get too long for browsers #1431 and Revisit search history #1419).

Additional steps for exposing visualization API
- [ ] Potentially, #1427

• Certainly: use common data format for visualization responses
• Document data format for visualizations
• Design visualization endpoint (task polling)
• Solution to save visualization requests & results: users can request results for their own tasks, not other tasks

[lukavdplas]

For this kind of advanced use, it may be more useful to give users an API key to make requests directly to the backend (through curl / python / etc), rather than send JSON queries in the interface.

[lukavdplas]

I know we discussed that having only a search API would be a logical "intermediary" step to having a visualisation API, but now I'm not so sure it would be beneficial to the application.

Just to note, one of the potential uses of exposing the API would be to support other applications using I-analyzer data, but we're not currently talking about supporting those cases (and instead requiring uses to generate temporary tokens in the client).

Instead, the intended use case would be users with programming skills who want to analyse I-analyzer corpora programmatically. But for those users, we already have an interoperable and machine-readable solution in the CSV download.

I think you'd have to justify that the time spent on improving the API, handling authentication, etc., would not be better spent on improving and maintaining downloads. (In terms of robustness, efficiency, making large-scale downloads available to more users, more file configurations, etc.) And that in the long term, spreading maintenance/support for quantitative analysis over two fronts (CSV download and API) is worth it because it offers significantly more.

A notable advantage of the CSV download as a way to start quantitative analysis is that it's more accessible to beginning programmers. If I would write a tutorial or workshop on analysing I-analyzer data with Python, I would absolutely explain CSV download and not even think to mention an API.

Also, an issue with a search API is that, in the application as it is now, that would be the only way to download the full data in a corpus with over 10.000 documents. But for our server, it's easier if users just make one download of the full corpus, rather than thousands of small requests - especially if we implement #947. So if we want to support large-scale downloading, it would be better to invest in the download functionality.

So if a search API is not supporting external applications, and we would only it recommend for advanced users, and it's not the recommended way to download data in bulk, its use cases would be very limited.

An API for visualisations would more uses, as it would be a way to request analysed data. I think the use case you mentioned was a researcher wanting to compare the term frequency of dozens of search terms. In that case, an API can indeed be quite useful. Perhaps it would be better to only implement these endpoints?

Even in a use case like that, an API isn't the only option. We have two "advanced" visualisations (term frequency and ngrams) that are not trivial to reproduce. Normally I would suggest outfactoring that analysis to a separate python package (so someone could run it themselves), but the analysis is very integrated with elasticsearch so that's not really useful.

We could, however, write a Python package that performs the same analysis on I-analyzer CSV files. (But analyses the CSV contents rather than elasticsearch output.) A nice side effect of this option is that we won't have to deal with users requesting heavy analysis in bulk from our server. Writing such a package would take some work, but not more than the implementation laid out here.

[lukavdplas]

Hm, this was written under the assumption that I-analyzer already had a working system to download the CSV of a full corpus, but apparently that's not the case 😢 See #1474 So this didn't age well.

I can see the reasoning that for large collections of data, an API to download bite-size chunks may be the best system, but I can't really judge this on a technical level. That said, I still think that downloading CSV data from the frontend is probably the preferred method for researchers who know a bit of Python/R.