Management of download size

[BeritJanssen]

@lukavdplas @Meesch @JeltevanBoheemen what are your two cents on the number of documents a user can download.

• should we implement a download limit per corpus or per server? Right now, we have the latter option, and it is set to 10000 for all servers, but perhaps we want to bring this down for corpora that have large documents?
• if a user requests permission to download more, should we remove limits (as suggested in change download limit to permission #1227) , or continue handling user-specific limits?

[BeritJanssen]

Thinking about this some more, I think a good setup would be the following:

• there is a server-wide download limit (typically, 10'000 documents)
• each corpus can have a download limit (default: 1'000'000 documents)

If we give a user permission to download more than the server-wide download limit, we can still cap it at a corpus level, e.g., to prevent overly long downloads (for which csv format may not be appropriate anyway), or to comply with potential requests of suppliers of corpus source data (still hypothetical at the moment, but I can imagine that some supplier may not be okay with downloads of more than a given number of documents). Perhaps exactly this solution was discussed at an earlier meeting, but I don't think this is formulated in an issue then just yet.

[lukavdplas]

This sounds good, just a few comments:

• There is an ultimate cap that is typically based on the server setup (e.g. configured apache timeout), the available memory on the server, physical limits like internet speed, etc.. This really belongs in the server settings, not on each individual corpus.
• The limit for a corpus might be lower than the default limit for the server.

My suggestion would be:

• The server settings configure both the default and ultimate limit for downloads, e.g. DEFAULT_DOWNLOAD_LIMIT = 10000 and MAX_DOWNLOAD_LIMIT = 1000000.
• Corpora have an optional field download_limit.

The limit on a download request could be calculated more or less like this:

corpus_limit = corpus.download_limit if corpus.download_limit != None else math.inf
user_limit = settings.DEFAULT_DOWNLOAD_LIMIT if not has_big_download_permission(user) else math.inf
server_limit = settings.MAX_DOWNLOAD_LIMIT
user_limit = min(corpus_limit, user_limit, server_limit)

[BeritJanssen]

Hm, but then we'd also still need the setting DIRECT_DOWNLOAD_LIMIT.

[lukavdplas]

DIRECT_DOWNLOAD_LIMIT controls a different (though analogous) behaviour, so I don't really see the connection; can you explain?