Improve upload handling

[stijnvanhoey]

In order to speed up the uploads towards the S3, the handling of multiple files at the same time would be a huge improvement.
A first option would be working with async, but as the boto3 libraryr currently not yet support async handling, this approach will yet fail to work. Working with multiple threads or working parallel would be an valid option to implement.

[peterdesmet]

@stijnvanhoey This is an old issue from data-repository. Is this implemented in any way in vptstools?

[stijnvanhoey]

The vpts-creation has been implemented parallel (using multiple processes, see https://github.com/enram/vptstools/blob/main/src/vptstools/vpts.py#L256-L264), the handling of multiple daily/monthly files at the same time has not. This was technically possible, but the foreseen server only had 1 core and very limited RAM. e.g. the vpts-creation uses the single core and parallelism is not used operationally.

This could be updated within the context of the new deployment. Each daily-file creation is self-sustained and could be parallelized instead of run in a for-loop. This would require some refactoring from the vph5-CLI though.