Discover missing files in GigaDB

[only1chunts]

User story

As a curator
I want to audit GigaDB files on the server
So that all data files that we host are included in the database

Acceptance criteria

Given I know there are files on the file server that are NOT listed in GigaDB
When we run the tool to find them
Then I am presented with a list of files and corresponding locations

Given I have a list of missing files (from above)
When I select add to GigaDB
Then the relevant file(s) are added to the correct dataset in GigaDB
And a history log item is added

Additional Info

I know for a fact that there are some datasets that have thousands of files that have not been added to GigaDB due to issues with display of so many files, so that issue needs to be fixed before those files are added (e.g. dataset 100191 camera trap dataset)

• The only things that need done on Wasabi server is to run the rclone command to get the list of file (curators could do that)
• generate the list of files from the database #2164
• Fix missing files in GigaDB #2184

Product Backlog Item Ready Checklist

• Business value is clearly articulated
• Item is understood enough by the IT team so it can make an informed decision as to whether it can complete this item
• Dependencies are identified and no external dependencies would block this item from being completed
• At the time of the scheduled sprint, the IT team has the appropriate composition to complete this item
• This item is estimated and small enough to comfortably be completed in one sprint
• Acceptance criteria are clear and testable
• Performance criteria, if any, are defined and testable
• The Scrum team understands how to demonstrate this item at the sprint review

Product Backlog Item Done Checklist

• Code is complete
• Automated tests related to the changes are implemented and passing
• All automated test suites are passing locally
• Code is refactored to best practices and coding standards
• Documentation is updated as needed
• A Pull Request has been created and review requested
• Pull Request is reviewed and approved
• The item has been merged to the develop branch
• All automated test suites are passing on continuous Integration pipeline and item is ready to release

[only1chunts]

this is related to #2131