Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upload: if any file/path is skipped due to not part of DANDI layout -- log WARNING upon completion or ask #1493

Open
yarikoptic opened this issue Aug 26, 2024 · 0 comments
Assignees

Comments

@yarikoptic
Copy link
Member

yarikoptic commented Aug 26, 2024

Might need to be implemented along with

and overall provide partial (UX) resolution to e.g.

where "non-DANDI layout" file is found to be not uploaded.

I think overall implementation, upon processing command upload for a specific (sub)path (could be top of dandiset), and having obtained list of paths which were uploaded, should then call find_unused_paths(path, used_paths), which would os.walk that path, and if encountering folder, check if any path among user_paths is under that folder path, if not -- add entire folder path into result, if some is, descend into that folder (recurse); if encountering a non-folder, see if was among used_paths and if not -- add to result.

Common known to be ignored paths (like vcs folders and some OSX trash folders) should be ignored altogether.

Then WARNING could report total number of skipped entities (folders of files) with some (up to 10?), and log the rest at DEBUG level (into log file).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants