-
Notifications
You must be signed in to change notification settings - Fork 295
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Benchmarking of all pre-trained weights #1792
Comments
I'd be interested in contributing to this - could I have this assigned to me? I'd be interested in experiments to help with the benchmarking effort. |
Hi @kvenkman, thanks for volunteering! How much compute do you have access to? This issue will likely require a lot of GPU time. |
Hi @adamjstewart , I have access to a 1070ti and 1080 on separate desktop computers. They're relatively older machines, but I do have uninterrupted access to them. |
Thanks @kvenkman. That's not a lot, but it should be possible to work on Sentinel-1, NAIP, and/or fMoW since there is only a single model per satellite that needs to be evaluated. Here are the steps I think you need to complete for each one of those:
This is obviously a big project, but even if you only manage to finish steps 1 and 2 for a model, that gets us a long way there. Let me know if you have any questions. |
Issue
In order to select the best pre-trained weights, users need reliable benchmark scores on a number of datasets. We have this for all Landsat weights and some Sentinel-2 weights but are still missing benchmark datasets and evaluation scores for most other weights.
Fix
For Sentinel-2, we should fill in the blanks of the table. For NAIP, Sentinel-1, fMoW, etc., we should decide on a set of benchmark datasets and evaluate all models.
The text was updated successfully, but these errors were encountered: