i.eodag: Adjust configuration to breaking changes in EODAG 3.0 #1263
Conversation
In EODAG 3.0, kwarg for download _outputs_prefix_ has been renamed to _output_dir_. See: https://eodag.readthedocs.io/en/stable/breaking_changes.html#b3 This PR adjust the kwarg name accordingly.
|
@HamedElgizery would you mind checking this, please? Looks fine to me, but I cannot test it right now. Thanks! 🙏 |
There was a problem hiding this comment.
Thanks for the fix, @ninsbl!
I tested the output_dir change and it is working fine.
Could you please consider the suggestions I left in the skip_existing function, other than that it is working fine.
| @@ -879,11 +879,7 @@ def skip_existing(output, search_result): | |||
| if scene_file.exists(): | |||
| creation_time = datetime.utcfromtimestamp(os.path.getctime(scene_file)) | |||
There was a problem hiding this comment.
| creation_time = datetime.utcfromtimestamp(os.path.getctime(scene_file)) | |
| creation_time = str( | |
| datetime.utcfromtimestamp(os.path.getctime(scene_file)) | |
| ) |
Converting to string to be able to normalize it below...
| @@ -879,11 +879,7 @@ def skip_existing(output, search_result): | |||
| if scene_file.exists(): | |||
| creation_time = datetime.utcfromtimestamp(os.path.getctime(scene_file)) | |||
| ingestion_time = scene.properties.get("modificationDate") | |||
There was a problem hiding this comment.
| ingestion_time = scene.properties.get("modificationDate") | |
| ingestion_time = scene.properties.get( | |
| "modificationDate", | |
| scene.properties.get( | |
| "publicationDate", scene.properties.get("creationDate") | |
| ), | |
| ) |
Rollback in case modificationDate doesn't exist...
| and datetime.fromisoformat(ingestion_time).replace(tzinfo=None) | ||
| <= creation_time | ||
| ): | ||
| if normalize_time(ingestion_time) <= creation_time: |
There was a problem hiding this comment.
There is some cases where ingestion_time woud be None, e.g. if you use the "onda" provider to search for S2_MSI_L2A products there will be no modificationDate parameter. So it is necessary to check if it is None or not before attempting to normalize.
If you run the following for example there won't be modificationDate.
i.eodag -j provider=onda producttype=S2_MSI_L2A output="Onda_S2" limit=1 start=2024-12-20
| if normalize_time(ingestion_time) <= creation_time: | |
| if ingestion_time and normalize_time(ingestion_time) <= normalize_time( | |
| creation_time | |
| ): |
| and datetime.fromisoformat(ingestion_time).replace(tzinfo=None) | ||
| <= creation_time | ||
| ): | ||
| if normalize_time(ingestion_time) <= creation_time: | ||
| # This is to check that the file was completely downloaded | ||
| # without interruptions. | ||
| # The reason this works: |
There was a problem hiding this comment.
There is also a change that needs to be done to line 892, in accordance with this EODAG change: https://github.com/CS-SI/eodag/pull/1023/files
Could you please change lines [891, 893] to be:
hashed_file = (
downloaded_dir / md5((scene.product_type+"-"+scene.properties['id']).encode()).hexdigest()
)To align with the new hashing method.
In EODAG 3.0, kwarg for download outputs_prefix has been renamed to output_dir. See: https://eodag.readthedocs.io/en/stable/breaking_changes.html#b3
In consequence, the output parameter of the module is not respected, and downloaded files end up in a local tempdir.
This PR adjust the kwarg name according to the changes in EODAG.