Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore inactive sources #3226

Merged

Conversation

maudetes
Copy link
Contributor

@maudetes maudetes commented Dec 9, 2024

active field wasn't actually used to deactivate a source.
This field was misleading and not doing anything.

Actual sources that are marked inactive (but not actually deactivated):

>>> from udata.harvest.models import HarvestSource
>>> for source in HarvestSource.objects(validation__state='accepted', active=False):
...     print(source.id, source.get_last_job().status)
... 
66713b4a2063a18d4f2c77c0 done
666854d2ae020ef75f404615 done
66632580d7365f18b2c81902 done-errors
6451227e111449ae4bff5531 done-errors
641477c66237d10651e97010 done
63dd6eec5f9e9f5cf63bd631 done
633c4314a184a25b2e68b5dd done
632094a92d19bc6c4b86a351 done
62bef8cc760c4fa031fef6ae failed
621e20edf3eeb9b91b4c385c done
61c473775299bbb5dc7af685 done
619fa44bc8e802460bca2faa done
617930065fe883dc16fd36a8 done
60ba37a6657399492704cf0a done-errors
609b98f009c75b3309329255 done-errors
607e9fcbc43657eaf1b0ad93 failed
605b3b19a97b6edbad874083 done
6017a714d475bada80963806 failed
6013cb4c2eb0904b6ab34f77 failed
6013ca70c5aa11960b3e48e1 failed
600fe3a8f921a2ce63619484 done
5fa985cc221828792d64fc3b done
5f89b201fb6b903f47f98892 done
5b45c89c88ee381dc9c83574 done

We should set active=True and we may want to contact them to inform them of their active harvesters.
➡️ Updated sources with active=True in production, except for the trouver.datasud.fr failing harvest sources.

@maudetes maudetes requested a review from magopian December 9, 2024 18:19
Copy link
Contributor

@magopian magopian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a hard one to review ;)
Good job!
🚢

@maudetes maudetes merged commit bde7a9b into opendatateam:master Dec 19, 2024
1 check passed
@maudetes maudetes deleted the fix/dont-harvest-inactive-source branch December 19, 2024 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants