-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stop storing FileError items from Kingfisher Collect in the database #366
Comments
|
Obviously, as part of this, we would also stop sending messages for file errors from Collect. Edit: There is also some logic in collectionstatus that we can remove As part of this, we can delete collection_note rows |
@yolile FYI, now that we are using scrapy-log-analyzer in production, we can progress this issue, which would simplify the Collect-Process integration. (Right how, HTTP errors are showing up in Process like at https://data.open-contracting.org/admin/data_registry/job/2416/change/, but Collect should simply log these itself, and scrapy-log-analyzer can report them in the warnings email.) |
open-contracting/kingfisher-collect#917 (comment)
SInce this version of Process stores the Scrapyd job ID, it's easy to use scrapy-log-analyzer to parse the log file itself. This avoids errors being introduced by Process, the network, etc.
The text was updated successfully, but these errors were encountered: