Objects missing from Lasair but present in other brokers

[genghisken]

Some alerts appear to be going missing. Either because we're not ingesting them, or we are not saving them.

For example, ZTF22abkyiyc is present in ALeRCE, but missing from Lasair.

https://alerce.online/object/ZTF22abkyiyc
https://lasair-ztf.lsst.ac.uk/object/ZTF22abkyiyc/

I've checked the Cassandra database, and no detections for that alert are present, hence I think we are either accidentally dropping them from the stream or the save is not happening for some reason. (I see occasional errors in the Lasair Notifications stream indicating that objects are not being saved - e.g. ERROR in ingest/ingestBatch: Cannot connect to Cassandra but the last time this error happened was on 23rd August 2022.)

[RoyWilliams]

On Oct 5, Julien Peleton wrote "we have received no alerts in the last 4 days, but I can see observations are carried out. Is this expected?". Lasair received alerts on Oct 1 and Oct 6 but nothing in between. There was a huge burst of alerts the evening of Oct 5, which I think is the delayed alerts from Oct 2, 3, and 4. But there is a new topic on kafka every day, and Lasair only listens to today's topic. So we didn't get the alert of Oct 4, that was delivered on Oct 5.

But Lasair was certainly receiving at 08.36 UTC on Oct 6, when the alert should have been received again. I don't know why that is missing, will do more investigations.

[RoyWilliams]

I have gone through the stored alerts that we received from ZTF, and I do not see this objectId

• we do not have ztf_20221003_programid1 see above
• we do not have ztf_20221004_programid1 see above
• ZTF22abkyiyc is not present in
  • 212171 alerts from ztf_20221005_programid1
  • 275513 alerts from ztf_20221006_programid1
  • 176654 alerts from ztf_20221007_programid1

[RoyWilliams]

I note that Fink also has this alert
https://fink-portal.org/ZTF22abkyiyc

[RoyWilliams]

When I change from our kafka 'lasair-ztf-kafka-0:9092' to the ZTF kafka 'public.alerts.ztf.uw.edu:9092', I see the topics
ztf_20221005_programid1 and 6 and 7, but there seem to be no alerts in any of them.

Is it possible that Mirrormaker is not keeping all the alerts?

[gpfrancis]

So we tracked some of this down to delayed processing and alerts showing up after the end of the night. This should now be fixed.

We also noted that some of the missing alerts were detections that happened to be coincident with a known solar system object.

Is there a remaining problem, or are we done with this?

[RoyWilliams]

I think we are done