Create an Indexer Auditing Mechanism To Run in K8s

[artntek]

See Indexer Reliability and Performance Epic

The audit should check short time window and to run frequently. It is used for handling the scenarios needing a quick response, such as the client's editing.

Steps:

• Create a script to find non-indexed objects within the previous (configurable) time window
• Log these to PVC so we can evaluate what's happening, before we...
• Optionally submit for reindexing

[mbjones]

Agreed with above, especially if non-indexed objects also excludes objects that are already scheduled to be indexed. Alternatively, we could remove any duplicate index requests for a pid when one gets picked up (i.e., when pid X is sent to an index worker, any duplicate entries for PID X in the queue are deleted). This would make it robust to scheduling the same pid many times.

[artntek]

Index audit cron has been running every 10 mins in log-only mode on arcticdata.io since 10/9/25. I have been checking weekly, and in that time it has flagged 14 objects as not being indexed. However, I checked them all, and they were all subsequently indexed successfully. (My assumption is that at the time of running, the script sometimes catches a couple of objects that are in the process of being indexed, or have not yet started).