[bug?] Batch API status stuck at `registering`

[rudiedirkx]

• Web request: c35786c3ddda46c2ade734ee5fd5b249
• Mail request: 1e96cbedc8f04955aec3ef87f672b729

We've been using the API for a few weeks, and it works almost perfectly, but tonight it's stuck. We created a batch for 384 domains at 18:02:14, and 1.5h later both requests are still registering. Usually the entire batch takes less than 1h to complete, results and report and all. Sometimes for 200 domains, sometimes for 500.

Maybe I'm too fast complaining, but this has never happened so I'm concerned.

[rudiedirkx]

I accidentally started another batch at 00:47 because I'm dumb 😆 Also stuck, still.

web request: 152946d5c6154c6996da92120ec86dcc
mail request: 68b427cc9e7a47049022806e61ffdfc7

[bwbroersma]

Maybe related, old finished requests are also stuck in "status": "generating".

@rudiedirkx: there was a migration to use docker on batch, there was a spike in Wednesday, and it's being investigated at the moment, related:

• Operations for upgrading batch to Docker  #1253

BTW I never tried it, but it should be able to cancel a request, see the OpenAPI specification.

[rudiedirkx]

The original 2 requests are "status": "done" now. Both finished around 2024-09-12 18:00. The newer 2 (accidental) requests are still "status": "registering".

Is it safe to start new requests (2 x ~250 domains every night), or should I use the old/individual/scraper checks for now? #1255 (comment)

[bwbroersma]

RCA:

• thanks for reporting @rudiedirkx since our normal monitoring did not trigger, since it does not use the batch API, action Add periodic batch test to monitor for operational problems #1499
• one user did nearly 294 614 http calls, mostly triggering report generation, no rate limiting is in place Add http rate limit on API requests #1500, probably this was related to the server/software migration somehow
• these generating reports clogged the queue, since they are not scheduled fairly over the users

The queue was cleared as a quick temporary solution, since the queue was only slowly moving. As a result, some requests in registering or generating are in limbo. We're looking into a solution, probably the generating status will be fixed over time (I have some experience with large crashing reports).

[rudiedirkx]

Tonight's batches of 374 domains hung again. Requested at 18:02, still "registering" just now. I cancelled them and they're going through internet.nl frontend now.

[bwbroersma]

Because of the option in the dashboard to scan once or twice a month, the 1st and 15th of each month have some extra load. We noticed a task queue building up from 15th of September 0 UTC until it was 0 again at 8:40 UTC (10:40) 16th of September. We're unsure if this behavior is new (because of the new setup) or was old, but we're just seeing it now because there is more monitoring. Sadly the report generation can 'clog' the queue, also the registering of domains. Please let us know it this happens again before the end of the month. It can be expected to be a problem on the first of the month. We're looking into improving the current situation.

[rudiedirkx]

Tonight's batch is stuck at running. They (web & mail, 227 domains) have been running for at least 2.5 hours. It's not registering, but it's not much better 😆 I've never seen stuck at running before.

Is there a service status page or something?

It would also be nice to get some more metadata in the Request status API call. Maybe there's a queue with # of waiting before me? Or a checked-on timestamp? Something to indicate it's moving ahead and not just stuck forever.

[aequitas]

@rudiedirkx we found the issue. It is probably not the same cause as issues mentioned earlier but at the moment I can't tell.

The cause of this issue was the beat scheduling process apparently being stuck. Restarting the processes resolved the issue. It will take a while for the backlog to clear and your batch request to finish.

[rudiedirkx]

Both status done at 23:44 👍

I don't know what the beat scheduler is, but does it make sense that it was stuck at running?

Is there a Batch Service status page or something? Or would this have been undetectable anyway?