Retry jobs which fail or have gone missing

[fredden]

This is a feature suggestion. When jobs fail or have been detected as 'gone away', it would be helpful to be able to automatically retry these jobs. I've set the default to ignore 'failed' jobs, but to automatically retry 'missing' jobs.

I'm happy to discuss this / receive feedback about this feature.

[Ethan3600]

@fredden Love this idea!

My only concern is the scenario where someone has a custom job that's failing for some arbitrary reason (let's say some I/O operation fails or something.. and it's very taxing on the system). Maybe there should be a max retry logic as well? Not sure how we'd want to (or even if we'd want to) persist that as we'd now have to keep track of how many times an arbitrary job runs.

We just don't want to bring an instance to a halt b/c we keep retrying something that's a super heavy lift. I think that's my only concern.

What do you think? Other than that, I think this is a great addition.

[fredden]

I've thought about this some more. I think these probably need to be configuration options for each group (not global), and be off by default for all. It also makes sense to have a retry limit per job; working out when to reset this counter may be tricky.

I agree that we don't want to create a situation where a resource-intensive process crashes on repeat and this feature leads to an outage / denial of service.