adds (optional) automatic retries with exponential backoff to jobs #777

jhilden · 2019-02-28T14:48:37Z

Define a job with your intended maximum number of retries and agenda will take care of automatically rerunning the job in case of a failure.

agenda.define('job with retries', { maxRetries: 2 },

The job is retried with an exponentially increasing delay to avoid too high load on your queue.

The formula for the backoff is copied from Sidekiq and includes a random element.

These would be some possible example values for the delay:

retry #	delay in s
1	27
2	66
3	118
4	346
6	727
7	1366
8	2460
9	4379
10	6613
11	10288
12	14977
13	20811
14	28636
15	38554
16	50830
17	65803
18	83625

Fixes #123

Would you consider adding such a feature to agenda?

Define a job with your intended maximum number of retries and agenda will take care of automatically rerunning the job in case of a failure. `agenda.define('job with retries', { maxRetries: 2 },` The job is retried with an exponentially increasing delay to avoid too high load on your queue. The formula for the backoff is copied from [Sidekiq](https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry) and includes a random element. These would be some possible example values for the delay: |retry #|delay in s| |---| --- | | 1 | 27 | | 2 | 66 | | 3 | 118 | | 4 | 346 | | 6 | 727 | | 7 | 1366 | | 8 | 2460 | | 9 | 4379 | | 10 | 6613 | | 11 | 10288 | | 12 | 14977 | | 13 | 20811 | | 14 | 28636 | | 15 | 38554 | | 16 | 50830 | | 17 | 65803 | | 18 | 83625 | Fixes agenda#123

santiq · 2019-02-28T15:30:23Z

Hey @jhilden interesting feature that you made it adds more stability and make jobs more robust, I'm relying on agenda.js for a couple of projects so this seems something that I would use :)

I hope that the maintainers can review it and possibly merge it. If they don't reply you, you may try to speak with them on the slack channel.
Beware that CI tests may be broken because of a small linter issue #701

jhilden · 2019-03-15T10:20:42Z

It would be great if I could get some feedback on this draft PR, so I can decide whether to put more time into it.

If there should be no interest in merging such a feature, I will instead spend that time on simply building a custom retry solution in our private project.

guillegette · 2019-03-28T23:04:07Z

I think this is pretty solid. How can we get some attention from the maintainers ? @dandv @santiq

alex-hall · 2019-04-04T18:01:45Z

@dandv @santiq We use Agenda heavily in production (30+ jobs that run hourly), and we're struggling with building a sane alerting policy that doesn't go off constantly in the middle of the night.

Having an intelligent retry would save our support teams a lot of headache.

santiq · 2019-04-05T00:41:17Z

@alex-hall @guillegette

I use agenda in production too, and with more than 100+ jobs running each day, so this concerns me as well.

But I'm afraid that I don't have merge access, and I asked to be a maintainer several times but they didn't reply or they don't want me, I don't know.

What you can do is enter the slack channel and ask there to the maintainers if they could have a look at this.

Or just fork the repo and use that on your package.json.

alex-hall · 2019-04-05T00:51:58Z

Thanks for the response @santiq. I actually posted into the slack at 2:40 PM EST today, so we'll see if I get a response.

Otherwise, I may have to jump ship to bull or something with this feature already :(

dandv · 2019-04-05T05:39:29Z

Before accepting any code PRs, we need to fix the CI. That would open the gates to resuming Agenda development.

guillegette · 2019-04-05T12:03:29Z

Thanks @dandv the fix for the CI seems to be a all lint problem, would you guys review a PR for that? Are you guys looking for sponsors ?

alex-hall · 2019-04-05T18:43:47Z

@gaboesquivel @dandv please see #798 which fixes CI linter errors

alex-hall · 2019-05-28T14:28:05Z

Bumping this. The PR above is merged and CI is fixed.

guillegette · 2019-06-24T06:13:24Z

Any update about this?

guillegette · 2019-06-28T17:48:23Z

@dandv @gaboesquivel is there anything we can do guys to help you move this forward? are you guys accepting other contributors? I understand that you may not have time but there is enough people interested in this community to help :)

ndraiman · 2019-07-06T15:22:13Z

Thank you @jhilden
Using code copied from the PR as a workaround while waiting for the PR to merge:

import * as agenda from 'agenda';
import * as moment from 'moment';

const maxRetries = 5;

const agenda = new Agenda();
agenda.define('MyJob', (job, done) => {...});
agenda.on('fail:MyJob', onJobFailure);

async function onJobFailure(error: any, job: Agenda.Job<any>) {
    const retryCount = job.attrs.failCount - 1;
    if (retryCount <= maxRetries) {
        job.attrs.nextRunAt = calcExponentialBackoff(retryCount);;
        await job.save();
    }
}

function calcExponentialBackoff(retryCount: number): Date {
    const waitInSeconds = Math.pow(retryCount, 4) + 15 + Math.random() * 30 * (retryCount + 1);
    return moment()
        .add(waitInSeconds, 'seconds')
        .toDate();
}

hosnas · 2019-10-27T10:30:21Z

Isnt this going to be merged? 🤔

hosnas · 2019-11-10T21:56:14Z

we need this pull request to be merged
@simison

AgustinQuetto · 2019-12-17T05:01:12Z

Hello guys. It will be merged? Because I want a feature like this. Temporary I'm making my own feature. Thanks.

salmanahmad94 · 2020-02-06T19:36:32Z

Any updates on this merge?

benheller · 2020-04-01T16:47:35Z

👍 I'm also tracking this PR, and would love to see it integrated into Agenda. Thanks!

ScreamZ · 2020-05-01T20:35:23Z

+1

@simison @OmgImAlexis @rschmukler

Guys could you at least give some vision about such feature implementation or confirm the pattern ?

Thanks a lot and stay safe :)

Thank you @jhilden
Using code copied from the PR as a workaround while waiting for the PR to merge:

import * as agenda from 'agenda';
import * as moment from 'moment';

const maxRetries = 5;

const agenda = new Agenda();
agenda.define('MyJob', (job, done) => {...});
agenda.on('fail:MyJob', onJobFailure);

async function onJobFailure(error: any, job: Agenda.Job<any>) {
    const retryCount = job.attrs.failCount - 1;
    if (retryCount <= maxRetries) {
        job.attrs.nextRunAt = calcExponentialBackoff(retryCount);;
        await job.save();
    }
}

function calcExponentialBackoff(retryCount: number): Date {
    const waitInSeconds = Math.pow(retryCount, 4) + 15 + Math.random() * 30 * (retryCount + 1);
    return moment()
        .add(waitInSeconds, 'seconds')
        .toDate();
}

jaschaio · 2020-05-25T19:41:33Z

Any reason this PR isn't getting any attention despite lots of interest? @simison @OmgImAlexis @rschmukler

Uzlopak · 2020-05-26T05:50:07Z

lib/job/fail.js

@@ -18,5 +19,15 @@ module.exports = function(reason) {
 this.attrs.failedAt = now;
 this.attrs.lastFinishedAt = now;
 debug('[%s:%s] fail() called [%d] times so far', this.attrs.name, this.attrs._id, this.attrs.failCount);
+ const retryCount = this.attrs.failCount - 1;
+ if (retryCount <= this.attrs.maxRetries) {


I think this is wrong. it should be < and not <= . in my adaptation if you modify your test to maxRetry 0 it would still pass the test even though the test should fail.

Uzlopak · 2020-05-26T13:56:49Z

imho: maxRetries should be part of the definition and not part of the task

zizzfizzix · 2020-05-28T14:30:56Z

.gitignore

@@ -3,3 +3,4 @@ coverage.html
 .idea
 .DS_Store
 docs
+package-lock.json


package-lock.json should be kept in VCS as per https://docs.npmjs.com/configuring-npm/package-lock-json.html

zizzfizzix · 2020-05-30T11:00:26Z

imho: maxRetries should be part of the definition and not part of the task

I think it's better to provide it when scheduling rather than as part of definition e.g. agenda.now('newTask', { maxRetries: 5 });

salmanahmad94 · 2020-06-04T07:45:00Z

imho: maxRetries should be part of the definition and not part of the task

I think it's better to provide it when scheduling rather than as part of definition e.g. agenda.now('newTask', { maxRetries: 5 });

Yup. That's what I'm doing.

CoolHandLuke88 · 2022-01-11T19:08:55Z

Hey all, first I would like to say thank you for all the hard work on this project!

Are there any updates regarding this feature? If not what would it take to get this across the finish line?

code-xhyun · 2024-05-05T15:37:58Z

It's a shame that these essential functions are missing.
Pulse, the fork project of agenda, provides that functionality.

Check out this docs page -> Docs

jhilden added 2 commits February 28, 2019 15:33

small fix

0c47280

jhilden mentioned this pull request Feb 28, 2019

How to restart job after failing? #123

Closed

Uzlopak reviewed May 26, 2020

View reviewed changes

zizzfizzix reviewed May 28, 2020

View reviewed changes

harisvsulaiman mentioned this pull request Aug 9, 2021

Retrying failed jobs #1354

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adds (optional) automatic retries with exponential backoff to jobs #777

adds (optional) automatic retries with exponential backoff to jobs #777

jhilden commented Feb 28, 2019 •

edited

santiq commented Feb 28, 2019

jhilden commented Mar 15, 2019

guillegette commented Mar 28, 2019

alex-hall commented Apr 4, 2019

santiq commented Apr 5, 2019

alex-hall commented Apr 5, 2019

dandv commented Apr 5, 2019

guillegette commented Apr 5, 2019

alex-hall commented Apr 5, 2019 •

edited

alex-hall commented May 28, 2019

guillegette commented Jun 24, 2019

guillegette commented Jun 28, 2019

ndraiman commented Jul 6, 2019

hosnas commented Oct 27, 2019

hosnas commented Nov 10, 2019

AgustinQuetto commented Dec 17, 2019

salmanahmad94 commented Feb 6, 2020

benheller commented Apr 1, 2020

ScreamZ commented May 1, 2020

jaschaio commented May 25, 2020

Uzlopak May 26, 2020

Uzlopak commented May 26, 2020

zizzfizzix May 28, 2020

zizzfizzix commented May 30, 2020 •

edited

salmanahmad94 commented Jun 4, 2020

CoolHandLuke88 commented Jan 11, 2022

code-xhyun commented May 5, 2024

adds (optional) automatic retries with exponential backoff to jobs #777

Are you sure you want to change the base?

adds (optional) automatic retries with exponential backoff to jobs #777

Conversation

jhilden commented Feb 28, 2019 • edited

santiq commented Feb 28, 2019

jhilden commented Mar 15, 2019

guillegette commented Mar 28, 2019

alex-hall commented Apr 4, 2019

santiq commented Apr 5, 2019

alex-hall commented Apr 5, 2019

dandv commented Apr 5, 2019

guillegette commented Apr 5, 2019

alex-hall commented Apr 5, 2019 • edited

alex-hall commented May 28, 2019

guillegette commented Jun 24, 2019

guillegette commented Jun 28, 2019

ndraiman commented Jul 6, 2019

hosnas commented Oct 27, 2019

hosnas commented Nov 10, 2019

AgustinQuetto commented Dec 17, 2019

salmanahmad94 commented Feb 6, 2020

benheller commented Apr 1, 2020

ScreamZ commented May 1, 2020

jaschaio commented May 25, 2020

Uzlopak May 26, 2020

Choose a reason for hiding this comment

Uzlopak commented May 26, 2020

zizzfizzix May 28, 2020

Choose a reason for hiding this comment

zizzfizzix commented May 30, 2020 • edited

salmanahmad94 commented Jun 4, 2020

CoolHandLuke88 commented Jan 11, 2022

code-xhyun commented May 5, 2024

jhilden commented Feb 28, 2019 •

edited

alex-hall commented Apr 5, 2019 •

edited

zizzfizzix commented May 30, 2020 •

edited