Dynamic recurring jobs

[rileymd88]

Hello,

Within my web app, I would like to add the ability for users to schedule their own jobs (also recurring) - I will have one job.js file which will be generic and receive an object by using the worker option.

I have seen good examples using a database as a queue for one off jobs but not sure how to handle this for recurring jobs. Any ideas or examples I can be linked to?

What I have tried looks like this but it does not seem to work and I am sure it is not the best way to do this:

service.js

import Bree from 'bree'
import Graceful from '@ladjs/graceful'
import ms from 'ms'

const bree = new Bree({
    jobs: [
    ]
});

const graceful = new Graceful({ brees: [bree] });
graceful.listen();
bree.start();

export async function addJob(id, interval, oneOff, trigger) {
    try {
        const job = {
            name: id,
            closeWorkerAfterMs: ms('1m'),
            path: './jobs/jobs.js',
            interval: interval,
            worker: {
                workerData: {
                    id: id,
                    trigger: trigger,
                    oneOff: oneOff,
                }
            }
        }
        bree.add(job)
        bree.start(job.name)
        if (oneOff) {
            bree.remove(job.name)
        }
    } catch (err) {
        throw new Error('Could not add job: ' + err.message)
    }
}

controller.js

export async function addJob(id, oneOff, trigger) {
    const interval = formatTimeTrigger(trigger)
    return await s.addJob(id, interval , oneOff, trigger)
}

route.js

import express from 'express'
var router = express.Router();
import * as cont from './controller.js'

const controllerHandler = (promise, params) => async (req, res, next) => {
    const boundParams = params ? params(req, res, next) : [];
    try {
        const result = await promise(...boundParams);
        return res.json(result || { message: 'OK' });
    } catch (error) {
        return res.status(500) && next(error);
    }
};
const c = controllerHandler;

// ******************** ROUTES ******************** //   
router.post('/addJob', c(cont.addJob, (req, res, next) => [req.body.data.id, req.body.data.oneOff, req.body.data.trigger]))
export default router;

server.js

import express from 'express';
import bodyParser from 'body-parser'
import router from './router.js'

const PORT = 5002;
const app = express();
app.use(bodyParser.json());
app.use('/api/v1', router)

app.listen(PORT, () => {
  console.info('Schedule connected!')
});

[titanism]

You should be using a database to store the jobs you want to add, then the job itself should poll the database.

[titanism]

To further elaborate - your controller should be writing to a DB. Then you should have a job that polls this DB for things that need to be run (that were created in the controller). This job should poll it in a way where the query does not run already completed jobs. Furthermore you may also want to implement locking (e.g. the job will lock a job - though this is not needed if you are in a single process environment). After running the job successfully, you should either remove it or mark it as completed.

[rileymd88]

Thanks for your quick reply @titanism!

I understand this concept for when I need to run my job in a one off fashion however lots of my jobs have recurring intervals like 'every Monday at 9AM' and I am not sure how I should persist these kind of jobs in my database.

[titanism]

Just make the job that polls the database run once a minute or once an hour. When you add to your database, set a date when you want it to run at. If the date hasn't passed yet (you'd query for this in your DB in your job), then it won't run it. We can't help you write your architecture, but suggest you look at the source for https://github.com/forwardemail/forwardemail.net to see how we set up jobs there.