Fix concurrency issue with .jobs

[jdkirkpatrick]

Every time I click "Generate" it returns an Unknown Error and takes me back to the editor. The only success I've had was cutting the audio down to a short clip (<10sec), but sometimes that still fails. It sometimes runs through generating frames and gets to near 70% before exiting, but other times it fails before it even starts.

My terminal output is:

SyntaxError: Unexpected end of input
at Object.parse (native)
at load (/home/jeff/audiogram/lib/transports/redis/fake.js:14:15)
at getJobList (/home/jeff/audiogram/lib/transports/redis/fake.js:65:19)
at start (/home/jeff/audiogram/node_modules/d3-queue/build/d3-queue.js:72:11)
at poke (/home/jeff/audiogram/node_modules/d3-queue/build/d3-queue.js:56:11)
at Queue.queue.defer (/home/jeff/audiogram/node_modules/d3-queue/build/d3-queue.js:33:5)
at module.exports (/home/jeff/audiogram/server/status.js:7:6)
at Layer.handle [as handle_request] (/home/jeff/audiogram/node_modules/express/lib/router/layer.js:95:5)
at next (/home/jeff/audiogram/node_modules/express/lib/router/route.js:131:13)
at Route.dispatch (/home/jeff/audiogram/node_modules/express/lib/router/route.js:112:3)

I have been tweaking my theme, which looks like this:

"Background image with top right text": {
"width": 960,
"height": 720,
"framesPerSecond": 20,
"maxDuration": 600,
"captionAlign": "right",
"captionTop": 16,
"captionRight": 944,
"captionLeft": 480,
"captionFont": "300 30px 'Source Sans Pro-Black'",
"captionLineHeight": 30,
"captionLineSpacing": 5,
"backgroundImage": "Barn.jpg",
"samplesPerFrame": 96,
"foregroundColor": "#e5e7e9",
"waveBottom": 320,
"waveTop": 180,
"waveLeft": 32,
"waveRight": 840,
"pattern": "roundBars"
},

I'm running the server on my local machine which is running Ubuntu 16.04 Mate.

Any guidance would be appreciated.

Thank you

[rjas4]

I've been getting the same error - almost always when processing a large file, but occasionally also with small files.

The problem seems to be that the JSON parsing in redis/fake.js fails when loading the '.jobs' file in load(). Judging by the fairly random occurrence of the error, I suspect there's a concurrency issue with reading/writing to the '.jobs' file. Catching the parsing error and retrying after a short wait seems to resolve the error.

[veltman]

Thanks for the details - it looks like the issue stems from the /status endpoint polling the job info, which can potentially happen mid-write while the renderer is updating the info. It seems like this would be very rare but I guess not (and it would make sense that it would be more likely on a large file). I will see if I can push a patch for this, but in the meantime, a more robust fix would be to enable Redis in your settings by installing Redis (see INSTALL.md) and adding:

redisHost: "127.0.0.1"

To your settings/index.js file. This ought to fix the problem.

[veltman]

Three possible options for resolving this once and for all:

1. Keep the vanilla .jobs file but add some sort of lockfile system to prevent collisions.
2. Require Redis for everyone
3. Use LevelDB as the "local" option instead of Redis. This would basically remove one dependency vs. Drag-and-drop uploads #2, and add complexity elsewhere.

[jdkirkpatrick]

Perfect, that fixed it (at least my first couple of jobs have worked). I installed Redis and restarted my audiogram server.

Thanks for your help.

[ooobo]

yeah weirdly I've been noticing this too -- only in the last few days! Installing and using Redis seemed to fix it. Hadn't touched the server for a few weeks. I'm on the smallest Ubuntu droplet on digitalocean, in case anyone else is and that's the factor, but sounds like everyone else is using different servers.

[iankevinmcdonald]

I'm having exactly the same issue - for all but the shortest clips. I'm running on an Ubuntu 12 box (Linux Mint maya installed in an Asus EeePC I got back in 2008). It might not help that it's an old (and hence slower) machine.

[iankevinmcdonald]

• and the Redis fix works. Thanks! @veltman if there was just a line about this issue in the troubleshooting section of INSTALL.md, that would have saved me (and possibly others) hours of trying to figure out the problem by myself! (Though it's my fault really for not just typing the error message into the search box on github.)

[rachelwilson]

For the record I also have this "Unexpected end of input" problem often. Maybe 1 in 10 times. I have been testing with a mp3 file that's 411KB. Retrying the same file nearly always works the next time. I'll try the redis fix, but +1 for a troubleshooting section.

[kylophone]

Same issue here. Fixed by switching to Redis.

[veltman]

I think the appropriate fix for this will be to replace .jobs with SQLite or LevelDB backed by a file.

[jeremycherfas]

Still happening even with redis and deleting .jobs

On another try, I noticed that .jobs was continuing to update, and at the end of the process contained

{"jobs":[],"f26ccae0-9229-11e7-86ef-4b62a4f0a8fe":{"status":"ready","numFrames":840,"framesComplete":840,"url":"/video/f26ccae0-9229-11e7-86ef-4b62a4f0a8fe.mp4"}}

And lo! When I looked in /media/video there was a finished mp4 video file that played perfectly.

So although I received the "Unknown error", quite by chance I discovered that Audiogram had produced the video, and of course I could move and rename it by hand.

[hank3rd]

Can someone explain how to add the code: js
redisHost: "127.0.0.1",
worker: true

to the independentx file? I cannot get it right.

[JoeWrightNC]

Very late to this party. Running this for our newsroom on a heroku server. Implemented Redis which is a good call anyway, but were still sometimes seeing the 500 occur when calling to the status endpoint during rendering. For what's it worth, a very easy fix that so far seems to be without issue was just changing the timeout call to give more time for the reading and writing to occur between status checks to decrease the likelihood of the concurrency instance. Might be useful for someone else so wanted to share.