[Enhancement]: Review Storage mechanics PLUS!

[KeithHanson]

Hello!

Firstly, I would like to commend you on the work you've done here. It's precisely what I need for a very important project for my city as well.

I am creating a fork of the repository now to begin diving into the codebase and attempt to determine what may be happening on our end, so please know I plan on rolling up my sleeves and helping contribute to a solution, NOT just log the bug :P

Anyhow, please see below:

ISSUE: Many hours after starting PM2 service for NVRJS, the timeline does not show video segments on the timeline.
Context:
We are looking to use NVRJS for the camera systems we've built utilizing Raspberry Pi's, PoE cams + switch, and USB harddrives.

We are so very close thanks to your work here. But after testing for roughly a week, we see timeline issues.

We DO see that it is properly restarting the ffmpeg processes and the files are logging to disk.

So everything seems to be working (ffmpeg, UI), except for some reason the segments stored to disk.

Also, NVRJS runs rock solid (haven't seen it rebooting over and over or anything for days at a time).

Once I DO end up restarting NVRJS, the timeline begins working normally, though is missing files that are definitely on disk.

I'll log here if I make progress on it!

Thank you for any insight you can help with though :)

[marcus-j-davies]

Hi @KeithHanson,

It's way passed my bed time where I am (2AM) but wanted to touch base before I hit sleep.

Thanks for the kind words btw :)

I must admit, this was a DIY project, but would welcome any help in making it more accessible.

On to the issue.

My initial thoughts are the way it stores the clip metadata.

It uses SQLITE which I'm not overly a fan of - I wonder if a different storage mechanism can be used?

To stop multiple writes happening, all writes are queued (FIFO), and an interval is happening to check for anything in the queue with the Commit() function (every 10s).

it's based on setTimeout, and if a write happens to error out, it may not reach calling setTimeout again for the next 10s, meaning there is lots in queue, but the Commit function is not being executed, as it crashed before setting the next 10s - the crash may have been due to some SQLITE lock issue, or file IO error, hence why i am not overly happy with my choice in using SQLite.

Another potential is the IO buffer with the pipes between the FFMPEG processs and it's self, if that gets full, it's self will stop receiving FFMPEG output, but I'm sure I ignore stdout...

[KeithHanson]

Thank you for responding so quickly! DIY is my approach to handling a fleet of pole cameras for our municipality. We have a decentralized, edge-based, DIY approach. So far, our Real Time Crime Center analysts like the interface :) And it fits my approach (if it can run decentralized straight from the pole, that's the best outcome).

Your response above all makes sense and what I was suspecting - likely some weird failure on the HD that gets taken care of by our other services on the Pi (decrypts and re-mounts the volume).

I will make a branch littered with debug statements to try to catch this (though, as you know, we will need to wait some time for whatever failure to occur).

Have you considered ditching the SQLite DB entirely? It's almost perfect as a use case to just look at the file system and draw the timeline since I only care about showing the user what's on disk. But you may have explored this already, which might lead me down the wrong track if you've already tried it.

If not, I may take a crack at a lightweight solution. It would be great if there were one less moving part to worry about (and no SQLite DB means not having to import gaps in the data, easy recovery, and other icing-on-the-cake so to speak :)

[KeithHanson]

Also, one last thought for now :P Converting from SQLite to something like PostgreSQL or MySQL might be pretty straightforward. I'm trying to keep as few moving parts on my setups as possible, though (since there's already so many), and keep processor utilization down as much as possible, so I liked the SQLite DB at first sight.

I think I will spend some time on the error not re-queueing the timeout. I'm not the best nodejs dev but I'm sure we're not the only ones who have had this issue with critical functions not being called / queued back up.

[marcus-j-davies]

To draw clips on the timeline, one could look to use ffinfo, along with the timestamp of the file being created/last modified, to accurately place them in time, the missing piece will be to associate any event data, and associate them and store the event data somewhere (other than SQLite :))

[marcus-j-davies]

I think I will spend some time on the error

Makes perfect sense at this stage, might be a simple overlook on my part 👍

[KeithHanson]

To draw clips on the timeline, one could look to use ffinfo, along with the timestamp of the file being created/last modified, to accurately place them in time, the missing piece will be to associate any event data, and associate them and store the event data somewhere (other than SQLite :))

Yes - that is pretty much the implementation I was thinking.

Ah - true true. I forgot about the events timeline (which was one of the reasons I chose this over others - that is very interesting down the road to me, and a simple API for it makes a lot of sense).

Perhaps it might be worth exploring an option for that. Since the timeline of segments is critical for our uses, it would make sense to store event data in an optional database. If you're not using that feature (like us), then you can disable it.

If you DO want that feature, event data is likely considered as non-critical compared to the video on disk itself.

So using a more "fail-proof" mechanism ("this is what is on disk and that's all I know") makes sense to me if it does to you.

[KeithHanson]

I think I will spend some time on the error

Makes perfect sense at this stage, might be a simple overlook on my part +1

Quick research suggests the "retry" module might be the ticket to patch it quickly :) Will give it a go in a branch.

I also think I will hit this issue again pretty quickly - I'm testing this on several Pi's on actual poles right now so... one of them is bound to trip up (something is causing problems regularly on our side - which is good for you and I :D )

[KeithHanson]

Progress: I've got debug statements running on a camera system right now via this fork/branch:
https://github.com/city-of-shreveport/nvr-js.git/(debug-statements)

Working on setting up a way to cause failure for the SQLite DB and test the retry module in another branch.

[KeithHanson]

Ok! I found it. Some error catching and logs went a long way. I am sorry I had to dirty up your clean code :D

But the problem, I think, is more basic than I initially thought. Since the FIFO dequeue wasn't checking for an error on the run, it just drops those segments. I added an enqueue and reset the timeout. I tested this by simply mving the .db file and moving it back.

I'll submit a pull request once I do more live testing on our setup. I've pushed my changes here and re-deployed on two test and live systems. Will report back with any findings :)

This is the deployed codebase:
https://github.com/city-of-shreveport/nvr-js/tree/retry-for-segment-writes

[marcus-j-davies]

Wow! You don't waste time 😅

Would you like to test MKV?

I did take a small look at the other PR - but your environment might be a more substantial test.

• can you view live footage
• can you view recorded segments
• all in the browser.

All whilst using mkv.

swap out the mp4 extension.

Here

nvr-js/NVRJS.js

Line 449 in ab9e402

CommandArgs.push(path.join(Path, '%Y-%m-%dT%H-%M-%S.mp4'));

And here

nvr-js/NVRJS.js

Line 508 in ab9e402

FileName.replace(/.mp4/g, ''),

[KeithHanson]

Certainly! That one is very selfishly interesting to me, ha!

One adjustment I'll need make to this issue's branch prior to that is to either revert to console.log or dig into making debug output to stdout instead of stderr - PM2 thought it was simply erroring out due to this but the logs themselves seem to indicate everything is fine.

Happy to say using 3 minute chunks on 3 connected cameras to the pi, it didn't hiccup once with the patch.

Oddly, at about 6:30am my time though (CST here), the process died in an errored state and didn't restart because of it, so I'll dig into that today.

Once I verify we are good to go with logging, I'll push the mkv patch too.

The MKV format is interesting to me because our analysts typically want to roll back the footage within a very short window (they pull footage and monitor on every 911 call).

And we've seen all kinds of things go wrong on a pole (power sag/power loss/cabling coming loose inside our box, disk space, mounted drive unmounted, etc). So, lots of opportunity for improperly closing the mp4. And MKV will allow us to view partially saved footage.

I've gotta go wear the suit and tie for a little while but once I can get some time to code I'll knock those two adjustments out (stdout and MKV testing).

[marcus-j-davies]

Ok - you have twisted my arm 😅
I need a break from other OSS projects

I have created a branch (2.1.0) - I will use this to merge any enhancements being worked on - so any PR's, address them to this branch for the time being.

I think we can use the video file(s) as a way to provide the timeline content, and do away from any DB based index of segments.
this way even MKV files still being written to will be in the timeline.

I'll wait to hear back from the MKV tests, before I venture down that road, but if the MKV files prove to work in the browser, this should be a smooth change - even event metadata could be made extremely simple also - a flat JSON file per event (that contains the linked segment file name) 😎

Quick Example (bf9d2a78-7425-4249-a559-39bc9ff85f6c-1660052517.json)

{
  "event": "Building Car Park Gates Opened",
  "cameraId": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b",
  "linkedSegment": "yyyy-mm-dd-hh:mm.mkv",
  "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
  "timestamp": 1660052517
}

[KeithHanson]

Awesome! I love the idea of a flat file metadata setup.

One thing I will need to is to checksum the video upon creation, for example, so as to provide proof that the file was not tampered with (if used in court in the U.S., chain of custody is a big deal when you can't prove authenticity of a file; with a checksum on creation, none of that matters much).

So that change could be interesting. Perhaps a video file + a metadata/events file along with each file on disk? IE: for any events coming in, they'd get appended to a file named the same as the video, but with a json extension.

This way, handing all details related to a clip is just two file downloads.

Re: 2.1.0 branch - will do!

[marcus-j-davies]

So that change could be interesting. Perhaps a video file + a metadata/events file along with each file on disk? IE: for any events coming in, they'd get appended to a file named the same as the video, but with a json extension.

That's the ticket!

A metadata file per segment.
The metadata file will contain the checksum +basic info + any events that occurred in that segment

52e5b562-1a1c-4ae8-88a9-c92dc94b497b.1660054102.json

{
  "segment": {
    "cameraId": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b",
    "fileName": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b.1660054102.mkv",
    "startTime": 1660054102,
    "endTime": 1660098736,
    "checksum": "sha256:xxxxxxxxxxxxxx"
  },
  "events": [
    {
      "event": "Building Car Park Gates Opened",
      "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
      "timestamp": 1660052517
    },
    {
      "event": "Building Car Park Gates Opened",
      "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
      "timestamp": 1660052517
    }
  ]
}

These files then drive the timeline UI.
let me know how the MKV use holds up. 👍

[KeithHanson]

PERFECT (for my use cases at least)!

I am happy to let you know that everything has been working perfectly with my branch's patch to re-enqueuing the writes to the SQLDB for over 12 hours now. I'll let it keep running for another 12 before I update that box to using MKV's.

I'm going to deploy this to another system now after making the MKV adjustment.

Question for my current branch, though - I do not submit pull requests often (ever), so I'm not all that versed on how best to handle this.

I have a pending branch that re-queues the SQLDB write. I actually need this in production atm, so am curious if you would:

1. Like me to submit a pull request for just that patch now that it's been fairly well tested in the wild?
2. Would you like another branch modifying the MP4 output to MKV from current main? Or should I spin up another branch based off of my current branch?

Thanks for the guidance!

[marcus-j-davies]

Hey @KeithHanson,

Believe it or not. - I have made massive progress, in removing the need for SQLite altogether (I also don't waste time 😄)

This is what will be coming (have written the logic for).....

• When a new video file is about to be created
  • A new metafile for it will be created (as above - give or take)
    • events created during this segment will be added to that file also.
      • at the end of the segment, its checksum is added to that metafile
        • a new video file is created + a new metafile
          • Rinse and repeat

File name examples:

 {CamerID}.{StartUnixTimstamp}.mkv
 {CamerID}.{StartUnixTimstamp}.json

These files will be used to drive the UI.

This should remove a truck load of surface area for problems to occur.
I think at this stage

• Stick to your "remix of the logic" - for lack of a better term
• Once you patch the files to use mkv (in place of mp4)
  • let me know of the experience
• A PR is not likely worth it at this stage

Once I am happy, you can migrate to the latest version.
does that sounds like a plan?

I have not touched this project in a few weeks - so thanks for getting me excited about it again 😄

Importantly, no difference in the UI, its all backend changes. - but the UI should benefit by being able to view recorded footage that is still being written to (well on the basis MKV works 😅 )

[KeithHanson]

Absolutely! And I'm excited you're excited :)

Agreed about the PR! Sounds like my changes won't be worth much since this will now be flat-file based.

Patching in and deploying the MKV now - will let you know after a couple hours of footage :)

[marcus-j-davies]

By all means, once the new (better) enhancements are in place, feel free to rebrand it - Its OSS after all 😉

Renamed to [Enhancement]: Review Storage mechanics PLUS! 😅

[KeithHanson]

I do think making that configurable would be a great update :) But I also don't mind proudly using open-source and the branding that comes with :)

So! I just patched in MKV files, deployed them to another system, and watched two segments hit the timeline.

1. Live Streaming Works (Chrome/Firefox)

2. I can view recorded segments on the timeline (Chrome/Firefox) but can only view the video from the timeline in Chrome

3. All works in Chrome (linux), but playback of a recorded segment DOES NOT work in Firefox with the following error:
   Uncaught (in promise) DOMException: The media resource indicated by the src attribute or assigned media provider object was not suitable.

A quick google seems to indicate that MKV support is just not in Firefox and from what I can see, not coming :/

Thoughts? This isn't a deal breaker for us here since we use Chrome anyways.

[marcus-j-davies]

Live Streaming Works (Chrome/Firefox)

That make sense, since live streaming uses the MP4 container (well fMP4 - with trickery using websockets)
One of the benefits with NVRJS - is that one FFMPEG instance caters for both the segment and live streaming.

See, I split the one input (camera) to 2 outputs [Live, File] - using one FFMPEG instance per camera,

IMO - The benefit to be had with MKV outweighs the use of Firefox, so at this point I can look past that 😅
Firefox prob doesn't work as it doesnt like MKV - but it works for Live as that is using the MP4 container, where as the segments use MKV.

I could test Fragmented MKV - but will likely be pushing my luck 😄

The live stream is passed via web sockets to a custom media file handler - the browser thinks its a single file (but really its live) it uses the MediaSource Bowser API

[marcus-j-davies]

My side of the world will be sleeping soon.
update here (for my Morning), to ensure MKV is still a go ahead, and I'll continue to work on the enhancements 👍

[KeithHanson]

Ok. I can say that: it works reliably on both test machines. BUT...

For whatever reason, loading MP4 segments are significantly faster than loading MKV segments.

I have both in my timeline and recorded a test for you to see the dramatic difference.
https://cityofshreveport.app.box.com/s/pjjd2y3chp8iepxloy3qn4rh8zw8hyh4 (1 minute .webm file)

Basically, MKV took about 4-5 seconds to load PER SEGMENT (with 3-minute segments - very small compared to regular operation of 15-minute segments), and MP4 took about half a second, reliably.

I have tested it during multiple periods today, and the behavior has been the same. MKV is significantly slower to load than MP4. I have no idea why, though :/

[marcus-j-davies]

Ok,

I have identified why MKV is slower.

As part of the recording chain, I move each files internal metadata to the start of the file (its usually at the end by default)
MP4 allows you to move this metadata to the start (and I do so)

This allows quicker loading of the file into the Browser (by only loading a small portion to get to the metadata and start playing whilst it still downloading), and guess what, MKV does not allow this, so even though I add the flag to the ffmpeg chain, it has no affect to MKV files.

Meaning the browser has to grab the entier file (to get to the meta info at the end of the file, before it can start playing)

Turns out also, that MKV is really only supported by Chrome, I'm going to opt out of using MKV, but if one wanted to, there is a value you can change in code.

I'm almost done with using flat files - so far its working great!
and the UI feels snappier with it

[KeithHanson]

Ahhh that makes sense!

I think I am with you. Snappier is better for realtime/in-the-moment lookups. I set the segments small enough that I don't think we will need to worry about that.

That is looking really good! I'm excited to begin testing it 😁

One thing I've noticed on our machines, but PM2 registers the NVRJS.js as errored after some time. When I check PM2 logs NVRJS, I don't see anything that would indicate it errored.

I'll try to dig in when I can today, but if you have some insight there as well, we'd appreciate it 😁

I can also open another issue on it to track if you'd like. But that is really the only problem I'm bumping into with my requeue patch in place while waiting for the flat file updates 😁

[marcus-j-davies]

Mmmm - I'm not sure how PM2 identifies something as errored, it might be picking up some FFMPEG debug output, and freaking out maybe.

I don't use PM2 myself these days (well... I did when I created this, hence the example start up 😅)
I mainly use systemd now.

One thing to note about the upcoming version (3.0.0)
Since it now uses a different storage mechanism, it won't pick up on previous recordings, as the DB has gone.

Unless you manually create the JSON files for each segment 😬
I should be able to push a test version out a bit later.

but I will also push it to the branch I created (2.1.0 - which I'll rename to 3.0.0 once uploaded)

3.0.0 also allows you to add events to the live view.
i.e a live stream is being viewed, you can tag anytime an event to it - which will then appear on the timeline view.

[marcus-j-davies]

@KeithHanson

Overhaul complete.
https://github.com/marcus-j-davies/nvr-js/tree/v3.0.0
https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/CHANGELOG.md

Its not published yet to NPM - but you should be able to pull down the branch and use it.

[KeithHanson]

Going to test this immediately :) THANK YOU! :)

I will keep digging into the PM2 issue. I love it due to PM2.io's interface, custom actions, metrics, etc. It's pretty nice when managing our 20+ camera systems that will eventually be 100+.

One question:

Since it now uses a different storage mechanism, it won't pick up on previous recordings, as the DB has gone.
Can you tell me more here? Is it because we have to generate the metadata file to know it exists? Perhaps we could create an "import" or a "backlog" sort of script that does that work with what's on disk?

Our storage is pretty much temp storage anyways - the important parts are the live monitoring and clip retrieval in the moment. So not having it show up in the timeline for previous isn't a huge deal breaker, though I may just blow it away and start fresh.

Not a huge deal, but could become important in the future to have some sort of "Catch up" script. Most important thing is the video is on disk and files get deleted over time when space or retention dictates.

[KeithHanson]

Ah, that makes a lot of sense why that's not as straightforward as I was hoping.

Yes - that would be fine - as long as we're able to recover automatically we are happy :)

[marcus-j-davies]

Just an update - I am moving the Meta Creation Logic over to listen for FFMPEG directly after all.
its WIP - so don't pull down any changes just yet.

[KeithHanson]

Roger that! Thanks for the update!

[KeithHanson]

@marcus-j-davies Any updates? :) :) :)

Current metadata bug is affecting us - I'm solving it with a cron-based restart of the service for now.

If you think it will be more than a few more days (no judgement! promise! just planning next steps), I'll go ahead and knock out the backfill task for us and have that on a periodic run & reboot service along with that every hour and I don't think we'll see any further metadata writing issues.

If I can help in any way further than that, please let me know! I don't want to step on your toes, though as I'm certain you'll produce a better outcome than I could :P

Again, many thanks for such a great tool and all the collaboration with us!

[marcus-j-davies]

Hi @KeithHanson ,

Only a couple of lines left really.
I'll see if I can knock it out this evening.

Will ping you shortly

[marcus-j-davies]

@KeithHanson

The patch is ready to test.

To Disable Login:
add this to your config file.

module.exports = {
	/* System Settings */
	system: {
		/* Disable Security - Know what your doing before changing this! */
		disableUISecurity: true,
		.....
        }
}

Changes

{timestamp}_placeholder.json
A file used to store a temp metadata payload (created on FFMPEG process start), this also allows to continue creating events, before the main metafile is written

{timestamp}.json
The main metafile, created at the end of each segment (as advised by FFMPEG).
any events created in {timestamp}_placeholder.json are moved into this file, then finally a new
{timestamp}_placeholder.json is created.

metafile creation is now tied to FMPEG activity, and is no longer 'watching' for new files, which I still believe seem to have problems recovering after IO errors.

[KeithHanson]

THANK YOOOOOU!!!! Will deploy tonight!

[KeithHanson]

Got excited. Deployed to one of our systems for testing and so far things are working :P

I've got about 30+ systems I'll deploy to tonight to verify.

[KeithHanson]

Ok - I've got this running on 10 of our live deployments as of about 20 minutes ago :)

Going to let this run for 24 hours or so and report back :)

[KeithHanson]

Oooh, as a bonus - I moved the files from one folder to the renamed camera folders and bam - everything was picked up. I did have to restart the service for it to show in the timeline, but suuuuper convenient!

[KeithHanson]

Ok, I have deployed to all systems we have under control. ~30.

Something else I thought of that is a win because of this is the way we want to backup our data.

I'm planning on having a second NVRJS running on 81, but it will pull the low quality streams to disk. We're designing an algorithm that will pick the most appropriate "buddy" to back up to and from and kick off some kind of copy process.

Because of these changes, we can easily obtain at least low quality footage by simply using the interface and update the config to display the backed up folder <3

This lets us focus on the hard/critical part for us (choosing the right pole to back up to with a variety of factors) while getting a UI for free! :) :) :)

[marcus-j-davies]

You are excited! 😆

There are APIs built in to fetch segments (metadata), current system utilisation and camera details, if you wanted to create a dashboard to monitor all instances (not footage, more health)- i have not spent a great deal documenting them but they do exist.

Also if I am reading your comments correctly - NVRJS has not been tested to run 80+ cameras (i.e pole 81) - just one to bare in mind.

Let me know if the recent patch fixes the missing meta files (I'm hoping they have)

[KeithHanson]

30 total systems - We'll get to 80 before end of year :)

Excellent - we will definitely tap into that - we have a heartbeat service that checks all kinds of things relevant to us (disks, encryption status, temps, software versions, etc).

[KeithHanson]

We did hit an issue on two systems.

If the metadata file is corrupted for any reason, parsing it fails and NVRJS goes into a boot loop.

2|NVRJS  | SyntaxError: Unexpected end of JSON input
2|NVRJS  |     at JSON.parse (<anonymous>)
2|NVRJS  |     at /home/pi/nvr-js/NVRJS.js:494:19
2|NVRJS  |     at Array.forEach (<anonymous>)
2|NVRJS  |     at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
2|NVRJS  |     at /home/pi/nvr-js/NVRJS.js:393:2
2|NVRJS  |     at Array.forEach (<anonymous>)
2|NVRJS  |     at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
2|NVRJS  |     at Module._compile (internal/modules/cjs/loader.js:1068:30)
2|NVRJS  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
2|NVRJS  |     at Module.load (internal/modules/cjs/loader.js:933:32)
PM2      | App [NVRJS:2] exited with code [1] via signal [SIGINT]
PM2      | App [NVRJS:2] starting in -fork mode-
PM2      | App [NVRJS:2] online

I am fairly sure we just need a try-catch here: https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/NVRJS.js#L494

I patched in a try/catch and things seemed to work:

$ node NVRJS.js 
 - Checking config.
 - Config loaded: /home/pi/nvrjs.config.js
 - Checking volumes and ffmpeg.
 - Creating express application.
 - Compiling pages.
 - Configuring camera: Camera 1
Unable to parse: 1660650121.json
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/pi/nvr-js/NVRJS.js:495:20
    at Array.forEach (<anonymous>)
    at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
    at /home/pi/nvr-js/NVRJS.js:393:2
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
    at Module._compile (internal/modules/cjs/loader.js:1068:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
    at Module.load (internal/modules/cjs/loader.js:933:32)
 - Configuring camera: Camera 2
Unable to parse: 1660650120.json
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/pi/nvr-js/NVRJS.js:495:20
    at Array.forEach (<anonymous>)
    at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
    at /home/pi/nvr-js/NVRJS.js:393:2
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
    at Module._compile (internal/modules/cjs/loader.js:1068:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
    at Module.load (internal/modules/cjs/loader.js:933:32)
 - Configuring camera: Camera 3
Unable to parse: 1660650120.json
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/pi/nvr-js/NVRJS.js:495:20
    at Array.forEach (<anonymous>)
    at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
    at /home/pi/nvr-js/NVRJS.js:393:2
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
    at Module._compile (internal/modules/cjs/loader.js:1068:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
    at Module.load (internal/modules/cjs/loader.js:933:32)
 - Strting purge timer.
 - NVR JS is Ready!
 - Purging data.

[marcus-j-davies]

Yeah - a try catch should be added in a couple of places really.
I am more interested in the root problem.

What did 1660650120.json & 1660650121.json look like?

Just want to make sure I am doing nothing silly, and its more IO Disk errors, when writing files.

[KeithHanson]

Almost undoubtedly disk error - we have to deal with it on our crappy power grid and recover automatically (sad, I know - drink one for me in commiseration :P).

Both files were empty.

[KeithHanson]

If you want I can add in this try catch for now and submit a pull request. I've patched the two systems that failed manually for now.

If you're already off to the races on the patch I can hang back ofc :)

[marcus-j-davies]

Cool - well not, but you know 😅

I'll add a try catch to any reading of a file (and writing for good measure).
I cant do anything special here, other than backing out of the current operation - in favour of not bringing down the instance in its entirety.

EDIT:

If you're already off to the races on the patch I can hang back ofc :)

Already on it

[KeithHanson]

Awesome :) And that is fine - I'm not sure if there is any magical code that could solve that lol. But that's also a reason I reduce things to 3 minute chunks (faster streaming, faster downloading, less risk of missing important things because of gremlins/dragons like this).

[KeithHanson]

Just did a count of all the systems. Deployed most recent commit to 41 raspberry pi's with 2TB and 4TB drives attached :)

I tested it on the previously failing system, and everything went smoothly.

Thank you! :) I'll report in if I find anything.

[KeithHanson]

Rock solid :D

[marcus-j-davies]

Nice!

This is with the recent patch to stop reading Corrupted JSON files?

The changes mode here.

5639de6

[KeithHanson]

Correct! I deployed it after I saw the update.

I see some gaps here and there, but that's from whatever failures happened.

But the only problem I've had is HDD space filling up at this point :D Just means I need to tune the retention.

I've spot checked about 10 of the 41 systems and those that didn't fill up their drive have a full 2 days of history!

Brilliant! :)

EDIT: they all have little gaps here and there (3-6 minutes), but it's obvious our code is recovering from the issue, and your code is handling the recovery gracefully :)

[marcus-j-davies]

Assuming all is well?
I am aiming to publish 3.0 soon.

[trc-turing]

i think this is a great project, and it gives me a good idea to make a NVR system. thank you very much.
i tested v2.0 and v3.0 version.
every .mp4 file record video for one minute. the record video part works well.
after i record video for 16 hours, if i scroll the mouse wheel many times to zoom in/out timeline quickly,
and then scroll the timeline from right to left a few seconds. it will crash, although i press the refresh button, it also doesn't work.
two versions all happened the same issue.

1. i tested the vis-timeline using the link: https://visjs.github.io/vis-timeline/examples/graph2d/08_performance.html, the issue didn't happen.
2. record video for 5 minutes(not many hours), the issue didn't happen.

i think when i scroll the mouse wheel, the timeline.on('rangechanged') will be called many times,
query SQLite or ReadMetaFile maybe have some delay.

[marcus-j-davies]

Hi @trc-turing,

Version 3 has removed SQLIte entirely, and its now based on a JSON file per segment, this seems to have removed a lot of problems, v3 is currently being used in a very large installation, that seems to be quite stable.

See v3 Change Log
https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/CHANGELOG.md

As for rangechanged - yes! this is what causes the query (previously SQLite), but now uses the file mech.
rangechanged will only continue if rangechange is not triggered within 500ms.

nvr-js/web/static/js/scripts.js

Line 78 in 5639de6

timeline.on('rangechange', (event) => {

in other other words, the scroll/zoom must be static for 500ms - try messing with this value.

[trc-turing]

thanks for your reply.
yes, you are right. if i change the settimout duration from 500 to 1000 or 3000.
it doesn't crash. but after i did it, when i click the timeline,
because the duration it will have a delay, or click event doesn't work.

[marcus-j-davies]

Yup, the timeline needs to load data based on the time span in view +- 2 hours.

As the NVR can be running for weeks/months/years at a time, I don't load everything - as the browser could be overloaded with data, can you imagine 1 min segments over a week? That's 10,080 segments in the timeline!

But then zooming out will do the same 😅

I therefore need to cap it based on the current view.

I load what timespan is in view (plus 2 hours)

You can override the 2 hour buffer by changing SearchTimeBufferHours in the scripts file.

At the moment, I don't have a method (or time) to stop the unnecessary loading.

I can probably improve it, but I need to find the time todo so, and it's currently not a priority of mine.

I welcome PRs if you want to contribute 😇

[baozi510]

你好@trc-turing,

版本 3 完全删除了 SQLIte，现在它基于每个段的 JSON 文件，这似乎消除了很多问题，v3 目前正在一个非常大的安装中使用，这似乎相当稳定。

请参阅 v3 更改日志 https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/CHANGELOG.md

至于rangechanged——是的！这就是导致查询的原因（以前是 SQLite），但现在使用文件 mech. 只有在 500 毫秒内未触发 rangechanged时才会继续。rangechange

nvr-js/web/static/js/scripts.js

Line 78 in 5639de6

timeline.on('rangechange', (event) => {

换句话说，滚动/缩放必须是静态的 500 毫秒 - 尝试弄乱这个值。

published yet to NPM

wait published to npm

[marcus-j-davies]

@baozi510

The 3.0 Pull Request is ready -> #13

I'll publish in a day or so.