[Sleep Log] Gadgetbridge sync support

[joserebelo]

I would like to implement sync support for the sleep logs to Gadgetbridge.

I see that we already send real-time events to Gadgetbridge, but they are currently not being handled. Aditionally, any events that happen while disconnected will be lost.

I was envisioning a mechanism similar to #2889, where we request all the sleep logs since a timestamp and update existing records in Gadgetbridge, along with a button in the preferences to trigger a full fetch of all data present on the Bangle, as that might take some time to execute.

@storm64 do you have any thoughts on this?

[gfwilliams]

I could be wrong here, but as far as I can see from https://github.com/espruino/BangleApps/blob/master/apps/sleeplog/boot.js#L27 sleep log mainly uses the 10 minute 'movement' value from the built-in health event.

This is great because it's not really using any extra power, but also the health app is already storing this information too, all the time. And it's synced already via the {t:'act', mov:...} - we may just not be storing it in Gadgetbridge?

... so in a way, we may just be able to run this algorithm on data that's already in gadgetbridge? Looks like sleeplog does check wear state and whether it's charging though, which we don't store (but charge state at the very least could be sent/stored trivially, and I guess if it's not worn we won't have a BPM reading for that 10 min health event).

That's one thought, but also sleeplog saves everything to a storage file. Perhaps we should consider having some kind of generic file sync built into Gadgetbridge rather than setting out to do it just for sleeplog?

• metadata.json already has a data field listing data files the apps make. Maybe if sync:true is set there, we could make sure that's pushed through to appname.info in the bangle
• when syncing Gadgetbridge could scan those files and figure out a list of files to sync, and could then automatically sync them

Espruino has two different storage types (one fixed length and binary, one appendable and text). There's already the code to handle syncing the appendable one for tracks thanks to @thyttan, but I guess we might need to add something to deal with the binary one.

[thyttan]

Off topic:

@gfwilliams

This is great because it's not really using any extra power

I also like this aspect of sleeplog. But every time I use it for a while I end up suspecting the app of being a big drain on battery somehow. Do you have this experince as well? Could it be writes to storage? I haven't looked at this deeply yet (and don't know if I ever will).

[gfwilliams]

I haven't used it I'm afraid - but it depends how big the sleeplog JSON file is - if it is big it could end up being a huge drain.

Using StorageFile for stuff that gets appended to would be the way to go... Or actually we could extend the 'health' app's file format with extra fields.

• battery and wear state would be a good one
• we could also have a field for 'activity type' which apps could update, so sleeplog could just update that and then it'd be saved automatically. Similarly the 'run' app could automatically set activity type when it's running, the cycle app, and so on.

I feel like having some kind of global activity state would probably fit better with what Gadgetbridge expects?

[thyttan]

If we extended the 'health' file format that way, would that mean we could refactor sleeplog to just display data from the 'health' file? Sounds pretty neat.

[gfwilliams]

would that mean we could refactor sleeplog to just display data from the 'health' file?

Yes - it'd still have to run over the data from the day and count up all the moments of sleep and how long and so on, but it'd just be a case of looking at whether activityType==DEEP_SLEEP and so on.

but it depends how big the sleeplog JSON file is - if it is big it could end up being a huge drain.

I just checked again and I was totally wrong here. It uses StorageFile already so that should really have no big effect on power usage no matter how big the file gets.

[joserebelo]

... so in a way, we may just be able to run this algorithm on data that's already in gadgetbridge?

One one hand, this would work for the current implementation. On the other, I'm left wondering if this would prevent the algorithm from being improved (eg. taking small bursts of real-time data like Sleep as Android does).

I feel like having some kind of global activity state would probably fit better with what Gadgetbridge expects?

I do like this idea, but I'm not sure we should just think about what Gadgetbridge expects :) its current implementation is driven by how the first gadgets that needed saved data, which is not always the most efficient or easier to work with format. It might also be a bit heavier on the Bangle side storage-wise, as we'd need to keep the sleep state for every 10-minute block, instead of a sleep range "from X time to Y" (I did not check how the current sleep log app saves it).

We have been having to merge the data from multiple sources for some gadgets (steps from a table, heart rate from another, sleep from another), so that's not really a problem.

[gfwilliams]

It might also be a bit heavier on the Bangle side storage-wise, as we'd need to keep the sleep state for every 10-minute block

Yes, however right now we store 4 bytes for every 10 minutes, so that's 18kB per month - which isn't a big deal as we have 8MB available. I was thinking maybe we add a bit to that.

• 1 byte could store charge state, wear state, and current activity easily.
• I also fancy storing battery level and altitude (packed into 2 bytes)

Not sure if there's anything else?

[thyttan]

Pinging @storm64 again in case you missed the first one. What's your thoughts on the discussion here?

[gfwilliams]

@storm64 ?

[thyttan]

Semi-monthly ping @storm64

[storm64]

Sorry for me not responding, the working life was a bit too demanding.
It's been a while since I've done the coding, but here are some thoughts on some points mentioned:

1. Recalculating sleep data on the smartphone side:
   Might be possible if we know the "wearing"-state and a movement value with timestamp.

2. Reciving live changes from sleeplog:
   To get changes while connected to Gadgetbridge you could simply add a trigger as described in #add-functions-triggered-by-status-changes-or-inside-a-specified-time-period.

3. Reciving sleep data from sleeplog:
   I'd implementet some utils to get the sleep data for a defined time period quiet fast (depending on the range).
   You could simply run require("sleeplog").getStats(0, {UNIX timestamp of last connection}); on reconnect.
   More informaton on this: #access-statistics

4. Battery usage and storage types:
   To be as gentle to the battery as possible (for my skils), I implemented the following features:

• Using the 10min-movement-values and only checking for "wearing" if no movement is detected.
• Append only status changes with some necessary additional data to a StorageFile.
• Moving the cleaned sleep data over to a normal file and clear the StorageFile every 14days, so it doesn't get to big. See 5. in #timestamps-and-files.

Thanks for working on this topic. I try to be faster on responding, now. ;)

[thyttan]

Nice to hear back from you @storm64 :)

ping @joserebelo @gfwilliams

[32bitx64bit]

I searched up for some reason this didn't show up. Glad to see work getting done on this. Hopefully development moves quickly and efficiently 😄

[32bitx64bit]

@storm64 just giving a ping to see if any progress has been made.

[32bitx64bit]

Just doing the monthly checkup.

[gfwilliams]

Just an update - https://codeberg.org/Freeyourgadget/Gadgetbridge/pulls/4323 is now merged in (but I haven't yet done a Play Store release). So there's nothing in the Bangle.js software yet but it should be possible to push activity data in to Gadgetbridge

[32bitx64bit]

Just an update - https://codeberg.org/Freeyourgadget/Gadgetbridge/pulls/4323 is now merged in (but I haven't yet done a Play Store release). So there's nothing in the Bangle.js software yet but it should be possible to push activity data in to Gadgetbridge

Awesome ❤️

[xxxserxxx]

Just an update - https://codeberg.org/Freeyourgadget/Gadgetbridge/pulls/4323 is now merged in (but I haven't yet done a Play Store release).

Will this be in the nightly F-Droid build? I think I saw that PR, along with a number of Sleep score related changes, in the nightly.

Edit looks like! Sleep detection had been iffy; I don't understand the variables, or the scalar values, well enough to tune it, but those are absolutely the sleep values from my Bangle right there in BangleJS!

Very nice! Thank you!

[32bitx64bit]

Just an update - https://codeberg.org/Freeyourgadget/Gadgetbridge/pulls/4323 is now merged in (but I haven't yet done a Play Store release).

Will this be in the nightly F-Droid build? I think I saw that PR, along with a number of Sleep score related changes, in the nightly.

Edit looks like! Sleep detection had been iffy; I don't understand the variables, or the scalar values, well enough to tune it, but those are absolutely the sleep values from my Bangle right there in BangleJS!

Very nice! Thank you!

Qurstion. I'm using the nightly build. How did you get the data in the gadget bridge app?
As it doesn't seem to appear for me, Im using the nightly versions of the Bangle JS version.

[gfwilliams]

@xxxserxxx how did you get the sleep values?

That code should be in the nightly build, yes. However there is no code in Bangle.js to actually transmit the current activity to Gadgetbridge as far as I'm aware. The line of code is https://github.com/espruino/BangleApps/blob/master/apps/android/boot.js#L35 and right now that's not sending an act field.

[xxxserxxx]

I'm not getting any actual report that I slept, either on my watch in the SleepLog or in the app; however, I do see the activity & heartrate metrics in the sleep graph in banglejs. All I did was install nightly.

[32bitx64bit]

Sleeplog probably has to implement the sending of data. Hopefully that happens soonish

[ilar]

What would be needed to implement that functionality?

[gfwilliams]

I've recently added the ability for the firmware to report back basic activity (worn/not worn/walking/lots of exercise) in the cutting edge builds (and 2v26 when it's released), and I just pushed some code to the Android integration app (0.40 in development app loader) to forward this to Gadgetbridge.

I think the next steps would be to make the Sleep Log app change the activity field that's in the health data in new firmwares before it gets sent to Gadgetbridge:

• Ensure we add the health event listener for sleeplog right at the start. So change on to prependListener at https://github.com/espruino/BangleApps/blob/master/apps/sleeplog/boot.js#L28
• When the health event is handled, at the end of https://github.com/espruino/BangleApps/blob/master/apps/sleeplog/boot.js#L153-L177 - add something like:

if (data.status==3) data.activity="LIGHT_SLEEP";
if (data.status==4) data.activity="DEEP_SLEEP";

And I think that's literally all you need. There would be some extra stuff required to get it to sync to Gadgetbridge if your watch wasn't always connected to the phone, but I guess that's for later.

The reason I haven't done it yet really is I don't wear the watch at night and don't use sleeplog, so I have no way to test if it's actually working.

[storm64]

I'm not sure if I got something wrong, but as I understand the activity value, it has to be set on the beginning of an activity.
This might be problematic with the sleep status, as the sleep detection is recognizing the time with less movement and then back dating the beginning of the sleep interval.

Is it possible to send the activity (change) to GB with a status and a timestamp?

[gfwilliams]

I'm not sure if I got something wrong, but as I understand the activity value, it has to be set on the beginning of an activity.

Ok, well that could be a bigger issue then - maybe something we need to sort out in Gadgetbridge. Because unless the user manually enters it, there's no way we can look into the future and give Gadgetbridge an inferred activity value based on data we don't have.

At the end of the day, I feel like we send data every 10 minutes. It's not a massive problem if the activity shown has a timestamp that is 10 minutes wrong, as long as the total amounts of activity per day are ok.

Is it possible to send the activity (change) to GB with a status and a timestamp?

I believe so, yes.

But I think that's too much of a faff. In Gadgetbridge, if that really is the case, when we receive the act packet we should put in the step/hrm data for the given timestamp, and then update the previous data with the current activity we have.

[ilar]

I would be willing to test it, if you help me get set up with it?

[gfwilliams]

If you've got Gadgetbridge installed from the Play Store and up to date, and you have up to date apps from the development app loader (https://espruino.github.io/BangleApps/) it should now work

[32bitx64bit]

If you've got Gadgetbridge installed from the Play Store and up to date, and you have up to date apps from the development app loader (https://espruino.github.io/BangleApps/) it should now work

The development app loader thing doesn't seem to work for me.
Doesn't automatically connect, and when I tap manually to connect, gets stuck on "Getting device info..."
(The normal stable loader works, which has the latest updates, can confirm it does detect sleep in the app, buut any old data doesn't get sent, so if your watch happens to disconnect from your phone, you get in turn, nothing.)

[ilar]

Same for me, would it be because I'm using firefox?

[gfwilliams]

Does the stable app loader still work? Because they should be identical now - I updated it yesterday morning.

would it be because I'm using firefox?

Yes - firefox doesn't have any bluetooth support, so it won't work.

It's a point though - I think the original app loader code would have popped up a window to warn you, but I made some changes recently and now I don't think it will. I'll see about getting that added.

[32bitx64bit]

Does the stable app loader still work? Because they should be identical now - I updated it yesterday morning.

would it be because I'm using firefox?

Yes - firefox doesn't have any bluetooth support, so it won't work.

It's a point though - I think the original app loader code would have popped up a window to warn you, but I made some changes recently and now I don't think it will. I'll see about getting that added.

Yeah the normal app loader works, maybe its the gadget bridge nightly app that is causing some weird issue?

Also tested the sleeplog thing overnight. Sent like one deep sleep bit of data, the rest was just "Watch not worn"

[gfwilliams]

@32bitx64bit did you try sleeplog with the changes from #3275 (comment) ?

It looks like nobody's added that yet so the watch would just be setting activity data itself (presumably looking at lack of movement) rather that having sleeplog update it to what it thinks

[gfwilliams]

I've just put the code in to sleeplog - you'll have to upload with the development app loader. It'd be great if someone would test it though.

the normal app loader works, maybe its the gadget bridge nightly app

If you were running it from Gadgetbridge you need to ensure the URL has /android.html on the end - but maybe just try disconnecting from gadgetbridge and using the web version?

[32bitx64bit]

I've just put the code in to sleeplog - you'll have to upload with the development app loader. It'd be great if someone would test it though.

the normal app loader works, maybe its the gadget bridge nightly app

If you were running it from Gadgetbridge you need to ensure the URL has /android.html on the end - but maybe just try disconnecting from gadgetbridge and using the web version?

Adding android.html seems to have fixed the development loader.
As for sleeplog, I'll have to wait till tonight. As doesn't send old data. (yet)

[gfwilliams]

I'll have to wait till tonight. As doesn't send old data. (yet)

As long as all your code is up to date, next month (April) a new 'health' file will be created in a new format that is able to store the current activity, and that should now include any sleep tracking info that was put in by the sleeplog change I did today 👍

[32bitx64bit]

I'll have to wait till tonight. As doesn't send old data. (yet)

As long as all your code is up to date, next month (April) a new 'health' file will be created in a new format that is able to store the current activity, and that should now include any sleep tracking info that was put in by the sleeplog change I did today 👍

Gotcha 👍

[ilar]

I was also able to get it loaded onto my watch.

[32bitx64bit]

I'll have to wait till tonight. As doesn't send old data. (yet)

As long as all your code is up to date, next month (April) a new 'health' file will be created in a new format that is able to store the current activity, and that should now include any sleep tracking info that was put in by the sleeplog change I did today 👍

Can confirm, new sleep data is sent (As long as the watch is connected). Good work!

[gfwilliams]

Great! And come April (or if you deleted the March 'health' data file) it should be able to store the activity as well even when not connected