Temperature spikes occurring frequently

[louisaw123]

Describe the bug
Temperature spikes occurring in fridge monitoring sensors frequently. Extremely high temperatures occurring that range 60 degrees and above. Spike temps recorded over 3 months:

Spike temperature is not accurate as other sensors are also kept near the affected sensor but show normal readings. Examples:

To Reproduce
Unknown

Expected behavior
Temperature should be reading accurately otherwise may cause inaccurate data analysis. It also makes it difficult for user data reliability particularly for those monitoring without a backup sensor.

Additional context
The issue is not isolated to a single site and has been noticed in other sites too. Manufacturer has said that they have not come across this issue on their end and suspect it might be something caused on our end - have been monitoring own sensors with CCA and tempo but have not come across any spikes yet to confirm this.

[adamdewey]

Weird that they're all in a similar temp range...

[adamdewey]

Not had any spikes in the last 90 days for any of the sensors on the vaccine server 🤔

[adamdewey]

I wonder if it could be caused by older sensors?

I think client could have sensors that are 2+ years old, hmm

[jmbrunskill]

@adamdewey - This reminds me that we might want some kind of filter for this in notify? Should we ignore temperatures that are over or under certain limits? We don't want people getting alerts saying their fridge is 70 degrees we it's actually fine...

[adamdewey]

Hey @jmbrunskill - yeah I agree, wouldn't want to alert for those fake spikes.

I think that could be a good idea to implement even before hiding the spikes in CCA, Open mSupply etc as alerts would be highly visible.

[adamdewey]

@jmbrunskill - do we need a separate issue in the Notify repo, or can we link multiple PRs from different repos to this issue?

[louisaw123]

umm hold your horses... i don't think you should filter any temps? if its high it might be high for a legitimate reason.. i have no idea what the alert plans are but are they not going to get an alert only after having lets say 15-20 mins of high readings - rather than immediately? if that is the case then a spike will not affect anything...

[jmbrunskill]

umm hold your horses... i don't think you should filter any temps? if its high it might be high for a legitimate reason.. i have no idea what the alert plans are but are they not going to get an alert only after having lets say 15-20 mins of high readings - rather than immediately? if that is the case then a spike will not affect anything...

The current plan it to alert as soon as it goes high so someone can go and check why, e.g. if someone left the door open, or the power's off so they need to start a generator or something.
The idea is to allow someone to respond quickly before there is a breach which would be when the temperature had been too high or two low for a long time.

[louisaw123]

hmm.. what is the temp considered "breach" here because temps can jump above 8 pretty quickly and then jump back down like from opening a fridge door to transfer stock. sometimes having too many alerts might make people desensitised.

usually fridges have an alarm that beeps if it happened in the first instance and you would generally try and do something if you were already there...

just weighing up instant notification vs notifying after x mins option anyway

[adamdewey]

yeah good points Louisa...

Maybe we need to think about it some more - i.e. how can we help them to prevent a breach happening in the first place?

[adamdewey]

Hey I passed this msg on to Andrei last week but have not had a chance to catch up with u guys about it. Had a small sensor temp blip last week. We assume it's more related to the sensor? and also assumed bcos it was running with AA alkaline..? Although technically did not look into the cause in depth. Just thought I'd keep u guys updated too

That last one is more worrying as it's a realistic temperature so harder to filter out

[adamdewey]

Couple more spikes:

[louisaw123]

top one is a spike the bottom one is not - can see the datapoints gradually going up and back down

[adamdewey]

top one is a spike the bottom one is not - can see the datapoints gradually going up and back down

Ah good spotting LDP!

[andreievg]

Another one, from customer:

[adamdewey]

This spike occurred while using the new AA battery cases

[adamdewey]

For the client above spikes have been seen on 3 out of 19 sensors (so far)

For another client they've been seen on 4 out of 12 sensors

[adamdewey]

Temporary fix will be to mask high outliers (above 55 C) in both the dashboard and notification service.

Full fix is likely to be to go with BLE packet advertising - added to backlog

[marthakeezy]

Just noticed another spike (in case you guys need more cases)

[louisaw123]

20.4 degree spike

[louisaw123]

and 57.6

[louisaw123]

[adamdewey]

Check out this for some weird spike behaviour:

Multiple super high spikes in a row

[louisaw123]

Experiment with finding BM Spikes:

METHOD:

LOL im gonna forget this ill write it:

1. take sensor. install it into cca app
2. turn off cca app
3. after 3 weeks, open BM app and connect to each sensor , download logs
4. open cca app and sync sensors
5. Compare data from BM app and CCA to see if any difference

(Notes re method: the sensors i use were all previously connected to my test tablet already at home so i am just disconnecting it from this tablet and installing it into the cca app on a new tablet for this test)

---

RESULTS:

Sensor 1: EA56ABB1

Sensor 2: CFE0C3B5

Sensor 3: F143B0EC

---

OBSERVATIONS:

1. Note that temperatures on the CCA App are recording 10 minutes behind from BM app when comparing on spreadsheet. This is due to the tablet I am using having the incorrect time (it is 10 minutes behind from real time) and the BM data being pulled from my phone which is on real time. So if a time on the CCA app data says 20:15, it is technically meant to be 20:25.

2. the orange table shows data from blue maestro app while blue table shows data from cold chain app. i have aligned all records next to each other, they all appear fine except for the first record

3. for sensor 1 @ 20:25 -> sensor read at 15.6 on cca while on bm it reads at 6.6...... sensor did not leave fridge during this time and can see this in the bm app from the records before this. With sensor 1 there are also records from 7pm which could have been due to the sensor not syncing to the old tablet hence extra records sitting in there. interestingly these records are picked up on bm app but are not picked up on cca app?

4. sensor 2 and sensor 3 both have different temperatures on cca app vs bm app for the first reading. no other temps were different.

Have added the excel files for the rest of the records...
For BM app: sensor 1 recorded 4628 records, while sensor 2 and 3 recorded 4614 records.

F143B0EC_1710921030505.xls

EA56ABB1_1710921337407.xls

CFE0C3B5_1710921108745.xls

[adamdewey]

I tried to repro this (spike on first reading) with 6000 logs for 3 sensors with a 1 min log interval (~3 days of data) but couldn't get the same result unfortunately

[jmbrunskill]

I think I might have seen a temperature spike on the blue maestro app today wasn't fast enough to screenshot it though. Maybe I confused a humidity reading...

[adamdewey]

hmm, you mean it showed and then you think they masked it after?

Because the historical data should be accessible on the blue maestro app?

[jmbrunskill]

Nah I just saw it on the home page of their app. Can try downloading logs...

[adamdewey]

Ooh, that would be interesting if you did manage to catch one!

[adamdewey]

Double spike seen on a client just now:

[adamdewey]

@louisaw123 caught a spike using the new 0.5.7 version.

Upon investigation it appears that CCA is occasionally taking the humidity reading instead of the temperature reading (😱)

Spike noticed on dashboard at 27.7 °C:

Spike seen on CCA at 27.7 °C:

Sensor logs showing humidity reading of 27.7 °C:

Export of sensor logs:
C1671B0D_1730231726948_log.csv

Export of CCA logs:
C1671B0D64C9-30-10-2024-09-33-53.csv

Debug logs for CCA:
com.msupplycoldchain-latest.3.log
com.msupplycoldchain-2024-10-29.14.log
com.msupplycoldchain-2024-10-29.15.log
com.msupplycoldchain-2024-10-29.16.log
com.msupplycoldchain-2024-10-29.17.log

[adamdewey]

Captured a negative spike today, the spike doesn't exist on the sensor logs, only in CCA.

Possibly has taken the dewpoint reading instead of temperature, although the logs don't line up exactly:

E1185AAF693B-31-10-2024-10-11-51.csv
E1185AAF_1730322650968_log.csv

[kahn-msupply]

Another spike recorded. v0.5.6.