Remove total sensors

[llamafilm]

I found my way here after having trouble setting up IotaWatt in Home Assistant. After a lot of reading, I've come to the conclusion that the Accumulated sensors in Home Assistant should never be used. They cause wildly inaccurate data in the HA energy dashboard. According to Bob, IotaWatt's integration feature makes this obsolete because:

• Integrators use 5 second granularity, much better than anything that can be done in Home Assistant
• These queries put excessive/unnecessary load on the IotaWatt

The Accumulated sensors in HA are known here as _total. So I suggest removing all these _total sensors, essentially reverting 4262ab2. If there is no dissent, I'll work on a PR for this and also for the HA integration.

[F13]

This repository hasn't seen any activity in a while. I would love to see improvements made to this library and the HA integration (or at least live somewhere where issues and PRs aren't ignored). Even if the official process is slow, forking the custom integration would at least allow people to have reliable IoTaWatt data in HA.

I'm happy to help as I can.

[llamafilm]

@F13 The HA integration was moved into core, so I believe that other repo is obsolete.

[F13]

@llamafilm That's what I'm saying - even if changes to core will be slow (since they're using this library, and getting core to switch to a new fork might take time), users can still benefit from updates if there is a custom integration they can use.

[F13]

Best case scenario in my mind is to fork this repo, deal with some of the issues, and get HA core to use the forked repo. But if that's not doable on a realistic time frame, a custom integration that uses the new library would let users (myself included) have a nearly-first-class installation experience through HACS.

[llamafilm]

Oh I see what you mean, yeah this repo seems abandoned. I would prefer to focus on improving core, not a custom component.
@gtdiehl @jyavenard @agners if you don't have time to maintain this, would you consider adding other collaborators?

[F13]

Already asked and ignored in #18 :) My point is, don't wait too long for a response. The best way forward is probably a fork and trying to get core to move to said fork.

[agners]

I found my way here after having trouble setting up IotaWatt in Home Assistant. After a lot of reading, I've come to the conclusion that the Accumulated sensors in Home Assistant should never be used.

Agreed, I came to the same conclusion, see home-assistant/core#65143 (comment).

The Accumulated sensors in HA are known here as _total. So I suggest removing all these _total sensors, essentially reverting 4262ab2. If there is no dissent, I'll work on a PR for this and also for the HA integration.

I think the opposite is true: The accumulated do not use what is known here as _total. They poll the API and sum the value for each period, which is inaccurate since a period might get missed for various reasons (e.g. at restart). Looking at the total is like looking at a good old power meter: It moves continously, and if you miss a read period the change will just be visible in the next period...

[agners]

@gtdiehl @jyavenard @agners if you don't have time to maintain this, would you consider adding other collaborators?

I don't have rights in this repository, but having IotAWatt integration relying on an unmaintained lib is not in my interested either 😅

I probably can create a fork of this in the home-assistant-libs org.

@llamafilm That's what I'm saying - even if changes to core will be slow (since they're using this library, and getting core to switch to a new fork might take time)

Since the fork will largely be the same, and in that case in a trusted org, I think it should be fairly quick thing to move Core to use the fork.

[llamafilm]

Oops, I thought you had rights because I saw some commits by you. I didn't know HA had an org for this but that sounds like the right place for this to live. I'll give Greg some time to respond to my email and then we can transfer ownership to your new repo according to PEP 541.

[agners]

I'll give Greg some time to respond to my email and then we can transfer ownership to your new repo according to PEP 541.

So we would be able to publish new releases under the same PyPI project?

My intention was to rename by pre- or postfixing the project name with home-assistant, but using the same project name would be nicer.

[jyavenard]

the _total sensors should be removed, and the library that output accumulated sensor should be modified too.

I added those when the integration sensors didn't exist yet. _total is no longer needed, should be using the integration sensors instead.

I've wanted to make that change for a while, but lack of free time always got in the way I'm sorry.

[llamafilm]

So we would be able to publish new releases under the same PyPI project?

Yes exactly. That's what happens when a project is abandoned by the owner.

@jyavenard no need to apologize, I know everyone is busy! Do you agree with moving this repo under the HA organization so that more people can contribute?

[jyavenard]

It's not my repo, so sure. having to get my changes pushed upstream and they officially released was a pain anyway

[agners]

@jyavenard we definitely agree on removing the accumulated sensor. This should be done in Core first. Do you have time to create a PR which does that?

[jyavenard]

I'll work on it this week-end.

[agners]

I've created the fork at https://github.com/home-assistant-libs/iotawattpy at created a 0.1.0 release (tagged the last commit, which seems to be what is on PyPI currently).

I'd like the first release to be as thin as possible, so we can easily switch to it from Core side. E.g. just addressing the NoneType issue which I see in my installation too (#16).

@llamafilm when do you think we can publish the first release on PyPI from that fork?

[llamafilm]

Before requesting ownership transfer on PyPi, we need to make some improvements on the new fork. I guess that None issue would be enough to start. I think we should wait at least a week for Greg to respond by email.

[jyavenard]

I found my way here after having trouble setting up IotaWatt in Home Assistant. After a lot of reading, I've come to the conclusion that the Accumulated sensors in Home Assistant should never be used. They cause wildly inaccurate data in the HA energy dashboard. According to Bob, IotaWatt's integration feature makes this obsolete because:

not quite.

accumulated (energy) sensor are perfectly fine to use when you want to get the energy used by a particular device.

[llamafilm]

@jyavenard what do you mean? I removed the accumulated sensors in my own fork and the regular Wh sensor is working fine. At any moment, it shows how much energy this device has used since midnight. Here's an example of my spa. There is one small bug — the sensor resets to zero 8 seconds after midnight, as shown in the second, zoomed in graph I don't know why it's late but I don't think it's a big deal.

[agners]

The new repository is up and running, I've also created a discussion section (which this thread probably should be continue at): home-assistant-libs#2.

Before requesting ownership transfer on PyPi, we need to make some improvements on the new fork. I guess that None issue would be enough to start. I think we should wait at least a week for Greg to respond by email.

I agree, the question is how we fix that exactly. Let's move discussion to the new home (specifically home-assistant-libs#1 in this case).

[agners]

As we all agree that the Accumulated sensors should not be used I went ahead and removed them in Home Assistant Core, see home-assistant/core#86611.

[jay-oswald]

How are you doing the energy dashboard now? Removing accumulated values has broken my energy dashboard, the release notes say to use integrators, but I can't get integrators from the IoTaWatt to work with Home Assistant? They do not appear under the integration after adding them and reloading the integration

[llamafilm]

@jay-oswald In HA, outputs and integrators do not appear in the UI to be associated with the IotaWatt device because they lack Unique IDs. I have a PR to fix that: home-assistant/core#86834

You should still be able to see these Wh entities and choose them for the energy dashboard.

[jyavenard]

That they do not appear has nothing to do with the lack of unique_id.
You can (or at least should) use the energy sensor.

[jyavenard]

This is showing the integration "export" ; no unique id

[jay-oswald]

This is showing the integration "export" ; no unique id

When I first set up iotawatt last year, each sensor had the sensor.name which showed Watts, then sensor.name_wh which showed a daily total, and then the sensor.name_wh_accumulated which showed an all time total. So wouldn't using the sensor.name_wh version be the daily resetting one, which the HA energy dashboard is not meant to work with? And its also in WH, now kwh, which I also had issues with and couldnt get to work previousally

[jyavenard]

So wouldn't using the sensor.name_wh version be the daily resetting one, which the HA energy dashboard is not meant to work with? And its also in WH, now kwh, which I also had issues with and couldnt get to work previousally

No. The sensors ending with _wh are energy sensors with a last_reset attribute. HA knows that the value got reset and will continue from there.

For an energy sensor to show in the energy setup it needs 3 attributes:
state_class to be either total or total_increasing
a last_reset with a date
a device_class that is energy

[F13]

More technically, @jyavenard, sensors do not need a last_reset if their state_class is total_increasing. When HA observes a significant (>10%) value reduction in these sensors, it assumes a counter reset.