integrated dual HomeMatic+homematicIP support for HmIP-RFUSB

[jens-maus]

This PR updates the generic_raw_uart kernel module package to its latest version.

This newer version of generic_raw_uart integrates support for the HmIP-RFUSB line of homematic RF USB sticks, which essentially is necessary to utilize the new 4.4.x dualcoprocessor firmware coming with the latest RaspberryMatic Add-on and which significantly enhances the capabilities of the HmIP-RFUSB to communicate and being compatible not only to homematicIP devices but also to the older BidCos-RF/HomeMatic devices by eQ3/ELV. In addition, this newer HmIP-RFUSB firmware also integrates support to link a HmIP-HAP or even HmIPW-DRAP homematicIP-Wired Gateway, which previously was only possible when using the GPIO-based RPI-RF-MOD rf module.

[alexreinert]

I'm not sure, but I see chances, that this PRwill break the old Homematic OCCU Addon as the new generic-raw-uart modules will be used for the HmIP-RFUSB sticks instead of the CP210x kernel modules.

[jens-maus]

I'm not sure, but I see chances, that this PRwill break the old Homematic OCCU Addon as the new generic-raw-uart modules will be used for the HmIP-RFUSB sticks instead of the CP210x kernel modules.

Ah yeah, you are right. I forgot about that in my PR description. This will be indeed the case. However, adding a simple modprobe cp210x followed by the new_id echo as in the previous udev rules should solve that in a new version of the old HomeMatic OCCU HA-Addon. Or the addon can be quite easily changed to also use /dev/raw-uart instead of /dev/ttyUSB0.

But then, what purpose does this old and largely outdated HomeMatic OCCU Addon serve these days anyway? AFAIK it is largely outdated, not up to date and largely limited in functionality anyway. And IMHO the bottom line should be, that HAos should not force the modprobe of cp210x proactively itself anyway but let the addons - which require a certain hardware - decide to either modprobe cp210x manually or use /dev/raw-uart.

[alexreinert]

It is not that easy: After the raw-uart device is created, a modprobe of the cp210x and the echo to the new_id will not claim the stick to a ttyUSB device, insteadit will remain as raw-uart device.
Of course it should be not a problem to use the raw-uart dev node in the OCCU addon, but for that, the user needs to change that by his own. I'm not aware how the rules for breaking changes in Home Assistant are, but I think at least a hint in the release notes should be done.

[agners]

that HAos should not force the modprobe of cp210x proactively itself anyway but let the addons - which require a certain hardware - decide to either modprobe cp210x manually or use /dev/raw-uart.

We don't do anything special TBH, whatever PID/VID the cp210x driver has registered will be claimed automatically by the kernel/modprobe infrastructure. And I don't think we should interfere with that.

I see, we have the kernel udev rule.

But then, what purpose does this old and largely outdated HomeMatic OCCU Addon serve these days anyway? AFAIK it is largely outdated, not up to date and largely limited in functionality anyway.

I am not familiar with the HomeMatic OCCU Add-on, but it sounds that maybe phase it out would make sense? From analytics it seems that there is still a fairly large user base:

curl https://analytics.home-assistant.io/addons.json | jq  .core_homematic
{
  "total": 251,
  "versions": {
    "11.3.0": 231,
    "11.2.2": 17,
    "11.0.4": 1,
    "9.3": 1
  },
  "protected": 251,
  "auto_update": 78
}

@pvizeli do you have thoughts on that?

[jens-maus]

And even if you don't want to phase put the old OCCU addon, please keep in mind that users only have to change their config from /dev/ttyUSB0 to /dev/raw-uart after integration of this PR and their installations will still continue to work.

However I still think it is fair to really think about phasing out the old addon in favour of joining forces on the RaspberryMatic add-on.

[pvizeli]

However I still think it is fair to really think about phasing out the old addon in favour of joining forces on the RaspberryMatic add-on. Yes. Is there a migration path?

By side of that, I think this should be fine.

[jens-maus]

Is there a migration path?

AFAIK there isn't a real migration path besides creating a normal *.sbk backup using the WebUI and then reimporting thus backup. However, I have never tried that myself since I never really used the old HomeMatic CCU add-on myself.

By side of that, I think this should be fine.

Great! Would then be really nice if this PR can be merge rather sooner than later and integrated into the next rel-7 release so that the RaspberryMatic add-on can then use the new dual copro firmware for the HmIP-RFUSB.

[pvizeli]

I don't think the backup function will work. The idea of the add-on was not to run a full CCU, just having some things like zwave.

[jens-maus]

Ok, then we probably would just need to document the migration path from old addon to the RaspberryMatic addon by stating that a re-teaching of all homematicIP devices would be required?

[agners]

Or the addon can be quite easily changed to also use /dev/raw-uart instead of /dev/ttyUSB0.

Is the driver binding to the device automatically or is there some user space help needed? Will /dev/raw-uart be available in the Add-on?

I guess that would mean we break HmIP-RFUSB users, they will have to change hmip -> device setting manually to /dev/raw-uart.

[jens-maus]

Or the addon can be quite easily changed to also use /dev/raw-uart instead of /dev/ttyUSB0.

Is the driver binding to the device automatically or is there some user space help needed? Will /dev/raw-uart be available in the Add-on?

/dev/raw-uart should be available right from plugging in the RFUSB and no user-space tool is required to init&access this dev node.

I guess that would mean we break HmIP-RFUSB users, they will have to change hmip -> device setting manually to /dev/raw-uart.

Indeed. However, if you could generate a dev build with this PR integrated I could more easily test this.

[agners]

What I essentially wonder is, if we would merge this PR, can the user then use /dev/raw-uart? E.g. supported by the subsystem=tty filter? https://github.com/home-assistant/addons/blob/master/homematic/config.yaml#L36

I guess that depends on the device type/major number of /dev/raw-uart.

[agners]

Indeed. However, if you could generate a dev build with this PR integrated I could more easily test this.

We can do this now with the test build label 🚀

[pvizeli]

Ok, then we probably would just need to document the migration path from old addon to the RaspberryMatic addon by stating that a re-teaching of all homematicIP devices would be required?

I think we can maybe generate the tarfile manually in the structure which we can import into your addon? So we release a version of the add-on that only downloads that file for the migration

[alexreinert]

A filter subsystem=tty will not work, as the driver creates no tty device because the additional tty stuff is not needed for the HmIP-RFUSB and the dev major of raw-uart is dynamic.

[jens-maus]

I think we can maybe generate the tarfile manually in the structure which we can import into your addon? So we release a version of the add-on that only downloads that file for the migration

That's exactly what I have in mind once the dev test build is ready so that I can test that procedure and give instructions on how to migrate from the old addon to the new one. My hope is, that I can work out a small backup procedure that will generate a compatible backup file (*.sbk) which can then be directly restored in the RaspberryMatic add-on.

[agners]

@jens-maus it seems that ODROID C4 failed, probably due to a race condition. But the rest is read and available:
https://os-builds.home-assistant.io/8.0.dev2021122901683/

[jens-maus]

@jens-maus it seems that ODROID C4 failed, probably due to a race condition. But the rest is read and available: https://os-builds.home-assistant.io/8.0.dev2021122901683/

Thx. However, how can I update an already running haos install and provide it a raucb file for updating to it? Is there some CLI magic I don't know? At least I didn't find something on the net yet.

[jens-maus]

Ok, I did a fresh 8.0.dev2021122901683 ova install in my Proxmox environment now and voila, I could verify that this PR perfectly works and immediately shows up a connected HmIP-RFUSB device with /dev/raw-uart within HAos. In addition, the RaspberryMatic Add-on perfectly shows up and works with traditional BidCos-RF (HomeMatic) and HmIP (homematicIP) teach-in mode. See:

Unfortunately @agners has released a new 7.1 HAos version just a few hours ago as I hoped to see this PR integrated before the next release. However, I will then see how we can document a migration route for the old/obsolete HomeMatic Add-on ASAP.

[agners]

I could verify that this PR perfectly works and immediately shows up a connected HmIP-RFUSB device with /dev/raw-uart within HAos.

Good! But that was to be expected since the RaspberryMatic CCU Add-on has more permissions.

I was more wondering if HomeMatic CCU can be used with /dev/raw-uart still as a stopgap. But as Alex already mentioned, it won't work, at least not with the current configuration.

Unfortunately @agners has released a new 7.1 HAos version just a few hours ago as I hoped to see this PR integrated before the next release. However, I will then see how we can document a migration route for the old/obsolete HomeMatic Add-on ASAP.

Sorry, I did not want to hold up 7.1 any longer. We can make 7.2 pretty soon once we feel things are ready.

[jens-maus]

However, I will then see how we can document a migration route for the old/obsolete HomeMatic Add-on ASAP.

Ok, I had a quick look over the current state of affairs regarding the old Homematic add-on. As @alexreinert pointed out before the /dev/raw-uart device is not belonging to the tty subsystem. Thus, it is currently not possible to simply replace /dev/ttyUSB0 with /dev/raw-uart in the user config of the add-on. So we either have to remove the subsystem filter from the config.json to allow to specify /dev/raw-uart instead. However, I am still unsure if this is all worth the trouble if we really consider throwing away the old Homematic add-on anyway?!?

Regarding a potential migration route from the old Homematic add-on to the newer RaspberryMatic add-on, I also had a quick look over the current data storage within the old add-on. In principle it should be indeed possible to get the WebUI based backup routines in the old add-on to export all the data in a somewhat compatible *.sbk backup format so that it can restored/imported by the RaspberryMatic add-on. But also here, I am still unsure if this is all worth the trouble (it would require some dedicated CCU WebUI patching just for that purpose) to work on such a migration patch for the old add-on, also given that it would be hard to test all this to really get a reliable data export routine which actually works as expected. So what do you @agners and @pvizeli think about that? Given the still limited number of users actually using this old add-on, do you really think it to be mandatory to actually retire the old add-on in favour of promoting the new one?

[alexreinert]

Maybe it would be an option to change the subsystem filter to "raw-uart"?

[agners]

The statistics above say that 251 users are using the Add-on (granted, only some are using HmIP-RFUSB, but that number is also just those who opt-in to send statistics). Just merging this and hence cause people's home break when they upgrade the OS is not nice IMHO.

At a bare minimum, I think we should mark the old Add-on as deprecated or similar and write a blog post. And then we should also wait for a while to give people the chance to migrate to the new Add-on. Ideally we want people using the new Add-on before breaking the old.

If we can come up with a solution which allows to continue using the old Add-on with a minimal change, then I think we can push out this change much quicker, and people can move to the new Add-on over a longer period of time.

Maybe it would be an option to change the subsystem filter to "raw-uart"?

Afaik, currently dynamic devices are not really supported by the subsystem filter. Using them requires the Add-on to disable the protection mode (there is that open issue home-assistant/supervisor#2690). Ideally we should get that sorted, then add raw-uart to the subsystem filter.

[jens-maus]

The statistics above say that 251 users are using the Add-on (granted, only some are using HmIP-RFUSB, but that number is also just those who opt-in to send statistics). Just merging this and hence cause people's home break when they upgrade the OS is not nice IMHO.

At a bare minimum, I think we should mark the old Add-on as deprecated or similar and write a blog post. And then we should also wait for a while to give people the chance to migrate to the new Add-on. Ideally we want people using the new Add-on before breaking the old.

Well, but this "wait period" could be of course quite a very long road. As it stands, the RaspberryMatic add-on really requires this PR for full HomeMatic+homematicIP support with the HmIP-RFUSB, so that I can move on. So holding back this PR just because we are afraid to introduce a regression is IMHO also not the smartest way given that the potential user numbers waiting for this functionality to appear in the RaspberryMatic add-on will also definitley raise over time and might be easily higher than then current old add-on user numbers.

If we can come up with a solution which allows to continue using the old Add-on with a minimal change, then I think we can push out this change much quicker, and people can move to the new Add-on over a longer period of time.

Can't we simply remove the whole subsystem filter in the config.json of the old addon so that users can freely set whatever device node they think might fit? Or would that introduce a requirement to remove the protection mode as you mentioned? If so, that should still be IMHO a valid compromise for the old add-on to continue to work after this PR has been merged so that users that can't migrate immediately are at least able to change their configuration from /dev/ttyUSB0 to /dev/raw-uart (which can be introduced with a warning note in the ChangeLog of the old add-on which should be visible when upgrading the add-on). Furthermore, we can then present the user the suggested deprecated warning message and I can (as another compromise) work on some minor migration path even for that few hundred users so that they can more easily export their current ccu config and re-import it into the new Raspberrymatic add-on WebUI.

Maybe it would be an option to change the subsystem filter to "raw-uart"?

Afaik, currently dynamic devices are not really supported by the subsystem filter. Using them requires the Add-on to disable the protection mode (there is that open issue home-assistant/supervisor#2690). Ideally we should get that sorted, then add raw-uart to the subsystem filter.

Besides this, finally solving home-assistant/supervisor#2690 would be great for the RaspberryMatic add-on alone as well. More and more users are coming up, complaining or raising security questions regarding the necessity to remove the protection mode (cf. jens-maus/RaspberryMatic#1642) and if solving this issue would remove the necessity to remove the protection mode would be indeed great for the new add-on as well. Nevertheless, I still think we shouldn't wait for this issue to be solved first, but perhaps simply remove the subsystem filter from the old add-on and instruct users via the changelog to change their device node to /dev/raw-uart once they moved to a newer HAos version (and potentially remove protection mode in that case). What do you think?

[pvizeli]

Yeah, fixing dynamically loaded kernel modules and updating cgroup dynamic is high on my list as the os-agent supports it.

I think a blogpost + describes a migration path with deprecating the old add-on looks like the path forward. @jens-maus is it possible to generate a tar file with the device paring data that can be imported without patching your add-on? Repairing all devices is quite a high pain if you have a lot of relays and other devices that are not directly exposed.

[jens-maus]

Yeah, fixing dynamically loaded kernel modules and updating cgroup dynamic is high on my list as the os-agent supports it.

This is great news! When do you think you will have that implemented so that the RaspberryMatic add-on can be used without having to disable protection mode. And at the same spot: What about the still open issue regarding missing multicast udp routing (cf. home-assistant/plugin-multicast#17 + jens-maus/RaspberryMatic#1373)? Because this also severely limits the current functionality of the RaspberryMatic add-on. If advice, I could at least help in that regard.

I think a blogpost + describes a migration path with deprecating the old add-on looks like the path forward. @jens-maus is it possible to generate a tar file with the device paring data that can be imported without patching your add-on? Repairing all devices is quite a high pain if you have a lot of relays and other devices that are not directly exposed.

Well, a straight tar file of the /usr/local path (which is used by a real CCU as well as RaspberryMatic) is not possible due to the way the old Homematic CCU add-on was designed/developed. But, if it would help, I could invest some time here to either create a shell script which will collect all the necessary files in the right order and then create a *.sbk file which can be directly restored/imported into the RaspberryMatic Add-on WebUI. And perhaps I can also add that routine to the WebUI of the old add-on for a more smoother migration. And of course, if you want I could also assist in the blogpost and documentation around that migration path.

But still, I think for a first step it would be great to get the config.json of the old add-on modified so that the subsystem filter is removed, thus users could then change /dev/ttyUSB0 to /dev/raw-uart themselves as a first migration step.

[jens-maus]

@pvizeli @agners
Please note home-assistant/addons#2352 which is my proposal for getting the old homematic add-on retired in favor of the RaspberryMatic add-on like we discussed above.

Please note that I only very briefly tested the backup export routines I implemented in the old homematic add-on. So help would be indeed appreciated from users actually using the old homematic add-on with some devices.

Looking forward to get this PR implemented so that we can proceed with this PR here!

[agners]

Now that the add-on is updated I'll have another look at this tomorrow and merge it so we have some dev builds. We then can backport it that it makes it into 7.3.

[jens-maus]

Now that the add-on is updated I'll have another look at this tomorrow and merge it so we have some dev builds. We then can backport it that it makes it into 7.3.

Just a short notice: With the latest 99.0.2 version of the old "Homematic CCU" add-on binding the HmIP-RFUSB via raw-uart seems to work now. Thus, the add-on migration process should be done now like we discussed. So please see if you can merge this PR ASAP and also backport it to 7.3 and generate some dev build of it so that I and others can test it accordingly. But using your last 8.0dev version, I could perfectly verify that this PR works fine.

[jens-maus]

@agners Thx for the quick merge. So the only thing missing is the os-agent improvements @pvizeli promised to work on so that I can potentially remove the necessity to disable protection mode (thus, remove full_access from the add-on config) of the RaspberryMatic Add-on.