Get Entities node is extremely slow and cause Home assiatnt to disconnect

[smarthomeaf]

Describe the bug

A simple query with a single entity, takes 2 seconds, while a more complex one with multiple queries or using an "in group" takes up to 10 seconds and causing connectivity issues and tons of Home assistant Disconnects. I rolled back to .65 and these queries are instant. Not sure exactly what version this started at, but my guess is that this is what is causing the majority of the issues that are listed on GitHub (ie disconnect, crashes etc)

To Reproduce

a simple flow as outlined in the screenshot

Expected behavior

In older version (.65 tested) this was instant

Screenshots

Example Flow

[
    {
        "id": "dfd6fc951c155ad8",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "70151ef02612fa4f",
        "type": "inject",
        "z": "dfd6fc951c155ad8",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 280,
        "y": 380,
        "wires": [
            [
                "1f31c4b6275e9762"
            ]
        ]
    },
    {
        "id": "1f31c4b6275e9762",
        "type": "ha-get-entities",
        "z": "dfd6fc951c155ad8",
        "name": "",
        "server": "5ea56d96.083004",
        "version": 1,
        "rules": [
            {
                "condition": "state_object",
                "property": "entity_id",
                "logic": "is",
                "value": "device_tracker.unifi_default_00_50_56_9f_bc_22",
                "valueType": "str"
            }
        ],
        "outputType": "array",
        "outputEmptyResults": false,
        "outputLocationType": "msg",
        "outputLocation": "payload",
        "outputResultsCount": 1,
        "x": 630,
        "y": 380,
        "wires": [
            [
                "19dab93174b6898f"
            ]
        ]
    },
    {
        "id": "19dab93174b6898f",
        "type": "debug",
        "z": "dfd6fc951c155ad8",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 960,
        "y": 380,
        "wires": []
    },
    {
        "id": "5ea56d96.083004",
        "type": "server",
        "name": "Home Assistant",
        "addon": true
    }
]

Environment Information

The button not in my nodered isnat and there is no way to download the diagnostic flow from your https://zachowj.github.io/node-red-contrib-home-assistant-websocket/guide/diagnostics.html#overview

Additional context

No response

[zachowj]

How many entities does your Home Assistant instance have? You can find out by running this template in the Developer Tools > Template section:

{{ states | count }}

[smarthomeaf]

7768

I can see this being a high count, but the problem did not exist 0.65

[zachowj]

Try updating to the latest release, v0.74.2, and check if there's any improvement in the output speed.

[smarthomeaf]

Looks much better, not sure it was still as fast as before as i still see a bit of a delay, but at least it is not causing nodered to crash and disconnect. The other thing I noticed that that I can use inject nodes again in the Home Assistant UI rather than having to go directly connecting to Node Red. So thank you for that as well.

Can we keep this open for a few more days to see if the issue resurfaces

[N3rdix]

I had observed weak performance for get_entities as well, but already with small groups ~20 members which took around 1 second each...
Version 0.74.2 is a lot faster, back down to what feels like milliseconds 👍

[chrhen]

In my setup with 1600 entities 0.74.2 is faster than 0.73, but still far from the state before get-entities was significantly extended (0.65.1 for me).

For comparison see the CPU load of my the VM where I run HA with the Node-RED addon. Not much in the setup changed except the version of the Node-RED addon and consequently the version of the ha-websocket-contrib:

• Pre October 10: node-red-contrib-home-assistant-websocket 0.65.1 with around 8% of CPU load. Get Entities node had no noticeable performance impact iterating through 1600 entities and matching some by regex name and state every 5 seconds via inject node.
• October 10: Upgrade from ha-websocket-contrib from 0.65.1 to 0.67.2. Drastic change of CPU load to average of 25%. Unchanged node setup.
• October 30: Switch inject node to every 2 seconds, CPU load goes to 50 on same node setup.
• Late November 1: Upgrade to node-red-contrib-home-assistant-websocket 0.74.2. CPU load decreases to average of 15%, but still far from the previous 8% where the Node-RED-addon was almost no contributor to CPU load at all.

If I disable my flow that regularly calls several Get Entities nodes my CPU load is back to 8% levels.

For me the situation improved but it is still far from 0.65.1.

Note: More details on my setup in above mentioned issue.

[smarthomeaf]

I agree with @chrhen, it's acceptable, but def much slower than it used to be.

[smarthomeaf]

seems to be related, but I just noticed that action nodes with action on todo.get_items is super slow and sometimes cause nodered to crash as well.