Add "hidden_by_default" configuration variable to entities

[jspiros]

Describe your core improvement

ESPHome entities support a disabled_by_default configuration variable that determines whether a client (usually Home Assistant) should require user interaction before the entity on the client side is enabled.

On the Home Assistant side, this relies on the entity_registry_enabled_default attribute. Home Assistant also has a neighboring entity_registry_visible_default attribute which allows an entity to be added and enabled, so that any controls are accessible from automations and data is recorded automatically, while still being "hidden" from the wider Home Assistant UI.

I propose adding a hidden_by_default configuration variable (to match the disabled_by_default precedent in naming it after the non-default state) in ESPHome to provide access to that functionality.

Especially for end-user/consumer devices that are built on ESPHome and are designed to be used with Home Assistant, this would allow those manufacturers and integrators to provide a more user-friendly out-of-the-box experience while still exposing valuable functionality by default.

Current limitations

Currently, there is no way to expose data sources or controls to Home Assistant that only make sense in automations or via the device info page without also cluttering the wider UI and potentially confusing users. Clearly, the entity_registry_visible_default attribute exists on the Home Assistant side, and is used by some native integrations, because sometimes this feature is precisely what's needed, so you need only search the Home Assistant code for entity_registry_visible_default to see existing examples of how/why it's used.

Technical benefits

It's not exactly a technical benefit, but it's a user-experience benefit and a new feature for developers, and I'm reporting this as a core functionality feature as the relevant code is in the core, and it would affect many different components that already offer the matching disabled_by_default configuration variable. (It also should be very easy to add for that reason, as it should end up implemented largely identically to how disabled_by_default is already implemented.)

Additional context

The existing disabled_by_default configuration variable in EntityBase:
https://github.com/esphome/esphome/blob/2adb99324227f5057a5ce2fbc4bbcbf3d6d242cd/esphome/core/entity_base.h#L42

entity_registry_enabled_default and entity_registry_visible_default in Home Assistant:
https://developers.home-assistant.io/docs/core/entity#registry-properties
https://github.com/home-assistant/core/blob/3b1b095b47aed28f195cb0c6ae5ae3d785f1943c/homeassistant/helpers/entity.py#L869
https://github.com/home-assistant/core/blob/3b1b095b47aed28f195cb0c6ae5ae3d785f1943c/homeassistant/helpers/entity.py#L881

I'm also requesting this on the Home Assistant side for the MQTT integration, for all of the same reasons, as the MQTT integration and ESPHome are both used by other products and projects that would surely benefit from the ability to refine the user experience with this feature.
https://github.com/orgs/home-assistant/discussions/750

[jesserockz]

I like this.

It also should be very easy to add for that reason

By all means, please implement it and put up some PRs =)

[jspiros]

I like this.

It also should be very easy to add for that reason

By all means, please implement it and put up some PRs =)

Great, thanks! I'll put it on my list of "as soon as I find some time".

I typically push my changes to any public fork as I go, and set up pull requests, with minimal delay. So if someone's reading this in the future and it's not done, and you're impatient and want to work on it yourself, please go ahead, it probably means I haven't found any time yet. I just want it, I don't care who implements it. 😄

@jesserockz Since you're with @OpenHomeFoundation, does this also suggest that PRs implementing my other proposal on the Home Assistant side regarding exposing this functionality via the MQTT integration as well (I mentioned it in the last line of additional context above) would be warmly received, or should I wait for someone to affirmatively respond over there too? For now, I'm assuming I should wait until I hear otherwise.