[SharedUX] Replacement solution for multiple UI components polling the server for updates

[tsullivan]

This topic comes up because of planned work to show banners in the UI that originate from the platform. How does the UI receive an update that there is a banner to show? The quick-and-dirty solution would be to create a component that polls the server for a signal to show the banner. This issue is to discuss solutions alternative solutions, because the quick-and-dirty solution is already used in many components throughout Kibana.

Where is polling used in Kibana (not an exhaustive list):

1. Kibana Reporting. When a user requests a report to be generated, a poller is triggered to look for changes in the status of the report job. If the status becomes complete or failed, a toast message appears.
2. Data plugin uses polling for fetching the status of async searches and search sessions
3. ES UI Plugin: consumers of useRequest can implement a polling integration

Many plugins also feature pages that list content, and use polling for auto-refreshing and keeping the list up-to-date: CCR, Enterprise Search, Fleet, ML, Security Solution, Snapshot Restore, Triggers Actions, Watcher, etc.

The Content Management project creates a new use case for this, as polling for changes in content attributes will also be a factor. The UI components for listing Saved Objects need up-to-date content that is annotated with CM attributes. Related: #153263

In this issue, I propose that SharedUX provide a client-side event bus for plugins to subscribe to. There would be an event router in place to send the appropriate response data to the individual subscribers. Plugins can register the queries and response formats with a subscriber, and deregister the subscriber at cleanup time. The benefit would be to minimize the number of connections from the browser to the Kibana webserver, and achieve consistency in handling timeouts, and implement request cancellation where it would be needed. The downside is that polling requests would be fired at a constant interval, even when there are no updates.

An alternative to the client-side poller with event bus would be to implement server push with WebSockets or Server-Sent Events. However, this solution would require more resources on the server. Events originate due to changes in the storage tier, which is Elasticsearch. Elasticsearch doesn't support publishing events to a message queue or have a pub/sub model, so we would need a layer (maybe Logstash) in between the Kibana server and Elasticsearch to handle the accumulation and publishing of events.

Let's use this issue to debate the pros and cons of polling vs a server push model. Participants, feel free to correct me on any part of my understanding.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[jloleysens]

Related #98881

[tsullivan]

@jloleysens Thanks for bringing that discussion in. I felt that issue was a little light on explaining the use cases where http2 / server push will help. There are reasons why I don't think that Server -> Client communications is a solution for a lot of our use cases, or one that can be generally suited to replace long polling in the client.

Polling in the client still seems like the best fit for most use cases in Kibana because Elasticsearch isn't an event-driven database. In other words, ES doesn't provide a way for subscribers to be notified when there is a change in data; it requires that clients poll the data through queries and track on their own if something has changed. If we're talking about using http2 to avoid polling in the browser generally, I don't think it will fundamentally help because we would just be talking about moving the burden of polling from the browser to the server.

Detailed concerns:

1. We still have the same concerns we have today with polling in the browser: how to batch the queries, handle timeouts, cancel previous pending requests, etc.
2. Having multiple Kibana server instances connected to the same cluster for load balancing creates complexity.
3. Knowing when there are no browser clients connected and when to stop polling on the server creates complexity.

Just now, I did a little research on the Elasticsearch enhancement roadmap, and found: elastic/elasticsearch#55358 and elastic/elasticsearch#1242. Having a Live Stream API or Changes API would alleviate my concerns and convince me that that server push could be a general solution to keeping state in Kibana UIs up-to-date. Let's give our thumbs-up on those issues!

[jloleysens]

that discussion

I have a suspicion that further progress on that issue will strongly overlap with the "shared bus" approach you were describing... Now, I'm not exactly sure when that progress will be occurring (9.0 👀 ) so it's good to hear you're also think about this problem.

The concrete reason we wanted to keep it open is for recent progress on AI assistants that may rely on SSE since use of SSE consumes 1 of our 6 concurrent request pool browser side (hence the shared bus idea).

Given that additional context and what you described, I'm not sure your shared-poller-bus idea should block on the investigations into SSE as it sounds like a more well-understood problem that helps reduce traffic without preventing SSE investigations.

ES doesn't provide a way for subscribers to be notified when there is a change in data...We still have the same concerns we have today with polling in the browser

That's a good point! I wasn't considering reacting to SO data changes.

One (smaller) benefit I thought of for driving state updates from the server is that we'll reduce ES's pollers from: nr connected browser > nr of connected Kibana nodes. If we can find a lighter connection model that works with loadbalancing. But you're totally correct that represents more engineering complexity for us than just polling.

did a little research on the Elasticsearch enhancement roadmap, and found: elastic/elasticsearch#55358 and elastic/elasticsearch#1242.

Those sound awesome but also pretty stale (one of them recently becoming a teenager 🥳 )

[tsullivan]

I think we can close this issue as there are no action items here. This issue has served its purpose as a discussion thread related to ongoing work at #98881