Implementing Dependency-Responsive Liveliness Probes in Frontend Pod

[ziemowit141]

Is your feature request related to a problem? Please describe.

We have identified a critical issue in our system architecture related to service interdependencies. Specifically, this problem arises when the matching pod encounters a failure, leading to a disruption in the connection. This disruption necessitates a manual restart of the frontend pod. The reason behind this is that the connections to dependent services are initialized during the startup phase of the frontend pod. Consequently, any interruption in these services post-launch results in the following error:

If that happens then we get this error:

Describe the solution you'd like

To mitigate this issue, we propose the implementation of advanced liveliness probes within the frontend pod. These probes will be responsible for continuously monitoring the health status of all dependent services. In the event of a service failure, these probes will automatically initiate a restart of the frontend pod, thereby restoring the system's operational status without manual intervention.

Additional context

I'm not sure about the root-cause of matching pod failing, if I establish that I will add those details to the issue.