Mixed deployment

[cdprete]

Hello.

In the company where I work we've several applications deployed on VMs and/or bare metal machines.
In particular, there are always (at least) two instances running in different availability zones per application.

Given that Spring Boot Admin is a Spring Boot application, I was thinking to deploy it in OpenShift with 2 instances up and running, as well, in different availability zones.

It's clear to me, by reading the documentation, that you're able - through Spring Cloud - to discover all the clients deployed in a k8s-like cluster but what about when the clients would need to discover the servers (2 in this case) in order to find out which URL to use?
The idea is that the clients register themselves to both the servers, so that the monitoring keeps up even if one of the 2 goes down.

I think to need something in the direction of https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/#spring-cloud-kubernetes-discoveryserver in the client applications, but how does this (and if at all) connect with Spring Boot Admin Client?

I can't find any information about this use-case in the documentation and your help would be, therefore, appreciated.

[erikpetzold]

hi @cdprete

in general, there are 2 different ways for connecting admin servers with monitored apps:

1. Server discoveres apps (via spring cloud)
2. Apps register themselfes on server

For 1) there are again different ways how to handle discovery in k8s, as there are multiple client implementations. The spring-cloud-kubernetes-discoveryserver can be used if you don't want to give too many of you pods access to the k8s api, but in the end it is also just a service registry (which would be used by admin server to discover apps). We have runnable examples for all k8s variants here: https://github.com/codecentric/spring-boot-admin-runtime-playground/tree/main/kubernetes

Option 2) is an alternative to discovery. The apps must use the client library and know the server urls. You can provide multiple server urls. See http://docs.spring-boot-admin.com/3.4.2/docs/client/
There is also a property spring.boot.admin.client.register-once to decide if apps should register on one or all servers, see docs for details.

When you run multiple admin servers, you can also connect them to a cluster, so they share their state, see http://docs.spring-boot-admin.com/3.4.2/docs/server/server#clustering

Hope this helps to sort your requirements, if you have more specific questions feel free to ask.

[cdprete]

Hello @erikpetzold.

Option 2) is an alternative to discovery. The apps must use the client library and know the server urls.

How can clients know the URLs where to connect to if the server instances are just managed by a Deployment and exposed through a Service? The clients may use the URL of the Service, but then only one of the servers would receive the registration of the client X and not both.

There is also a property spring.boot.admin.client.register-once to decide if apps should register on one or all servers, see docs for details.

Isn't this property just to instruct the client to send the registration once and (e.g.: similar to the Eureka) - and it's independent on the number of servers which are configured - or have I misunderstood the documenation?

For 1) there are again different ways how to handle discovery in k8s, as there are multiple client implementations. The spring-cloud-kubernetes-discoveryserver can be used if you don't want to give too many of you pods access to the k8s api, but in the end it is also just a service registry (which would be used by admin server to discover apps). We have runnable examples for all k8s variants here: https://github.com/codecentric/spring-boot-admin-runtime-playground/tree/main/kubernetes

I'm failing to find a suitable example on the above.
If I would be just be using the discoveryserver by myself then I would need to know the internals of Spring Boot Admin and its API in order replicate it rather than leveraging what the client library already provides out of the box.

The spring-cloud-kubernetes-discoveryserver can be used if you don't want to give too many of you pods access to the k8s api

The point is more that my applications are deployed on-premise while Spring Boot Admin will be deployed on OpenShift.
Therefore, my applications would need to use the k8s API as well as the Spring Boot Admin API together (k8s API: find the hostname and port of the Spring Boot Admin servers; sba-client API: find the path to the REST resources to use).
Actually, the real blocker it's just the registration call since it's the one initiated by the clients and by their need to know the server URL(s) in advance. Once the clients are registered the server(s) should be able to grab their statuses with no issues.

[erikpetzold]

Ok, I think now I understand the problem.

• You have Spring Boot Admin running in k8s and monitored apps outside k8s.
• Are threre also monitored services inside k8s?
• SBA cannot find the apps as they are not available for discovery (they are unknown to k8s).
• Apps cannot find SBA as they don't know how to access pods inside k8s.

I think the cleanest solution would be to use a service registry (like eureka), make the apps register themselfes there and use the discovery mechanism inside SBA Server to let the server discover the apps via the registry.

If you want to stick to the "app registers at the SBA Server" pattern, then that should also work.

• make sure the server is reachable from the outside, you probably need an ingress or something similar
• that would mean that the load balancer decides which of your admin instances receives the registration request
• as registration can be done again and again, the apps would probably register at both instances over time
• but deregistration is happening only once, so that would reach only one server
• therefore the clustering mechanism could be used to sync both admin instances

[cdprete]

I think the cleanest solution would be to use a service registry (like eureka), make the apps register themselfes there and use the discovery mechanism inside SBA Server to let the server discover the apps via the registry.

Ideally yes, but we also have applications in Perl, PHP, C++,... for which we would like to leverage the same monitoring solution.
My idea for those is to build a small agent in Spring Boot that would expose then their health status and whatever else is needed.
Having to build and maintain then also one for Eureka or any other service registry it would be quite costly.

as registration can be done again and again, the apps would probably register at both instances over time

Why is it so?
From the apps point of view, only one server would then be configured (the URL to the ingress), so how could they register twice?

• if the server goes down, the other one will pick up the querying for the information due to the clustering
• if the app goes down, the server should detect it (because the poll for the information will fail) and somehow unregister it
  • on the next registration, the broken instance would/should then be replaced in the server

Of course, there is still the edge case that an app goes down and comes back up in the interval between one poll by the server and the next one. But, even in this case, for the server should still be transparent (ignore the registration) because - according to the last state it has about the app - this is already registered.

[erikpetzold]

Ideally yes, but we also have applications in Perl, PHP, C++,... for which we would like to leverage the same monitoring solution.
My idea for those is to build a small agent in Spring Boot that would expose then their health status and whatever else is needed.
Having to build and maintain then also one for Eureka or any other service registry it would be quite costly.

Does it make a big difference if your agent is registering at Spring Boot Admin or at eureka? Should be the same effort.

• Agent registers at eureka (integration provided by spring, no Spring Boot Admin Client needed) (does not need to be eureka, other supported registries would also work)
• Admin server queries the registry (via spring cloud)
• Admin Server polls the actuator endpoints

That would be the more generic solution in contrast to

• Agent registers at Admin Server via SBA Client (but which one? How to configure?)
• Admin Server polls the actuator endpoints

From the apps point of view, only one server would then be configured (the URL to the ingress), so how could they register twice?

The app will register again and again by default (because it may be possible that the Admin Server got restartet and lost its state). So if your ingress distributes the requests to different SBA Server instances, all of them will get the registration requests over time.
But this also means, that all server instances poll your application. While this should work, I still think that registry or clustering approaches would be better.

[cdprete]

Does it make a big difference if your agent is registering at Spring Boot Admin or at eureka? Should be the same effort.

I wouldn't say so if you don't have a service registry at all to begin with.
I need then to maintain and keep running two applications: the service registry and the SBA server.
Or am I missing something from your message/suggestion?

Of course, there is also the approach that the SBA server can, at the same time, be also a service registry.
In that case, SBA server would query localhost as registry.
Given the internals of SBA server which are, for me, unknown so far, would it work in your opinion?

I still think that registry or clustering approaches would be better.

In agree, in general and from a design point of view.