Error while reading JSON-RPC during agent enrolment

[max0013]

Describe the bug

Hello,

When I install and configure an agent to speak with his satellite, I'm facing an issue with the CSR.
Ramdomly (sometimes it work on first try) the CSR is not waiting for validation on master, so the agent enrollment end but there is no communication between agent and satellite.
NB : Before writing here, I opened a topic on community forum : https://community.icinga.com/t/question-about-csr-auto-signing/14403
NB2 : I thought that the version 2.14.5 might fix my issue regarding this pull : #10337

Expected behavior

Enrollement succes with secure communication between agent and satellite.

To Reproduce

• API call to get ticket with CN=[agent_fqdn]
• Generate autosign certificate with :

icinga2 pki new-cert
    –cn [agent_fqdn]
    –key /var/lib/icinga2/certs/[agent_fqdn].key
    –cert /var/lib/icinga2/certs/[agent_fqdn].crt

• Ask for parent certificate

icinga2 pki save-cert
   –host [satellite_fqdn]
   –port 5665
   –key /var/lib/icinga2/certs/[agent_fqdn].key
   –cert /var/lib/icinga2/certs/[agent_fqdn].crt
   –trustedcert /var/lib/icinga2/certs/master.crt

• Node setup

 icinga2 node setup
    –zone [agent_fqdn]
    –endpoint [satellite_fqdn],[satellite_fqdn],5665
    –endpoint [satellite2_fqdn],[satellite_fqdn],5665
    –parent_host [satellite_fqdn],5665
    –parent_zone [satellite_zone]
    –cn agent_fqdn
    –accept-config
    –accept-commands
    –disable-confd
    –trustedcert /var/lib/icinga2/certs/master.crt
    –ticket [ticket_from_master_requested_before]

• Restart icinga2 service

Actual behavior

No visible error during the installation process but there is no communication between agent and satellite.
On GUI I can see :
Remote Icinga instance '[agent_fqdn]' is not connected to '[stallite_fqdn]'

On satellite log :
information/ApiListener: New client connection for identity '[agent_fqdn]' from [::ffff:xxx.xxx.xxx.xxx]:52882 (certificate validation failed: code 18: self-signed certificate)

I enabled debuglog feature on agent, satellites and master and I found those messages in logs :

On agent :

notice/JsonRpcConnection: Received 'icinga::Hello' message from identity '[satellite_fqdn_1]'.
debug/JsonRpcConnection: Error while reading JSON-RPC message for identity '[satellite_fqdn_1]': Error: End of file

Stacktrace:
 0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x00005AE3823B6241 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 2# icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 4# 0x00005AE3826540FF in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# 0x00005AE382691035 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# make_fcontext in /lib/x86_64-linux-gnu/libboost_context.so.1.74.0

warning/JsonRpcConnection: API client disconnected for identity '[satellite_fqdn_1]'
notice/JsonRpcConnection: Received 'icinga::Hello' message from identity '[satellite_fqdn_2]'.
debug/JsonRpcConnection: Processed JSON-RPC 'icinga::Hello' message for identity '[satellite_fqdn_2]' (took total 0ms).
notice/JsonRpcConnection: Error while reading JSON-RPC message for identity '[satellite_fqdn_2]': Error: End of file

Stacktrace:
 0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x00005AE3823B6241 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 2# icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 4# 0x00005AE3826540FF in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# 0x00005AE382691035 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# make_fcontext in /lib/x86_64-linux-gnu/libboost_context.so.1.74.0

notice/JsonRpcConnection: Disconnecting API client for identity '[satellite_fqdn_1]'

On satellites

notice/JsonRpcConnection: Error while reading JSON-RPC message for identity '[agent_fqdn]': Error: End of file [asio.misc:2]
notice/JsonRpcConnection: Error while reading JSON-RPC message for identity '[agent_fqdn]': Error: stream truncated [asio.ssl.stream:1]
debug/JsonRpcConnection: Error while reading JSON-RPC message for identity '[agent_fqdn]': Error: Operation canceled [system:125 at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:134 in function 'do_complete']

Your Environment

• Version used (icinga2 --version): r2.14.5-1
• Operating System and version:
  • Master : Linux Ubuntu 22.04
  • Satellites : Linux Ubuntu 24.04
  • Agents : Linux Ubuntu 22.04 / Ubuntu 24.04 / Debian 12 / Rocky Linux 9
• Enabled features (icinga2 feature list): api checker icingadb mainlog
• Icinga Web 2 version and modules (System - About):
  • Icinga Web 2 Version 2.12.2
  • Loaded Modules
  • setup 2.12.2
  • businessprocess 2.5.1
  • director 1.11.3
  • icingadb 1.1.3
  • incubator 0.22.0
  • netbox 3.6.1.2
  • teamdashboards 1.0.0
• Config validation (icinga2 daemon -C):

satellite 1

    [2025-02-18 17:02:05 +0100] information/cli: Icinga application loader (version: r2.14.5-1)
    [2025-02-18 17:02:05 +0100] information/cli: Loading configuration file(s).
    [2025-02-18 17:02:05 +0100] information/ConfigItem: Committing config item(s).
    [2025-02-18 17:02:05 +0100] information/ApiListener: My API identity: [satellit1_fqdn]
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 11 NotificationCommands.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 24 HostGroups.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 237 Hosts.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1 Downtime.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 2 Comments.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1 FileLogger.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 144 Zones.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 143 Endpoints.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1 ApiUser.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1 ApiListener.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 781 CheckCommands.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 5 TimePeriods.
    [2025-02-18 17:02:06 +0100] information/ConfigItem: Instantiated 1923 Services.
    [2025-02-18 17:02:06 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
    [2025-02-18 17:02:06 +0100] information/cli: Finished validating the configuration file(s).

satellite 2

    [2025-02-18 17:04:43 +0100] information/cli: Icinga application loader (version: r2.14.5-1)
    [2025-02-18 17:04:43 +0100] information/cli: Loading configuration file(s).
    [2025-02-18 17:04:43 +0100] information/ConfigItem: Committing config item(s).
    [2025-02-18 17:04:43 +0100] information/ApiListener: My API identity: [satellite2_fqdn]
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 11 NotificationCommands.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 24 HostGroups.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 237 Hosts.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1 Downtime.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 3 Comments.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1 FileLogger.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 144 Zones.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 143 Endpoints.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1 ApiUser.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1 ApiListener.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 781 CheckCommands.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 5 TimePeriods.
    [2025-02-18 17:04:44 +0100] information/ConfigItem: Instantiated 1923 Services.
    [2025-02-18 17:04:44 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
    [2025-02-18 17:04:44 +0100] information/cli: Finished validating the configuration file(s).

master1

    [2025-02-18 17:07:53 +0100] information/cli: Icinga application loader (version: r2.14.5-1)
    [2025-02-18 17:07:53 +0100] information/cli: Loading configuration file(s).
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Committing config item(s).
    [2025-02-18 17:07:54 +0100] information/ApiListener: My API identity: [master1_fqdn]
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 11 NotificationCommands.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 2439 Notifications.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 24 HostGroups.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 261 Hosts.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 1 Downtime.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 5 Comments.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 1 FileLogger.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 155 Zones.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 155 Endpoints.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 4 ApiUsers.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 2 Users.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 1 ApiListener.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 781 CheckCommands.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 5 TimePeriods.
    [2025-02-18 17:07:54 +0100] information/ConfigItem: Instantiated 2178 Services.
    [2025-02-18 17:07:54 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
    [2025-02-18 17:07:54 +0100] information/cli: Finished validating the configuration file(s).

master2

    [2025-02-18 17:13:15 +0100] information/cli: Icinga application loader (version: r2.14.5-1)
    [2025-02-18 17:13:15 +0100] information/cli: Loading configuration file(s).
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Committing config item(s).
    [2025-02-18 17:13:15 +0100] information/ApiListener: My API identity: [master2_fqdn]
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 11 NotificationCommands.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 7 HostGroups.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 1 FileLogger.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 3 Zones.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 2 Endpoints.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 1 ApiUser.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 1 ApiListener.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 760 CheckCommands.
    [2025-02-18 17:13:15 +0100] information/ConfigItem: Instantiated 5 TimePeriods.
    [2025-02-18 17:13:15 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
    [2025-02-18 17:13:15 +0100] information/cli: Finished validating the configuration file(s).

zones.conf

• master1 & 2

object Endpoint "[master1_fqdn]" {
        host = "[master1_fqdn]"
        port = "5665"
}

object Endpoint "[master2_fqdn]" {
        host = "[master2_fqdn]"
        port = "5665"
}

object Endpoint "[satellite1_fqdn]" {
        host = "[satellite1_fqdn]"
        port = "5665"
}

object Endpoint "[satellite2_fqdn]" {
        host = "[satellite2_fqdn]"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "[master2_fqdn]" ,"[master1_fqdn]" ]
}

object Zone "satellite-cust-marg01" {
        endpoints = [ "[satellite1_fqdn]","[satellite2_fqdn]" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

satellite1 :

object Endpoint "[master1_fqdn]" {
        host = "[master1_fqdn]"
        port = "5665"
}

object Endpoint "[master2_fqdn]" {
        host = "[master2_fqdn]"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "[master1_fqdn]","[master2_fqdn]" ]
}

object Endpoint "[satellite1_fqdn]" {
}

object Zone "satellite-cust-marg01" {
        endpoints = [ "[satellite1_fqdn]" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Satellite2

object Endpoint "[master1_fqdn]" {
        host = "[master1_fqdn]"
        port = "5665"
}

object Endpoint "[master2_fqdn]" {
        host = "[master2_fqdn]"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "[master1_fqdn]","[master2_fqdn]" ]
}

object Endpoint "[satellite2_fqdn]" {
}

object Zone "satellite-cust-marg01" {
        endpoints = [ "[satellite2_fqdn]" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

[yhabteab]

Hi, thanks for reporting!

so the agent enrollment end but there is no communication between agent and satellite.

What does no communication exactly mean? If the icinga2 node setup ... command finished without errors then your actual issue shouldn’t be related to the agent enrolment process. Can you please share the logs with timestamps from both of your satellites and your agent?

Apart from that, the zone configuration of your satellites does not seem to be consistent. It looks like this on your masters, ...

object Zone "satellite-cust-marg01" {
       endpoints = [ "[satellite1_fqdn]","[satellite2_fqdn]" ]
       parent = "master"
}

... but both satellites are configured in non-HA mode, i.e. they do not know one another.