Floating reputation 100-250 [wrn][hpc] Failed to enqueue comm msg.

[mayro1982]

Hello,

On some of my servers, the reputation keeps fluctuating between 100 and 250. What could be the issue? The hp.log file is filled with the following messages:
hp.6.log
hp.7.log
hp.8.log
hp.log

20241217 05:28:42.051 [inf][hpc] HotPocket 0.6.4
20241217 05:28:42.059 [inf][hpc] Role: Validator
20241217 05:28:42.059 [inf][hpc] Public key: edda1946dc9d4af7b43aa092ea1a812d90e26d84f6e458ec615d3c90a7970ac88a
20241217 05:28:42.059 [inf][hpc] Contract: 211a7f44-a86f-405a-91df-4ca201875e69 (1.0)
20241217 05:28:42.075 [inf][hpc] Initial state: 5652d393 | patch: 1d89604b
20241217 05:28:42.075 [inf][hpc] Hpfs cont server started.
20241217 05:28:42.075 [inf][hpc] Hpfs cont sync: Worker started.
20241217 05:28:42.116 [inf][hpc] Ledger primary:0-00000000 | raw:0-00000000
20241217 05:28:42.117 [inf][hpc] Hpfs ldgr server started.
20241217 05:28:42.117 [inf][hpc] Read request thread pool manager started.
20241217 05:28:42.117 [inf][hpc] Hpfs ldgr sync: Worker started.
20241217 05:28:42.117 [inf][hpc] Consensus processor started.
20241217 05:28:42.133 [inf][hpc] Started listening for peer connections on 22861
20241217 05:28:42.133 [inf][hpc] Started peer managing thread.
20241217 05:28:42.135 [inf][hpc] Started listening for user connections on 26201
20241217 05:28:47.525 [inf][hpc] Not enough peers proposing to perform consensus. votes:5 needed:33
20241217 05:28:49.311 [inf][hpc] Reported violation '2' from 81.103.13.204
20241217 05:28:52.206 [inf][hpc] Reported violation '1' from 162.249.169.18
20241217 05:28:57.520 [inf][hpc] Not enough peers proposing to perform consensus. votes:24 needed:33
20241217 05:28:58.774 [inf][hpc] Reported violation '1' from 81.103.13.204
20241217 05:29:07.526 [inf][hpc] Not enough peers proposing to perform consensus. votes:29 needed:33
20241217 05:29:20.019 [inf][hpc] Not enough peers proposing to perform consensus. votes:3 needed:33
20241217 05:29:30.020 [inf][hpc] Not enough peers proposing to perform consensus. votes:11 needed:33
20241217 05:29:40.019 [inf][hpc] Not enough peers proposing to perform consensus. votes:15 needed:33
20241217 05:29:55.023 [inf][hpc] Cannot close ledger. Possible fork condition. won:8 needed:33
20241217 05:29:58.384 [wrn][hpc] Became weakly connected.
20241217 05:30:05.022 [inf][hpc] Cannot close ledger. Possible fork condition. won:17 needed:33
20241217 05:30:15.025 [inf][hpc] Cannot close ledger. Possible fork condition. won:19 needed:33
20241217 05:30:25.025 [inf][hpc] Cannot close ledger. Possible fork condition. won:20 needed:33
20241217 05:30:35.026 [inf][hpc] Cannot close ledger. Possible fork condition. won:23 needed:33
20241217 05:30:45.020 [inf][hpc] Cannot close ledger. Possible fork condition. won:21 needed:33
20241217 05:30:55.019 [inf][hpc] Cannot close ledger. Possible fork condition. won:24 needed:33
20241217 05:30:57.901 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.903 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.904 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.904 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.904 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.904 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.904 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.905 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.905 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.905 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.905 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.906 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.906 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.906 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.907 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.907 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.907 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.907 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.908 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.908 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.908 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.908 [wrn][hpc] Failed to enqueue comm msg.
20241217 05:30:57.909 [wrn][hpc] Failed to enqueue comm msg.
.............................................
hp.2.log
hp.3.log
hp.4.log
hp.5.log

20241217 05:57:57.519 [inf][hpc] Not enough peers proposing to perform consensus. votes:21 needed:33
20241217 05:58:07.520 [inf][hpc] Not enough peers proposing to perform consensus. votes:22 needed:33
20241217 05:58:17.519 [inf][hpc] Not enough peers proposing to perform consensus. votes:22 needed:33
20241217 05:58:27.523 [inf][hpc] Not enough peers proposing to perform consensus. votes:22 needed:33
20241217 05:58:37.520 [inf][hpc] Not enough peers proposing to perform consensus. votes:22 needed:33
20241217 05:58:47.519 [inf][hpc] Not enough peers proposing to perform consensus. votes:22 needed:33

Such messages in the logs are observed on various servers with fluctuating reputations, and the total log size is around 50 MB. I reinstalled Ubuntu 20.04 with Ubuntu 24.04, but this did not solve the problem.
This problem started about a month ago. Before that, all my servers worked perfectly, and the reputation was always 250.
Server specifications: Ryzen 9 5950X CPU, 6 vCPUs, 10GB RAM, SSD 100GB DISK.
The internet on the servers works perfectly, with stable pings and high speed of 500 Mbps. Performance is excellent, with 99% of server resources free, and the disk system speed is around 500 MB/s.

evernode.6x9FCtxRb.log

[du1ana]

have you observed these logs continuously over an extended period, or were they only noticed within a specific 1 or 2 hour timeframe? also what xahaud server(s) are you using?

[mayro1982]

I have 8 servers, 5 are running stably, and 3 have fluctuating reputation between 100-250. All servers operate on xahaud wss://wss.ripplered.info. I just checked the logs on other servers, even those with a reputation of 240, and a similar error is present.
Judging by the logs, I suspect that the network is somewhat unstable, with many connection errors, disconnects, and other inconsistencies.

Here is the log from a server with a reputation of 252 and a similar error:

hp.log
rw.stdout.log
rw.stderr.log

ChatGPT:

The error is that the system is unable to gather enough votes to perform consensus (process transactions or make decisions about blocks). In each log entry, it says "Not enough peers proposing to perform consensus," meaning there are not enough participants (peers) proposing to carry out the consensus. The required number of votes for consensus is 33, but fewer votes are received (e.g., 14 or 16).

Other issues mentioned in the logs include:

"Cannot close ledger. Possible fork condition" — a possible issue with a fork (split in the blockchain).
"Failed to enqueue comm msg" — an error when trying to enqueue a message.
This may indicate network issues or insufficient activity from other peers in the network. It is important to ensure that enough peers are connected and functioning correctly.

[mayro1982]

Here is another server with a reputation of 122:

Execution lcl 1-2e12751484bd39e6a95a38181825d5470f37b02d67875b94c1e33bf16ea3b08c
Killed
Execution lcl 2-5789a4fb9d38e410323468ff55ef4dd95805ec81b5546b2bb753387346a38fca
Killed
Execution lcl 3-8c33d67b0ebdf1c085522f3fb5a70a2788ceb526087f89e64032db6b00d12df7
Killed
Execution lcl 5-9a78d9ed6d1a52dc42a0d55e33c897b6b1024727ba9369d077657eb4f5cda587
[GPUDPClient] wss://n20.1g13.deluxenode.xyz:41960 - Max timeout 15000 reached
Max timeout 15000 reached
[GPUDPClient] wss://n20.1g13.deluxenode.xyz:41961 - Max timeout 15000 reached
Max timeout 15000 reached
Killed
Execution lcl 6-14971ea0b3ec718d8e050b163d095f7bf41cac5f3d1fbe7fd2913e0290010333
[GPUDPClient] wss://n20.1g13.deluxenode.xyz:41960 - Max timeout 15000 reached
Max timeout 15000 reached
[GPUDPClient] wss://n20.1g13.deluxenode.xyz:41961 - Max timeout 15000 reached
Max timeout 15000 reached
[GPTCPClient] wss://n453.bisdaknode451-500.ovh:46100 - Max timeout 15000 reached
Max timeout 15000 reached
[GPTCPClient] wss://n453.bisdaknode451-500.ovh:46101 - Max timeout 15000 reached
Max timeout 15000 reached
Killed
Execution lcl 7-d42ee965d68df0c6c1aeed23f8a5867734fe4cb5f339d0d238c96ccf9b108de2
[GPUDPClient] wss://n20.1g13.deluxenode.xyz:41960 - Max timeout 15000 reached
Max timeout 15000 reached
[GPUDPClient] wss://n20.1g13.deluxenode.xyz:41961 - Max timeout 15000 reached
Max timeout 15000 reached
Killed

20241217 08:28:41.521 [inf][hpc] HotPocket 0.6.4
20241217 08:28:41.521 [inf][hpc] Role: Validator
20241217 08:28:41.521 [inf][hpc] Public key: ed666cef8a09fc340c05e38af74ada5e59b6ee7b395f1879e5eb55471ceba73039
20241217 08:28:41.521 [inf][hpc] Contract: 08e99409-5ce0-4c96-bac1-75ce492e55ba (1.0)
20241217 08:28:41.551 [inf][hpc] Initial state: 5652d393 | patch: 74c5a20b
20241217 08:28:41.552 [inf][hpc] Hpfs cont sync: Worker started.
20241217 08:28:41.552 [inf][hpc] Hpfs cont server started.
20241217 08:28:41.555 [inf][hpc] Ledger primary:0-00000000 | raw:0-00000000
20241217 08:28:41.555 [inf][hpc] Hpfs ldgr sync: Worker started.
20241217 08:28:41.555 [inf][hpc] Hpfs ldgr server started.
20241217 08:28:41.555 [inf][hpc] Consensus processor started.
20241217 08:28:41.556 [inf][hpc] Read request thread pool manager started.
20241217 08:28:41.562 [inf][hpc] Started listening for peer connections on 22861
20241217 08:28:41.562 [inf][hpc] Started peer managing thread.
20241217 08:28:41.565 [inf][hpc] Started listening for user connections on 26201
20241217 08:28:50.849 [inf][hpc] Not enough peers proposing to perform consensus. votes:14 needed:33
20241217 08:28:51.194 [inf][hpc] Reported violation '1' from 142.44.137.219
20241217 08:29:00.852 [inf][hpc] Not enough peers proposing to perform consensus. votes:24 needed:33
20241217 08:29:10.850 [inf][hpc] Not enough peers proposing to perform consensus. votes:27 needed:33
20241217 08:29:20.848 [inf][hpc] Not enough peers proposing to perform consensus. votes:31 needed:33
20241217 08:29:33.348 [inf][hpc] Not enough peers proposing to perform consensus. votes:16 needed:33
20241217 08:29:48.349 [inf][hpc] Cannot close ledger. Possible fork condition. won:30 needed:33
20241217 08:29:58.407 [inf][hpc] Ledger created (lcl:1-2e127514 state:5652d393 patch:74c5a20b)
20241217 08:30:07.947 [wrn][hpc] Became weakly connected.
20241217 08:32:15.859 [inf][hpc] Not enough peers proposing to perform consensus. votes:14 needed:33
20241217 08:32:16.869 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.955 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.955 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.956 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.956 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.956 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.957 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.957 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.958 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.960 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.960 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.960 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.960 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.961 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.961 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.962 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.962 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.963 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.964 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:16.964 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.051 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.051 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.051 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.052 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.052 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.052 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.053 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.053 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.053 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.053 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.053 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.054 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.054 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.054 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.055 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.055 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.056 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.056 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.056 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.056 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.056 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.056 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.057 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.057 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.057 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.057 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.057 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.058 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.058 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.058 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.058 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.059 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.059 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.060 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.061 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.061 [wrn][hpc] Failed to enqueue comm msg.
20241217 08:32:17.061 [wrn][hpc] Failed to enqueue comm msg.

[kithminisg]

Are you using LetsEncrypt SSL Certificates for those hosts?

[mayro1982]

Are you using LetsEncrypt SSL Certificates for those hosts?

Two days ago, I reinstalled all servers from Ubuntu 20.04 to 24.04, but the problem was not resolved. All Let's Encrypt SSL certificates are valid. There were similar errors on Ubuntu 20.04.

[evernodeau]

I have the same issue