Occasional 500 errors / ajp_read_header: ajp_ilink_receive failed messages have started appearing

[opoplawski]

We are occasionally getting 500 error responses to CA queries on our IPA servers.

access_log:

10.30.20.100 - - [15/Apr/2024:12:56:52 -0700] "POST /ca/ocsp HTTP/1.1" 500 527 "-" "Microsoft-CryptoAPI/10.0"
10.20.0.37 - - [16/Apr/2024:10:00:01 -0700] "GET /ca/ee/ca/getCRL?op=getCRL&crlIssuingPoint=MasterCRL HTTP/1.1" 500 527 "-" "curl/7.76.1"

error_log:

[Mon Apr 15 12:57:52.497337 2024] [proxy_ajp:error] [pid 313979:tid 139718817662720] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
[Mon Apr 15 12:57:52.520477 2024] [proxy_ajp:error] [pid 313979:tid 139718817662720] [client 10.30.20.100:63548] AH00992: ajp_read_header: ajp_ilink_receive failed
[Mon Apr 15 12:57:52.520531 2024] [proxy_ajp:error] [pid 313979:tid 139718817662720] (70007)The timeout specified has expired: [client 10.30.20.100:63548] AH00878: read response failed from [::1]:8009 (localhost)
[Tue Apr 16 10:01:01.772286 2024] [proxy_ajp:error] [pid 313979:tid 139718876411648] (70007)The timeout specified has expired: AH01030: ajp_ilink_receive() can't receive header
[Tue Apr 16 10:01:01.777629 2024] [proxy_ajp:error] [pid 313979:tid 139718876411648] [client 10.20.0.37:58930] AH00992: ajp_read_header: ajp_ilink_receive failed
[Tue Apr 16 10:01:01.803097 2024] [proxy_ajp:error] [pid 313979:tid 139718876411648] (70007)The timeout specified has expired: [client 10.20.0.37:58930] AH00878: read response failed from [::1]:8009 (localhost)

pki-tomcat/ca/debug log:

2024-04-15 12:54:05 [Timer-0] INFO: SessionTimer: checking security domain sessions
2024-04-15 12:59:05 [Timer-0] INFO: SessionTimer: checking security domain sessions
2024-04-16 10:00:01 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-3] INFO: AAclAuthz: Granting read permission for certServer.ee.crl
2024-04-16 10:00:01 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-3] INFO: LDAPSession: reading cn=MasterCRL,ou=crlIssuingPoints,ou=ca,o=ipaca
2024-04-16 10:00:02 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-6] INFO: AAclAuthz: Granting read permission for certServer.ee.crl
2024-04-16 10:00:02 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-6] INFO: LDAPSession: reading cn=MasterCRL,ou=crlIssuingPoints,ou=ca,o=ipaca
2024-04-16 10:00:02 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO: AAclAuthz: Granting read permission for certServer.ee.crl
2024-04-16 10:00:02 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] INFO: LDAPSession: reading cn=MasterCRL,ou=crlIssuingPoints,ou=ca,o=ipaca
2024-04-16 10:00:02 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-4] SEVERE: An I/O error was encountered while outputting CRL results.

They are very infrequent (8 times over 6 servers in 4 weeks) so perhaps not a big deal, but it feels like I'm seeing more of them lately.

ipa-server-4.9.12-11.module+el8.9.0+20824+f2605038.x86_64