pki.server.healthcheck.certs.trustflags.CASystemCertTrustFlagCheck is slow and triggers ipa-healthcheck timeout

[flo-renaud]

With the PKI version shipped in the copr repo @pki/master (dogtag-pki-base-11.9.0~alpha1^20251107211925.325851d0-1.fc42.noarch), the check pki.server.healthcheck.certs.trustflags.CASystemCertTrustFlagCheck is slower than before and triggers a timeout in ipa-healthcheck.

Example of failing runs in
https://$ARTIFACTS_SERVER/idm-ci/freeipa_upstream_nightly/Nightly-latest-pki/master/2025-11-08_11-19/pki/test_ipahealthcheck/2/

2025-11-10T09:28:48+0000 [ipatests.pytest_ipa.integration.host.Host.master.cmd110] DEBUG RUN ['ipa-healthcheck', '--output-type', 'human', '--failures-only']
2025-11-10T09:29:20+0000 [ipatests.pytest_ipa.integration.host.Host.master.cmd110] DEBUG ERROR: pki.server.healthcheck.certs.trustflags.CASystemCertTrustFlagCheck: Request timed out
2025-11-10T09:29:20+0000 [ipatests.pytest_ipa.integration.host.Host.master.cmd110] DEBUG Exit code: 1

We can see the checks took more than 10s.

I tried manually and also see a long time for CASystemCertTrustFlagCheck:

# time ipa-healthcheck --source pki.server.healthcheck.certs.trustflags --check CASystemCertTrustFlagCheck
[
  {
    "source": "pki.server.healthcheck.certs.trustflags",
    "check": "CASystemCertTrustFlagCheck",
    "result": "ERROR",
    "uuid": "acdd3f10-9376-474d-9b63-65c0177f3e23",
    "when": "20251110143423Z",
    "duration": "10.000795",
    "kw": {
      "key": "audit_signing",
      "nssdbDir": "/var/lib/pki/pki-tomcat/conf/alias",
      "msg": "Unable to load cert from NSSDB: Request timed out"
    }
  }
]

real	0m12.035s
user	0m14.452s
sys	0m1.116s

Running ipa-healthcheck --debug, we can see that the failing command is

# ipa-healthcheck --debug  --source pki.server.healthcheck.certs.trustflags --check CASystemCertTrustFlagCheck
...
NSSDatabase.get_trust(auditSigningCert cert-pki-ca)
Command: pki -d /var/lib/pki/pki-tomcat/conf/alias -f /var/lib/pki/pki-tomcat/conf/password.conf nss-cert-show auditSigningCert cert-pki-ca
stdout:
b'  Nickname: auditSigningCert cert-pki-ca\n  Serial Number: 0x1dff4bfe3484aa29617700327fa797b4\n  Subject DN: CN=CA Audit,O=IPA.TEST\n  Issuer DN: CN=Certificate Authority,O=IPA.TEST\n  Not Valid Before: Mon Nov 10 09:14:07 EST 2025\n  Not Valid After: Sun Oct 31 09:14:07 EDT 2027\n  Trust Flags: u,u,Pu\n'
NSSDatabase.get_cert_info(auditSigningCert cert-pki-ca) ends
NSSDatabase.get_trust(auditSigningCert cert-pki-ca)
Command: pki -d /var/lib/pki/pki-tomcat/conf/alias -f /var/lib/pki/pki-tomcat/conf/password.conf nss-cert-show auditSigningCert cert-pki-ca
Unable to load cert from NSSDB: Request timed out
[
  {
    "source": "pki.server.healthcheck.certs.trustflags",
    "check": "CASystemCertTrustFlagCheck",
    "result": "ERROR",
    "uuid": "581a76f6-7a2b-42bf-8886-6c7c1baa5f84",
    "when": "20251110144605Z",
    "duration": "10.000600",
    "kw": {
      "key": "audit_signing",
      "nssdbDir": "/var/lib/pki/pki-tomcat/conf/alias",
      "msg": "Unable to load cert from NSSDB: Request timed out"
    }
  }
]

Running the command manually succeeds:

# pki -d /var/lib/pki/pki-tomcat/conf/alias -f /var/lib/pki/pki-tomcat/conf/password.conf nss-cert-show 'auditSigningCert cert-pki-ca'
  Nickname: auditSigningCert cert-pki-ca
  Serial Number: 0x1dff4bfe3484aa29617700327fa797b4
  Subject DN: CN=CA Audit,O=IPA.TEST
  Issuer DN: CN=Certificate Authority,O=IPA.TEST
  Not Valid Before: Mon Nov 10 09:14:07 EST 2025
  Not Valid After: Sun Oct 31 09:14:07 EDT 2027
  Trust Flags: u,u,Pu

so it looks like the failure is just related to the total time taken to execute the checks. Increasing ipa-healthcheck timeout in /etc/ipahealthcheck/ipahealthcheck.conf fixes the issue but it's unexpected that this command takes so much time. The default timeout for each check is 10s.