`RecordPagedList: Error to get a new page` after upgrading from 11.5.1 to 11.6

[haerdib]

After upgrading from dogtag v 11.5.1 to v 11.6 / 11.7 we're experiencing the following error upon a submitting a ca-cert-request;

Error:

com.netscape.certsrv.base.PKIException: Unable to create enrollment request: RecordPagedList: Error to get a new page
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at com.netscape.certsrv.client.PKIClient.handleErrorResponse(PKIClient.java:254)
        at com.netscape.certsrv.client.PKIClient.getEntity(PKIClient.java:285)
        at com.netscape.certsrv.client.PKIClient.post(PKIClient.java:393)
        at com.netscape.certsrv.client.Client.post(Client.java:136)
        at com.netscape.certsrv.ca.CACertRequestClient.enrollRequest(CACertRequestClient.java:57)
        at com.netscape.certsrv.ca.CACertClient.enrollRequest(CACertClient.java:119)
        at com.netscape.cmstools.ca.CACertRequestSubmitCLI.execute(CACertRequestSubmitCLI.java:379)
        at org.dogtagpki.cli.CommandCLI.execute(CommandCLI.java:58)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at com.netscape.cmstools.cli.SubsystemCLI.execute(SubsystemCLI.java:79)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:694)
        at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:733)

We have not changed anything on the LDAP server side, our own queries still produce the expected results. Do you have any idea where we could start debugging?

Log on the LDAP server side:

[17/Nov/2025:11:03:06.515451670 +0100] conn=235 op=23 SRCH base="ou=People,o=pki-tls-client-CA,dc=test,dc=de" scope=2 filter="(description=2;3527;CN=Testing SECNET TLS-Client CA,O=TEST AG,C=DE;CN=PKI Administrator,E=caadmin@test.de,OU=pki-tls-client,O=test.de Security Domain)" attrs=ALL
[17/Nov/2025:11:03:06.515664939 +0100] conn=235 op=23 RESULT err=0 tag=101 nentries=1 wtime=0.000169096 optime=0.000214711 etime=0.000381483
[17/Nov/2025:11:03:06.517308778 +0100] conn=235 op=24 SRCH base="uid=caadmin,ou=People,o=pki-tls-client-CA,dc=test,dc=de" scope=0 filter="(objectClass=*)" attrs=ALL
[17/Nov/2025:11:03:06.517419866 +0100] conn=235 op=24 RESULT err=0 tag=101 nentries=1 wtime=0.000097883 optime=0.000112029 etime=0.000207768
[17/Nov/2025:11:03:06.518919796 +0100] conn=235 op=25 SRCH base="ou=Groups,o=pki-tls-client-CA,dc=test,dc=de" scope=1 filter="(&(objectClass=groupofuniquenames)(uniqueMember=uid=caadmin,ou=people,o=pki-tls-client-CA,dc=test,dc=de))" attrs="cn description"
[17/Nov/2025:11:03:06.519400214 +0100] conn=235 op=25 RESULT err=0 tag=101 nentries=9 wtime=0.000089557 optime=0.000481719 etime=0.000569613
[17/Nov/2025:11:03:06.633568237 +0100] conn=233 op=42 SRCH base="ou=ca,ou=requests,o=pki-tls-client-CA,dc=test,dc=de" scope=1 filter="(&(requestId<=0810000000)(requestState=*))" attrs=ALL
[17/Nov/2025:11:03:06.637207874 +0100] conn=233 op=42 SORT -requestId (*)
[17/Nov/2025:11:03:06.637287773 +0100] conn=233 op=42 RESULT err=53 tag=101 nentries=0 wtime=0.000102111 optime=0.003720598 etime=0.003820665 notes=P details="Paged Search" pr_idx=1 pr_cookie=-1

Full debug output:

DEBUG: Command: ca-cert-request-submit
DEBUG: Command: /usr/lib/jvm/jre-17-openjdk/bin/java -cp /usr/share/pki/lib/* -Dcom.redhat.fips=false -Djava.util.logging.config.file=/usr/share/pki/etc/logging.properties com.netscape.cmstools.cli.MainCLI -C secret.txt -n caadmin_tls-client -U https://localhost:8943 --skip-revocation-check --debug ca-cert-request-submit --profile caTlsClientCert --csr-file webservice.csr
INFO: Server URL: https://localhost:8943
INFO: Loading NSS password from secret.txt
INFO: NSS database: /root/.dogtag/nssdb
FINE: Message format: null
FINE: Command: ca-cert-request-submit --profile caTlsClientCert --csr-file webservice.csr
FINE: Module: ca
FINE: Initializing NSS
FINE: Logging into internal token
FINE: Using internal token
INFO: Connecting to https://localhost:8943
INFO: PKIConnection: Socket factory: org.dogtagpki.client.JSSSocketFactory
INFO: HTTP request: GET /pki/rest/info HTTP/1.1
FINE: - Host: localhost:8943
FINE: - Connection: Keep-Alive
FINE: - User-Agent: Apache-HttpClient/4.5.13 (Java/17.0.17)
FINE: - Accept-Encoding: gzip,deflate
FINE: Request:

INFO: Creating SSL socket with existing socket
INFO: Client certificate: caadmin_tls-client
INFO: Server certificate:
INFO: - subject:[..]
INFO: - issuer: [..]
WARNING: BAD_CERT_DOMAIN encountered on [..]' indicates a common-name mismatch
INFO: HTTP response: HTTP/1.1 200
FINE: - Set-Cookie: JSESSIONID=8654B25632BE712F607C992899464F80; Path=/pki; Secure; HttpOnly
FINE: - Content-Type: application/json
FINE: - Content-Length: 50
FINE: - Date: Mon, 17 Nov 2025 06:52:39 GMT
FINE: - Keep-Alive: timeout=300
FINE: - Connection: keep-alive
FINE: Response:
{"Version":"11.2.1","Attributes":{"Attribute":[]}}
INFO: Server Name: null
INFO: Server Version: 11.2.1
INFO: HTTP request: GET /ca/rest/account/login HTTP/1.1
FINE: - Host: localhost:8943
FINE: - Connection: Keep-Alive
FINE: - User-Agent: Apache-HttpClient/4.5.13 (Java/17.0.17)
FINE: - Accept-Encoding: gzip,deflate
FINE: Request:

INFO: HTTP response: HTTP/1.1 200
FINE: - Cache-Control: private
FINE: - Set-Cookie: JSESSIONID=6C9CFCD0089B7937449EC029F5E1B10D; Path=/ca; Secure; HttpOnly
FINE: - Content-Type: application/json
FINE: - Content-Length: 374
FINE: - Date: Mon, 17 Nov 2025 06:52:39 GMT
FINE: - Keep-Alive: timeout=300
FINE: - Connection: keep-alive
FINE: Response:
{"id":"caadmin","FullName":"caadmin","Email":"[...]","Roles":["Administrators","Certificate Manager Agents","Enterprise CA Administrators","Enterprise KRA Administrators","Enterprise OCSP Administrators","Enterprise RA Administrators","Enterprise TKS Administrators","Enterprise TPS Administrators","Security Domain Administrators"],"Attributes":{"Attribute":[]}}
INFO: Account:
INFO: - ID: caadmin
INFO: - Full Name: caadmin
INFO: - Email: [...]
INFO: Roles:
INFO: - Administrators
INFO: - Certificate Manager Agents
INFO: - Enterprise CA Administrators
INFO: - Enterprise KRA Administrators
INFO: - Enterprise OCSP Administrators
INFO: - Enterprise RA Administrators
INFO: - Enterprise TKS Administrators
INFO: - Enterprise TPS Administrators
INFO: - Security Domain Administrators
FINE: Module: cert
FINE: Module: request
FINE: Module: submit
INFO: Retrieving caTlsClientCert profile
INFO: HTTP request: GET /ca/rest/certrequests/profiles/caTlsClientCert HTTP/1.1
FINE: - Host: localhost:8943
FINE: - Connection: Keep-Alive
FINE: - User-Agent: Apache-HttpClient/4.5.13 (Java/17.0.17)
FINE: - Cookie: JSESSIONID=6C9CFCD0089B7937449EC029F5E1B10D
FINE: - Accept-Encoding: gzip,deflate
FINE: Request:

INFO: HTTP response: HTTP/1.1 200
FINE: - Content-Type: application/json
FINE: - Content-Length: 864
FINE: - Date: Mon, 17 Nov 2025 06:52:39 GMT
FINE: - Keep-Alive: timeout=300
FINE: - Connection: keep-alive
FINE: Response:
{"ProfileID":"caTlsClientCert","Renewal":false,"RemoteHost":"","RemoteAddress":"","Input":[{"id":"i1","ClassID":"certReqInputImpl","Name":"Certificate Request Input","ConfigAttribute":[],"Attribute":[{"name":"cert_request_type","Value":"","Descriptor":{"Syntax":"cert_request_type","Description":"Certificate Request Type"}},{"name":"cert_request","Value":"","Descriptor":{"Syntax":"cert_request","Description":"Certificate Request"}}]},{"id":"i2","ClassID":"genericInputImpl","Name":"Generic Input","ConfigAttribute":[],"Attribute":[{"name":"remote_user","Value":"","Descriptor":{"Syntax":"string","Description":"remote_user"}},{"name":"client_ip","Value":"","Descriptor":{"Syntax":"string","Description":"client_ip"}},{"name":"pki_component","Value":"","Descriptor":{"Syntax":"string","Description":"pki_component"}}]}],"Output":[],"Attributes":{"Attribute":[]}}
INFO: Request type: pkcs10
FINE: CSR:
-----BEGIN CERTIFICATE REQUEST-----
[...]
-----END CERTIFICATE REQUEST-----

INFO: Request:
com.netscape.certsrv.cert.CertEnrollmentRequest@9441aa39
INFO: DNS names: null
INFO: Requestor: null
INFO: HTTP request: POST /ca/rest/certrequests HTTP/1.1
FINE: - Content-Length: 1899
FINE: - Content-Type: application/json
FINE: - Host: localhost:8943
FINE: - Connection: Keep-Alive
FINE: - User-Agent: Apache-HttpClient/4.5.13 (Java/17.0.17)
FINE: - Cookie: JSESSIONID=6C9CFCD0089B7937449EC029F5E1B10D
FINE: - Accept-Encoding: gzip,deflate
FINE: Request:
{
  "ProfileID" : "caTlsClientCert",
  "Renewal" : false,
  "RemoteHost" : "",
  "RemoteAddress" : "",
  "Input" : [ {
    "id" : "i1",
    "ClassID" : "certReqInputImpl",
    "Name" : "Certificate Request Input",
    "ConfigAttribute" : [ ],
    "Attribute" : [ {
      "name" : "cert_request_type",
      "Value" : "pkcs10",
      "Descriptor" : {
        "Syntax" : "cert_request_type",
        "Description" : "Certificate Request Type"
      }
    }, {
      "name" : "cert_request",
      "Value" : "-----BEGIN CERTIFICATE REQUEST-----[..]-----END CERTIFICATE REQUEST-----\n",
      "Descriptor" : {
        "Syntax" : "cert_request",
        "Description" : "Certificate Request"
      }
    } ]
  }, {
    "id" : "i2",
    "ClassID" : "genericInputImpl",
    "Name" : "Generic Input",
    "ConfigAttribute" : [ ],
    "Attribute" : [ {
      "name" : "remote_user",
      "Value" : "",
      "Descriptor" : {
        "Syntax" : "string",
        "Description" : "remote_user"
      }
    }, {
      "name" : "client_ip",
      "Value" : "",
      "Descriptor" : {
        "Syntax" : "string",
        "Description" : "client_ip"
      }
    }, {
      "name" : "pki_component",
      "Value" : "",
      "Descriptor" : {
        "Syntax" : "string",
        "Description" : "pki_component"
      }
    } ]
  } ],
  "Output" : [ ],
  "Attributes" : {
    "Attribute" : [ ]
  }
}
SEVERE: WARNING: SSL alert received: CLOSE_NOTIFY
INFO: HTTP response: HTTP/1.1 500
FINE: - Content-Type: application/xml;charset=UTF-8
FINE: - Content-Length: 292
FINE: - Date: Mon, 17 Nov 2025 06:52:39 GMT
FINE: - Connection: close
SEVERE: WARNING: SSL alert sent: CLOSE_NOTIFY
FINE: Response:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<PKIException>
    <ClassName>com.netscape.certsrv.base.PKIException</ClassName>
    <Attributes/>
    <Code>500</Code>
    <Message>Unable to create enrollment request: RecordPagedList: Error to get a new page</Message>
</PKIException>

com.netscape.certsrv.base.PKIException: Unable to create enrollment request: RecordPagedList: Error to get a new page
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at com.netscape.certsrv.client.PKIClient.handleErrorResponse(PKIClient.java:254)
        at com.netscape.certsrv.client.PKIClient.getEntity(PKIClient.java:285)
        at com.netscape.certsrv.client.PKIClient.post(PKIClient.java:393)
        at com.netscape.certsrv.client.Client.post(Client.java:136)
        at com.netscape.certsrv.ca.CACertRequestClient.enrollRequest(CACertRequestClient.java:57)
        at com.netscape.certsrv.ca.CACertClient.enrollRequest(CACertClient.java:119)
        at com.netscape.cmstools.ca.CACertRequestSubmitCLI.execute(CACertRequestSubmitCLI.java:379)
        at org.dogtagpki.cli.CommandCLI.execute(CommandCLI.java:58)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at com.netscape.cmstools.cli.SubsystemCLI.execute(SubsystemCLI.java:79)
        at org.dogtagpki.cli.CLI.execute(CLI.java:353)
        at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:694)
        at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:733)
ERROR: Command: /usr/lib/jvm/jre-17-openjdk/bin/java -cp /usr/share/pki/lib/* -Dcom.redhat.fips=false -Djava.util.logging.config.file=/usr/share/pki/etc/logging.properties com.netscape.cmstools.cli.MainCLI -C secret.txt -n caadmin_tls-client -U https://localhost:8943 --skip-revocation-check --debug ca-cert-request-submit --profile caTlsClientCert --csr-file webservice.csr
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pki/cli/main.py", line 423, in <module>
    cli.execute(sys.argv[1:])
  File "/usr/lib/python3.9/site-packages/pki/cli/main.py", line 407, in execute
    self.execute_java(args.remainder)
  File "/usr/lib/python3.9/site-packages/pki/cli/main.py", line 331, in execute_java
    subprocess.check_call(cmd, stdout=stdout)
  File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/lib/jvm/jre-17-openjdk/bin/java', '-cp', '/usr/share/pki/lib/*', '-Dcom.redhat.fips=false', '-Djava.util.logging.config.file=/usr/share/pki/etc/logging.properties', 'com.netscape.cmstools.cli.MainCLI', '-C', 'secret.txt', '-n', 'caadmin_tls-client', '-U', 'https://localhost:8943', '--skip-revocation-check', '--debug', 'ca-cert-request-submit', '--profile', 'caTlsClientCert', '--csr-file', 'webservice.csr']' returned non-zero exit status 255.

[haerdib]

Interestingly, some DB calls succeed, some fail.

In the same CA we found the following Debug Logs:

Error:

2025-11-17 17:02:43 [SerialNumberUpdateTask] INFO: LDAPPagedSearch: Searching ou=certificateRepository, ou=ca,o=pki-sss-
client-CA,dc=test,dc=de
2025-11-17 17:02:43 [SerialNumberUpdateTask] INFO: LDAPPagedSearch: filter: (&(certStatus=*)(serialno<=09268435456))
2025-11-17 17:02:43 [SerialNumberUpdateTask] WARNING: CertificateRepository: updateCounter: RecordPagedList: Error to ge
t a new page
java.lang.RuntimeException: RecordPagedList: Error to get a new page
        at com.netscape.cmscore.dbs.RecordPagedList.<init>(RecordPagedList.java:
[...]

No Error:

2025-11-17 17:02:39 [CertStatusUpdateTask] INFO: LDAPPagedSearch: Searching ou=certificateRepository, ou=ca,o=pki-sss-cl
ient-CA,dc=test,dc=de
2025-11-17 17:02:39 [CertStatusUpdateTask] INFO: LDAPPagedSearch: filter: (&(certStatus=VALID)(notAfter<=20251117170239Z
))
2025-11-17 17:02:39 [CertStatusUpdateTask] INFO: CertStatusUpdateTask: Updating revoked certs to expired

No Error2:

025-11-17 17:02:43 [SerialNumberUpdateTask] INFO: CAEngine: Updating serial number counter
2025-11-17 17:02:43 [SerialNumberUpdateTask] INFO: Repository: Getting last serial number in range 425961..268435456
2025-11-17 17:02:43 [SerialNumberUpdateTask] INFO: LDAPSession: Searching ou=certificateRepository, ou=ca,o=pki-sss-client-CA,dc=test,dc=de for (&(serialno>=06425961)(serialno<=09268435456))
2025-11-17 17:02:43 [SerialNumberUpdateTask] INFO: DBSearchResults: - cn=425961,ou=certificateRepository,ou=ca,o=pki-sss-client-CA,dc=test,dc=de
[.....]

[fmarco76]

Hi @haerdib, it is possible the error is related to paged search default limit. Could you take a look at #5133 for the problem and possible workaround?

[haerdib]

It didn't help. But after some debugging, it seems the server side sort is the problem. Has this been newly introduced in v11.6? If so, it's clearly a server fault on our side.

[fmarco76]

@haerdib for several reason the VLV query will be gradually dropped and replaced with paged search. This process is started in 11.6.
In existing instances with a large number of entries, the DS default limits could create problems to handle the queries so these limits need to be updated.

[haerdib]

Is somehow possible to find the exact query of the dogtag CA? Because according to the LDAP server logs, it seems that a Paged Search is combined with a VLV Query - something that our server (= DS389) does not support.

[fmarco76]

I do not think all the queries are logged. Anyway, if PKI run in debug mode (set debug.level=0 in CS.cfg) it should be possible to identify the code generating the problem.

[haerdib]

Because of missing logs, we used a tcpdump for further investigation. For the cli-request sudo pki --debug --skip-revocation-check -U https://localhost:8943/ -n caadmin_tls-client -C secret.txt ca-cert-request-submit --profile caTlsClientCert --csr-file webservice.csr, which results in the RecordPagedList: Error to get a new page error. we got the following tcpdump:

The interesting bytes are underlined red. According to the LDAP doc, 81 is a length Encoding: The byte after the 81 states the length.
So we have the following bytes after the request id:

• 81 -> type boolean to our interpretation
• 02 -> msg of 2 bytes length incoming
• 00 -> boolean: msg of 1 byte length
• 81 -> invalid boolean msg, but expected due to length of 2

I'm guessing the 00 should be the reverseOrder bool, but the preceding length byte is wrong. According to the stacktrace, the error is happening in LDAPPagedSearch. Our guess is that something goes wrong during setting the ServerControls in this part:

https://github.com/dogtagpki/pki/blob/master/base/server/src/main/java/com/netscape/cmscore/dbs/LDAPPagedSearch.java#L133-L140

Do you think this is plausible?

[fmarco76]

@haerdib very good investigation. It is possible there is a problem in the code.
However, the same CLI command is used in github CI (this is an example) and internal tests without getting the error so it is something weird and need more investigation.
Could you share the configuration and the steps to reproduce the error in a new deployment (even including the update step)?

[haerdib]

It is very strange. We will try to reproduce with a minimal example. We'll keep you updated.

[kimettog]

Created IDM-4335