errors with curl SChannel backend on Windows

[mbjones]

Some users report problems with curl on Windows recently failing to connect, with the following error:

> library(dataone)
> cn <- CNode("PROD")
Error in curl::curl_fetch_memory(url, handle = handle) : 
  Failure when receiving data from the peer [cn.dataone.org]:
schannel: failed to read data from server: SEC_E_CONTEXT_EXPIRED (0x80090317) - The context has expired and can no longer be used.

This seems to be associated with a recent change to the Schannel SSL backend on windows. Switching to using the OpenSSL backend (as described here: https://cran.r-project.org/web/packages/curl/vignettes/windows.html) seems to fix the problem but is probably not a great long-term solution. Here's a workaround by setting an environment variable to tell curl to use the OpenSSL backend:

write('CURL_SSL_BACKEND=openssl', file = "~/.Renviron", append = TRUE)

Restart the R session and verify that OpenSSL is now active (no longer in parenthesis):

> curl::curl_version()$ssl_version
Initiating curl with CURL_SSL_BACKEND: openssl
[1] "OpenSSL/3.5.0 (Schannel)"

Now, try the code again and it should work:

> library(dataone)
> cn <- CNode("PROD")

This doesn't identify why exactly this SSL error is occurring with SChannel, which would be good to solve, so leaving this ticket open. At a minimum we should document the workaround.

• Document workaround
• Fix problems with SChannel

[mbjones]

One possibility is that SChannel is complaining because it can't find the revocation URLS that are published in the SSL certs. If that is the case, we can see if it works skipping looking for the revocation certs with:

library(curl)
handle <- new_handle(ssl_options = curl::curl_symbols("CURLSSLOPT_NO_REVOKE")$value)
req <- curl_fetch_memory("https://cn.dataone.org/cn/v2/node", handle = handle)

If that produces no errror, but the curl call without the handle errors, then its a revocation URL problem.

It could also be other problems with not being able to validate the trust chain, or other SSL validation issues.

[datadavev]

Possibly relevant: curl/curl#18029

[mbjones]

I tested the CURLSSLOPT_NO_REVOKE option with Rani but it didn't change anything. I also had her print out verbose SSL output, but it provided no helpful details.

Dave pointed out in the ticket that this may be a bug in curl that they have identified and are working on a a fix, so it may go away with a windows patch sometime.... see the ticket that @datadavev linked for details.

[mbjones]

This SChannel issue is now causing windows failures on tests, so interferes with releases too. See for example https://github.com/DataONEorg/rdataone/actions/runs/18238425581/job/51936225604 which has errors like so:

── Error ('test.AuthenticationManager.R:7:3'): AuthenticationManager isAuthValid() for v2 node works ──
<curl_error_recv_error/curl_error/error/condition>
Error in `curl::curl_fetch_memory(url, handle = handle)`: Failure when receiving data from the peer [cn-stage.test.dataone.org]:
schannel: failed to read data from server: SEC_E_CONTEXT_EXPIRED (0x80090317) - The context has expired and can no longer be used.
Backtrace:
    ▆
 1. ├─dataone::CNode("STAGING") at test.AuthenticationManager.R:7:3
 2. ├─dataone::CNode("STAGING") at dataone/R/CNode.R:72:3
 3. │ └─dataone (local) .local(x, ...)
 4. │   └─httr::GET(CN_URI) at dataone/R/CNode.R:115:3
 5. │     └─httr:::request_perform(req, hu$handle$handle)
 6. │       ├─httr:::request_fetch(req$output, req$url, handle)
 7. │       └─httr:::request_fetch.write_memory(req$output, req$url, handle)
 8. │         └─curl::curl_fetch_memory(url, handle = handle)
 9. └─curl:::raise_libcurl_error(...)
── Error ('test.AuthenticationManager.R:25:3'): AuthenticationManager getAuthMethod(), getToken(), getCert() work ──
<curl_error_recv_error/curl_error/error/condition>
Error in `curl::curl_fetch_memory(url, handle = handle)`: Failure when receiving data from the peer [cn-stage.test.dataone.org]:
schannel: failed to read data from server: SEC_E_CONTEXT_EXPIRED (0x80090317) - The context has expired and can no longer be used.

The same tests pass on 4 other OS platform/version combinations on MacOS and Linux.

[mbjones]

TL;DR: Enabling TLSv1.3 on the server while also disabling client-cert verification with SSLVerifyClient none seems to make things work for both windows and mac curl clients. Need to test with Java.

After some configuration exploration, it appears we have TLS 1.3 disabled on the server due to problems on the Mac with client-side certificate verification. By enabling TLSv1.3 on the server, it seems the the schannel errors may go away. Here's the mozilla recommended config today that I tried and seems to work with TLSv1.3:

# Mozilla recommended intermediate settings from 2025-10-09                                                               #
# https://ssl-config.mozilla.org/#server=apache&version=2.4.52&config=intermediate&openssl=3.0.2&guideline=5.7            #
###########################################################################################################################

SSLProtocol             -all +TLSv1.2 +TLSv1.3
SSLOpenSSLConfCmd       Curves X25519:prime256v1:secp384r1
SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
SSLHonorCipherOrder     off
SSLSessionTickets       off

SSLUseStapling On
SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"

However, that config causes mac curl clients to fail when we have SSLVerifyClient optional set. This requests a client cert from the mac, and then triggers Post-Handshake Authentication, which causes a failure if the underlying SSL lib doesn't support PHA. The error manifests as a 403 Forbidden response, as shown in the following curl output:

❯ /usr/bin/curl -s -vv https://cn-sandbox.test.dataone.org/cn
* Host cn-sandbox.test.dataone.org:443 was resolved.
* IPv6: (none)
* IPv4: 128.111.85.177
*   Trying 128.111.85.177:443...
* Connected to cn-sandbox.test.dataone.org (128.111.85.177) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=cn-sandbox-orc-1.test.dataone.org
*  start date: Aug 17 19:48:12 2025 GMT
*  expire date: Nov 15 19:48:11 2025 GMT
*  subjectAltName: host "cn-sandbox.test.dataone.org" matched cert's "cn-sandbox.test.dataone.org"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /cn HTTP/1.1
> Host: cn-sandbox.test.dataone.org
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 403 Forbidden
< Date: Fri, 10 Oct 2025 02:13:42 GMT
< Server: Apache
< Content-Length: 258
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.Reason: Cannot perform Post-Handshake Authentication.<br /></p>
</body></html>
* Connection #0 to host cn-sandbox.test.dataone.org left intact

If I disable client-side verification entirely on the server side with SSLVerifyClient none, then suddenly the connection succeeds:

❯ /usr/bin/curl -s -v https://cn-sandbox.test.dataone.org/cn
* Host cn-sandbox.test.dataone.org:443 was resolved.
* IPv6: (none)
* IPv4: 128.111.85.177
*   Trying 128.111.85.177:443...
* Connected to cn-sandbox.test.dataone.org (128.111.85.177) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=cn-sandbox-orc-1.test.dataone.org
*  start date: Aug 17 19:48:12 2025 GMT
*  expire date: Nov 15 19:48:11 2025 GMT
*  subjectAltName: host "cn-sandbox.test.dataone.org" matched cert's "cn-sandbox.test.dataone.org"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
* using HTTP/1.x
> GET /cn HTTP/1.1
> Host: cn-sandbox.test.dataone.org
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 200
< Date: Fri, 10 Oct 2025 02:45:40 GMT
< Server: Apache
< Vary: User-Agent,Accept-Encoding,Origin
< Content-Language: en-US
< Content-Length: 1676
< Access-Control-Allow-Origin:
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: Authorization, Content-Type, Location, Content-Length, x-annotator-auth-token, Cache-Control
< Access-Control-Expose-Headers: Content-Length, Content-Type, Location
< Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
< Content-Type: text/xml;charset=UTF-8
<
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="/cn/xslt/dataone.types.v2.xsl" ?>
<ns3:node xmlns:ns2="http://ns.dataone.org/service/types/v1" xmlns:ns3="http://ns.dataone.org/service/types/v2.0" replicate="false" synchronize="false" type="cn" state="up">
    <identifier>urn:node:cnSandboxUCSB1</identifier>
    <name>cn-sandbox-ucsb-1</name>
    <description>University of California Santa Barbara Sandbox Coordinating Node</description>
    <baseURL>https://cn-sandbox-ucsb-1.test.dataone.org/cn</baseURL>
    <services>
        <service name="CNCore" version="v1" available="true"/>
        <service name="CNRead" version="v1" available="true"/>
        <service name="CNAuthorization" version="v1" available="true"/>
        <service name="CNIdentity" version="v1" available="true"/>
        <service name="CNReplication" version="v1" available="true"/>
        <service name="CNRegister" version="v1" available="true"/>
        <service name="CNCore" version="v2" available="true"/>
        <service name="CNRead" version="v2" available="true"/>
        <service name="CNAuthorization" version="v2" available="true"/>
        <service name="CNIdentity" version="v2" available="true"/>
        <service name="CNReplication" version="v2" available="true"/>
        <service name="CNRegister" version="v2" available="true"/>
        <service name="CNDiagnostic" version="v2" available="true"/>
        <service name="CNView" version="v2" available="true"/>
    </services>
    <subject>CN=urn:node:cnSandboxUCSB1,DC=dataone,DC=org</subject>
    <contactSubject>CN=Robert P Waltz A904,O=Google,C=US,DC=cilogon,DC=org</contactSubject>
* Connection #0 to host cn-sandbox.test.dataone.org left intact
</ns3:node>

So, the challenge is how to handle this with client-side certs, which DataONE relies on for various MN-to-MN and MN-to-CN API calls, including replication. To be discussed with the team, but I think disabling SSLVerifyClient is our path forward.

[mbjones]

I've now tested the above TLSv1.3 changes pretty carefully, along with a change to SSLClientVerify by adding curl to the conditional exclusion in this block:

    SSLVerifyClient none
    <Location "/cn">
        <If " ! ( %{HTTP_USER_AGENT} =~ /(windows|chrome|mozilla|safari|webkit|curl)/i )">
            SSLVerifyClient optional
        </If>
    </Location>

With that curl addition to the conditional, curl-based clients can no longer use client-side SSL certificates to authenticate. But it also allows the TLSv1.3 protocol to be enabled and tests successfully on the mac and linux platforms, but Windows is still failing:

My hypothesis is that tests fully pass with both TLSv1.2 and TLSv1.3 on mac and linux now, but on Windows tests still fail on TLSv1.2 -- while I updated the testing hosts to TLSv1.3, the production hosts are still running 1.2 and causing the failures, as summarized here:

Type	host	TLS	Mac	Linux	Windows
cn	PROD	1.2	🟢	🟢	🔴
mn	KNB	1.2	🟢	🟢	🔴
cn	STAGING	1.3	🟢	🟢	🟢
cn	SANDBOX	1.3	🟢	🟢	🟢
mn	mnTestKNB	1.3	🟢	🟢	🟢
mn	mnStageUCSB2	1.3	🟢	🟢	🟢

This is hard to determine for sure, because the test output is somewhat ambiguous as to what the root cause is for many of the tests (PROD and KNB are referenced in a lot of the tests). I'd like to try enabling TLSv1.3 on PROD and KNB and see how we do.

[artntek]

Java issue

Notification Service gets a 403 error when trying to authenticate:

response:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.Reason: Cannot perform Post-Handshake Authentication.<br /></p>
</body></html>

Using the Java 21 version of java.net.HttpURLConnection. This worked previously (TLS 1.2). (note, however, that Metacat admin login works OK on test.adc)

@artntek investigating

[mbjones]

After examining the Java issue with @artntek , we determined the same PHA error that is triggered for Mac clients was being triggered for the notification service linux client using Java. @artntek changed the notification service to use a unique user agent string (DataONE notification-service/1.0.0) which I then used in the CN apache config to exclude that client from SSLVerify for client-side certificates. This allowed the SSL negotiation to complete successfully. We may want to switch out apache config around to use an explicit user-agent set that we include in the config, rather than an exclude list.

[taojing2002]

Here is the list of all user agent in cn-ucsb-1.

user-agent-in-cn.txt