HANDSHAKE_SSL_ERROR with RDS and 3.9.3+

[mattmbrightside]

using mysql2 with AWS nodejs lambdas and AWS RDS.

As of March 2024, our RDS instance was still on the 2019 certificate bundle. Versions of mysql2 <3.9.3 were working fine with calls to createConnection. We updated a project to 3.9.3 and started getting these errors:

{"errorType":"Error","errorMessage":"unable to get local issuer certificate","code":"HANDSHAKE_SSL_ERROR","message":"unable to get local issuer certificate","stack":["Error: unable to get local issuer certificate"," at fn (/private/var/folders/48/8n9w9qzn17s1gpq7tjs3cnv40000gp/T/tmplakzpggq/node_modules/mysql2/promise.js:253:31)

I noticed the reference to #2131 and instructed my team to update the mysql certificate bundle so we could retest. We updated RDS to bundle rds-ca-rsa2048-g1. Trying mysql2 3.9.3 gives the same error.

I noticed the mysql2 issue: #2542 regarding the certificates and waited for the fix and for 3.9.4. We have tried again with 3.9.4 and get the same error on calls to createConnection.

If anyone is successfully using this package 3.9.3 or 3.9.4 with RDS, please let me know what steps are required to get past this error. Thanks!

[mattmbrightside]

a bit more info, we are using the AWS RDS Proxy service between the lambda and RDS and our connection options passed to createConnection are

{
  "host": "<rds proxy>.<region>.rds.amazonaws.com",
  "user": "username",
  "password": "password",
  "database": "dbname",
  "ssl": "Amazon RDS",
  "charset": "utf8mb4"
}

[wellwelwel]

Thanks, @mattmbrightside. That clarifies things 🙋🏻‍♂️
This issue probably can be fixed by joining old and new certificates, but this will create an even larger file.

Unfortunately, even with a RDS in hand, I wouldn't be able to reproduce this error, since it will be created with one of the new certificates.

---

@sidorares, I think it's important to move this responsibility to a separate dependency soon for more flexibility. Users will soon be alerted to update to version ^3.9.4.

I thought of these possibilities:

A) Create a first release with the old certificates, then create a second release with the new certificates. In this way, the users who need to use old versions could force the version they want from the certificate dependency and we would document it (not yet considering a major because of this) and, by default, the new certificates would be loaded.

B) Just combine all the certificates into a larger file until we think about a major release (this option is independent of a new dependency).

[jeffrey-mutual]

I would love to help reproduce this error (and create a PR with a fix), but I seem to be having some difficulty doing so.

Amazon RDS
MySQL version 8.0.35
Certificate authority is rds-ca-2019
[email protected]

Config is

{
  host: '<details>.rds.amazonaws.com',
  database: 'db',
  user: 'user',
  password: 'password',
  ssl: 'Amazon RDS'
}

This works fine and I'm able to query the database through mysql2. Anything else you'd like me to check? Could be related to the proxy.

[sidorares]

@wellwelwel I created https://github.com/mysqljs/aws-ssl-profiles and added you as a maintainer

Intended use when it's ready would be something along the lines

import awsCaBundle from 'aws-ssl-profiles';

const connection = mysql.createConnection({
  //...
  ssl: awsCaBundle
})l

[dougwilson]

If only RDS would have just used public certificates like Azure 🤦‍♂️

[wellwelwel]

Really thanks, @jeffrey-mutual 🤝

When looking into 675dd04, it's possible to see that not all old certificates have been changed, which would explain why only some servers return the authentication error.

[jeffrey-mutual]

Really thanks, @jeffrey-mutual 🤝

When looking into 675dd04, it's possible to see that not all old certificates have been changed, which would explain why only some servers return the authentication error.

I guess I got lucky then. Both of my databases work on 3.9.4. Without a repro I probably won't work on the PR since I wouldn't easily be able to validate the fix.

[EasyKoen]

We have the same issue, we also use RDS Proxy.

I found the following information from AWS: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.howitworks.html#rds-proxy-security.tls:

When using a client with --ssl-mode VERIFY_CA or VERIFY_IDENTITY, specify the --ssl-ca option pointing to a CA in .pem format. For the .pem file to use, download all root CA PEMs from Amazon Trust Services and place them into a single .pem file.

https://www.amazontrust.com/repository/ has 5 root CA's, I am going to add them manually to our setup and see if that resolves the problem.

[EasyKoen]

Well, that actually did the trick.

So there you have it. When you connect to RDS through RDS Proxy, you need the Amazon Root CA's which you can find on the website I posted in the comment above.

[pedrovanzella]

I can also confirm that adding those certs to the certificate chain fixes the issues for me, using RDS Proxy. Maybe they can just be added to the chain in this repo while the certificates package isn't done yet?

[arturjeber]

Same problem with Azure when using Nextjs 14

const dbConfig = {
  host: process.env.DB_HOST,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_DATABASE,
  ssl: { 
     cert:  process.env.DB_SSL.replace(/\\n/g, '\n')
  },
};

{
  status: 401,
  results: [],
  error: [Error: 00417A49F87F0000:error:0A00010B:SSL routines:tls_validate_record_header:wrong version number:ssl/record/methods/tlsany_meth.c:81:
  ] {
    library: 'SSL routines',
    reason: 'wrong version number',
    code: 'HANDSHAKE_SSL_ERROR',
    fatal: true
  }
}

[robert-pitt-foodhub]

We have updated our version from 3.1.1 to 3.9.7 in our SIT environment and this error was introduced across hundreds of lambdas, we are going to revert back to 3.1.1, but thought I would share here to help confirm that the recent changes in 3.9.7 are failing RDS connections