SMIME decryption failing using openssl smime

[christof-brungraeber]

Hi there,

I'm unable to decrypt S/MIME encrypted messages using openssl smime - which I would have expected. It works using openssl cms, tough.

I generated a key-pair using the available example

cat test.crt
-----BEGIN CERTIFICATE-----
MIIC0zCCAb2gAwIBAgIBATALBgkqhkiG9w0BAQUwHjEcMAkGA1UEBhMCUlUwDwYD
VQQDHggAVABlAHMAdDAeFw0yMjEwMjEwOTA2MDRaFw0yMzEwMjEwOTA2MDRaMB4x
HDAJBgNVBAYTAlJVMA8GA1UEAx4IAFQAZQBzAHQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC4uB/cg0Im+hig+rWwTtupKeggjPovc9m3oNiap+dUqFaL
IOs+KeTFF79vaR+tkWYW9uVJ7AyjQghb4ogEaaUnnD1xRaFc1trzCms0IUr6HUvS
RqqfPM2fasoYJ92I0ekSkcuk20L+XAWc2u0/8cyZjFBEhnRrJE0+wi0NrRbDNhWR
H6H+eP1CPSSfi/TiRwuqxawuUWZhoisy2XLLPSPI1ZgUic4xMPZAbJ7GCnDn3tB6
bgWL3ZBneIOkO2521dI+zF+p64vloe4vaxhi6S2kiuxiAL8mJAHR5kL0s/5r9WxB
YlgSy4wqfqIg02vtQtD56QxIn2ROLHYVXWGyzDUJAgMBAAGjIDAeMA8GA1UdEwQI
MAYBAf8CAQMwCwYDVR0PBAQDAgAGMAsGCSqGSIb3DQEBBQOCAQEAFOqmEnVvyZfC
7f84Jgb5HzdzKIqnCZZWHxq2w4s7aiO1C2Gmnh7rR/C2VMAIPipspZllkIK3b6RB
N23Mgsi+WQuCovrYqZgriP/QJBdwbS1JXNrbdfHimHI3r0o+tTUiw7NrAd+tKSDH
q0B535EVaapLrkLKjvCU1k7AMBg4itrfMaNiKgf3S0I24HPZJ4VP+cy4Mr7JUHfq
936mo9BNxJmtHeF7xusW1e81OFqaZDZxvR20qGw43bp0QGKKn2vBoRbZrX7kdrQY
Q18sxYmnMFbBnneNBKs22YLurpEN2OYN30lgPcNjm7CpEAer+Hx1EF42/LhoTiW1
l/jtPQV+xA==
-----END CERTIFICATE-----

cat test.key
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC4uB/cg0Im+hig
+rWwTtupKeggjPovc9m3oNiap+dUqFaLIOs+KeTFF79vaR+tkWYW9uVJ7AyjQghb
4ogEaaUnnD1xRaFc1trzCms0IUr6HUvSRqqfPM2fasoYJ92I0ekSkcuk20L+XAWc
2u0/8cyZjFBEhnRrJE0+wi0NrRbDNhWRH6H+eP1CPSSfi/TiRwuqxawuUWZhoisy
2XLLPSPI1ZgUic4xMPZAbJ7GCnDn3tB6bgWL3ZBneIOkO2521dI+zF+p64vloe4v
axhi6S2kiuxiAL8mJAHR5kL0s/5r9WxBYlgSy4wqfqIg02vtQtD56QxIn2ROLHYV
XWGyzDUJAgMBAAECggEAURKfCU2izzCddqoS7FrwFgMKmp7Ff+Y8/FdgOiDPa1ym
/1FnblU8zzRpFSDygl96i0G5yQQNV+o3ePyn7ifgEM6GORzygajbWCiOfMbP/Y62
JIhXgqRRK1LXeFqm5NIclcXft290q+b/n6SG1LXvssTexfStTAkWziCzXXlGsE5f
NaO0eMpQNVPlEHzA3QtP+xGiqeKHYSXGReHWMIK0HFB7giKx8zIydhVdKCK2SquN
lR9eblGtz+v2pxAvLTWmDbebSMZ/I5NfQHUek9/4wz0Gp0Np4+QOTM06KMwWlqs0
zSkSO+KMdOKBmSrCGuRyHezQgFdoymuSvtMBqcyJcQKBgQD/XbKOTdVTHvDIGXOr
jVTvQ/2cKpQ15SO4OErsU1X4mfnfUmFs/v+JkoqjhQ39OXY3CXvZCS3R3rjMBNbE
Grn9IPn7mDfq1WUqYfLVxRDKhaOx0LJEc48yluuETvXmIkgbzzt8jUP4iwVg0GFC
ICIJytRYkaoiuL5t04ZaiF7dQwKBgQC5LYawroy6sfJg+G0S4iJN5mSv+E0Am2uM
+EcBDdetWHgvfzXDuF5NK7HkffhfMu5h0/D3qjy4JcPS78C+LTdSgWqAQtBb8f2V
zxGG7JsBzBTjBkUbMFbGUp9DBqlYcz3tJ3I6bD8go8pdlWZr95OCECdn8o2mYwRT
R+BZ0md5wwKBgCGFkDnj97volNz/klKTw/GAsFr+r+P/R1gYc6VgLynqNIXfT5C6
TJsnlHFwtuxlzaHd6dcMwGoLXaDShcrKY13aJVaknbqC9rvAjdsH+nNLZbii17Pq
o4KVkycUIm1qVKLp6jooZSLMRRF3aTz/8NfuOz/BJ19VdxH9sWZS/nj3AoGAIYw2
55Fn23bbg2feUtp2/ofr29lvEdoWXP6p1m4MkdICTBCOy/t90htFajDMK2sNttlv
wV6tC2uEE5Xt5EZKzH63ra7tEos/tsFB2qDI4EV5zztWj7ttLRbQ1ZLCYUzXfjx0
PFNaCG7D/bpQSOUcaybEDNfrg6ZSpUfTRFKS/oMCgYEAgW9lFBQlUDHrBXDJbXQa
YWRxey0IIZVc4BYOtNcrddlEZ8QNGysBzNZvvK1OQX+lIlsYZpg0xslzX5oxVN+d
0juWmFAr9r/mGiTzGEovpv/2PN9+GVo+xrvv/oXSH77VYjYiol+4cm8+opJAk0C2
zLpIGfp3onz99LtpKZONbuY=
-----END PRIVATE KEY-----

cat test.txt
Content-Type: application/pkcs7-mime; name=smime.p7m;
 smime-type=enveloped-data; charset=binary
Content-Description: Enveloped Data
Content-Disposition: attachment; filename=smime.p7m
Content-Transfer-Encoding: base64
From: sender@example.com
To: recipient@example.com
Subject: Example S/MIME encrypted message
Date: Fri, 21 Oct 2022 09:06:37 +0000
Message-Id: <1666343197457-97020db4-e71abae1-7f511e1e@example.com>
MIME-Version: 1.0

MIAGCSqGSIb3DQEHA6CAMIACAQIxggE/MIIBOwIBADAjMB4xHDAJBgNVBAYTAlJVMA8GA1UEAx4I
AFQAZQBzAHQCAQEwDQYJKoZIhvcNAQEHMAAEggEAXCm3sGcqmoJf5gXpbUTQ7WK+x/zg88Z9eHJS
tlLQJuzHg2XH6ZYp9svD9U4sNusWuKTu4SddO+KfDHwDmxqWR10A1YFaW82wlfncgFnjBjW/yewp
Gfq04QNDeDLMobvPLB0z0LM+7FArkIHfXLuia877DHLr06jygJnyqDvxa7jrmGEzGPqodp+50pm9
SUdRRS6iRpUKuEbUFZZpTnMQ6S1ltZUQVbqOKoMciz6BZSaQNOSm3koAJzuQoxbJzwBGNCyCjk6I
61uCsrauppdyFS/NQYGiwM0QCMVwftHq08bauYSO1aiiF1HLktp4gqY/ZNATEuKLoikvfQDUwKzc
9DCABgkqhkiG9w0BBwEwHQYJYIZIAWUDBAECBBAQsZQORAPzTr0N25iOGLiEoIAEEMjPgHGy9srs
hIdw21l6X7gAAAAAAAAAAAAA

I'm unable to decrypt using vanilla openssl that comes with the OS (that doesn't come with cms):

/usr/bin/openssl version
LibreSSL 2.8.3

/usr/bin/openssl smime -decrypt -aes256 -recip test.crt -inkey test.key -in test.txt
Error decrypting PKCS#7 structure
4731367084:error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt:/System/Volumes/Data/SWE/macOS/BuildRoots/880a0f6e74/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.4/libressl-2.8/crypto/evp/evp_enc.c:521:

Using openssl@3:

/usr/local/opt/openssl/bin/openssl version
OpenSSL 3.0.5 5 Jul 2022 (Library: OpenSSL 3.0.5 5 Jul 2022)

/usr/local/opt/openssl/bin/openssl smime -decrypt -aes-256-cbc -recip test.crt -inkey test.key -in test.txt
Error decrypting PKCS#7 structure
000E8C0D01000000:error:1C800064:Provider routines:ossl_cipher_unpadblock:bad decrypt:providers/implementations/ciphers/ciphercommon_block.c:124:

/usr/local/opt/openssl/bin/openssl cms -decrypt -aes-256-cbc -recip test.crt -inkey test.key -in test.txt
Test message.

I've been trying to disable the useOAEP flag for addRecipientByCertificate but without success.

[microshine]

This script works fine. Decrypted message is Test message.

// Decode input certificate
const certificateBuffer = fromPEM(`-----BEGIN CERTIFICATE-----
MIIC0zCCAb2gAwIBAgIBATALBgkqhkiG9w0BAQUwHjEcMAkGA1UEBhMCUlUwDwYD
VQQDHggAVABlAHMAdDAeFw0yMjEwMjEwOTA2MDRaFw0yMzEwMjEwOTA2MDRaMB4x
HDAJBgNVBAYTAlJVMA8GA1UEAx4IAFQAZQBzAHQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC4uB/cg0Im+hig+rWwTtupKeggjPovc9m3oNiap+dUqFaL
IOs+KeTFF79vaR+tkWYW9uVJ7AyjQghb4ogEaaUnnD1xRaFc1trzCms0IUr6HUvS
RqqfPM2fasoYJ92I0ekSkcuk20L+XAWc2u0/8cyZjFBEhnRrJE0+wi0NrRbDNhWR
H6H+eP1CPSSfi/TiRwuqxawuUWZhoisy2XLLPSPI1ZgUic4xMPZAbJ7GCnDn3tB6
bgWL3ZBneIOkO2521dI+zF+p64vloe4vaxhi6S2kiuxiAL8mJAHR5kL0s/5r9WxB
YlgSy4wqfqIg02vtQtD56QxIn2ROLHYVXWGyzDUJAgMBAAGjIDAeMA8GA1UdEwQI
MAYBAf8CAQMwCwYDVR0PBAQDAgAGMAsGCSqGSIb3DQEBBQOCAQEAFOqmEnVvyZfC
7f84Jgb5HzdzKIqnCZZWHxq2w4s7aiO1C2Gmnh7rR/C2VMAIPipspZllkIK3b6RB
N23Mgsi+WQuCovrYqZgriP/QJBdwbS1JXNrbdfHimHI3r0o+tTUiw7NrAd+tKSDH
q0B535EVaapLrkLKjvCU1k7AMBg4itrfMaNiKgf3S0I24HPZJ4VP+cy4Mr7JUHfq
936mo9BNxJmtHeF7xusW1e81OFqaZDZxvR20qGw43bp0QGKKn2vBoRbZrX7kdrQY
Q18sxYmnMFbBnneNBKs22YLurpEN2OYN30lgPcNjm7CpEAer+Hx1EF42/LhoTiW1
l/jtPQV+xA==
-----END CERTIFICATE-----`);

const certSimpl = pkijs.Certificate.fromBER(certificateBuffer);

const privateKeyBuffer = fromPEM(`-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC4uB/cg0Im+hig
+rWwTtupKeggjPovc9m3oNiap+dUqFaLIOs+KeTFF79vaR+tkWYW9uVJ7AyjQghb
4ogEaaUnnD1xRaFc1trzCms0IUr6HUvSRqqfPM2fasoYJ92I0ekSkcuk20L+XAWc
2u0/8cyZjFBEhnRrJE0+wi0NrRbDNhWRH6H+eP1CPSSfi/TiRwuqxawuUWZhoisy
2XLLPSPI1ZgUic4xMPZAbJ7GCnDn3tB6bgWL3ZBneIOkO2521dI+zF+p64vloe4v
axhi6S2kiuxiAL8mJAHR5kL0s/5r9WxBYlgSy4wqfqIg02vtQtD56QxIn2ROLHYV
XWGyzDUJAgMBAAECggEAURKfCU2izzCddqoS7FrwFgMKmp7Ff+Y8/FdgOiDPa1ym
/1FnblU8zzRpFSDygl96i0G5yQQNV+o3ePyn7ifgEM6GORzygajbWCiOfMbP/Y62
JIhXgqRRK1LXeFqm5NIclcXft290q+b/n6SG1LXvssTexfStTAkWziCzXXlGsE5f
NaO0eMpQNVPlEHzA3QtP+xGiqeKHYSXGReHWMIK0HFB7giKx8zIydhVdKCK2SquN
lR9eblGtz+v2pxAvLTWmDbebSMZ/I5NfQHUek9/4wz0Gp0Np4+QOTM06KMwWlqs0
zSkSO+KMdOKBmSrCGuRyHezQgFdoymuSvtMBqcyJcQKBgQD/XbKOTdVTHvDIGXOr
jVTvQ/2cKpQ15SO4OErsU1X4mfnfUmFs/v+JkoqjhQ39OXY3CXvZCS3R3rjMBNbE
Grn9IPn7mDfq1WUqYfLVxRDKhaOx0LJEc48yluuETvXmIkgbzzt8jUP4iwVg0GFC
ICIJytRYkaoiuL5t04ZaiF7dQwKBgQC5LYawroy6sfJg+G0S4iJN5mSv+E0Am2uM
+EcBDdetWHgvfzXDuF5NK7HkffhfMu5h0/D3qjy4JcPS78C+LTdSgWqAQtBb8f2V
zxGG7JsBzBTjBkUbMFbGUp9DBqlYcz3tJ3I6bD8go8pdlWZr95OCECdn8o2mYwRT
R+BZ0md5wwKBgCGFkDnj97volNz/klKTw/GAsFr+r+P/R1gYc6VgLynqNIXfT5C6
TJsnlHFwtuxlzaHd6dcMwGoLXaDShcrKY13aJVaknbqC9rvAjdsH+nNLZbii17Pq
o4KVkycUIm1qVKLp6jooZSLMRRF3aTz/8NfuOz/BJ19VdxH9sWZS/nj3AoGAIYw2
55Fn23bbg2feUtp2/ofr29lvEdoWXP6p1m4MkdICTBCOy/t90htFajDMK2sNttlv
wV6tC2uEE5Xt5EZKzH63ra7tEos/tsFB2qDI4EV5zztWj7ttLRbQ1ZLCYUzXfjx0
PFNaCG7D/bpQSOUcaybEDNfrg6ZSpUfTRFKS/oMCgYEAgW9lFBQlUDHrBXDJbXQa
YWRxey0IIZVc4BYOtNcrddlEZ8QNGysBzNZvvK1OQX+lIlsYZpg0xslzX5oxVN+d
0juWmFAr9r/mGiTzGEovpv/2PN9+GVo+xrvv/oXSH77VYjYiol+4cm8+opJAk0C2
zLpIGfp3onz99LtpKZONbuY=
-----END PRIVATE KEY-----`);
// Parse S/MIME message to get CMS enveloped content
const parser = parse(`Content-Type: application/pkcs7-mime; name=smime.p7m;
smime-type=enveloped-data; charset=binary
Content-Description: Enveloped Data
Content-Disposition: attachment; filename=smime.p7m
Content-Transfer-Encoding: base64
From: sender@example.com
To: recipient@example.com
Subject: Example S/MIME encrypted message
Date: Fri, 21 Oct 2022 09:06:37 +0000
Message-Id: <1666343197457-97020db4-e71abae1-7f511e1e@example.com>
MIME-Version: 1.0

MIAGCSqGSIb3DQEHA6CAMIACAQIxggE/MIIBOwIBADAjMB4xHDAJBgNVBAYTAlJVMA8GA1UEAx4I
AFQAZQBzAHQCAQEwDQYJKoZIhvcNAQEHMAAEggEAXCm3sGcqmoJf5gXpbUTQ7WK+x/zg88Z9eHJS
tlLQJuzHg2XH6ZYp9svD9U4sNusWuKTu4SddO+KfDHwDmxqWR10A1YFaW82wlfncgFnjBjW/yewp
Gfq04QNDeDLMobvPLB0z0LM+7FArkIHfXLuia877DHLr06jygJnyqDvxa7jrmGEzGPqodp+50pm9
SUdRRS6iRpUKuEbUFZZpTnMQ6S1ltZUQVbqOKoMciz6BZSaQNOSm3koAJzuQoxbJzwBGNCyCjk6I
61uCsrauppdyFS/NQYGiwM0QCMVwftHq08bauYSO1aiiF1HLktp4gqY/ZNATEuKLoikvfQDUwKzc
9DCABgkqhkiG9w0BBwEwHQYJYIZIAWUDBAECBBAQsZQORAPzTr0N25iOGLiEoIAEEMjPgHGy9srs
hIdw21l6X7gAAAAAAAAAAAAA`);

// Make all CMS data
const cmsContentSimpl = pkijs.ContentInfo.fromBER(parser.content.buffer);
const cmsEnvelopedSimp = new pkijs.EnvelopedData({ schema: cmsContentSimpl.content });

const result = await cmsEnvelopedSimp.decrypt(0,
  {
    recipientCertificate: certSimpl,
    recipientPrivateKey: privateKeyBuffer
  });

console.log(Convert.ToString(result));

[christof-brungraeber]

Hi @microshine,
many thanks for following up and for your example. Appreciated! I was able to successfully and smoothly integrate your library into a nodejs backend service. Unit tests in terms of decrypting what was previously encrypted with pki.js indeed do succeed, as well as integration tests using openssl cms on top of testcontainers.
What doesn't work, though, is decrypting using "standard tooling" such as vanilla openssl smime and - more importantly - Outlook. The service is supposed to be sending S/MIME encrypted emails. But unfortunately, I can't get this functionality working end-to-end - due to the receiving end simply not being able to decrypt.
I could imagine that pki.js is using more recent ciphers or envelopes than these tools allow for, but to be honest I lack the in-depth knowledge. I could imagine it has to do with the RSA OAEP hashing algorithm. If I disable the corresponding flag for addRecipientByCertificate I can't even decrypt using openssl cms anymore, though (what at least works if I leave OAEP enabled).

Would there be a way to get this going - decrypting S/MIME encrypted using "well-seasoned" tooling what was previously generated using pki.js?