Skip to content

ssl: Keep shared socket options in atomics instead of an ETS table - #11689

Open
zmstone wants to merge 1 commit into
erlang:masterfrom
zmstone:ssl/shared-opts-atomics
Open

zmstone wants to merge 1 commit into
erlang:masterfrom
zmstone:ssl/shared-opts-atomics

Conversation

@zmstone

@zmstone zmstone commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Since 03593c7, ssl_gen_statem:init/1 creates one ETS table per TLS and DTLS connection to share the packet type and the send watermarks between the connection process, ssl:send/2 in the caller and tls_sender. This replaces the table with a three-word atomics array.

Measured on 28.4.1, per connection: the table plus one row costs 2,861 bytes and ets:lookup_element/3 takes 680 ns; the array costs 58 bytes and atomics:get/2 takes 248 ns. On a node with 5,000 idle TLS connections that is 14 MB and 5,000 of its 5,247 ETS tables.

ssl:send/2 used the table dying with the connection process as its closed-socket signal. That catch is gone; tls_sender:call/2 already maps the noproc exit to {error, closed}, and a new test case covers it. Negative watermarks are now rejected in setopts, as inet does.

Tests: ssl_socket_SUITE (3 new cases), ssl_packet_SUITE, ssl_payload_SUITE, ssl_api_SUITE, ssl_basic_SUITE, tls_1_3_version_SUITE, dtls_api_SUITE pass.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

CT Test Results

    2 files     67 suites   24m 48s ⏱️
  850 tests   804 ✅  46 💤 0 ❌
4 373 runs  3 406 ✅ 967 💤 0 ❌

Results for commit 881239b.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

// Erlang/OTP Github Action Bot

@zmstone
zmstone force-pushed the ssl/shared-opts-atomics branch from b1eda44 to 6bd8b10 Compare September 23, 2026 05:58
@zmstone
zmstone marked this pull request as ready for review September 23, 2026 06:48
@zmstone

zmstone commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

the ci failures do not seem related.

@garazdawi

Copy link
Copy Markdown
Member

I re-triggered it to see if it was a one-off or not.

@zmstone
zmstone force-pushed the ssl/shared-opts-atomics branch 4 times, most recently from b4712b0 to 866b00c Compare September 23, 2026 18:55
@zmstone

zmstone commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

A quick bench ref: 2.1 GB measured saving at 180k connections.

@dgud

dgud commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Nice, the idea was to move (more later) stuff out of the state to reduce memory and GC time, but that has not
happened yet.

Should we allow signed numbers? If we want to move other data out of the state into the atomics later,
it should be possible to find more numbers that can be moved to atomics.

I think we should target master instead of maint, though.

@dgud dgud added the testing currently being tested, tag is used by OTP internal CI label Oct 2, 2026
@zmstone

zmstone commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Nice, the idea was to move (more later) stuff out of the state to reduce memory and GC time, but that has not happened yet.

Should we allow signed numbers? If we want to move other data out of the state into the atomics later, it should be possible to find more numbers that can be moved to atomics.

I think we should target master instead of maint, though.

I suppose it's trivial to support.
I can rebase it on to master.

@zmstone
zmstone force-pushed the ssl/shared-opts-atomics branch from 866b00c to 9fae6c1 Compare October 2, 2026 09:39
@zmstone
zmstone changed the base branch from maint to master October 2, 2026 09:40
Since commit 03593c7, ssl_gen_statem:init/1 creates an ETS table
for every TLS and DTLS connection. It holds at most three entries: the
packet type, which ssl:send/2 reads in the caller's process to encode
the packet header, and the high and low watermarks, which tls_sender
reads when it sets up asynchronous sending. Three processes read or
write these values, so they cannot live in the state machine's state.

A table is an expensive container for three integers. Measured on
OTP 28.4.1, per connection: the table plus one row costs 2,861 bytes,
and reading it with ets:lookup_element/3 takes 680 ns. An atomics
array of three words costs 58 bytes and atomics:get/2 takes 248 ns.
On a node with 5,000 idle TLS connections the tables were 5,000 of
its 5,247 ETS tables and 14 MB.

Replace the table with an atomics array, created in
ssl_gen_statem:init/1 and carried in the fields that held the table,
whose names are unchanged. The new module
ssl_shared_opts owns the encoding: the packet type maps to a small
integer (raw and 0 share a code), and each watermark is stored as its
value plus one so that zero can mean "not set", because 0 is a legal
watermark. DTLS gets the same array; it never read the table, and a
shared constructor keeps the two paths alike.

The table doubled as a closed-socket signal: it died with the
connection process, and ssl:send/2 turned the resulting badarg into
{error, closed}. An atomics array is refcounted and outlives the
process, so that catch is removed. The sender call already covers the
case: tls_sender:call/2 turns noproc, normal and shutdown exits into
{error, closed}. What remains of the old behaviour is one wasted
header encode when sending on a closed socket.

Setting a negative watermark used to be accepted into the table and
now returns {error, {options, {socket_options, Opt}}}; inet rejects
negative watermarks, and an unsigned atomics word cannot hold one.

New cases in ssl_socket_SUITE: setopts/getopts of every packet type,
with a send in {packet, 2} mode to show the encoder uses the value set
last; the watermarks on the tls_socket_tcp transport (defaults, set,
0, and negative rejected); and send after the owner closed the socket.
@zmstone
zmstone force-pushed the ssl/shared-opts-atomics branch from 9fae6c1 to 881239b Compare October 2, 2026 09:48
@Mikaka27 Mikaka27 removed the testing currently being tested, tag is used by OTP internal CI label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team:PS Assigned to OTP team PS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants