Skip to content

Commit badf570

Browse files
committedMar 19, 2025··
docs: add asyncio vs. threading
1 parent c4c4ec9 commit badf570

File tree

2 files changed

+56
-59
lines changed

2 files changed

+56
-59
lines changed
 

‎docs/asyncio.rst

-21
Original file line numberDiff line numberDiff line change
@@ -184,24 +184,3 @@ such as `GenericNetlinkSocket`, or using custom wrappers, like in
184184
`IPRoute`. The plan is to refactor all components to provide an asynchronous
185185
API, keeping the synchronous API for compatibility with existing projects
186186
that use pyroute2.
187-
188-
Thread safety
189-
-------------
190-
191-
Is the current core thread-safe? Yes and no at the same time. While there
192-
are no locks in the core, components like sockets and message queues are
193-
now thread-local.
194-
195-
This means that the same pyroute2 socket object manages as many
196-
underlying netlink sockets as there are threads accessing it.
197-
198-
Pros:
199-
200-
* Simplicity and absence of mutexes, which eliminates the risk of deadlocks.
201-
202-
Cons:
203-
204-
* Race conditions are still possible if shared data is not thread-local.
205-
* Debugging existing netlink flows at runtime is limited because any
206-
debugger session will create its own underlying netlink socket. This
207-
makes logging and post-mortem analysis more important.

‎docs/threading.rst

+56-38
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,72 @@
11
.. _threading:
22

3-
Using library in threaded environments
4-
======================================
3+
Using the library in threaded environments
4+
==========================================
55

66
Network namespaces
77
------------------
88

9-
In order to run a separate socket in network namespace `A` while keeping
10-
the Python process in namespace `B`, the library performs these steps:
9+
To run a separate socket in one network namespace while keeping the
10+
Python process in another namespace, the library follows these steps:
1111

12-
1. spawn a child process
13-
2. execute `netns.setns()` in the child
14-
3. create a socket
15-
4. send the file descriptor using `socket.send_fds()` back to the parent
16-
5. terminate the child process
17-
6. create a socket with `socket(fileno=...)`
12+
1. Spawn a child process.
13+
2. Execute `netns.setns()` in the child.
14+
3. Create a socket.
15+
4. Send the file descriptor back to the parent using `socket.send_fds()`.
16+
5. Terminate the child process.
17+
6. Create a socket in the parent using `socket(fileno=...)`.
1818

19-
As a result, in the parent process we get a socket that belongs to another
20-
network namespace, but we can use it natively like any other socket, both in
21-
sync and async way.
19+
As a result, the parent process obtains a socket belonging to
20+
another network namespace. However, it can be used natively like
21+
any other socket, both synchronously and asynchronously.
2222

23-
Start a child: os.fork()
24-
------------------------
23+
Starting a child process: os.fork()
24+
-----------------------------------
2525

26-
By default, pyroute2 uses `os.fork()` to create the child. When using it
27-
in multithreaded processes, no threads will be recreated, and the child
28-
will only continue the thread where `os.fork()` was called in. This leaves
29-
the GC in a corrupted state.
26+
By default, pyroute2 uses `os.fork()` to create the child process.
27+
In multithreaded environments, `os.fork()` does not recreate threads;
28+
the child process continues only from the thread where `os.fork()`
29+
was called. This can leave the garbage collector in a corrupted state.
3030

31-
This should not be an issue since the socket creation routine stops the GC,
32-
and doesn't rely on any shared data. But still there is some risk.
31+
While this is generally not an issue -- since the socket creation routine
32+
stops the garbage collector, and does not rely on shared data -- there
33+
is still some risk.
3334

34-
That's why pyroute2 provides a configuration option
35-
`config.child_process_mode`. By default it is `"fork"`, but you can change
36-
the option to `"mp"`, and then pyroute2 will use `multiprocessing` to
37-
create and control the child process.
35+
To address this, pyroute2 provides a configuration option:
36+
`config.child_process_mode`. The default value is `"fork"`, but you
37+
can change it to `"mp"` to use the `multiprocessing` module for creating
38+
and managing the child process.
3839

39-
Start a child: multiprocessing
40-
------------------------------
40+
Starting a child process: multiprocessing
41+
-----------------------------------------
4142

42-
Using `multiprocessing` may or may not rely on `os.fork()`, it depends on
43-
`multiprocessing.set_start_method()`. In Python version < 3.14 the default
44-
is `"fork"`, but starting with 3.14, the default is `"spawn"`.
43+
The `multiprocessing` module may or may not rely on `os.fork()`, depending
44+
on the method set via `multiprocessing.set_start_method()`. In Python
45+
versions earlier than 3.14, the default method is `"fork"`, but starting
46+
from Python 3.14, the default is `"spawn"`.
4547

46-
Using `"spawn"` is safer, but significantly slower. Beside of that, `"spawn"`
47-
implies additional limitations by pickling the target method and its
48-
arguments. This prevents passing lambdas as the target, and make impossible
49-
to pass the `libc` instance to the child process.
48+
Using `"spawn"` is safer but significantly slower. Additionally, `"spawn"`
49+
introduces limitations due to pickling:
5050

51-
The `multiprocessing` start method is out of scope for pyroute2, thus no
52-
way to set it using `config.child_process_mode`, where you can specify
53-
only `"mp"`. You have to run `multiprocessing.set_start_method()` by
54-
yourself somewhere else in your program then.
51+
* The target function and its arguments must be pickleable.
52+
* Passing lambda functions as the target is not possible.
53+
* The libc instance cannot be passed to the child process.
54+
55+
Since pyroute2 does not manage the `multiprocessing` start method, the
56+
start method cannot be configured via `config.child_process_mode`. If you
57+
set `config.child_process_mode` to `"mp"`, and need to explicitly specify
58+
the start method, you must call `multiprocessing.set_start_method()`
59+
manually elsewhere in your program.
60+
61+
Threading and asyncio
62+
---------------------
63+
64+
An asyncio event loop can only run in the thread where it was started.
65+
In a multithreaded environment, the library creates a local event loop
66+
and a local netlink socket for each thread that accesses the object.
67+
While this approach is safe, it complicates object termination.
68+
Although an event loop can be stopped from another thread, it cannot
69+
be closed.
70+
71+
The best solution is to call `close()` in every thread where you
72+
call `bind()`.

0 commit comments

Comments
 (0)
Please sign in to comment.