|
1 | 1 | .. _threading:
|
2 | 2 |
|
3 |
| -Using library in threaded environments |
4 |
| -====================================== |
| 3 | +Using the library in threaded environments |
| 4 | +========================================== |
5 | 5 |
|
6 | 6 | Network namespaces
|
7 | 7 | ------------------
|
8 | 8 |
|
9 |
| -In order to run a separate socket in network namespace `A` while keeping |
10 |
| -the Python process in namespace `B`, the library performs these steps: |
| 9 | +To run a separate socket in one network namespace while keeping the |
| 10 | +Python process in another namespace, the library follows these steps: |
11 | 11 |
|
12 |
| -1. spawn a child process |
13 |
| -2. execute `netns.setns()` in the child |
14 |
| -3. create a socket |
15 |
| -4. send the file descriptor using `socket.send_fds()` back to the parent |
16 |
| -5. terminate the child process |
17 |
| -6. create a socket with `socket(fileno=...)` |
| 12 | +1. Spawn a child process. |
| 13 | +2. Execute `netns.setns()` in the child. |
| 14 | +3. Create a socket. |
| 15 | +4. Send the file descriptor back to the parent using `socket.send_fds()`. |
| 16 | +5. Terminate the child process. |
| 17 | +6. Create a socket in the parent using `socket(fileno=...)`. |
18 | 18 |
|
19 |
| -As a result, in the parent process we get a socket that belongs to another |
20 |
| -network namespace, but we can use it natively like any other socket, both in |
21 |
| -sync and async way. |
| 19 | +As a result, the parent process obtains a socket belonging to |
| 20 | +another network namespace. However, it can be used natively like |
| 21 | +any other socket, both synchronously and asynchronously. |
22 | 22 |
|
23 |
| -Start a child: os.fork() |
24 |
| ------------------------- |
| 23 | +Starting a child process: os.fork() |
| 24 | +----------------------------------- |
25 | 25 |
|
26 |
| -By default, pyroute2 uses `os.fork()` to create the child. When using it |
27 |
| -in multithreaded processes, no threads will be recreated, and the child |
28 |
| -will only continue the thread where `os.fork()` was called in. This leaves |
29 |
| -the GC in a corrupted state. |
| 26 | +By default, pyroute2 uses `os.fork()` to create the child process. |
| 27 | +In multithreaded environments, `os.fork()` does not recreate threads; |
| 28 | +the child process continues only from the thread where `os.fork()` |
| 29 | +was called. This can leave the garbage collector in a corrupted state. |
30 | 30 |
|
31 |
| -This should not be an issue since the socket creation routine stops the GC, |
32 |
| -and doesn't rely on any shared data. But still there is some risk. |
| 31 | +While this is generally not an issue -- since the socket creation routine |
| 32 | +stops the garbage collector, and does not rely on shared data -- there |
| 33 | +is still some risk. |
33 | 34 |
|
34 |
| -That's why pyroute2 provides a configuration option |
35 |
| -`config.child_process_mode`. By default it is `"fork"`, but you can change |
36 |
| -the option to `"mp"`, and then pyroute2 will use `multiprocessing` to |
37 |
| -create and control the child process. |
| 35 | +To address this, pyroute2 provides a configuration option: |
| 36 | +`config.child_process_mode`. The default value is `"fork"`, but you |
| 37 | +can change it to `"mp"` to use the `multiprocessing` module for creating |
| 38 | +and managing the child process. |
38 | 39 |
|
39 |
| -Start a child: multiprocessing |
40 |
| ------------------------------- |
| 40 | +Starting a child process: multiprocessing |
| 41 | +----------------------------------------- |
41 | 42 |
|
42 |
| -Using `multiprocessing` may or may not rely on `os.fork()`, it depends on |
43 |
| -`multiprocessing.set_start_method()`. In Python version < 3.14 the default |
44 |
| -is `"fork"`, but starting with 3.14, the default is `"spawn"`. |
| 43 | +The `multiprocessing` module may or may not rely on `os.fork()`, depending |
| 44 | +on the method set via `multiprocessing.set_start_method()`. In Python |
| 45 | +versions earlier than 3.14, the default method is `"fork"`, but starting |
| 46 | +from Python 3.14, the default is `"spawn"`. |
45 | 47 |
|
46 |
| -Using `"spawn"` is safer, but significantly slower. Beside of that, `"spawn"` |
47 |
| -implies additional limitations by pickling the target method and its |
48 |
| -arguments. This prevents passing lambdas as the target, and make impossible |
49 |
| -to pass the `libc` instance to the child process. |
| 48 | +Using `"spawn"` is safer but significantly slower. Additionally, `"spawn"` |
| 49 | +introduces limitations due to pickling: |
50 | 50 |
|
51 |
| -The `multiprocessing` start method is out of scope for pyroute2, thus no |
52 |
| -way to set it using `config.child_process_mode`, where you can specify |
53 |
| -only `"mp"`. You have to run `multiprocessing.set_start_method()` by |
54 |
| -yourself somewhere else in your program then. |
| 51 | +* The target function and its arguments must be pickleable. |
| 52 | +* Passing lambda functions as the target is not possible. |
| 53 | +* The libc instance cannot be passed to the child process. |
| 54 | + |
| 55 | +Since pyroute2 does not manage the `multiprocessing` start method, the |
| 56 | +start method cannot be configured via `config.child_process_mode`. If you |
| 57 | +set `config.child_process_mode` to `"mp"`, and need to explicitly specify |
| 58 | +the start method, you must call `multiprocessing.set_start_method()` |
| 59 | +manually elsewhere in your program. |
| 60 | + |
| 61 | +Threading and asyncio |
| 62 | +--------------------- |
| 63 | + |
| 64 | +An asyncio event loop can only run in the thread where it was started. |
| 65 | +In a multithreaded environment, the library creates a local event loop |
| 66 | +and a local netlink socket for each thread that accesses the object. |
| 67 | +While this approach is safe, it complicates object termination. |
| 68 | +Although an event loop can be stopped from another thread, it cannot |
| 69 | +be closed. |
| 70 | + |
| 71 | +The best solution is to call `close()` in every thread where you |
| 72 | +call `bind()`. |
0 commit comments