-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
937 lines (749 loc) · 275 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Herry's blog</title>
<link href="/atom.xml" rel="self"/>
<link href="https://www.songcser.com/"/>
<updated>2017-12-07T07:07:34.000Z</updated>
<id>https://www.songcser.com/</id>
<author>
<name>songcser</name>
<email>[email protected]</email>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>asyncio之调试</title>
<link href="https://www.songcser.com/2017/11/21/debugging-with-asyncio/"/>
<id>https://www.songcser.com/2017/11/21/debugging-with-asyncio/</id>
<published>2017-11-21T01:44:14.000Z</published>
<updated>2017-12-07T07:07:34.000Z</updated>
<content type="html"><![CDATA[<p>asyncio构建了一些有用的调试特性。</p>
<p>首先，事件循环在运行时使用logging来发送状态信息。在应用中启用日志就会获取到一些信息。其他的可以打开告诉事件循环来发送更多的调试信息。通过调用set_debug()传递boolean值来表明是否启动调试。</p>
<p>由于构建在asyncio上的应用程序对贪婪的协程很敏感而无法返回控制权，因此对于检查内置于事件循环中的慢回调是有帮助的。通过打开启用调试，并且通过将loop的slow_callback_duration的属性设置为发出警告的秒数来控制”slow”的定义。</p>
<p>最后，如果应用程序使用asyncio退出但没有清理一些协程或者其他的资源，这或许意味着有一个逻辑错误阻止了一些应用代码运行。在程序退出时，启用ResourceWarning警告会报告这些情况。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_debug.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> argparse</div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"><span class="keyword">import</span> time</div><div class="line"><span class="keyword">import</span> warnings</div><div class="line"></div><div class="line">parser = argparse.ArgumentParser(<span class="string">'debugging asyncio'</span>)</div><div class="line">parser.add_argument(</div><div class="line"> <span class="string">'-v'</span>,</div><div class="line"> dest=<span class="string">'verbose'</span>,</div><div class="line"> default=<span class="keyword">False</span>,</div><div class="line"> action=<span class="string">'store_true'</span>,</div><div class="line">)</div><div class="line">args = parser.parse_args()</div><div class="line"></div><div class="line">logging.basicConfig(</div><div class="line"> level=logging.DEBUG,</div><div class="line"> format=<span class="string">'%(levelname)7s: %(message)s'</span>,</div><div class="line"> stream=sys.stderr,</div><div class="line">)</div><div class="line">LOG = logging.getLogger(<span class="string">''</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">inner</span><span class="params">()</span>:</span></div><div class="line"> LOG.info(<span class="string">'inner starting'</span>)</div><div class="line"> <span class="comment"># Use a blocking sleep to simulate</span></div><div class="line"> <span class="comment"># doing work inside the function.</span></div><div class="line"> time.sleep(<span class="number">0.1</span>)</div><div class="line"> LOG.info(<span class="string">'inner completed'</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">outer</span><span class="params">(loop)</span>:</span></div><div class="line"> LOG.info(<span class="string">'outer starting'</span>)</div><div class="line"> <span class="keyword">await</span> asyncio.ensure_future(loop.create_task(inner()))</div><div class="line"> LOG.info(<span class="string">'outer completed'</span>)</div><div class="line"></div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">if</span> args.verbose:</div><div class="line"> LOG.info(<span class="string">'enabling debugging'</span>)</div><div class="line"></div><div class="line"> <span class="comment"># Enable debugging</span></div><div class="line"> event_loop.set_debug(<span class="keyword">True</span>)</div><div class="line"></div><div class="line"> <span class="comment"># Make the threshold for "slow" tasks very very small for</span></div><div class="line"> <span class="comment"># illustration. The default is 0.1, or 100 milliseconds.</span></div><div class="line"> event_loop.slow_callback_duration = <span class="number">0.001</span></div><div class="line"></div><div class="line"> <span class="comment"># Report all mistakes managing asynchronous resources.</span></div><div class="line"> warnings.simplefilter(<span class="string">'always'</span>, ResourceWarning)</div><div class="line"></div><div class="line">LOG.info(<span class="string">'entering event loop'</span>)</div><div class="line">event_loop.run_until_complete(outer(event_loop))</div></pre></td></tr></table></figure>
<p>当没有启用调试运行时，所有都看起来是好的。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_debug.py</div><div class="line"></div><div class="line">DEBUG: Using selector: KqueueSelector</div><div class="line"> INFO: entering event loop</div><div class="line"> INFO: outer starting</div><div class="line"> INFO: inner starting</div><div class="line"> INFO: inner completed</div><div class="line"> INFO: outer completed</div></pre></td></tr></table></figure>
<p>打开调试将暴露一些问题，包括虽然inner()方法完成，但是由于设置了slow_callback_duration，将需要更多的时间，并且程序退出时事件循环未被正确关闭的事实。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_debug.py -v</div><div class="line">DEBUG: Using selector: KqueueSelector</div><div class="line"> INFO: enabling debugging</div><div class="line"> INFO: entering event loop</div><div class="line"> INFO: outer starting</div><div class="line"> INFO: inner starting</div><div class="line"> INFO: inner completed</div><div class="line">WARNING: Executing <Task finished coro=<inner() done, defined at asyncio_debug.py:34> result=None created at asyncio_debug.py:44></div><div class="line">took 0.102 seconds</div><div class="line"> INFO: outer completed</div><div class="line">.../lib/python3.5/async/base_events.py:429: ResourceWarning:unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=Trur></div><div class="line"> DEBUG: Close <_UnixSelectorEventLoop running=False closed=False debug=True></div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/debugging.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>asyncio构建了一些有用的调试特性。</p>
<p>首先，事件循环在运行时使用logging来发送状态信息。在应用中启用日志就会获取到一些信息。其他的可以打开告诉事件循环来发送更多的调试信息。通过调用set_debug()传递boolean值来表明是否启动调试。</p>
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之协程和线程进程组合</title>
<link href="https://www.songcser.com/2017/11/13/combining-coroutines-with-threads-and-processes/"/>
<id>https://www.songcser.com/2017/11/13/combining-coroutines-with-threads-and-processes/</id>
<published>2017-11-13T07:32:01.000Z</published>
<updated>2017-12-13T05:56:05.000Z</updated>
<content type="html"><![CDATA[<p>许多已经存在的库还没有使用原生的asyncio。可能造成阻塞或者依赖并发特性的模块不可用。还是有可能在基于asyncio的应用中使用这些库，通过使用concurrent.futures的executor可以在一个分离的线程或者进程中运行代码。</p>
<h2 id="线程"><a href="#线程" class="headerlink" title="线程"></a>线程</h2><p>事件循环的run_in_executor()方法生成executor实例，一个常规的函数调用，并且传递一些参数。方法返回Future对象，常常用来等待函数完成任务和返回值。如果没有传入执行executor，就会创建ThreadPoolExecutor实例。这个例子明确的创建一个executor，来限制可获得工作线程的数量。</p>
<p>ThreadPoolExecutor启动工作线程，然后在线程中调用每一个提供的函数。这个例子显示了如何组合run_in_executor()和wait()方法在一个协程中，当阻塞函数在一个分离的线程中运行时将控制权返还给事件循环，并且在阻塞函数完成时再唤醒。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_executor_thread.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> concurrent.futures</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"><span class="keyword">import</span> time</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">blocks</span><span class="params">(n)</span>:</span></div><div class="line"> log = logging.getLogger(<span class="string">'blocks({})'</span>.format(n))</div><div class="line"> log.info(<span class="string">'running'</span>)</div><div class="line"> time.sleep(<span class="number">0.1</span>)</div><div class="line"> log.info(<span class="string">'done'</span>)</div><div class="line"> <span class="keyword">return</span> n ** <span class="number">2</span></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">run_blocking_tasks</span><span class="params">(executor)</span>:</span></div><div class="line"> log = logging.getLogger(<span class="string">'run_blocking_tasks'</span>)</div><div class="line"> log.info(<span class="string">'starting'</span>)</div><div class="line"></div><div class="line"> log.info(<span class="string">'creating executor tasks'</span>)</div><div class="line"> loop = asyncio.get_event_loop()</div><div class="line"> blocking_tasks = [</div><div class="line"> loop.run_in_executor(executor, blocks, i)</div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">6</span>)</div><div class="line"> ]</div><div class="line"> log.info(<span class="string">'waiting for executor tasks'</span>)</div><div class="line"> completed, pending = <span class="keyword">await</span> asyncio.wait(blocking_tasks)</div><div class="line"> results = [t.result() <span class="keyword">for</span> t <span class="keyword">in</span> completed]</div><div class="line"> log.info(<span class="string">'results: {!r}'</span>.format(results))</div><div class="line"></div><div class="line"> log.info(<span class="string">'exiting'</span>)</div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</div><div class="line"> <span class="comment"># Configure logging to show the name of the thread</span></div><div class="line"> <span class="comment"># where the log message originates.</span></div><div class="line"> logging.basicConfig(</div><div class="line"> level=logging.INFO,</div><div class="line"> format=<span class="string">'%(threadName)10s %(name)18s: %(message)s'</span>,</div><div class="line"> stream=sys.stderr,</div><div class="line"> )</div><div class="line"></div><div class="line"> <span class="comment"># Create a limited thread pool</span></div><div class="line"> executor = concurrent.futures.ThreadPoolExecutor(</div><div class="line"> max_workers=<span class="number">3</span>,</div><div class="line"> )</div><div class="line"></div><div class="line"> event_loop = asyncio.get_event_loop()</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(</div><div class="line"> run_blocking_tasks(executor)</div><div class="line"> )</div><div class="line"> <span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>asyncio_executor_thread.py使用logging可以方便的表明哪一个线程和函数产生的每一条日志信息。由于在每一个blocks()函数中使用分离的logger,以下输出清晰的显示了重复使用相同的线程以不同的参数调用函数的多个副本。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_executor_thread.py</div><div class="line"></div><div class="line">MainThread run_blocking_tasks: starting</div><div class="line">MainThread run_blocking_tasks: creating executor tasks</div><div class="line"> Thread-1 blocks(0): running</div><div class="line"> Thread-2 blocks(1): running</div><div class="line"> Thread-3 blocks(2): running</div><div class="line">MainThread run_blocking_tasks: waiting for executor tasks</div><div class="line"> Thread-1 blocks(0): done</div><div class="line"> Thread-3 blocks(2): done</div><div class="line"> Thread-1 blocks(3): running</div><div class="line"> Thread-2 blocks(1): done</div><div class="line"> Thread-3 blocks(4): running</div><div class="line"> Thread-2 blocks(5): running</div><div class="line"> Thread-1 blocks(3): done</div><div class="line"> Thread-2 blocks(5): done</div><div class="line"> Thread-3 blocks(4): done</div><div class="line">MainThread run_blocking_tasks: results: [16, 4, 1, 0, 2, 25, 9]</div><div class="line">MainThread run_blocking_tasks: exiting</div></pre></td></tr></table></figure>
<h2 id="进程"><a href="#进程" class="headerlink" title="进程"></a>进程</h2><p>ProcessPoolExecutor以相同的方式工作，代替线程创建一些进程工作。使用分离的进程需要更多的系统资源，但是对于计算密集型操作可以合理的在每一个CPU运行单独的任务。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_executor_process.py</span></div><div class="line"></div><div class="line"><span class="comment"># changes from asyncio_executor_thread.py</span></div><div class="line"></div><div class="line"><span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:</div><div class="line"> <span class="comment"># Configure logging to show the id of the process</span></div><div class="line"> <span class="comment"># where the log message originates.</span></div><div class="line"> logging.basicConfig(</div><div class="line"> level=logging.INFO,</div><div class="line"> format=<span class="string">'PID % (process)5s %(name)18s: %(message)s'</span>,</div><div class="line"> stream=sys.stderr,</div><div class="line"> )</div><div class="line"></div><div class="line"> <span class="comment"># Create a limited process pool.</span></div><div class="line"> executor = concurrent.futures.ProcessPoolExecutor(</div><div class="line"> max_workers=<span class="number">3</span>,</div><div class="line"> )</div><div class="line"></div><div class="line"> event_loop = asyncio.get_event_loop()</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(</div><div class="line"> run_blocking_tasks(executor)</div><div class="line"> )</div><div class="line"> <span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>将线程换成进程唯一的改变是创建一个不同类型的executor。这个例子也改变了日志格式字符串，将线程名字替换成了进程ID，来表明任务事实上运行在单独的进程中。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_executor_process.py</div><div class="line"></div><div class="line">PID 16429 run_blocking_tasks: starting</div><div class="line">PID 16429 run_blocking_tasks: creating executor tasks</div><div class="line">PID 16429 run_blocking_tasks: waiting for executor tasks</div><div class="line">PID 16430 blocks(0): running</div><div class="line">PID 16431 blocks(1): running</div><div class="line">PID 16332 blocks(2): running</div><div class="line">PID 16430 blocks(0): done</div><div class="line">PID 16432 blocks(2): done</div><div class="line">PID 16431 blocks(1): done</div><div class="line">PID 16430 blocks(3): running</div><div class="line">PID 16432 blocks(4): running</div><div class="line">PID 16431 blocks(5): running</div><div class="line">PID 16431 blocks(5): done</div><div class="line">PID 16432 blocks(4): done</div><div class="line">PID 16430 blocks(3): done</div><div class="line">PID 16429 run_blocking_tasks: results: [4, 0, 16, 1, 9, 25]</div><div class="line">PID 16429 run_blocking_tasks: exiting</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/executors.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>许多已经存在的库还没有使用原生的asyncio。可能造成阻塞或者依赖并发特性的模块不可用。还是有可能在基于asyncio的应用中使用这些库，通过使用concurrent.futures的executor可以在一个分离的线程或者进程中运行代码。</p>
<h2 id="线程"
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之接收Unix信号</title>
<link href="https://www.songcser.com/2017/11/06/receiving-unix-signals/"/>
<id>https://www.songcser.com/2017/11/06/receiving-unix-signals/</id>
<published>2017-11-06T01:12:54.000Z</published>
<updated>2017-12-12T07:32:00.000Z</updated>
<content type="html"><![CDATA[<p>通常Unix系统事件通知需要中断应用来触发执行。当使用asyncio时，信号处理回调会与事件循环管理的其他协程的回调交错进行。这导致中断的功能更少，因此需要提供安全防护来清理不完整的操作。</p>
<p>信号处理必须是常规的函数调用，不是协程。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_signal.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> functools</div><div class="line"><span class="keyword">import</span> os</div><div class="line"><span class="keyword">import</span> signal</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">signal_handler</span><span class="params">(name)</span>:</span></div><div class="line"> print(<span class="string">'signal_handler({!r})'</span>.format(name))</div></pre></td></tr></table></figure>
<p>使用add_signal_handler()注册信号处理。第一个参数是信号，第二个参数是回调。回调函数是没有参数的，所以如果函数需要参数可以使用functools.partial()。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"></div><div class="line">event_loop.add_signal_handler(</div><div class="line"> signal.SIGHUP,</div><div class="line"> functools.partial(signal_handler, name=<span class="string">"SIGHUP"</span>),</div><div class="line">)</div><div class="line">event_loop.add_signal_handler(</div><div class="line"> signal.SIGUSR1,</div><div class="line"> functools.partial(signal_handler, name=<span class="string">'SIGUSR1'</span>),</div><div class="line">)</div><div class="line">event_loop.add_signal_handler(</div><div class="line"> signal.SIGINT,</div><div class="line"> functools.partial(signal_handler, name=<span class="string">'SIGINT'</span>),</div><div class="line">)</div></pre></td></tr></table></figure>
<p>这个示例使用协程通过os.kill()发送信号给它自己。信号发送之后，协程让出控制来让处理程序运行。在一般的应用中，有更多的地方是应用代码返还给事件循环而不是像这样人为的返还。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">send_signals</span><span class="params">()</span>:</span></div><div class="line"> pid = os.getpid()</div><div class="line"> print(<span class="string">'starting send_signals for {}'</span>.format(pid))</div><div class="line"></div><div class="line"> <span class="keyword">for</span> name <span class="keyword">in</span> [<span class="string">'SIGHUP'</span>, <span class="string">'SIGHUP'</span>, <span class="string">'SIGUSR1'</span>, <span class="string">'SIGINT'</span>]:</div><div class="line"> print(<span class="string">'sending {}'</span>.format(name))</div><div class="line"> os.kill(pid, getattr(signal, name))</div><div class="line"> <span class="comment"># Yield control to allow the signal handler to run,</span></div><div class="line"> <span class="comment"># since the signal does not interrupt the program</span></div><div class="line"> <span class="comment"># flow otherwise.</span></div><div class="line"> print(<span class="string">'yielding control'</span>)</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.01</span>)</div><div class="line"> <span class="keyword">return</span></div></pre></td></tr></table></figure>
<p>主程序运行send_signals()直到所有的信号发送完。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(send_signals())</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>输出显示当发送信号之后send_signals()让出控制权，处理函数被调用。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_signal.py</div><div class="line"></div><div class="line">starting send_signals for 21772</div><div class="line">sending SIGHUP</div><div class="line">yielding control</div><div class="line">signal_handler('SIGHUP')</div><div class="line">sending SIGHUP</div><div class="line">yielding control</div><div class="line">signal_handler('SIGHUP')</div><div class="line">sending SIGUSR1</div><div class="line">yielding control</div><div class="line">yield_handler('SIGUSR1')</div><div class="line">sending SIGINT</div><div class="line">yielding control</div><div class="line">signal_handler('SIGINT')</div></pre></td></tr></table></figure>
<blockquote>
<blockquote>
<blockquote>
<p><a href="https://pymotw.com/3/signal/index.html#module-signal" target="_blank" rel="external">signal</a> - Receive notification of asynchronous system events</p>
</blockquote>
</blockquote>
</blockquote>
<p><a href="https://pymotw.com/3/asyncio/unix_signals.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>通常Unix系统事件通知需要中断应用来触发执行。当使用asyncio时，信号处理回调会与事件循环管理的其他协程的回调交错进行。这导致中断的功能更少，因此需要提供安全防护来清理不完整的操作。</p>
<p>信号处理必须是常规的函数调用，不是协程。</p>
<figure cl
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之子进程交互</title>
<link href="https://www.songcser.com/2017/10/26/working-with-subprocesses/"/>
<id>https://www.songcser.com/2017/10/26/working-with-subprocesses/</id>
<published>2017-10-26T07:58:09.000Z</published>
<updated>2017-12-12T07:17:22.000Z</updated>
<content type="html"><![CDATA[<p>经常需要和其他的一些程序和进程一起工作，以便使用已经存在的代码而不用重写，或者访问Python中已经不可用的库或特性。和网络IO一样，asyncio包含两个抽象概念来启动其他程序然后进行交互。</p>
<h2 id="对于Subprocesses使用Protocol-Abstraction"><a href="#对于Subprocesses使用Protocol-Abstraction" class="headerlink" title="对于Subprocesses使用Protocol Abstraction"></a>对于Subprocesses使用Protocol Abstraction</h2><p>这个例子使用协程来启动一个进程运行Unix命令df，寻找磁盘的剩余空间。使用subprocess_exec()启动进程并把它绑定到protocol类，这样就知道如何读取df的命令输出并分析。当subprocess的IO事件发生时protocol类的方法被自动调用。由于stdin和stderr参数都设置为Nore，这些通信管道没有连接到新的进程。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_subprocess_protocol.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> functools</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">run_df</span><span class="params">(loop)</span>:</span></div><div class="line"> print(<span class="string">'in run_df'</span>)</div><div class="line"></div><div class="line"> cmd_done = asyncio.Future(loop=loop)</div><div class="line"> factory = functools.partial(DFProtocol, cmd_done)</div><div class="line"> proc = loop.subprocess_exec(</div><div class="line"> factory,</div><div class="line"> <span class="string">'df'</span>, <span class="string">'-hl'</span>,</div><div class="line"> stdin=<span class="keyword">None</span>,</div><div class="line"> stderr=<span class="keyword">None</span>,</div><div class="line"> )</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> print(<span class="string">'launching process'</span>)</div><div class="line"> transport, protocol = <span class="keyword">await</span> proc</div><div class="line"> print(<span class="string">'waiting for process to complete'</span>)</div><div class="line"> <span class="keyword">await</span> cmd_done</div><div class="line"> <span class="keyword">finally</span>:</div><div class="line"> transport.close()</div><div class="line"></div><div class="line"> <span class="keyword">return</span> cmd_done.result()</div></pre></td></tr></table></figure>
<p>DFProtocol类由SubprocessProtocol派生出，定义类API和其他进程通过管道进行通信。done参数期望是Future对象，这样调用者可以监听到进程的完成。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">DFProtocol</span><span class="params">(asyncio.SubprocessProtocol)</span>:</span></div><div class="line"> </div><div class="line"> FD_NAMES = [<span class="string">'stdin'</span>, <span class="string">'stdout'</span>, <span class="string">'stderr'</span>]</div><div class="line"></div><div class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, done_future)</span>:</span></div><div class="line"> self.done = done_future</div><div class="line"> self.buffer = bytearray()</div><div class="line"> super().__init__()</div></pre></td></tr></table></figure>
<p>和socket连接一样，当输入管道有新的进程建立时，connection_made()方法被调用。transport参数是BaseSubprocessTransport子类的实例。如果进程配置成可以接收输入，可以通过进程读取数据输出，并且写入数据到进程的输入流。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">connection_made</span><span class="params">(self, transport)</span>:</span></div><div class="line"> print(<span class="string">'process started {}'</span>.format(transport.get_pid()))</div><div class="line"> self.transport = transport</div></pre></td></tr></table></figure>
<p>当进程产生输出，pipe_data_received()方法被调用，获取存储数据的文件描述符，并且从管道中读取真实数据。protocol类将进程的标准输出管道的输出保存在缓冲区中供以后处理。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">pipe_data_received</span><span class="params">(self, fd, data)</span>:</span></div><div class="line"> print(<span class="string">'read {} bytes from {}'</span>.format(len(data), self.FD_NAMES[fd]))</div><div class="line"> <span class="keyword">if</span> fd == <span class="number">1</span>:</div><div class="line"> self.buffer.extend(data)</div></pre></td></tr></table></figure>
<p>当进程中断，process_exited()被调用。transport对象通过调用get_returncode()方法可以获取进程的退出码。在这种情况下，如果没有错误报告，则在通过Future实例返回之前，可用的输出将被解码分析。如果有错误，则结果被假定为空。进程退出时告诉run_df()方法设置future的结果，所有该方法清理并且返回结果。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">process_exited</span><span class="params">(self)</span>:</span></div><div class="line"> print(<span class="string">'process exited'</span>)</div><div class="line"> return_code = self.transport.get_returncode()</div><div class="line"> print(<span class="string">'return code {}'</span>.format(return_code))</div><div class="line"> <span class="keyword">if</span> <span class="keyword">not</span> return_code:</div><div class="line"> cmd_output = bytes(self.buffer).decode()</div><div class="line"> results = self._parse_results(cmd_output)</div><div class="line"> <span class="keyword">else</span>:</div><div class="line"> results = []</div><div class="line"> self.done.set_result((return_code, results))</div></pre></td></tr></table></figure>
<p>将命令输出解析成一系列字典，将每个输出行的标题名映射到它的值，并返回结果列表。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">_parse_results</span><span class="params">(self, output)</span>:</span></div><div class="line"> print(<span class="string">'parsing results'</span>)</div><div class="line"> <span class="comment"># Output has one row of headers, all single words.</span></div><div class="line"> <span class="comment"># remaining rows are one per filesystem, with columns</span></div><div class="line"> <span class="comment"># matching the headers (assuming that none of the</span></div><div class="line"> <span class="comment"># mount points have whitespace in the names)</span></div><div class="line"> <span class="keyword">if</span> <span class="keyword">not</span> output:</div><div class="line"> <span class="keyword">return</span> []</div><div class="line"> lines = output.splitlines()</div><div class="line"> headers = lines[<span class="number">0</span>].split()</div><div class="line"> devices = lines[<span class="number">1</span>:]</div><div class="line"> results = [</div><div class="line"> dict(zip(headers, line.split()))</div><div class="line"> <span class="keyword">for</span> line <span class="keyword">in</span> devices</div><div class="line"> ]</div><div class="line"> <span class="keyword">return</span> results</div></pre></td></tr></table></figure>
<p>run_df()协程使用run_until_complete()方法运行，检测每一个设备的剩余空间并打印。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> return_code, results = event_loop.run_until_complete(</div><div class="line"> run_df(event_loop)</div><div class="line"> )</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div><div class="line"></div><div class="line"><span class="keyword">if</span> return_code:</div><div class="line"> print(<span class="string">'error exit {}'</span>.format(return_code))</div><div class="line"><span class="keyword">else</span>:</div><div class="line"> print(<span class="string">'\nFree space:'</span>)</div><div class="line"> <span class="keyword">for</span> r <span class="keyword">in</span> results:</div><div class="line"> print(<span class="string">'{Mounted:25}: {Avail}'</span>.format(**r))</div></pre></td></tr></table></figure>
<p>下面的输出显示执行步骤的顺序，和运行系统上三个设备的剩余空间。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_subprocess_protocol.py</div><div class="line"></div><div class="line">in run_df</div><div class="line">launching process</div><div class="line">process started 49675</div><div class="line">waiting for process to complete</div><div class="line">read 332 bytes from stdout</div><div class="line">process exited</div><div class="line">return code 0</div><div class="line">parsing results</div><div class="line"></div><div class="line">Free space:</div><div class="line">/ : 233Gi</div><div class="line">/Volumes/hubertinternal : 157Gi</div><div class="line">/Volumes/hubert-tm : 2.3Ti</div></pre></td></tr></table></figure>
<h2 id="使用协程和流调用子进程"><a href="#使用协程和流调用子进程" class="headerlink" title="使用协程和流调用子进程"></a>使用协程和流调用子进程</h2><p>使用协程直接运行进程，而不是通过Protocol子类访问，请调用create_subprocess_exec()并且指定要连接到的管道stdout, stderr和stdin。协程生成子进程的结果是一个Process实例，可用于管理子进程或与之通信。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_subprocess_coroutine.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> asyncio.subprocess</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">run_df</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in run_df'</span>)</div><div class="line"></div><div class="line"> buffer = bytearray()</div><div class="line"></div><div class="line"> create = asyncio.create_subprocess_exec(</div><div class="line"> <span class="string">'df'</span>, <span class="string">'-hl'</span>,</div><div class="line"> stdout=asyncio.subprocess.PIPE,</div><div class="line"> )</div><div class="line"> print(<span class="string">'launching process'</span>)</div><div class="line"> proc = <span class="keyword">await</span> create</div><div class="line"> print(<span class="string">'process started {}'</span>.format(proc.pid))</div></pre></td></tr></table></figure>
<p>在这个例子中， 除了命令行参数外，df不需要任何输入，所以下一步就是读取所有的输出。Protocol没有控制同一时间读取多少数据。这个例子使用readline()，但是也可以直接调用read()方法读取数据而不是面向行级的。和protocol例子一样，命令的输出进行了缓存，稍后打印出来。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">while</span> <span class="keyword">True</span>:</div><div class="line"> line = <span class="keyword">await</span> proc.stdout.readline()</div><div class="line"> print(<span class="string">'read {!r}'</span>.format(line))</div><div class="line"> <span class="keyword">if</span> <span class="keyword">not</span> line:</div><div class="line"> print(<span class="string">'no more output from command'</span>)</div><div class="line"> <span class="keyword">break</span></div><div class="line"> buffer.extend(line)</div></pre></td></tr></table></figure>
<p>因为程序完成，没有更多输出时，readline()方法返回空字符串。为了保证正确清理进程，下一步是等待进程完全退出。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">print(<span class="string">'waiting for process to complete'</span>)</div><div class="line"><span class="keyword">await</span> proc.wait()</div></pre></td></tr></table></figure>
<p>此时可以检查退出状态，以确定是解析输出还是错误处理，因为它不产生输出。逻辑解析和前面的例子是一样的，但是是独立的函数（这里没有显示），因为没有protocol类来隐藏它。数据解析之后，结果和退出码返回给调用者。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">return_code = proc.returncode</div><div class="line">print(<span class="string">'return code {}'</span>.format(return_code))</div><div class="line"><span class="keyword">if</span> <span class="keyword">not</span> return_code:</div><div class="line"> cmd_output = bytes(buffer).decode()</div><div class="line"> results = _parse_results(cmd_output)</div><div class="line"><span class="keyword">else</span>:</div><div class="line"> results = []</div><div class="line"></div><div class="line"><span class="keyword">return</span> (return_code, results)</div></pre></td></tr></table></figure>
<p>主程序和基于protocol的示例看起来相似，因为实现的不同部分都是独立在run_df()方法中。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> return_code, results = event_loop.run_until_complete(</div><div class="line"> run_df()</div><div class="line"> )</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div><div class="line"></div><div class="line"><span class="keyword">if</span> return_code:</div><div class="line"> print(<span class="string">'error exit {}'</span>.format(return_code))</div><div class="line"><span class="keyword">else</span>:</div><div class="line"> print(<span class="string">'\nFree space:'</span>)</div><div class="line"> <span class="keyword">for</span> r <span class="keyword">in</span> results:</div><div class="line"> print(<span class="string">'{Mounted:25}: {Avail}'</span>.format(**r))</div></pre></td></tr></table></figure>
<p>由于df的输出每次读取一行，所以可以不断的显示程序的进度。否则，输出看起来和以前的例子相似。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_subprocess_coroutine.py</div><div class="line"></div><div class="line">in run_df</div><div class="line">launching process</div><div class="line">process started 49678</div><div class="line">read b'Filesystem Size Used Avail Capacity iusedifree %iused Mounted on\n'</div><div class="line">read b'/dev/disk2s2 446Gi 213Gi 233Gi 48% 5595508261015132 48% /\n'</div><div class="line">read b'/dev/disk1 465Gi 307Gi 157Gi 67% 8051492241281172 66% /Volumes/hubertinternal\n'</div><div class="line">read b'/dev/disk3s2 3.6Ti 1.4Ti 2.3Ti 38% 181837749306480579 37% /Volumes/hubert-tm\n'</div><div class="line">read b''</div><div class="line">no more output from command</div><div class="line">waiting for process to complete</div><div class="line">return code 0</div><div class="line">parsing results</div><div class="line"></div><div class="line">Free space:</div><div class="line">/ : 233Gi</div><div class="line">/Volumes/hubertinternal : 157Gi</div><div class="line">/Volumes/hubert-tm : 2.3Ti</div></pre></td></tr></table></figure>
<h2 id="向子进程发送数据"><a href="#向子进程发送数据" class="headerlink" title="向子进程发送数据"></a>向子进程发送数据</h2><p>以前的例子都是使用一个通信管道从第二个进程读取数据。经常需要向一个命令的进程发送数据。这个例子定义一个协程执行Unix命令tr来转换输入流中的字符。在这个示例中，tr用于将小写字母转换为大写字母。</p>
<p>to_upper()协程方法采用事件循环和字符串作为参数。它产生第二个进程运行”tr [:lower:] [:upper:]”</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_subprocess_coroutine_write.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> asyncio.subprocess</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">to_upper</span><span class="params">(input)</span>:</span></div><div class="line"> print(<span class="string">'in to_upper'</span>)</div><div class="line"></div><div class="line"> create = asyncio.create_subprocess_exec(</div><div class="line"> <span class="string">'tr'</span>, <span class="string">'[:lower:]'</span>, <span class="string">'[:upper:]'</span>,</div><div class="line"> stdout=asyncio.subprocess.PIPE,</div><div class="line"> stdin=asyncio.subprocess.PIPE,</div><div class="line"> )</div><div class="line"> print(<span class="string">'launching process'</span>)</div><div class="line"> proc = <span class="keyword">await</span> create</div><div class="line"> print(<span class="string">'pid {}'</span>.format(proc.pid))</div></pre></td></tr></table></figure>
<p>下面to_upper()方法使用Process的communicate()方法发送输入字符串到命令行并且异步的读取所有输出结果。和subprocess.Popen版本相同的方法，communicate()方法返回完整的字符串字节输出。如果一个命令产生的数据不适合存放到内存中，那么输入不能一次产生或者输出必须立即处理掉，可以直接使用Process的stdin，stdout和stderr处理而不是调用communicate()。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">print(<span class="string">'communicating with process'</span>)</div><div class="line">stdout, stderr = <span class="keyword">await</span> proc.communicate(input.encode())</div></pre></td></tr></table></figure>
<p>IO完成之后，等待进程完全退出确保进程清理干净。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">print(<span class="string">'waiting for process to complete'</span>)</div><div class="line"><span class="keyword">await</span> proc.wait()</div></pre></td></tr></table></figure>
<p>检查退出码，解码输出字符串，准备好协程的返回值。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">return_code = proc.returncode</div><div class="line">print(<span class="string">'return code {}'</span>.format(return_code))</div><div class="line"><span class="keyword">if</span> <span class="keyword">not</span> return_code:</div><div class="line"> results = bytes(stdout).decode()</div><div class="line"><span class="keyword">else</span>:</div><div class="line"> results = <span class="string">''</span></div><div class="line"></div><div class="line"><span class="keyword">return</span> (return_code, results)</div></pre></td></tr></table></figure>
<p>主程序创建message字符串进行转换，并且使用事件循环运行to_upper()和打印结果。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line">MESSAGE = <span class="string">"""</span></div><div class="line"><span class="string">This message will be converted to all caps.</span></div><div class="line"><span class="string">"""</span></div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> return_code, results = event_loop.run_until_complete(</div><div class="line"> to_upper(MESSAGE)</div><div class="line"> )</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div><div class="line"></div><div class="line"><span class="keyword">if</span> return_code:</div><div class="line"> print(<span class="string">'error exit {}'</span>.format(return_code))</div><div class="line"><span class="keyword">else</span>:</div><div class="line"> print(<span class="string">'Original: {!r}'</span>.format(MESSAGE))</div><div class="line"> print(<span class="string">'Changed : {!r}'</span>.format(results))</div></pre></td></tr></table></figure>
<p>输出显示了一系列操作和如何进行简单文字信息的转换。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_subprocess_coroutine_write.py</div><div class="line"></div><div class="line">in to_upper</div><div class="line">launching process</div><div class="line">pid 49684</div><div class="line">communicating with process</div><div class="line">waiting for process to complete</div><div class="line">return code 0</div><div class="line">Original: '\nThis message will be converted\nto all caps.\n'</div><div class="line">Changed : '\nTHIS MESSAGE WILL BE CONVERTED\nTO ALL CAPS.\n'</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/subprocesses.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>经常需要和其他的一些程序和进程一起工作，以便使用已经存在的代码而不用重写，或者访问Python中已经不可用的库或特性。和网络IO一样，asyncio包含两个抽象概念来启动其他程序然后进行交互。</p>
<h2 id="对于Subprocesses使用Protocol-Abs
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之域名服务交互</title>
<link href="https://www.songcser.com/2017/10/25/interacting-with-domain-name-services/"/>
<id>https://www.songcser.com/2017/10/25/interacting-with-domain-name-services/</id>
<published>2017-10-25T02:41:41.000Z</published>
<updated>2017-12-12T03:16:31.000Z</updated>
<content type="html"><![CDATA[<p>应用程序使用网络与服务器进行通信时可能需要域名服务(DNS)，如在主机名和IP地址之间进行转换。asyncio事件循环有方便的方法在后台进行处理，以避免在查询时阻塞。</p>
<h2 id="根据域名查找地址"><a href="#根据域名查找地址" class="headerlink" title="根据域名查找地址"></a>根据域名查找地址</h2><p>使用getaddrinfo()协程方法将主机名和端口号转换成IP或IPv6地址。和在socket模块中的方法版本一样，返回值是一个包含五个信息的tuple列表。</p>
<ol>
<li>地址族</li>
<li>地址类型</li>
<li>协议</li>
<li>服务器的规范名称</li>
<li>套接字地址元组适用于打开原始指定的端口上的服务器连接</li>
</ol>
<p>查询可以通过协议进行过滤，在这个例子中，只返回TCP的响应。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_getaddrinfo.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> socket</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"></div><div class="line">TARGETS = [</div><div class="line"> (<span class="string">'pymotw.com'</span>, <span class="string">'https'</span>),</div><div class="line"> (<span class="string">'doughellmann.com'</span>, <span class="string">'https'</span>),</div><div class="line"> (<span class="string">'python.org'</span>, <span class="string">'https'</span>),</div><div class="line">]</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop, targets)</span>:</span></div><div class="line"> <span class="keyword">for</span> target <span class="keyword">in</span> targets:</div><div class="line"> info = <span class="keyword">await</span> loop.getaddrinfo(</div><div class="line"> *target,</div><div class="line"> proto=socket.IPPROTO_TCP,</div><div class="line"> )</div><div class="line"> <span class="keyword">for</span> host <span class="keyword">in</span> info:</div><div class="line"> print(<span class="string">'{:20}: {}'</span>.format(target[<span class="number">0</span>], host[<span class="number">4</span>][<span class="number">0</span>]))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop, TARGETS))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>这个例子将主机名和协议名转换成IP地址和端口号</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_getaddrinfo.py</div><div class="line"></div><div class="line">pymotw.com : 66.33.211.242</div><div class="line">doughellmann.com : 66.33.211.240</div><div class="line">python.org : 23.253.135.79</div><div class="line">python.org : 2001:4802:7901::e60a:1375:0:6</div></pre></td></tr></table></figure>
<h2 id="根据地址查找域名"><a href="#根据地址查找域名" class="headerlink" title="根据地址查找域名"></a>根据地址查找域名</h2><p>getnameinfo()协程方法以相反的方向工作，将IP地址转换成主机名并且可能将端口号转换成协议名。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_getnameinfo.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> socket</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"></div><div class="line">TARGETS = [</div><div class="line"> (<span class="string">'66.33.211.242'</span>, <span class="number">443</span>),</div><div class="line"> (<span class="string">'104.130.43.121'</span>, <span class="number">443</span>)</div><div class="line">]</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop, targets)</span>:</span></div><div class="line"> <span class="keyword">for</span> target <span class="keyword">in</span> targets:</div><div class="line"> info = <span class="keyword">await</span> loop.getnameinfo(target)</div><div class="line"> print(<span class="string">'{:15}: {} {}'</span>.format(target[<span class="number">0</span>], *info))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop, TARGETS))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>这个例子显示从域名公司DreamHost的服务器返回了pymotw.com的IP地址。第二个IP地址检查是python.org, 没有返回主机名。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_getnameinfo.py</div><div class="line"></div><div class="line">66.33.211.242 : apache2-echo.catalina.dreamhost.com https</div><div class="line">104.130.43.121 : 104.130.43.121 https</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/dns.html" target="_blank" rel="external">原文地址</a></p>
]]></content>
<summary type="html">
<p>应用程序使用网络与服务器进行通信时可能需要域名服务(DNS)，如在主机名和IP地址之间进行转换。asyncio事件循环有方便的方法在后台进行处理，以避免在查询时阻塞。</p>
<h2 id="根据域名查找地址"><a href="#根据域名查找地址" class="head
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
<category term="DNS" scheme="https://www.songcser.com/tags/DNS/"/>
</entry>
<entry>
<title>asyncio之使用SSL</title>
<link href="https://www.songcser.com/2017/10/23/using-ssl/"/>
<id>https://www.songcser.com/2017/10/23/using-ssl/</id>
<published>2017-10-23T09:50:35.000Z</published>
<updated>2017-12-12T03:11:04.000Z</updated>
<content type="html"><![CDATA[<p>asyncio内置支持在套接字上启用SSL通信。通过将SSLContext实例传递给协程来创建服务器或者客户端连接，可支持并确保在将套接字交给应用程序使用之前启用SSL协议。</p>
<p>比以前章节中以协程为基础的服务端和客户端更新了一些小变化。首先创建证书和key文件。使用下面命令生成自签名证书。</p>
<figure class="highlight sh"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ openssl req -newkey rsa:2048 -nodes -keyout pymotw.key -x509 -days 365 -out pymotw.crt</div></pre></td></tr></table></figure>
<p>openssl命令将提示生成证书的几个值，然后生成需要的输出文件。</p>
<p>在以前的服务端例子中，不安全的socket使用start_server()来创建监听的套接字。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">factory = asyncio.start_server(echo, *SERVER_ADDRESS)</div><div class="line">server = event_loop.run_until_complete(factory)</div></pre></td></tr></table></figure>
<p>为了加入编码认证，需要创建带证书和带刚生成key的SSLContext，然后传递给start_server()方法。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># The certificate is created with pymotw.com as the hostname,</span></div><div class="line"><span class="comment"># which will not match when the example code runs elsewhere,</span></div><div class="line"><span class="comment"># so disable hostname verification</span></div><div class="line">ssl_context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)</div><div class="line">ssl_context.check_hostname = <span class="keyword">False</span></div><div class="line">ssl_context.load_cert_chain(<span class="string">'pymotw.crt'</span>, <span class="string">'pymotw.key'</span>)</div><div class="line"></div><div class="line"><span class="comment"># Create the server and let the loop finish the coroutine before</span></div><div class="line"><span class="comment"># starting the real event loop.</span></div><div class="line">factory = asyncio.start_server(echo, *SERVER_ADDRESS, ssl=ssl_context)</div></pre></td></tr></table></figure>
<p>客户端需要相似的改变。老版本使用open_connection()来创建套接字链接到服务端。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">reader, writer = <span class="keyword">await</span> asyncio.open_connection(*address)</div></pre></td></tr></table></figure>
<p>也需要SSLContext来保护客户端的socket。客户端身份不需要强制，只需要加载证书。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># The certificate is created with pymotw.com as the hostname,</span></div><div class="line"><span class="comment"># which will not match when the example code runs</span></div><div class="line"><span class="comment"># elsewhere, so disable hostname verification.</span></div><div class="line">ssl_context = ssl.create_default_context(</div><div class="line"> ssl.Purpose.SERVER_AUTH,</div><div class="line">)</div><div class="line">ssl_context.check_hostname = <span class="keyword">False</span></div><div class="line">ssl_context.load_verify_locations(<span class="string">'pymotw.crt'</span>)</div><div class="line">reader, writer = <span class="keyword">await</span> asyncio.open_connection(</div><div class="line"> *server_address, ssl=ssl_context</div><div class="line">)</div></pre></td></tr></table></figure>
<p>在客户端需要一个其他小的改变。因为SSL连接不支持发送end-of-file(EOF)，客户端使用NULL字节作为消息终止。</p>
<p>老版本的客户端使用write_eof()发送给loop。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># This could be writer.writelines() except that</span></div><div class="line"><span class="comment"># would make it harder to show each part of the message</span></div><div class="line"><span class="comment"># being sent.</span></div><div class="line"><span class="keyword">for</span> msg <span class="keyword">in</span> messages:</div><div class="line"> writer.write(msg)</div><div class="line"> log.debug(<span class="string">'sending {!r}'</span>.format(msg))</div><div class="line"><span class="keyword">if</span> writer.can_write_eof():</div><div class="line"> writer.write_eof()</div><div class="line"><span class="keyword">await</span> writer.drain()</div></pre></td></tr></table></figure>
<p>新版本发送零字节(b’\x00’).</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># This could be writer.writelines() except that</span></div><div class="line"><span class="comment"># would make it harder to show each part of the message</span></div><div class="line"><span class="comment"># being sent.</span></div><div class="line"><span class="keyword">for</span> msg <span class="keyword">in</span> messages:</div><div class="line"> writer.write(msg)</div><div class="line"> log.debug(<span class="string">'sending {!r}'</span>.format(msg))</div><div class="line"><span class="comment"># SSL does not support EOF, so send a null byte to indicate</span></div><div class="line"><span class="comment"># the end of the message.</span></div><div class="line">writer.write(<span class="string">b'\x00'</span>)</div><div class="line"><span class="keyword">await</span> writer.drain()</div></pre></td></tr></table></figure>
<p>服务端的echo()协程方法必须寻找NULL字节，在接收到时关闭客户端连接。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">echo</span><span class="params">(reader, writer)</span>:</span></div><div class="line"> address = writer.get_extra_info(<span class="string">'peername'</span>)</div><div class="line"> log = logging.getLogger(<span class="string">'echo_{}_{}'</span>.format(*address))</div><div class="line"> log.debug(<span class="string">'connection accepted'</span>)</div><div class="line"> <span class="keyword">while</span> <span class="keyword">True</span>:</div><div class="line"> data = <span class="keyword">await</span> reader.read(<span class="number">128</span>)</div><div class="line"> terminate = data.endswith(<span class="string">b'\x00'</span>)</div><div class="line"> data = data.rstrip(<span class="string">b'\x00'</span>)</div><div class="line"> <span class="keyword">if</span> data:</div><div class="line"> log.debug(<span class="string">'received {!r}'</span>.format(data))</div><div class="line"> writer.write(data)</div><div class="line"> <span class="keyword">await</span> writer.drain()</div><div class="line"> log.debug(<span class="string">'sent {!r}'</span>.format(data))</div><div class="line"> <span class="keyword">if</span> <span class="keyword">not</span> data <span class="keyword">or</span> terminate:</div><div class="line"> log.debug(<span class="string">'message terminated, closing connection'</span>)</div><div class="line"> writer.close()</div><div class="line"> <span class="keyword">return</span></div></pre></td></tr></table></figure>
<p>在一个窗口启动服务端，在另一个窗口运行客户端，如下输出。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_echo_server_ssl.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">main: starting up on localhost port 10000</div><div class="line">echo_::1_53957: connection accepted</div><div class="line">echo_::1_53957: received b'This is the message. '</div><div class="line">echo_::1_53957: sent b'This is the message. '</div><div class="line">echo_::1_53957: received b'It will be sent in parts. '</div><div class="line">echo_::1_53957: sent b'It will be sent in parts.'</div><div class="line">echo_::1_53957: message terminated, closing connection</div><div class="line"></div><div class="line">$ python3 asyncio_echo_client_ssl.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">echo_client: connecting to localhost port 10000</div><div class="line">echo_client: sending b'This is the message.'</div><div class="line">echo_client: sending b'It will be sent '</div><div class="line">echo_client: sending b'in parts.'</div><div class="line">echo_client: waiting for response</div><div class="line">echo_client: received b'This is the message.'</div><div class="line">echo_client: received b'It will be sent in parts.'</div><div class="line">echo_client: closing</div><div class="line">main: closing event loop</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/ssl.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>asyncio内置支持在套接字上启用SSL通信。通过将SSLContext实例传递给协程来创建服务器或者客户端连接，可支持并确保在将套接字交给应用程序使用之前启用SSL协议。</p>
<p>比以前章节中以协程为基础的服务端和客户端更新了一些小变化。首先创建证书和key文件。
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之协程异步IO流</title>
<link href="https://www.songcser.com/2017/10/20/asynchronous-IO-using-coroutines-and-streams/"/>
<id>https://www.songcser.com/2017/10/20/asynchronous-IO-using-coroutines-and-streams/</id>
<published>2017-10-20T01:05:15.000Z</published>
<updated>2017-12-12T02:54:21.000Z</updated>
<content type="html"><![CDATA[<p>这节练习两个例程实现简单的服务端和客户端的替换版本。使用协程和asyncio流API代替protocol和transport抽象类。这节操作较低级别的抽象层而不是以前讨论的Protocol API，但是产生的事件是相似的。</p>
<h2 id="服务端"><a href="#服务端" class="headerlink" title="服务端"></a>服务端</h2><p>首先导入需要的模块asyncio和logging,然后创建事件循环对象。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_echo_server_coroutine.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"></div><div class="line">SERVER_ADDRESS = (<span class="string">'localhost'</span>, <span class="number">10000</span>)</div><div class="line">logging.basicConfig(</div><div class="line"> level=logging.DEBUG,</div><div class="line"> format=<span class="string">'%(name)s: %(messages)s'</span>,</div><div class="line"> stream=sys.stderr,</div><div class="line">)</div><div class="line">log = logging.getLogger(<span class="string">'main'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div></pre></td></tr></table></figure>
<p>定义协程处理连接。每次进行客户端连接，一个新的协程实例将被调用以便功能代码同一时间只连接一个客户端。Python的语言运行时管理每个协程实例的状态，所以应用代码不需要管理额外的数据结构来跟踪不同的客户端。</p>
<p>协程的参数是StreamReader和StreamWriter实例来连接新的连接。和Transport一样，客户端的地址能够通过writer的方法get_extra_info()方法获得。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">echo</span><span class="params">(reader, writer)</span>:</span></div><div class="line"> address = writer.get_extra_info(<span class="string">'peername'</span>)</div><div class="line"> log = logging.getLogger(<span class="string">'echo_{}_{}'</span>.format(*address))</div><div class="line"> log.debug(<span class="string">'connection accepted'</span>)</div></pre></td></tr></table></figure>
<p>虽然当连接建立时协程被调用，但是可能没有数据读取。当读取时为了避免阻塞，协程使用await关键字来调用read()方法，允许事件循环继续处理其他的任务直到数据读取到。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">while</span> <span class="keyword">True</span>:</div><div class="line"> data = <span class="keyword">await</span> reader.read(<span class="number">128</span>)</div></pre></td></tr></table></figure>
<p>如果客户端发送数据，将会在await中返回数据并且通过writer把数据返还给客户端。经常多次调用write()会缓存输出的数据，然后drain()方法来刷新数据。由于刷新网络IO会阻塞，所以使用await关键字将控制权返还给事件循环，该事件循环监控着写socket并在可以发送更多数据时调用writer。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">if</span> data:</div><div class="line"> log.debug(<span class="string">'received {!r}'</span>.format(data))</div><div class="line"> writer.write(data)</div><div class="line"> <span class="keyword">await</span> writer.drain()</div><div class="line"> log.debug(<span class="string">'sent {!r}'</span>.format(data))</div></pre></td></tr></table></figure>
<p>如果客户端没有发送数据，read()方法返还空字串来表明连接关闭了。服务端需要关闭套接字返还给客户端，然后协程返回表明完成。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">else</span>:</div><div class="line"> log.debug(<span class="string">'closing'</span>)</div><div class="line"> writer.close()</div><div class="line"> <span class="keyword">return</span></div></pre></td></tr></table></figure>
<p>两步来启动服务端。首先应用程序调用事件循环，使用协程和要监听的hostname和socket来创建新的服务端对象。start_server()方法本身是个协程，所以结果必须由事件循环产生，并且启动server。完成协程会产生一个绑定到事件循环的asyncio.Server实例。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># Create the server and let the loop finish the coroutine before </span></div><div class="line"><span class="comment"># starting the real event loop.</span></div><div class="line">factory = asyncio.start_server(echo, *SERVER_ADDRESS)</div><div class="line">server = event_loop.run_until_complete(factory)</div><div class="line">log.debug(<span class="string">'starting up on {} port {}'</span>.format(*SERVER_ADDRESS))</div></pre></td></tr></table></figure>
<p>需要运行事件循环才能处理事件和客户端请求。对于长时间运行的服务，run_forever()方法是最简单的方法。当事件循环停止，无论通过应用代码或者通过发信号给进程，都可以关闭server能够并且正确清理socket，然后事件循环在程序退出之前被关闭以便来处理其他协程。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># Enter the event loop permanently to handle all connections</span></div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_forever()</div><div class="line"><span class="keyword">except</span> KeyboradInterrupt:</div><div class="line"> <span class="keyword">pass</span></div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> log.debug(<span class="string">'closing server'</span>)</div><div class="line"> server.close()</div><div class="line"> event_loop.run_until_complete(server.wait_closed())</div><div class="line"> log.debug(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<h2 id="客户端"><a href="#客户端" class="headerlink" title="客户端"></a>客户端</h2><p>使用协程构造客户端和构造服务端很相似。代码首先也是从导入需要的asyncio和logging模块开始，然后创建事件循环对象。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_echo_client_coroutine.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"></div><div class="line">MESSAGES = [</div><div class="line"> <span class="string">b'This is the message. '</span>,</div><div class="line"> <span class="string">b'It will be sent '</span></div><div class="line"> <span class="string">b'in parts.'</span></div><div class="line">]</div><div class="line">SERVER_ADDRESS = (<span class="string">'localhost'</span>, <span class="number">10000</span>)</div><div class="line"></div><div class="line">logging.basicConfig(</div><div class="line"> level=logging.DEBUG,</div><div class="line"> format=<span class="string">'%(name)s: %(message)s'</span>,</div><div class="line"> stream=sys.stderr,</div><div class="line">)</div><div class="line">log = logging.getLogger(<span class="string">'main'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div></pre></td></tr></table></figure>
<p>echo_client协程的参数是服务端的地址和发送的信息。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">echo_client</span><span class="params">(address, messages)</span>:</span></div></pre></td></tr></table></figure>
<p>当任务开始时协程被调用，但是还没有活跃的连接工作。所以首先要建立自己的连接。当open_connection()协程运行时使用await关键字来避免阻塞其他活跃的连接。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">log = logging.getLogger(<span class="string">'echo_client'</span>)</div><div class="line"></div><div class="line">log.debug(<span class="string">'connection to {} port {}'</span>.format(*address))</div><div class="line">reader, writer = <span class="keyword">await</span> asyncio.open_connection(*address)</div></pre></td></tr></table></figure>
<p>open_connection()协程返回StreamReader和StreamWriter实例来连接新的套接字。下一步是使用writer来发送数据给服务端。在服务端，writer将缓存输出的数据直到socket准备好或者使用drain()方法刷新结果。由于刷新网络IO会阻塞，再次使用await关键字将控制权返还给事件循环，事件循环监控着socket写入并且当可以发送更多的数据时调用write。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># This could be writer.writelines() except that</span></div><div class="line"><span class="comment"># would make it harder to show each part of the message</span></div><div class="line"><span class="comment"># being sent.</span></div><div class="line"><span class="keyword">for</span> msg <span class="keyword">in</span> messages:</div><div class="line"> writer.write(msg)</div><div class="line"> log.debug(<span class="string">'sending {!r}'</span>.format(msg))</div><div class="line"><span class="keyword">if</span> writer.can_write_eof():</div><div class="line"> writer.write_eof()</div><div class="line"><span class="keyword">await</span> writer.drain()</div></pre></td></tr></table></figure>
<p>接下来，客户端通过尝试读取数据来等待服务端的响应直到没有数据读取。为了避免在单个read()调用上阻塞，await将控制权返还给事件循环。如果服务端发送数据，记录日志。如果服务端没有发送数据，read()方法返还空字串来表明连接关闭了。客户端需要关闭socket才能发送给服务端然后返回表明已经完成。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">log.debug(<span class="string">'waiting for response'</span>)</div><div class="line"><span class="keyword">while</span> <span class="keyword">True</span>:</div><div class="line"> data = <span class="keyword">await</span> reader.read(<span class="number">128</span>)</div><div class="line"> <span class="keyword">if</span> data:</div><div class="line"> log.debug(<span class="string">'received {!r}'</span>.format(data))</div><div class="line"> <span class="keyword">else</span>:</div><div class="line"> log.debug(<span class="string">'closing'</span>)</div><div class="line"> writer.close()</div><div class="line"> <span class="keyword">return</span></div></pre></td></tr></table></figure>
<p>要启动客户端，使用协程调用事件循环创建客户端。使用run_until_complete()方法避免在客户端程序中产生死循环。不像protocol的例子，当协程完成之后不需要向future单独发送信号，因为echo_client()包含所有的客户端处理逻辑，并且只有在收到响应和关闭服务器连接之后才会返回。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(</div><div class="line"> echo_client(SERVER_ADDRESS, MESSAGES)</div><div class="line"> )</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> log.debug(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<h2 id="输出"><a href="#输出" class="headerlink" title="输出"></a>输出</h2><p>在一个窗口启动服务器，在另一个窗口启动客户端产生下面输出。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_echo_client_coroutine.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">echo_client: connecting to localhost port 10000</div><div class="line">echo_client: sending b'This is the message. '</div><div class="line">echo_client: sending b'It will be sent '</div><div class="line">echo_client: sending b'in parts.'</div><div class="line">echo_client: waiting for response</div><div class="line">echo_client: received b'This is the message. It will be sent in parts.'</div><div class="line">echo_client: closing</div><div class="line">main: closing event loop</div><div class="line"></div><div class="line">$ python3 asyncio_echo_client_coroutine.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">echo_client: connecting to localhost port 10000</div><div class="line">echo_client: sending b'This is the message. '</div><div class="line">echo_client: sending b'It will be sent '</div><div class="line">echo_client: sending b'in parts.'</div><div class="line">echo_client: waiting for response</div><div class="line">echo_client: received b'This is the message. It will be sent in parts.'</div><div class="line">echo_client: closing</div><div class="line">main: closing event loop</div><div class="line"></div><div class="line">$ python3 asyncio_echo_client_coroutine.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">echo_client: connecting to localhost port 10000</div><div class="line">echo_client: sending b'This is the message. '</div><div class="line">echo_client: sending b'It will be sent '</div><div class="line">echo_client: sending b'in parts.'</div><div class="line">echo_client: waiting for response</div><div class="line">echo_client: received b'This is the message. It will be sent '</div><div class="line">echo_client: received b'in parts.'</div><div class="line">echo_client: closing</div><div class="line">main: closing event loop</div></pre></td></tr></table></figure>
<p>尽管客户端一直单独的发送数据，客户端首先运行了两次，服务端接收了大量的信息并且输出返回给客户端。这些结果在后续运行中有所不同，这取决于网络的繁忙程度以及是否释放刷新网络缓冲区之前所有数据准备好了。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_echo_server_coroutine.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">main: starting up on localhost port 10000</div><div class="line">echo_::1_64624: connection accepted</div><div class="line">echo_::1_64624: received b'This is the message. It will be sent in parts.'</div><div class="line">echo_::1_64624: sent b'This is the message. It will be sent in parts.'</div><div class="line">echo_::1_64624: closing</div><div class="line"></div><div class="line">echo_::1_64626: connection accepted</div><div class="line">echo_::1_64626: received b'This is the message. It will be sent in parts.'</div><div class="line">echo_::1_64626: sent b'This is the message. It will be sent in parts.'</div><div class="line">echo_::1_64626: closing</div><div class="line"></div><div class="line">echo_::1_64627: connection accepted</div><div class="line">echo_::1_64627: received b'This is the message. It will be sent '</div><div class="line">echo_::1_64627: sent b'This is the message. It will be sent '</div><div class="line">echo_::1_64627: received b'in parts.'</div><div class="line">echo_::1_64627: sent b'in parts.'</div><div class="line">echo_::1_64627: closing</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/io_coroutine.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>这节练习两个例程实现简单的服务端和客户端的替换版本。使用协程和asyncio流API代替protocol和transport抽象类。这节操作较低级别的抽象层而不是以前讨论的Protocol API，但是产生的事件是相似的。</p>
<h2 id="服务端"><a href=
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之抽象类Protocol异步IO</title>
<link href="https://www.songcser.com/2017/10/13/asynchronous-IO-with-protocol-class-abstractions/"/>
<id>https://www.songcser.com/2017/10/13/asynchronous-IO-with-protocol-class-abstractions/</id>
<published>2017-10-13T07:54:30.000Z</published>
<updated>2017-12-11T09:18:34.000Z</updated>
<content type="html"><![CDATA[<p>到目前为止，那些例子都避免混合并发和IO操作，以便每次都关注一个概念。当IO阻塞时进行上下文切换是asyncio一个主要的用例。<br>基于已经介绍的并发概念，这节验证两个例程，一个简单的服务端和客户端，和在socket和socketserver那节中的例子相似。客户端能够连接服务端，发送数据然后接收相同的数据。每一次遇到IO操作时，执行代码就会放弃控制给事件循环，允许其他的任务运行直到IO操作完成。</p>
<h2 id="服务端"><a href="#服务端" class="headerlink" title="服务端"></a>服务端</h2><p>服务端首先导入asyncio和logger模块，然后创建事件循环对象。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_echo_server.protocol.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"></div><div class="line">SERVER_ADDRESS = (<span class="string">'localhost'</span>, <span class="number">10000</span>)</div><div class="line"></div><div class="line">logging.basicConfig(</div><div class="line"> level=logging.DEBUG,</div><div class="line"> format=<span class="string">'%(name)s: %(message)s'</span>,</div><div class="line"> stream=sys.stderr,</div><div class="line">)</div><div class="line">log = logging.getLogger(<span class="string">'main'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div></pre></td></tr></table></figure>
<p>定义一个asyncio.Protocol的子类处理客户端的连接。基于事件的服务端socket连接发生时，protocol对象的方法被调用。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">EchoServer</span><span class="params">(asyncio.Protocol)</span>:</span></div></pre></td></tr></table></figure>
<p>每一个新的客户端连接都要触发connection_made()方法调用。transport的参数是asyncio.Transport实例，它提供了使用socket的抽象异步IO。不同的连接类型提供了不同的transport接口，都使用相同的API。例如，有单独的transport类用于处理socket和用于处理子进程的管道。客户端的地址可以通过tansport特定实现的接口get_extra_info()方法获得。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">connection_made</span><span class="params">(self, transport)</span>:</span></div><div class="line"> self.transport = transport</div><div class="line"> self.address = transport.get_extra_info(<span class="string">'peername'</span>)</div><div class="line"> self.log = logging.getLogger(</div><div class="line"> <span class="string">'EchoServer_{}_{}'</span>.format(*self.address)</div><div class="line"> )</div><div class="line"> self.log.debug(<span class="string">'connection accepted'</span>)</div></pre></td></tr></table></figure>
<p>链接建立之后，客户端传来的数据通过调用protocol的data_received()方法获得。数据以字节类型传输，到达应用时以合适的方法进行解码。日志记录下结果，然后通过调用tansport.write()将response立即返回给客户端。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">data_received</span><span class="params">(self, data)</span>:</span></div><div class="line"> self.log.debug(<span class="string">'received {!r}'</span>.format(data))</div><div class="line"> self.transport.write(data)</div><div class="line"> self.log.debug(<span class="string">'sent {!r}'</span>.format(data))</div></pre></td></tr></table></figure></p>
<p>一些transport支持特殊的端到文件(“EOF”)。当遇到EOF，eof_received()方法被调用。在这种实现中，EOF被返回给客户端表明数据已经被接收。由于不是所有的transport明确支持EOF，所以protocol首先询问transport发送EFO是否是安全的。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">eof_received</span><span class="params">(self)</span>:</span></div><div class="line"> self.log.debug(<span class="string">'reveived EOF'</span>)</div><div class="line"> <span class="keyword">if</span> self.transport.can_write_eof():</div><div class="line"> self.transport.write_eof()</div></pre></td></tr></table></figure>
<p>当链接关闭时，无论正常的或者错误的结果，protocol的connection_lost()方法都被调用。如果发生错误，参数中包含一个相关的异常对象，否则是None</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">connection_lost</span><span class="params">(self, error)</span>:</span></div><div class="line"> <span class="keyword">if</span> error:</div><div class="line"> self.log.error(<span class="string">'ERROR: {}'</span>.format(error))</div><div class="line"> <span class="keyword">else</span>:</div><div class="line"> self.log.debug(<span class="string">'closing'</span>)</div><div class="line"> super().connection_lost(error)</div></pre></td></tr></table></figure>
<p>需要两步启动服务。首先应用程序调用调用事件循环,使用protocol类以及要监听的hostname和socket来创建一个新的server对象。create_server()方法是协程，所以返回结果必须通过事件循环产生，并且启动server。完成协程会产生绑定到事件循环上的asyncio.Server实例。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># Create the server and let the loop finish the coroutine before starting the real event loop.</span></div><div class="line">factory = event_loop.create_server(EchoServer, *SERVER_ADDRESS)</div><div class="line">server = event_loop.run_until_complete(factory)</div><div class="line">log.debug(<span class="string">'starting up on {} prot {}'</span>.format(*SERVER_ADDRESS))</div></pre></td></tr></table></figure>
<p>然后，需要运行事件循环才能处理事件和处理客户端请求。对于长时间运行的服务，run_forever()方法是最简单的方式。当事件循环停止， 无论通过应用代码或者通过发信号给进程，都可以关闭server并且正确清理socket，然后在程序退出之前关闭事件循环以完成处理其他的协程。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># Enter the event loop permanently to handle all connections.</span></div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_forever()</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> log.debug(<span class="string">'closing server'</span>)</div><div class="line"> server.close()</div><div class="line"> event_loop.run_until_complete(server.wait_closed())</div><div class="line"> log.debug(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<h2 id="客户端"><a href="#客户端" class="headerlink" title="客户端"></a>客户端</h2><p>使用protocol类创建客户端和创建服务端很相似。代码还是从导入asyncio和logging模块开始，然后创建事件循环对象。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_echo_client_protocol.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> functools</div><div class="line"><span class="keyword">import</span> logging</div><div class="line"><span class="keyword">import</span> sys</div><div class="line"></div><div class="line">MESSAGES = [</div><div class="line"> <span class="string">b'This is the message. '</span>,</div><div class="line"> <span class="string">b'It will be sent'</span>,</div><div class="line"> <span class="string">b'in parts. '</span>,</div><div class="line">]</div><div class="line">SERVER_ADDRESS = (<span class="string">'localhost'</span>, <span class="number">10000</span>)</div><div class="line"></div><div class="line">logging.basicConfig(</div><div class="line"> level=logging.DEBUG,</div><div class="line"> format=<span class="string">'%(name)s: %(message)s'</span>,</div><div class="line"> stream=sys.stderr,</div><div class="line">)</div><div class="line">log = logging.getLogger(<span class="string">'main'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div></pre></td></tr></table></figure>
<p>protocol类的客户端定义了和服务端相同的方法，使用了不同的接口。构造类接收两个参数，即需要发送的消息列表,以及Futrue实例，用来从服务端接收响应以通知客户端已经完成一个任务循环。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="class"><span class="keyword">class</span> <span class="title">EchoClient</span><span class="params">(asyncio.Protocol)</span>:</span></div><div class="line"> </div><div class="line"> <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self, messages, future)</span>:</span></div><div class="line"> super().__init__()</div><div class="line"> self.messages = messages</div><div class="line"> self.log = logging.getLogger(<span class="string">'EchoClient'</span>)</div><div class="line"> self.f = future</div></pre></td></tr></table></figure>
<p>当客户端成功链接上服务端，立即进行通信。一系列的消息同时发送，不过在网络代码之下可能组合多个消息一次发送。当所有的消息发送完，最后发送EOF。</p>
<p>虽然看起来数据是被立即发送出去的，事实上transport对象先缓存需要发出去的数据，并且建立回调，当socket缓存准备好发送数据时才实际发送。这些处理都是透明的，所以应用代码可以像刚才进行IO操作时一样。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">connection_made</span><span class="params">(self, transport)</span>:</span></div><div class="line"> self.transport = transport</div><div class="line"> self.address = transport.get_extra_info(<span class="string">'peername'</span>)</div><div class="line"> self.log.debug(</div><div class="line"> <span class="string">'connecting to {} port {}'</span>.format(*self.address)</div><div class="line"> )</div><div class="line"> <span class="comment"># This could be transport.writelines() except that</span></div><div class="line"> <span class="comment"># would make it harder to show each part of the message</span></div><div class="line"> <span class="comment"># bing sent.</span></div><div class="line"> <span class="keyword">for</span> msg <span class="keyword">in</span> self.messages:</div><div class="line"> transport.write(msg)</div><div class="line"> self.log.debug(<span class="string">'sending {!r}'</span>.format(msg))</div><div class="line"> <span class="keyword">if</span> transport.can_write_eof():</div><div class="line"> transport.write_eof()</div></pre></td></tr></table></figure>
<p>当接收到服务端的响应时，打印日志。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">data_received</span><span class="params">(self, data)</span>:</span></div><div class="line"> self.log.debug(<span class="string">'received {!r}'</span>.format(data))</div></pre></td></tr></table></figure>
<p>当接收到EOF标记或者服务端关闭了链接，本地的transport对象也被关闭，future对象通过设置结果被标记完成。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">eof_received</span><span class="params">(self)</span>:</span></div><div class="line"> self.log.debug(<span class="string">'received EOF'</span>)</div><div class="line"> self.transport.close()</div><div class="line"> <span class="keyword">if</span> <span class="keyword">not</span> self.f.done():</div><div class="line"> self.f.set_result(<span class="keyword">True</span>)</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">connection_lost</span><span class="params">(self, exc)</span>:</span></div><div class="line"> self.log.debug(<span class="string">'server closed connection'</span>)</div><div class="line"> self.transport.close()</div><div class="line"> <span class="keyword">if</span> <span class="keyword">not</span> self.f.done():</div><div class="line"> self.f.set_result(<span class="keyword">True</span>)</div><div class="line"> super().connection_lost(exc)</div></pre></td></tr></table></figure>
<p>通常protocol类被传递给事件循环来创建连接。在这个例子中，因为事件循环传递其他的参数给protocol构造器，所以需要创建一个partial来包裹client类和发送的消息列表和Future实例。当调用create_connection()建立客户端链接时，使用新的可调用的类来代替。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">client_completed = asyncio.Future()</div><div class="line"></div><div class="line">client_factory = functools.partial(</div><div class="line"> EchoClient,</div><div class="line"> messages=MESSAGES,</div><div class="line"> future=client_completed,</div><div class="line">)</div><div class="line">factory_coroutine = event_loop.create_connection(</div><div class="line"> client_factory,</div><div class="line"> *SERVER_ADDRESS,</div><div class="line">)</div></pre></td></tr></table></figure>
<p>为了触发客户端运行，使用协程创建客户端时调用事件循环一次，然后在客户端完成时使用Future实例完成通信。像这样使用两个调用是避免客户端程序在完成服务端连接退出时进入死循环。如果只使用第一个调用等待协程创建客户端，则不可能处理所有的返回数据和清理服务端的链接。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">log.debug(<span class="string">'waiting for client to complete'</span>)</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(factory_coroutine)</div><div class="line"> event_loop.run_until_complete(client_completed)</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> log.debug(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>在一个窗口运行服务端，在另一个窗口运行客户端产生下面的输出。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_echo_client_protocol.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">main: waiting for client to complete</div><div class="line">EchoClient: connecting to ::1 port 10000</div><div class="line">EchoClient: sending b'This is the message. '</div><div class="line">EchoClient: sending b'It will be sent '</div><div class="line">EchoClient: sending b'in parts.'</div><div class="line">EchoClient: received b'This is the message. It will be sent in parts.'</div><div class="line">EchoClient: received EOF</div><div class="line">EchoClient: server closed connection</div><div class="line">main: closing event loop</div><div class="line"></div><div class="line">$python3 asyncio_echo_client_protocol.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">main: waiting for client to complete</div><div class="line">EchoClient: connectiong to ::1 port 10000</div><div class="line">EchoClient: sending b'This is the message. '</div><div class="line">EchoClient: sending b'It will be sent '</div><div class="line">EchoClient: sending b'in parts.'</div><div class="line">EchoClient: received b'This is the message. It will be sent in parts'</div><div class="line">EchoClient: received EOF</div><div class="line">EchoClient: server closed connection</div><div class="line">main: closing event loop</div><div class="line"></div><div class="line">$ python3 asyncio_echo_client_protocol.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">main: waiting for client to complete</div><div class="line">EchoClient: connecting to ::1 prot 10000</div><div class="line">EchoClient: sending b'This is the message. '</div><div class="line">EchoClient: sending b'It will be sent '</div><div class="line">EchoClient: sending b'in parts.'</div><div class="line">EchoClient: received b'This is the message. It will be sent in parts.'</div><div class="line">EchoClient: received EOF</div><div class="line">EchoClient: server closed connection</div><div class="line">main: closing event loop</div></pre></td></tr></table></figure>
<p>尽管客户端一直独立的发送消息，但是客户端第一次运行接收到服务端返回的大量数据。这些响应数据在后续运行中有所不同, 这取决于网络拥挤和在所有数据准备好之后进行网络缓存刷新。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_echo_server_protocol.py</div><div class="line">asyncio: Using selector: KqueueSelector</div><div class="line">main: starting up on localhost port 10000</div><div class="line">EchoServer_::1_63347: connection accepted</div><div class="line">EchoServer_::1_63347: received b'This is the message. It will be sent in parts.'</div><div class="line">EchoServer_::1_63347: received EOF</div><div class="line">EchoServer_::1_63347: closing</div><div class="line"></div><div class="line">EchoServer_::1_63387: connection accepted</div><div class="line">EchoServer_::1_63387: received b'This is the message. '</div><div class="line">EchoServer_::1_63387: sent b'This is the message. '</div><div class="line">EchoServer_::1_63387: received b'It will be sent in parts.'</div><div class="line">EchoServer_::1_63387: sent b'It will be sent in parts.'</div><div class="line">EchoServer_::1_63387: received EOF</div><div class="line">EchoServer_::1_63387: closing</div><div class="line"></div><div class="line">EchoServer_::1_63389: connection accepted</div><div class="line">EchoServer_::1_63389: received b'This is the message. It will be sent '</div><div class="line">EchoServer_::1_63389: sent b'This is the message. It will be sent '</div><div class="line">EchoServer_::1_63389: received b'in parts.'</div><div class="line">EchoServer_::1_63389: sent b'in parts.'</div><div class="line">EchoServer_::1_63389: received EOF</div><div class="line">EchoServer_::1_63389: closing</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/io_protocol.html#output" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>到目前为止，那些例子都避免混合并发和IO操作，以便每次都关注一个概念。当IO阻塞时进行上下文切换是asyncio一个主要的用例。<br>基于已经介绍的并发概念，这节验证两个例程，一个简单的服务端和客户端，和在socket和socketserver那节中的例子相似。客户端能够
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之同步原语</title>
<link href="https://www.songcser.com/2017/10/11/synchronization-primitives/"/>
<id>https://www.songcser.com/2017/10/11/synchronization-primitives/</id>
<published>2017-10-11T02:00:55.000Z</published>
<updated>2017-12-11T07:44:21.000Z</updated>
<content type="html"><![CDATA[<p>尽管asyncio应用经常是单线程进程运行，但是也同样可以构建同步应用。基于延迟、IO中断和其他事件的每一个coroutine或task可能在不可预测的顺序执行。为了支持安全同步，asyncio提供了和threading和multiprocessing模块相同的低水平原始接口。</p>
<h2 id="锁"><a href="#锁" class="headerlink" title="锁"></a>锁</h2><p>锁可以保证安全访问共享资源。只有加锁的才能使用资源。多个试图获取锁的将会阻塞住以便同一时间只有一个获取锁。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_lock.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> functools</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">unlock</span><span class="params">(lock)</span>:</span></div><div class="line"> print(<span class="string">'callback releasing lock'</span>)</div><div class="line"> lock.release()</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">coro1</span><span class="params">(lock)</span>:</span></div><div class="line"> print(<span class="string">'coro1 waiting for the lock'</span>)</div><div class="line"> <span class="keyword">with</span> <span class="keyword">await</span> lock:</div><div class="line"> print(<span class="string">'coro1 acquired lock'</span>)</div><div class="line"> print(<span class="string">'coro1 released lock'</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">coro2</span><span class="params">(lock)</span>:</span></div><div class="line"> print(<span class="string">'coro2 waiting for the lock'</span>)</div><div class="line"> <span class="keyword">await</span> lock</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> print(<span class="string">'coro2 acquired lock'</span>)</div><div class="line"> <span class="keyword">finally</span>:</div><div class="line"> print(<span class="string">'coro2 released lock'</span>)</div><div class="line"> lock.release()</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> <span class="comment"># Create and acquire a shared lock.</span></div><div class="line"></div><div class="line"> lock = asyncio.Lock()</div><div class="line"> print(<span class="string">'acquiring the lock before starting coroutines'</span>)</div><div class="line"> <span class="keyword">await</span> lock.acquire()</div><div class="line"> print(<span class="string">'lock acquired: {}'</span>.format(lock.locked()))</div><div class="line"></div><div class="line"> <span class="comment"># Schedule a callback to unlock the lock.</span></div><div class="line"> loop.call_later(<span class="number">0.1</span>, functools.partial(unlock, lock))</div><div class="line"></div><div class="line"> <span class="comment"># Run the coroutines that want to use the lock.</span></div><div class="line"> print(<span class="string">'waiting for coroutines'</span>)</div><div class="line"> <span class="keyword">await</span> asyncio.wait([coro1(lock), coro2(lock)]),</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>在这个例子中，lock可以直接调用，在coro2()方法中使用await获取锁，使用release()方法来释放锁。在coro1中可以使用with await关键字作为同步上下文管理。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_lock.py</div><div class="line"></div><div class="line">acquiring the lock before starting coroutines</div><div class="line">lock acquired: True</div><div class="line">waiting for coroutines</div><div class="line">coro1 waiting for the lock</div><div class="line">coro2 waiting for the lock</div><div class="line">callback releasing lock</div><div class="line">coro1 acquired lock</div><div class="line">coro1 released lock</div><div class="line">coro2 acquired lock</div><div class="line">coro2 released lock</div></pre></td></tr></table></figure>
<h2 id="事件"><a href="#事件" class="headerlink" title="事件"></a>事件</h2><p>asyncio.Event基于threading.Event, 经常被用来允许多个消费者等待相关联的通知发生，而不需要特定的值。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_event.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> functools</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">set_event</span><span class="params">(event)</span>:</span></div><div class="line"> print(<span class="string">'setting event in callback'</span>)</div><div class="line"> event.set()</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">coro1</span><span class="params">(event)</span>:</span></div><div class="line"> print(<span class="string">'coro1 waiting for event'</span>)</div><div class="line"> <span class="keyword">await</span> event.wait()</div><div class="line"> print(<span class="string">'coro1 triggered'</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">coro2</span><span class="params">(event)</span>:</span></div><div class="line"> print(<span class="string">'coro2 waiting for event'</span>)</div><div class="line"> <span class="keyword">await</span> event.wait()</div><div class="line"> print(<span class="string">'coro2 triggered'</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> <span class="comment"># Create a shared event</span></div><div class="line"> event = sayncio.Event()</div><div class="line"> print(<span class="string">'event start state: {}'</span>.format(event.is_set()))</div><div class="line"> </div><div class="line"> loop.call_later(</div><div class="line"> <span class="number">0.1</span>, functools.partial(set_event, event)</div><div class="line"> )</div><div class="line"> <span class="keyword">await</span> asyncio.wait([coro1(event), coro2(event)])</div><div class="line"> print(<span class="string">'event end state: {}'</span>.format(event.is_set()))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>就像锁一样，coro1()和coro2()都等待事件的发生。不同的是只要事件状态变化就都开始执行。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_event.py</div><div class="line"></div><div class="line">event start state: False</div><div class="line">coro2 waiting for event</div><div class="line">coro1 waiting for event</div><div class="line">setting event in callback</div><div class="line">coro2 triggered</div><div class="line">coro1 triggered</div><div class="line">event end state: True</div></pre></td></tr></table></figure>
<h2 id="条件变量"><a href="#条件变量" class="headerlink" title="条件变量"></a>条件变量</h2><p>Condition工作方式和Event相似，但是不是通知所有的等待的协程，而是根据传给notify()的参数只是唤醒一部分等待者。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_condition.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">consumer</span><span class="params">(condition, n)</span>:</span></div><div class="line"> <span class="keyword">with</span> <span class="keyword">await</span> condition:</div><div class="line"> print(<span class="string">'consumer {} if waiting'</span>.format(n))</div><div class="line"> <span class="keyword">await</span> condition.wait()</div><div class="line"> print(<span class="string">'consumer {} triggered'</span>.format(n))</div><div class="line"> print(<span class="string">'ending consumer {}'</span>.format(n))</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">manipulate_condition</span><span class="params">(condition)</span>:</span></div><div class="line"> print(<span class="string">'starting manipulate_condition'</span>)</div><div class="line"></div><div class="line"> <span class="comment"># pause to let consumers start</span></div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.1</span>)</div><div class="line"></div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">1</span>, <span class="number">3</span>):</div><div class="line"> <span class="keyword">with</span> <span class="keyword">await</span> condition:</div><div class="line"> print(<span class="string">'notifying {} consumers'</span>.format(i))</div><div class="line"> condition.notify(n=i)</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.1</span>)</div><div class="line"></div><div class="line"> <span class="keyword">with</span> <span class="keyword">await</span> condition:</div><div class="line"> print(<span class="string">'notifying remaining consumers'</span>)</div><div class="line"> condition.notify_all()</div><div class="line"></div><div class="line"> print(<span class="string">'ending maipulate_condition'</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> <span class="comment"># Create a condition</span></div><div class="line"> condition = asyncio.Condition()</div><div class="line"></div><div class="line"> <span class="comment"># Set up tasks watching the condition</span></div><div class="line"> consumers = [</div><div class="line"> consumer(condition, i)</div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">5</span>)</div><div class="line"> ]</div><div class="line"></div><div class="line"> <span class="comment"># Schedule a task to manipulate the condition variable</span></div><div class="line"> loop.create_task(manipulate_condition(condition))</div><div class="line"></div><div class="line"> <span class="comment"># Wait for the consumers to be done</span></div><div class="line"> <span class="keyword">await</span> asyncio.wait(consumers)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> result = event_loop.run_until_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>这个例子中启动了5个Condition的消费者。每一个都使用wait()方法等待通知以便继续进行。manipulate_condition()方法中通知一个消费者，然后两个，最后是所有剩下的消费者。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_condition.py</div><div class="line"></div><div class="line">starting manipulate_condition</div><div class="line">consumer 3 is waiting</div><div class="line">consumer 1 is waiting</div><div class="line">consumer 2 is waiting</div><div class="line">consumer 0 is waiting</div><div class="line">consumer 4 is waiting</div><div class="line">notifying 1 consumers</div><div class="line">consumer 3 triggered</div><div class="line">ending consumer 3</div><div class="line">notifying 2 consumers</div><div class="line">consumer 1 triggered</div><div class="line">ending consumer 1</div><div class="line">consumer 2 triggered</div><div class="line">ending consumer 2</div><div class="line">notifying remaining consumers</div><div class="line">ending manipulate_condition</div><div class="line">consumer 0 triggered</div><div class="line">ending consumer 0</div><div class="line">consumer 4 triggered</div><div class="line">ending consumer 4</div></pre></td></tr></table></figure>
<h2 id="队列"><a href="#队列" class="headerlink" title="队列"></a>队列</h2><p>asyncio.Queue提供了先进先出的协程数据结构，就像线程中的queue.Queue和进程中的multiprocessing.Queue。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_queue.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">consumer</span><span class="params">(n, q)</span>:</span></div><div class="line"> print(<span class="string">'consumer {}: starting'</span>.format(n))</div><div class="line"> <span class="keyword">while</span> <span class="keyword">True</span>:</div><div class="line"> print(<span class="string">'consumer {}: waiting for item'</span>.format(n))</div><div class="line"> item = <span class="keyword">await</span> q.get()</div><div class="line"> print(<span class="string">'consumer {}: has item {}'</span>.format(n, item))</div><div class="line"> <span class="keyword">if</span> item <span class="keyword">is</span> <span class="keyword">None</span>:</div><div class="line"> <span class="comment"># None is the signal to stop</span></div><div class="line"> q.task_done()</div><div class="line"> <span class="keyword">break</span></div><div class="line"> <span class="keyword">else</span>:</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.01</span> * item)</div><div class="line"> q.task_done()</div><div class="line"> print(<span class="string">'consumer {}: ending'</span>.format(n))</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">producer</span><span class="params">(q, num_workers)</span>:</span></div><div class="line"> print(<span class="string">'producer: starting'</span>)</div><div class="line"> <span class="comment"># Add some numbers to the queue to simulate jobs</span></div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(num_workers * <span class="number">3</span>):</div><div class="line"> <span class="keyword">await</span> q.put(i)</div><div class="line"> print(<span class="string">'producter: added task {} to the queue'</span>.format(i))</div><div class="line"> <span class="comment"># Add None entries in the queue</span></div><div class="line"> <span class="comment"># to signal the consumers to exit</span></div><div class="line"> print(<span class="string">'producer: add stop signals to the queue'</span>)</div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(num_workers):</div><div class="line"> <span class="keyword">await</span> q.put(<span class="keyword">None</span>)</div><div class="line"> print(<span class="string">'producer: waiting for queue to empty'</span>)</div><div class="line"> <span class="keyword">await</span> q.join()</div><div class="line"> print(<span class="string">'producer: ending'</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop, num_consumers)</span>:</span></div><div class="line"> <span class="comment"># Create the queue with a fixed size so the producer</span></div><div class="line"> <span class="comment"># will block until the consumers pull some items out.</span></div><div class="line"> q = asyncio.Queue(maxsize=num_consumers)</div><div class="line"></div><div class="line"> <span class="comment"># Scheduled the consumer tasks</span></div><div class="line"> consumers = [</div><div class="line"> loop.create_task(consumer(i, q))</div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(num_consumers)</div><div class="line"> ]</div><div class="line"></div><div class="line"> <span class="comment"># Schedule the producer task.</span></div><div class="line"> prod = loop.create_task(producer(q, num_consumers))</div><div class="line"></div><div class="line"> <span class="comment"># Wait for all of the coroutines to finish</span></div><div class="line"> <span class="keyword">await</span> asyncio.wait(consumers + [prod])</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop, <span class="number">2</span>))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>使用put()添加对象或者使用get()移除对象都是异步操作，直到队列数量固定（阻塞除外）或者队列为空（阻塞调用获取对象）</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_queue.py</div><div class="line"></div><div class="line">consumer 0: starting</div><div class="line">consumer 0: waiting for item</div><div class="line">consumer 1: starting</div><div class="line">consumer 1: waiting for item</div><div class="line">producer: starting</div><div class="line">producer: added task 0 to the queue</div><div class="line">producer: added task 1 to the queue</div><div class="line">consumer 0: has item 0</div><div class="line">consumer 1: has item 1</div><div class="line">producer: added task 2 to the queue</div><div class="line">producer: added task 3 to the queue</div><div class="line">consumer 0: waiting for item</div><div class="line">consumer 0: has item 2</div><div class="line">producer: added task 4 to the queue</div><div class="line">consumer 1: waiting for item</div><div class="line">consumer 1: has item 3</div><div class="line">producer: added task 5 to the queue</div><div class="line">producer: adding stop signals to the queue</div><div class="line">consumer 0: waiting for item</div><div class="line">consumer 0: has item 4</div><div class="line">consumer 1: waiting for item</div><div class="line">consumer 1: has item 5</div><div class="line">producer: waiting for queue to empty</div><div class="line">consumer 0: waiting for item</div><div class="line">consumer 0: has item None</div><div class="line">consumer 0: ending</div><div class="line">consumer 1: waiting for item</div><div class="line">consumer 1: has item None</div><div class="line">consumer 1: ending</div><div class="line">producer: ending</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/synchronization.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>尽管asyncio应用经常是单线程进程运行，但是也同样可以构建同步应用。基于延迟、IO中断和其他事件的每一个coroutine或task可能在不可预测的顺序执行。为了支持安全同步，asyncio提供了和threading和multiprocessing模块相同的低水平原始接
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之协程控制结构</title>
<link href="https://www.songcser.com/2017/10/10/composing-coroutines-with-control-structures/"/>
<id>https://www.songcser.com/2017/10/10/composing-coroutines-with-control-structures/</id>
<published>2017-10-10T01:13:17.000Z</published>
<updated>2017-12-12T03:04:56.000Z</updated>
<content type="html"><![CDATA[<p>使用语言内置的关键字await,一些协程之间的线性控制流是很容易进行管理的。更加复杂的结构使用asyncio的工具也可以做到允许一个协程等待其他协程的并发完成。</p>
<h2 id="等待多个协程运行"><a href="#等待多个协程运行" class="headerlink" title="等待多个协程运行"></a>等待多个协程运行</h2><p>经常会分割一个操作为多个部分分离执行。例如，下载多个远程资源或者查询远程API。在这种情况下，执行的顺序是无关紧要的并且可以有任意数量的操作，wait()方法能够停止一个协程等待其他协程完成后台操作。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_wait.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">phase</span><span class="params">(i)</span>:</span></div><div class="line"> print(<span class="string">'in phase {}'</span>.format(i))</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.1</span> * i)</div><div class="line"> print(<span class="string">'done with phase {}'</span>.format(i))</div><div class="line"> <span class="keyword">return</span> <span class="string">'phase {} result'</span>.format(i)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(num_phases)</span>:</span></div><div class="line"> print(<span class="string">'starting main'</span>)</div><div class="line"> phases = [</div><div class="line"> phase(i)</div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(num_phases)</div><div class="line"> ]</div><div class="line"> print(<span class="string">'waiting for phases to complete'</span>)</div><div class="line"> completed, pending = <span class="keyword">await</span> asyncio.wait(phases)</div><div class="line"> results = [t.result() <span class="keyword">for</span> t <span class="keyword">in</span> completed]</div><div class="line"> print(<span class="string">'results: {!r}'</span>.format(results))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(<span class="number">3</span>))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>在里面wait()使用set来装载Task实例。Task启动和结束都是在不可预料的顺序中，wait()函数的返回值是tuple类型，包含了完成和未完成的两个set集。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_wait.py</div><div class="line"></div><div class="line">starting main</div><div class="line">waiting <span class="keyword">for</span> phases to complete</div><div class="line"><span class="keyword">in</span> phase 0</div><div class="line"><span class="keyword">in</span> phase 1</div><div class="line"><span class="keyword">in</span> phase 2</div><div class="line"><span class="keyword">done</span> with phase 0</div><div class="line"><span class="keyword">done</span> with phase 1</div><div class="line"><span class="keyword">done</span> with phase 2</div><div class="line">results: [<span class="string">'phase 1 result'</span>, <span class="string">'phase 0 result'</span>, <span class="string">'phase 2 result'</span>]</div></pre></td></tr></table></figure>
<p>如果wait()函数加上超时时间，将只剩下未完成的操作。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_wait_timeout.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">phase</span><span class="params">(i)</span>:</span></div><div class="line"> print(<span class="string">'in phase {}'</span>.format(i))</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.1</span>*i)</div><div class="line"> <span class="keyword">except</span> asyncio.CancelledError:</div><div class="line"> print(<span class="string">'phase {} canceled'</span>.format(i))</div><div class="line"> <span class="keyword">raise</span></div><div class="line"> <span class="keyword">else</span>:</div><div class="line"> print(<span class="string">'done with phase {}'</span>.format(i))</div><div class="line"> <span class="keyword">return</span> <span class="string">'phase {} result'</span>.format(i)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(num_phases)</span>:</span></div><div class="line"> print(<span class="string">'starting main'</span>)</div><div class="line"> phases = [</div><div class="line"> phase(i)</div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(num_phases)</div><div class="line"> ]</div><div class="line"> print(<span class="string">'waiting 0.1 for phases to complete'</span>)</div><div class="line"> completed, pending = <span class="keyword">await</span> asyncio.wait(phases, timeout=<span class="number">0.1</span>)</div><div class="line"> print(<span class="string">'{} completed and {} pending'</span>.format(</div><div class="line"> len(completed), len(pending)</div><div class="line"> ))</div><div class="line"> <span class="comment"># Cancel remaining tasks so they do not generate errors</span></div><div class="line"> <span class="comment"># as we exit without finishing them.</span></div><div class="line"> <span class="keyword">if</span> pending:</div><div class="line"> print(<span class="string">'canceling tasks'</span>)</div><div class="line"> <span class="keyword">for</span> t <span class="keyword">in</span> pending:</div><div class="line"> t.cancel()</div><div class="line"> print(<span class="string">'exiting main'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(<span class="number">3</span>))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>剩下的后台任务应该被取消或者等待完成。当事件循环继续执行时挂起的任务将继续执行，如果全部的操作被终止了，这可能被认为是不可取的。在进程结束时还有挂起的任务将报警告信息。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># python3 asyncio_wait_timeout.py</span></div><div class="line"></div><div class="line">starting main</div><div class="line">waiting 0.1 <span class="keyword">for</span> phases to complete</div><div class="line"><span class="keyword">in</span> phase 1</div><div class="line"><span class="keyword">in</span> phase 0</div><div class="line"><span class="keyword">in</span> phase 2</div><div class="line"><span class="keyword">done</span> with phase 0</div><div class="line">1 completed and 2 pending</div><div class="line">cancelling tasks</div><div class="line">exiting main</div><div class="line">phase 1 cancelled</div><div class="line">phase 2 cancelled</div></pre></td></tr></table></figure>
<h2 id="获取协程的值"><a href="#获取协程的值" class="headerlink" title="获取协程的值"></a>获取协程的值</h2><p>如果执行语句已经定义好，只关心它们的结果，使用gather()函数来等待多个操作可能更加有用。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_gather.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">phase1</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in phase1'</span>)</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">2</span>)</div><div class="line"> print(<span class="string">'done with phase1'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'phase1 result'</span></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">phase2</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in phase2'</span>)</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">1</span>)</div><div class="line"> print(<span class="string">'done with phase2'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'pahse2 result'</span></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'starting main'</span>)</div><div class="line"> print(<span class="string">'waiting for phases to complete'</span>)</div><div class="line"> results = <span class="keyword">await</span> asyncio.gather(</div><div class="line"> phase1(),</div><div class="line"> phase2(),</div><div class="line"> )</div><div class="line"> print(<span class="string">'results: {!r}'</span>.format(results))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main())</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>由gather创建的task没有暴露出来，因此task不能被取消。返回值在一个列表里和传给gather()函数的参数的顺序相同，不管后台任务执行完成的顺序。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_gather.py</div><div class="line"></div><div class="line">starting main</div><div class="line">waiting <span class="keyword">for</span> phases to complete</div><div class="line"><span class="keyword">in</span> phase2</div><div class="line"><span class="keyword">in</span> phase1</div><div class="line"><span class="keyword">done</span> with phase2</div><div class="line"><span class="keyword">done</span> with phase1</div><div class="line">results: [<span class="string">'phase1 result'</span>, <span class="string">'phase2 result'</span>]</div></pre></td></tr></table></figure>
<h2 id="后台执行完成处理"><a href="#后台执行完成处理" class="headerlink" title="后台执行完成处理"></a>后台执行完成处理</h2><p>as_completed()是一个生成器，管理一个链表里的协程，在协程执行完之后立即消费它的结果。和wait()一样，as_completed()是不保证顺序的，但是没有必要等待所有的后台执行完成再执行其他的行为。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_as_completed.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">phase</span><span class="params">(i)</span>:</span></div><div class="line"> print(<span class="string">'in phase {}'</span>.format(i))</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.5</span> - (<span class="number">0.1</span> * i))</div><div class="line"> print(<span class="string">'done with phase {}'</span>.format(i))</div><div class="line"> <span class="keyword">return</span> <span class="string">'phase {} result'</span>.format(i)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(num_phases)</span>:</span></div><div class="line"> print(<span class="string">'starting main'</span>)</div><div class="line"> phases = [</div><div class="line"> phase(i)</div><div class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> range(num_phases)</div><div class="line"> ]</div><div class="line"> print(<span class="string">'waiting for phases to complete'</span>)</div><div class="line"> results = []</div><div class="line"> <span class="keyword">for</span> next_to_complete <span class="keyword">in</span> asyncio.as_completed(phases):</div><div class="line"> answer = <span class="keyword">await</span> next_to_complete</div><div class="line"> print(<span class="string">'received answer {!r}'</span>.format(answer))</div><div class="line"> results.append(answer)</div><div class="line"> print(<span class="string">'results: {!r}'</span>.format(results))</div><div class="line"> <span class="keyword">return</span> results</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(<span class="number">3</span>))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>这个例子中先开始的执行语句却是相反的顺序完成。当生成器被消费，loop使用await等待协程的结果。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_as_completed.py</div><div class="line"></div><div class="line">starting main</div><div class="line">waiting <span class="keyword">for</span> phases to complete</div><div class="line"><span class="keyword">in</span> phase 0</div><div class="line"><span class="keyword">in</span> phase 2</div><div class="line"><span class="keyword">in</span> phase 1</div><div class="line"><span class="keyword">done</span> with phase 2</div><div class="line">received answer <span class="string">'phase 2 result'</span></div><div class="line"><span class="keyword">done</span> with phase 1</div><div class="line">received answer <span class="string">'phase 1 result'</span></div><div class="line"><span class="keyword">done</span> with phase 0</div><div class="line">received answer <span class="string">'phase 0 result'</span></div><div class="line">results: [<span class="string">'phase 2 result'</span>, <span class="string">'phase 1 result'</span>, <span class="string">'phase 0 result'</span>]</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/control.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>使用语言内置的关键字await,一些协程之间的线性控制流是很容易进行管理的。更加复杂的结构使用asyncio的工具也可以做到允许一个协程等待其他协程的并发完成。</p>
<h2 id="等待多个协程运行"><a href="#等待多个协程运行" class="headerl
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之执行并发任务</title>
<link href="https://www.songcser.com/2017/09/26/executing-tasks-concurrently/"/>
<id>https://www.songcser.com/2017/09/26/executing-tasks-concurrently/</id>
<published>2017-09-26T22:23:40.000Z</published>
<updated>2017-12-08T09:48:35.000Z</updated>
<content type="html"><![CDATA[<p>Task是与事件循环交互的主要方式之一。 Task包装协程并跟踪它的完成。 Tasks是Future的子类，所以其他协程等待他们并且每一个协程都会获得task完成时返回的结果。</p>
<h2 id="启动Task"><a href="#启动Task" class="headerlink" title="启动Task"></a>启动Task</h2><p>使用create_task()创建Task实例启动任务。只要loop正在运行，并且协程没有返回，task将做为事件循环并发操作的一部分一直运行。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_create_task.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">task_func</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in task_func'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'the result'</span></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> print(<span class="string">'createing task'</span>)</div><div class="line"> task = loop.create_task(task_func())</div><div class="line"> print(<span class="string">'waiting for {!r}'</span>.format(task))</div><div class="line"> return_value = <span class="keyword">await</span> task</div><div class="line"> print(<span class="string">'task completed {!r}'</span>.format(task))</div><div class="line"> print(<span class="string">'return value: {!r}'</span>.format(return_value))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>这个例子中main()函数退出之前需要等待task返回结果。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># python3 asyncio_create_task.py</span></div><div class="line"></div><div class="line">creating task</div><div class="line">waiting <span class="keyword">for</span> <Task pending coro=<task_func() running at asyncio_create_task.py:<span class="number">12</span>>></div><div class="line"><span class="keyword">in</span> task_func</div><div class="line">task completed <Task finished coro=<task_func() done, defined at asyncio_create_task.py:<span class="number">12</span>> result=<span class="string">'the result'</span>></div><div class="line"><span class="keyword">return</span> value: <span class="string">'the result'</span></div></pre></td></tr></table></figure>
<h2 id="取消Task"><a href="#取消Task" class="headerlink" title="取消Task"></a>取消Task</h2><p>通过create_task()可以返回Task对象，这样就可以在task完成之前取消操作。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_cancel_task.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">task_func</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in task_func'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'the result'</span></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> print(<span class="string">'createing task'</span>)</div><div class="line"> task = loop.create_task(task_func())</div><div class="line"></div><div class="line"> print(<span class="string">'canceling task'</span>)</div><div class="line"> task.cancel()</div><div class="line"></div><div class="line"> print(<span class="string">'canceled task {!r}'</span>.format(task))</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> <span class="keyword">await</span> task</div><div class="line"> <span class="keyword">except</span> asyncio.CancelledError:</div><div class="line"> print(<span class="string">'caught error from canceled task'</span>)</div><div class="line"> <span class="keyword">else</span>:</div><div class="line"> print(<span class="string">'task result: {!r}'</span>.format(task.result()))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>这个例子中在事件循环开始之前创建task然后取消，会从run_until_complete()函数中抛出CancelledError异常。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_cancel_task.py</div><div class="line"></div><div class="line">creating task</div><div class="line">canceling task</div><div class="line">canceled task <Task cancelling coro=<task_func() running at asyncio_cancel_task.py:12>></div><div class="line">caught error from canceled task</div></pre></td></tr></table></figure>
<p>如果task在等待其他并发操作时被取消，它会在它等待的地方抛出CancelledError异常来通知task被取消。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_cancel_task2.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">task_func</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in task_func, sleeping'</span>)</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">1</span>)</div><div class="line"> <span class="keyword">except</span> asyncio.CancelledError:</div><div class="line"> print(<span class="string">'task_func was canceled'</span>)</div><div class="line"> <span class="keyword">raise</span></div><div class="line"> <span class="keyword">return</span> <span class="string">'the result'</span></div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">task_canceller</span><span class="params">(t)</span>:</span></div><div class="line"> print(<span class="string">'in task_canceller'</span>)</div><div class="line"> t.cancel()</div><div class="line"> print(<span class="string">'canceled the task'</span>)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> print(<span class="string">'createing task'</span>)</div><div class="line"> task = loop.create_task(task_func())</div><div class="line"> loop.call_soon(task_canceller, task)</div><div class="line"> <span class="keyword">try</span>:</div><div class="line"> <span class="keyword">await</span> task</div><div class="line"> <span class="keyword">except</span> asyncio.CancelledError:</div><div class="line"> print(<span class="string">'main() also sees task as canceled'</span>)</div><div class="line"></div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_until_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>如果需要，截获异常能够有机会来清理已经完成的任务。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_cancel_task2.py</div><div class="line"></div><div class="line">creating task</div><div class="line">in task_func, sleeping</div><div class="line">in task_canceller</div><div class="line">canceled the task</div><div class="line">task_func was canceled</div><div class="line">main() also sees task as canceled</div></pre></td></tr></table></figure>
<h2 id="由协程生成Task"><a href="#由协程生成Task" class="headerlink" title="由协程生成Task"></a>由协程生成Task</h2><p>ensure_future()函数返回的Task可以绑定到协程上。然后Task实例可以传递给其他代码，这些代码可以在不知道原始协程是如何构造或调用的情况下等待它。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_ensure_future.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">wrapped</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'wrapped'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'result'</span></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">inner</span><span class="params">(task)</span>:</span></div><div class="line"> print(<span class="string">'inner: starting'</span>)</div><div class="line"> print(<span class="string">'inner: waiting for {!r}'</span>.format(task))</div><div class="line"> result = <span class="keyword">await</span> task</div><div class="line"> print(<span class="string">'inner: task returned {!r}'</span>.format(result))</div><div class="line"></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">starter</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'starter: creating task'</span>)</div><div class="line"> task = asyncio.ensure_future(wrapped())</div><div class="line"> print(<span class="string">'starter: waiting for inner'</span>)</div><div class="line"> <span class="keyword">await</span> inner(task)</div><div class="line"> print(<span class="string">'starter: inner returned'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> print(<span class="string">'entering event loop'</span>)</div><div class="line"> result = event_loop.run_until_complete(starter())</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>记住ensure_future()函数生成的协程不会执行,直到使用await允许它执行。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_ensure_future.py</div><div class="line"></div><div class="line">entering event loop</div><div class="line">starter: creating task</div><div class="line">starter: waiting for inner</div><div class="line">inner: starting</div><div class="line">inner: waiting for <Task pending coro=<wrapped() running at asyncio_ensure_future.py:12>></div><div class="line">wrapped</div><div class="line">inner: task returned 'result'</div><div class="line">starter: inner returned</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/tasks.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>Task是与事件循环交互的主要方式之一。 Task包装协程并跟踪它的完成。 Tasks是Future的子类，所以其他协程等待他们并且每一个协程都会获得task完成时返回的结果。</p>
<h2 id="启动Task"><a href="#启动Task" class="hea
</summary>
</entry>
<entry>
<title>asyncio之异步生成结果</title>
<link href="https://www.songcser.com/2017/09/26/producing-results-asynchronously/"/>
<id>https://www.songcser.com/2017/09/26/producing-results-asynchronously/</id>
<published>2017-09-26T18:01:09.000Z</published>
<updated>2017-12-08T09:49:20.000Z</updated>
<content type="html"><![CDATA[<p>Future代表任务的结果还没有生成。事件循环能够监控Future对象的状态来标示完成，并且允许应用的一部分等待另一部分完成任务。</p>
<h2 id="等待Future"><a href="#等待Future" class="headerlink" title="等待Future"></a>等待Future</h2><p>Future的行为和协程一样，因此一些有用的等待协程的方法也可以用来等待Future的完成。这个例子将future传递给事件循环的run_until_complete()方法。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_future_event_loop.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">mark_done</span><span class="params">(future, result)</span>:</span></div><div class="line"> print(<span class="string">'setting future result to {!r}'</span>.format(result))</div><div class="line"> future.set_result(result)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> all_done = asyncio.Future()</div><div class="line"></div><div class="line"> print(<span class="string">'acheduling mark_done'</span>)</div><div class="line"> event_loop.call_soon(mark_done, all_done, <span class="string">'the result'</span>)</div><div class="line"></div><div class="line"> print(<span class="string">'entering event loop'</span>)</div><div class="line"> result = event_loop.run_untile_complete(all_done)</div><div class="line"> print(<span class="string">'returned result: {!r}'</span>.format(result))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> print(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div><div class="line"></div><div class="line">print(<span class="string">'future result: {!r}'</span>.format(all_done.result()))</div></pre></td></tr></table></figure>
<p>当set_result()方法被调用时Future的state改变了，并且Future实例保存结果以便最后返回。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_future_event_loop.py</div><div class="line"></div><div class="line">scheduling mark_done</div><div class="line">entering event loop</div><div class="line">setting future result to 'the result'</div><div class="line">returned result: 'the result'</div><div class="line">closing event loop</div><div class="line">future result: 'the result'</div></pre></td></tr></table></figure>
<p>Future也可以使用await关键字，如下所示。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asycnio_future_await.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">mark_done</span><span class="params">(future, result)</span>:</span></div><div class="line"> print(<span class="string">'setting future result to {!r}'</span>.format(result))</div><div class="line"> future.set_result(result)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> all_done = asyncio.Future()</div><div class="line"></div><div class="line"> print(<span class="string">'scheduling mark_done'</span>)</div><div class="line"> loop.call_soon(mark_done, all_done, <span class="string">'the result'</span>)</div><div class="line"></div><div class="line"> result = <span class="keyword">await</span> all_done</div><div class="line"> print(<span class="string">'returned result: {!r}'</span>.format(result))</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> event_loop.run_untile_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>Future的结果通过await返回，因此经常有可能普通协程函数和Future实例使用相同的代码。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">$ python3 asyncio_future_await.py</div><div class="line"></div><div class="line">scheduling mark_done</div><div class="line">setting future result to 'the result'</div><div class="line">returned result: 'the result'</div></pre></td></tr></table></figure>
<h2 id="Future回调函数"><a href="#Future回调函数" class="headerlink" title="Future回调函数"></a>Future回调函数</h2><p>除了像协程一样工作，Future也可以在执行结束之后调用回调函数。回调函数按照注册的顺序进行调用。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_future_callback.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> functools</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(future, n)</span>:</span></div><div class="line"> print(<span class="string">'{}: future done: {}'</span>.format(n, future.result()))</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">register_callbacks</span><span class="params">(all_done)</span>:</span></div><div class="line"> print(<span class="string">'registering callbacks on future'</span>)</div><div class="line"> all_done.add_done_callback(functools.partial(callback, n=<span class="number">1</span>))</div><div class="line"> all_done.add_done_callback(functools.partial(callback, n=<span class="number">2</span>))</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(all_done)</span>:</span></div><div class="line"> <span class="keyword">await</span> register_callbacks(all_done)</div><div class="line"> print(<span class="string">'setting result of future'</span>)</div><div class="line"> all_done.set_result(<span class="string">'the result'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> all_done = asyncio.Future()</div><div class="line"> event_loop.run_untile_complete(main(all_done))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>回调函数要求Future实例为第一个参数，传给回调的额外参数使用functools.partial()进行包裹</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_future_callback.py</div><div class="line"></div><div class="line">registering callbacks on future</div><div class="line">setting result of future</div><div class="line">1: future done: the result</div><div class="line">2: future done: the result</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/futures.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>Future代表任务的结果还没有生成。事件循环能够监控Future对象的状态来标示完成，并且允许应用的一部分等待另一部分完成任务。</p>
<h2 id="等待Future"><a href="#等待Future" class="headerlink" title="等待F
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之常规函数调用</title>
<link href="https://www.songcser.com/2017/09/26/scheduling-calls-to-regular-functions/"/>
<id>https://www.songcser.com/2017/09/26/scheduling-calls-to-regular-functions/</id>
<published>2017-09-26T10:48:23.000Z</published>
<updated>2017-12-08T09:49:54.000Z</updated>
<content type="html"><![CDATA[<p>除了管理协程和IO回调，asyncio事件循环可以根据loop中保留的定时器来调度常规函数。</p>
<h2 id="调度回调函数”Soon”"><a href="#调度回调函数”Soon”" class="headerlink" title="调度回调函数”Soon”"></a>调度回调函数”Soon”</h2><p>如果回调的时间无关紧要，可以使用call_soon()方法在loop的下次迭代中进行调度。方法被调用时可以将一些额外的位置参数传递给回调。可以使用functools模块的partial()方法将关键字参数传给回调。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_call_soon.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> functools</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(arg, *, kwarg=<span class="string">'default'</span>)</span>:</span></div><div class="line"> print(<span class="string">'callback invoked with {} and {}'</span>.format(arg, kwarg))</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> print(<span class="string">'registering callbacks'</span>)</div><div class="line"> loop.call_soon(callback, <span class="number">1</span>)</div><div class="line"> wrapped = functools.partial(callback, kwarg=<span class="string">'not default'</span>)</div><div class="line"> loop.call_soon(wrapped, <span class="number">2</span>)</div><div class="line"></div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.1</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> print(<span class="string">'entering event loop'</span>)</div><div class="line"> event_loop.run_untile_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> print(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>回调函数按顺序调度执行。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_call_soon.py</div><div class="line"></div><div class="line">entering event loop</div><div class="line">registering callbacks</div><div class="line">callback invoked with 1 and defalut</div><div class="line">callback invoked with 2 and not defalut</div><div class="line">closing event loop</div></pre></td></tr></table></figure>
<h2 id="延迟调度回调"><a href="#延迟调度回调" class="headerlink" title="延迟调度回调"></a>延迟调度回调</h2><p>使用call_later()推迟一段时间调用回调。第一个参数是延迟的秒数，第二个参数是回调函数。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_call_later.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(n)</span>:</span></div><div class="line"> print(<span class="string">'callback {} invoked'</span>.format(n))</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> print(<span class="string">'registering callbacks'</span>)</div><div class="line"> loop.call_later(<span class="number">0.2</span>, callback, <span class="number">1</span>)</div><div class="line"> loop.call_later(<span class="number">0.1</span>, callback, <span class="number">2</span>)</div><div class="line"> loop.call_soon(callback, <span class="number">3</span>)</div><div class="line"></div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">0.4</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> print(<span class="string">'entering event loop'</span>)</div><div class="line"> event_loop.run_untile_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> print(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>在这个例子中，一些回调函数在不同的时间使用不同的参数被调用。最后的实例中使用call_soon()调用带着参数3的回调函数,要在时间延迟调度实例的前面获取结果，显然”soon”通常暗示最小的延迟。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_call_later.py</div><div class="line"></div><div class="line">entering event loop</div><div class="line">registering callbacks</div><div class="line">callback 3 invoked</div><div class="line">callback 2 invoked</div><div class="line">callback 1 invoked</div><div class="line">closing event loop</div></pre></td></tr></table></figure>
<h2 id="在具体时间调度回调函数"><a href="#在具体时间调度回调函数" class="headerlink" title="在具体时间调度回调函数"></a>在具体时间调度回调函数</h2><p>也可以在具体时间调用回调函数。loop使用单调时间(monotonic clock)而不是挂钟时间(wall-clock time)，以确保”now”的值不会后退。选择调度回调函数的时间需要从内部状态开始，要使用loop的time()函数。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_call_at.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"><span class="keyword">import</span> time</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">callback</span><span class="params">(n, loop)</span>:</span></div><div class="line"> print(<span class="string">'callback {} invoked at {}'</span>.format(n, loop.time()))</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">main</span><span class="params">(loop)</span>:</span></div><div class="line"> now = loop.time()</div><div class="line"> print(<span class="string">'clock time: {}'</span>.format(time.time()))</div><div class="line"> print(<span class="string">'loop time: {}'</span>.format(now))</div><div class="line"></div><div class="line"> print(<span class="string">'registering callbacks'</span>)</div><div class="line"> loop.call_at(now + <span class="number">0.2</span>, callback, <span class="number">1</span>, loop)</div><div class="line"> loop.call_at(now + <span class="number">0.1</span>, callback, <span class="number">2</span>, loop)</div><div class="line"> loop.call_soon(callback, <span class="number">3</span>, loop)</div><div class="line"></div><div class="line"> <span class="keyword">await</span> asyncio.sleep(<span class="number">1</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> print(<span class="string">'entering event loop'</span>)</div><div class="line"> event_loop.run_untile_complete(main(event_loop))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> print(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>注意: 时间按照loop的时间而不是time.time()返回的值。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_call_at.py</div><div class="line"></div><div class="line">entering event loop</div><div class="line">clock time: 1479050248.66192</div><div class="line">loop time: 1008846.13856885</div><div class="line">registering callbacks</div><div class="line">callback 3 invoked at 1008846.13867956</div><div class="line">callback 2 invoked at 1008846.239931555</div><div class="line">callback 1 invoked at 1008846.343480996</div><div class="line">closing event loop</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/scheduling.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>除了管理协程和IO回调，asyncio事件循环可以根据loop中保留的定时器来调度常规函数。</p>
<h2 id="调度回调函数”Soon”"><a href="#调度回调函数”Soon”" class="headerlink" title="调度回调函数”Soon”">
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之协程的多任务处理</title>
<link href="https://www.songcser.com/2017/09/25/cooperative-multitasking-with-coroutines/"/>
<id>https://www.songcser.com/2017/09/25/cooperative-multitasking-with-coroutines/</id>
<published>2017-09-25T10:52:24.000Z</published>
<updated>2017-12-08T09:47:17.000Z</updated>
<content type="html"><![CDATA[<p>协程是一种用于并发操作的语言结构。协程函数在被调用时创建协程对象，调用者可以使用协程的send()的方法来运行代码。协程通过await可以停止其他协程的执行。停止后的协程的状态是meaintained,在下次被唤醒时从这个地方继续运行。</p>
<h2 id="开始一个协程"><a href="#开始一个协程" class="headerlink" title="开始一个协程"></a>开始一个协程</h2><p>有几种不同的方法可以使用asyncio事件循环启动协程。最简单的方式是使用run_until_complete()方法，直接运行协程。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_coroutine.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">coroutine</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in coroutine'</span>)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> print(<span class="string">'starting coroutine'</span>)</div><div class="line"> coro = coroutine()</div><div class="line"> print(<span class="string">'entering event loop'</span>)</div><div class="line"> event_loop.run_untile_complete(coro)</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> print(<span class="string">'closing event loop'</span>)</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>首先获取事件循环的引用，使用默认的loop或者指定特定的loop类。在这个例子中，使用类默认的loop。run_untile_complete()方法使用协程对象启动loop，当协程返回退出时停止loop。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_coroutine.py</div><div class="line"></div><div class="line">starting coroutine</div><div class="line">entering event loop</div><div class="line">in coroutine</div><div class="line">closing event loop</div></pre></td></tr></table></figure>
<h2 id="协程返回值"><a href="#协程返回值" class="headerlink" title="协程返回值"></a>协程返回值</h2><p>协程的返回值是从它开始启动并且等待运行结束的代码处返回的。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_coroutine_return.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">coroutine</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in coroutine'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'result'</span></div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> retrun_value = event_loop.run_untile_complete(</div><div class="line"> coroutine()</div><div class="line"> )</div><div class="line"> print(<span class="string">'it returned: {!r}'</span>.format(return_value))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>在这个例子中，run_untile_complete()方法返回了协程的值。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_coroutine_return.py</div><div class="line"></div><div class="line">in coroutine</div><div class="line">it returned: 'result'</div></pre></td></tr></table></figure>
<h2 id="协程链"><a href="#协程链" class="headerlink" title="协程链"></a>协程链</h2><p>一个协程可以启动另一个协程并且等待返回值。 这样更容易将任务分解成可重用的部分。下面的例子中有两个阶段必须按顺序执行，但可以和其他操作同时运行。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_coroutine_chain.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">outer</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in outer'</span>)</div><div class="line"> print(<span class="string">'waiting for result1'</span>)</div><div class="line"> result1 = <span class="keyword">await</span> phase1()</div><div class="line"> print(<span class="string">'waiting for result2'</span>)</div><div class="line"> result2 = <span class="keyword">await</span> phase2(result1)</div><div class="line"> <span class="keyword">return</span> (result1, result2)</div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">phase1</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in phase1'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'result1'</span></div><div class="line"></div><div class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">def</span> <span class="title">phase2</span><span class="params">(arg)</span>:</span></div><div class="line"> print(<span class="string">'in phase2'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'result2 derived from {}'</span>.format(arg)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> return_value =event_loop.run_untile_complete(outer())</div><div class="line"> print(<span class="string">'return value: {!r}'</span>.format(return_value))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>使用await关键字而不是添加新的协程到loop，因为被loop管理的控制流已经进入协程内部，所以不需要告诉loop去管理新的协程。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_coroutine_chain.py</div><div class="line"></div><div class="line">in outer</div><div class="line">waiting for result1</div><div class="line">in phase1</div><div class="line">waiting for result2</div><div class="line">in phase2</div><div class="line">return value: ('result', 'result2 derived from result1')</div></pre></td></tr></table></figure>
<h2 id="协程代替生成器"><a href="#协程代替生成器" class="headerlink" title="协程代替生成器"></a>协程代替生成器</h2><p>协程函数是asyncio的关键组件。它提供了语言级别的特性，可以停止程序执行了一部分，并且保存调用的状态，和最后重新进入的状态，这些都是并发框架具备的重要功能。</p>
<p>Python 3.5加入了新的语言特性，使用async def来定义原生的协程和使用await获取控制权。asyncio的例子都使用了新的特性。较早的Python3版本使用生成器函数，asyncio.coroutine()装饰器和yield from完成了相同的功能。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># asyncio_generator.py</span></div><div class="line"></div><div class="line"><span class="keyword">import</span> asyncio</div><div class="line"></div><div class="line"><span class="meta">@asyncio.coroutine</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">outer</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in outer'</span>)</div><div class="line"> print(<span class="string">'waiting for result1'</span>)</div><div class="line"> result1 = <span class="keyword">yield</span> <span class="keyword">from</span> phase1()</div><div class="line"> print(<span class="string">'waiting for result2'</span>)</div><div class="line"> result2 = <span class="keyword">yield</span> <span class="keyword">from</span> phase2(result1)</div><div class="line"> <span class="keyword">return</span> (result1, result2)</div><div class="line"></div><div class="line"><span class="meta">@asyncio.coroutine</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">phase1</span><span class="params">()</span>:</span></div><div class="line"> print(<span class="string">'in phase1'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'result1'</span></div><div class="line"></div><div class="line"><span class="meta">@asyncio.coroutine</span></div><div class="line"><span class="function"><span class="keyword">def</span> <span class="title">phase2</span><span class="params">(arg)</span>:</span></div><div class="line"> print(<span class="string">'in phase2'</span>)</div><div class="line"> <span class="keyword">return</span> <span class="string">'result2 derived from {}'</span>.format(arg)</div><div class="line"></div><div class="line">event_loop = asyncio.get_event_loop()</div><div class="line"><span class="keyword">try</span>:</div><div class="line"> return_value = event_loop.run_untile_complete(outer())</div><div class="line"> print(<span class="string">'return value: {!r}'</span>.format(return_value))</div><div class="line"><span class="keyword">finally</span>:</div><div class="line"> event_loop.close()</div></pre></td></tr></table></figure>
<p>上面的例子使用生成器代替原生协程模仿asyncio_coroutine_chain.py</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"># python3 asyncio_generator.py</div><div class="line"></div><div class="line">in outer</div><div class="line">waiting for result1</div><div class="line">in phase1</div><div class="line">waiting for result2</div><div class="line">in phase2</div><div class="line">return value: ('result1', 'result2 derived from result1')</div></pre></td></tr></table></figure>
<p><a href="https://pymotw.com/3/asyncio/coroutines.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<p>协程是一种用于并发操作的语言结构。协程函数在被调用时创建协程对象，调用者可以使用协程的send()的方法来运行代码。协程通过await可以停止其他协程的执行。停止后的协程的状态是meaintained,在下次被唤醒时从这个地方继续运行。</p>
<h2 id="开始一个协程
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>asyncio之异步协程概念</title>
<link href="https://www.songcser.com/2017/09/24/asynchronous-concurrency-concepts/"/>
<id>https://www.songcser.com/2017/09/24/asynchronous-concurrency-concepts/</id>
<published>2017-09-24T09:42:10.000Z</published>
<updated>2017-12-13T13:06:14.000Z</updated>
<content type="html"><![CDATA[<blockquote>
<p>最近在学习python3中新的异步模块asyncio, <a href="https://pymotw.com/3/" target="_blank" rel="external">PyMOTW-3</a>中对asyncio有详细的介绍，所以借此机会将相关的文章翻译出来，供大家一起学习。<br>由于英文水平有限，翻译难免有些错误，希望大家指正。</p>
</blockquote>
<p>现在大多数并发模型是线性写入的，并且依赖于语言运行时或操作系统的底层线程或进程进行管理,来适当的改变程序运行时的上下文。而基于asyncio的应用程序是由应用代码处理上下文的切换，使用正确的技术取决于几个相关的概念。</p>
<p>asyncio框架是以事件循环为中心，是负责高效处理IO事件，系统事件和应用上下文切换的第一类对象。并且提供了几个循环实现，以有效地利用操作系统功能。虽然通常可以自动的选择合理的默认值，但应用程序也可以选择特定的事件循环。这在Windows下是有用的，例如，一些支持外部程序的循环类在进行网络IO时可以提高效率。</p>
<p>应用程序和事件循环之间的交互需要注册代码运行，在资源可用时，事件循环可以调用应用程序的代码。例如，网络服务器在打开socket时进行注册以便接收到输入事件。当有新的连接接入或者有数据读取时，事件循环将提醒服务端代码。应用程序代码在当前上下文中没有任务做的短期时间内会重新让出控制运行。例如，没有更多的数据从套接字中读取时，服务端将控制权返还给事件循环。</p>
<p>出让控制权给事件循环的机制依赖于Python的协程。协程是一种特殊的函数能放弃控制权给调用者但不会丢失状态。协程和生成器很相似，事实上在Python 3.5版本以前没有原生的协程对象支持时生成器被用作实现协程。asycnio提供了protocols和transports的基类抽象层使用回调编写代码代替直接写协程。在基类和协程模块这两种方法都通过重新进入事件循环来明确的改变上下文运行环境，代替了Python的线程实现中隐式的上下文切换。</p>
<p>数据结构Future代表了还没有完成的任务结果。事件循环会监控Future对象直到完成，应用程序的一部分代码允许等待另一部分任务完成。除了features，asyncio还有以前的其他同步操作例如锁（locks）和信号量（semaphores）。</p>
<p>Task是Future的子类，能够包装和管理协程的执行。Task在它获取到需要的资源时，会被事件循环调度运行，并且生成结果给其他协程消费。</p>
<h2 id="文章目录"><a href="#文章目录" class="headerlink" title="文章目录"></a>文章目录</h2><ol>
<li><a href="https://www.songcser.com/2017/09/25/cooperative-multitasking-with-coroutines/">协程的多任务处理</a></li>
<li><a href="https://www.songcser.com/2017/09/26/scheduling-calls-to-regular-functions/">常规函数调用</a></li>
<li><a href="https://www.songcser.com/2017/09/26/producing-results-asynchronously/">异步生成结果</a></li>
<li><a href="https://www.songcser.com/2017/09/26/executing-tasks-concurrently/">执行并发任务</a></li>
<li><a href="https://www.songcser.com/2017/10/10/composing-coroutines-with-control-structures/">协程控制结构</a></li>
<li><a href="https://www.songcser.com/2017/10/11/synchronization-primitives/">同步原语</a></li>
<li><a href="https://www.songcser.com/2017/10/13/asynchronous-IO-with-protocol-class-abstractions/">抽象类Protocol异步IO</a></li>
<li><a href="https://www.songcser.com/2017/10/20/asynchronous-IO-using-coroutines-and-streams/">协程异步IO流</a></li>
<li><a href="https://www.songcser.com/2017/10/23/using-ssl/">使用SSL</a></li>
<li><a href="https://www.songcser.com/2017/10/25/interacting-with-domain-name-services/">域名服务交互</a></li>
<li><a href="https://www.songcser.com/2017/10/26/working-with-subprocesses/">子进程交互</a></li>
<li><a href="https://www.songcser.com/2017/11/06/receiving-unix-signals/">接收Unix信号</a></li>
<li><a href="https://www.songcser.com/2017/11/13/combining-coroutines-with-threads-and-processes/">协程和线程进程组合</a></li>
<li><a href="https://www.songcser.com/2017/11/21/debugging-with-asyncio/">调试</a></li>
</ol>
<p><a href="https://pymotw.com/3/asyncio/concepts.html" target="_blank" rel="external">原文链接</a></p>
]]></content>
<summary type="html">
<blockquote>
<p>最近在学习python3中新的异步模块asyncio, <a href="https://pymotw.com/3/" target="_blank" rel="external">PyMOTW-3</a>中对asyncio有详细的介绍，所以借此机
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="异步" scheme="https://www.songcser.com/tags/%E5%BC%82%E6%AD%A5/"/>
<category term="协程" scheme="https://www.songcser.com/tags/%E5%8D%8F%E7%A8%8B/"/>
</entry>
<entry>
<title>基于Sanic的微服务基础架构</title>
<link href="https://www.songcser.com/2017/09/20/sanic-microservice/"/>
<id>https://www.songcser.com/2017/09/20/sanic-microservice/</id>
<published>2017-09-20T01:25:50.000Z</published>
<updated>2017-12-29T06:45:53.000Z</updated>
<content type="html"><![CDATA[<p>基于sanic的微服务基础架构</p>
<h1 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h1><p>使用python做web开发面临的一个最大的问题就是性能，在解决C10K问题上显的有点吃力。有些异步框架Tornado、Twisted、Gevent 等就是为了解决性能问题。这些框架在性能上有些提升，但是也出现了各种古怪的问题难以解决。</p>
<p>在python3.6中，官方的异步协程库asyncio正式成为标准。在保留便捷性的同时对性能有了很大的提升,已经出现许多的异步框架使用asyncio。</p>
<p>使用较早的异步框架是aiohttp，它提供了server端和client端，对asyncio做了很好的封装。但是开发方式和最流行的微框架flask不同，flask开发简单，轻量，高效。</p>
<p>微服务是最近最火开发模式，它解决了复杂性问题，提高开发效率，便于部署等优点。</p>
<p>正是结合这些优点, 以Sanic为基础，集成多个流行的库来搭建微服务。 Sanic框架是和Flask相似的异步协程框架，简单轻量，并且性能很高。</p>
<p>本项目就是以Sanic为基础搭建的微服务框架。</p>
<h1 id="特点"><a href="#特点" class="headerlink" title="特点"></a>特点</h1><ul>
<li><strong>使用sanic异步框架，简单，轻量，高效。</strong></li>
<li><strong>使用uvloop为核心引擎，使sanic在很多情况下单机并发甚至不亚于Golang。</strong></li>
<li><strong>使用asyncpg为数据库驱动，进行数据库连接，执行sql语句执行。</strong></li>
<li><strong>使用aiohttp为Client，对其他微服务进行访问。</strong></li>
<li><strong>使用peewee为ORM，但是只是用来做模型设计和migration。</strong></li>
<li><strong>使用opentracing为分布式追踪系统。</strong></li>
<li><strong>使用unittest做单元测试，并且使用mock来避免访问其他微服务。</strong></li>
<li><strong>使用swagger做API标准，能自动生成API文档。</strong></li>
</ul>
<h1 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h1><h4 id="项目地址"><a href="#项目地址" class="headerlink" title="项目地址"></a>项目地址</h4><p><a href="https://github.com/songcser/sanic-ms" target="_blank" rel="external">sanic-ms</a></p>
<p><a href="https://github.com/songcser/sanic-ms/tree/master/examples" target="_blank" rel="external">Example</a></p>
<h4 id="Swagger-API"><a href="#Swagger-API" class="headerlink" title="Swagger API"></a>Swagger API</h4><p><img src="https://github.com/songcser/sanic-ms/raw/master/examples/images/1514528294957.jpg" alt="image"></p>
<h4 id="Zipkin-Server"><a href="#Zipkin-Server" class="headerlink" title="Zipkin Server"></a>Zipkin Server</h4><p><img src="https://github.com/songcser/sanic-ms/raw/master/examples/images/1514528423339.jpg" alt="image"><br><img src="https://github.com/songcser/sanic-ms/raw/master/examples/images/1514528479787.jpg" alt="image"></p>
<h1 id="服务端"><a href="#服务端" class="headerlink" title="服务端"></a>服务端</h1><blockquote>
<p>使用sanic异步框架，有较高的性能，但是使用不当会造成blocking, 对于有IO请求的都要选用异步库。<strong>添加库要慎重</strong>。<br>sanic使用uvloop异步驱动，uvloop基于libuv使用Cython编写，性能比nodejs还要高。</p>
</blockquote>
<p>功能说明：</p>
<h4 id="启动前"><a href="#启动前" class="headerlink" title="启动前"></a>启动前</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">@app.listener('before_server_start')</div><div class="line">async def before_srver_start(app, loop):</div><div class="line"> queue = asyncio.Queue()</div><div class="line"> app.queue = queue</div><div class="line"> loop.create_task(consume(queue, app.config.ZIPKIN_SERVER))</div><div class="line"> reporter = AioReporter(queue=queue)</div><div class="line"> tracer = BasicTracer(recorder=reporter)</div><div class="line"> tracer.register_required_propagators()</div><div class="line"> opentracing.tracer = tracer</div><div class="line"> app.db = await ConnectionPool(loop=loop).init(DB_CONFIG)</div></pre></td></tr></table></figure>
<ul>
<li>创建DB连接池</li>
<li>创建Client连接</li>
<li>创建queue, 消耗span，用于日志追踪</li>
<li>创建opentracing.tracer进行日志追踪</li>
</ul>
<h4 id="中间件"><a href="#中间件" class="headerlink" title="中间件"></a>中间件</h4><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line">@app.middleware('request')</div><div class="line">async def cros(request):</div><div class="line"> if request.method == 'POST' or request.method == 'PUT':</div><div class="line"> request['data'] = request.json</div><div class="line"> span = before_request(request)</div><div class="line"> request['span'] = span</div><div class="line"></div><div class="line"></div><div class="line">@app.middleware('response')</div><div class="line">async def cors_res(request, response):</div><div class="line"> span = request['span'] if 'span' in request else None</div><div class="line"> if response is None:</div><div class="line"> return response</div><div class="line"> result = {'code': 0}</div><div class="line"> if not isinstance(response, HTTPResponse):</div><div class="line"> if isinstance(response, tuple) and len(response) == 2:</div><div class="line"> result.update({</div><div class="line"> 'data': response[0],</div><div class="line"> 'pagination': response[1]</div><div class="line"> })</div><div class="line"> else:</div><div class="line"> result.update({'data': response})</div><div class="line"> response = json(result)</div><div class="line"> if span:</div><div class="line"> span.set_tag('http.status_code', "200")</div><div class="line"> if span:</div><div class="line"> span.set_tag('component', request.app.name)</div><div class="line"> span.finish()</div><div class="line"> return response</div></pre></td></tr></table></figure>
<ul>
<li>创建span, 用于日志追踪</li>
<li>对response进行封装，统一格式</li>
</ul>
<h4 id="异常处理"><a href="#异常处理" class="headerlink" title="异常处理"></a>异常处理</h4><p>对抛出的异常进行处理，返回统一格式</p>
<h4 id="任务"><a href="#任务" class="headerlink" title="任务"></a>任务</h4><p>创建task消费queue中对span，用于日志追踪</p>
<h4 id="异步处理"><a href="#异步处理" class="headerlink" title="异步处理"></a>异步处理</h4><p>由于使用的是异步框架，可以将一些IO请求并行处理</p>
<p>Example:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line">async def async_request(datas):</div><div class="line"> # async handler request</div><div class="line"> results = await asyncio.gather(*[data[2] for data in datas])</div><div class="line"> for index, obj in enumerate(results):</div><div class="line"> data = datas[index]</div><div class="line"> data[0][data[1]] = results[index]</div><div class="line"></div><div class="line">@user_bp.get('/<id:int>')</div><div class="line">@doc.summary("get user info")</div><div class="line">@doc.description("get user info by id")</div><div class="line">@doc.produces(Users)</div><div class="line">async def get_users_list(request, id):</div><div class="line"> async with request.app.db.acquire(request) as cur:</div><div class="line"> record = await cur.fetch(</div><div class="line"> """ SELECT * FROM users WHERE id = $1 """, id)</div><div class="line"> datas = [</div><div class="line"> [record, 'city_id', get_city_by_id(request, record['city_id'])]</div><div class="line"> [record, 'role_id', get_role_by_id(request, record['role_id'])]</div><div class="line"> ]</div><div class="line"> await async_request(datas)</div><div class="line"> return record</div></pre></td></tr></table></figure>
<p>get_city_by_id, get_role_by_id是并行处理。</p>
<h4 id="相关连接"><a href="#相关连接" class="headerlink" title="相关连接"></a>相关连接</h4><p><a href="https://github.com/channelcat/sanic" target="_blank" rel="external">sanic</a></p>
<h1 id="模型设计-amp-ORM"><a href="#模型设计-amp-ORM" class="headerlink" title="模型设计 & ORM"></a>模型设计 & ORM</h1><blockquote>
<p>Peewee is a simple and small ORM. It has few (but expressive) concepts, making it easy to learn and intuitive to use。</p>
<p>ORM使用peewee, 只是用来做模型设计和migration, 数据库操作使用asyncpg。</p>
</blockquote>
<p>Example:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div></pre></td><td class="code"><pre><div class="line"># models.py</div><div class="line"></div><div class="line">class Users(Model):</div><div class="line"> id = PrimaryKeyField()</div><div class="line"> create_time = DateTimeField(verbose_name='create time',</div><div class="line"> default=datetime.datetime.utcnow)</div><div class="line"> name = CharField(max_length=128, verbose_name="user's name")</div><div class="line"> age = IntegerField(null=False, verbose_name="user's age")</div><div class="line"> sex = CharField(max_length=32, verbose_name="user's sex")</div><div class="line"> city_id = IntegerField(verbose_name='city for user', help_text=CityApi)</div><div class="line"> role_id = IntegerField(verbose_name='role for user', help_text=RoleApi)</div><div class="line"></div><div class="line"> class Meta:</div><div class="line"> db_table = 'users'</div><div class="line"></div><div class="line"></div><div class="line"># migrations.py</div><div class="line"></div><div class="line">from sanic_ms.migrations import MigrationModel, info, db</div><div class="line"></div><div class="line">class UserMigration(MigrationModel):</div><div class="line"> _model = Users</div><div class="line"></div><div class="line"> # @info(version="v1")</div><div class="line"> # def migrate_v1(self):</div><div class="line"> # migrate(self.add_column('sex'))</div><div class="line"></div><div class="line">def migrations():</div><div class="line"> try:</div><div class="line"> um = UserMigration()</div><div class="line"> with db.transaction():</div><div class="line"> um.auto_migrate()</div><div class="line"> print("Success Migration")</div><div class="line"> except Exception as e:</div><div class="line"> raise e</div><div class="line"></div><div class="line">if __name__ == '__main__':</div><div class="line"> migrations()</div></pre></td></tr></table></figure>
<ul>
<li>运行命令 python migrations.py</li>
<li>migrate_v1函数添加字段sex, 在BaseModel中要先添加name字段</li>
<li>info装饰器会创建表migrate_record来记录migrate，version每个model中必须唯一，使用version来记录是否执行过，还可以记录author，datetime</li>
<li>migrate函数必须以<strong>migrate_</strong>开头</li>
</ul>
<h4 id="相关连接-1"><a href="#相关连接-1" class="headerlink" title="相关连接"></a>相关连接</h4><p><a href="http://docs.peewee-orm.com/en/latest/" target="_blank" rel="external">peewee</a></p>
<h1 id="数据库操作"><a href="#数据库操作" class="headerlink" title="数据库操作"></a>数据库操作</h1><blockquote>
<p><strong>asyncpg is the fastest driver among common Python, NodeJS and Go implementations</strong></p>
<p>使用asyncpg为数据库驱动, 对数据库连接进行封装, 执行数据库操作。</p>
<p>不使用ORM做数据库操作，一个原因是性能，ORM会有性能的损耗，并且无法使用asyncpg高性能库。另一个是单个微服务是很简单的，表结构不会很复杂，简单的SQL语句就可以处理来，没必要引入ORM。使用peewee只是做模型设计</p>
</blockquote>
<p>Example:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">sql = "SELECT * FROM users WHERE name=$1"</div><div class="line">name = "test"</div><div class="line">async with request.app.db.acquire(request) as cur:</div><div class="line"> data = await cur.fetchrow(sql, name)</div><div class="line"></div><div class="line">async with request.app.db.transaction(request) as cur:</div><div class="line"> data = await cur.fetchrow(sql, name)</div></pre></td></tr></table></figure>
<ul>
<li>acquire() 函数为非事务, 对于只涉及到查询的使用非事务，可以提高查询效率</li>
<li>tansaction() 函数为事务操作，对于增删改必须使用事务操作</li>
<li>传入request参数是为了获取到span，用于日志追踪</li>
<li><strong>TODO</strong> 数据库读写分离</li>
</ul>
<h4 id="相关连接-2"><a href="#相关连接-2" class="headerlink" title="相关连接"></a>相关连接</h4><p><a href="https://github.com/MagicStack/asyncpg" target="_blank" rel="external">asyncpg</a><br><a href="https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/" target="_blank" rel="external">benchmarks</a></p>
<h1 id="客户端"><a href="#客户端" class="headerlink" title="客户端"></a>客户端</h1><blockquote>
<p>使用aiohttp中的client，对客户端进行了简单的封装，用于微服务之间访问。</p>
<p>Don’t create a session per request. Most likely you need a session per application which performs all requests altogether.<br>A session contains a connection pool inside, connection reusage and keep-alives (both are on by default) may speed up total performance.</p>
</blockquote>
<p>Example: </p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">@app.listener('before_server_start')</div><div class="line">async def before_srver_start(app, loop):</div><div class="line"> app.client = Client(loop, url='http://host:port')</div><div class="line"></div><div class="line">async def get_role_by_id(request, id):</div><div class="line"> cli = request.app.client.cli(request)</div><div class="line"> async with cli.get('/cities/{}'.format(id)) as res:</div><div class="line"> return await res.json()</div><div class="line"></div><div class="line">@app.listener('before_server_stop')</div><div class="line">async def before_server_stop(app, loop):</div><div class="line"> app.client.close()</div></pre></td></tr></table></figure>
<p>对于访问不同的微服务可以创建多个不同的client，这样每个client都会keep-alives</p>
<h4 id="相关连接-3"><a href="#相关连接-3" class="headerlink" title="相关连接"></a>相关连接</h4><p><a href="http://aiohttp.readthedocs.io/en/stable/client.html" target="_blank" rel="external">aiohttp</a></p>
<h2 id="日志-amp-分布式追踪系统"><a href="#日志-amp-分布式追踪系统" class="headerlink" title="日志 & 分布式追踪系统"></a>日志 & 分布式追踪系统</h2><blockquote>
<p>使用官方logging, 配置文件为logging.yml, sanic版本要0.6.0及以上。JsonFormatter将日志转成json格式，用于输入到ES</p>
<p>Enter OpenTracing: by offering consistent, expressive, vendor-neutral APIs for popular platforms, OpenTracing makes it easy for developers to add (or switch) tracing implementations with an O(1) configuration change. OpenTracing also offers a lingua franca for OSS instrumentation and platform-specific tracing helper libraries. Please refer to the Semantic Specification.</p>
</blockquote>
<h3 id="装饰器logger"><a href="#装饰器logger" class="headerlink" title="装饰器logger"></a>装饰器logger</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">@logger(type='method', category='test', detail='detail', description="des", tracing=True, level=logging.INFO)</div><div class="line">async def get_city_by_id(request, id):</div><div class="line"> cli = request.app.client.cli(request)</div></pre></td></tr></table></figure>
<ul>
<li>type: 日志类型，如 method, route</li>
<li>category: 日志类别，默认为app的name</li>
<li>detail: 日志详细信息</li>
<li>description: 日志描述，默认为函数的注释</li>
<li>tracing: 日志追踪，默认为True</li>
<li>level: 日志级别，默认为INFO</li>
</ul>
<h3 id="分布式追踪系统"><a href="#分布式追踪系统" class="headerlink" title="分布式追踪系统"></a>分布式追踪系统</h3><ul>
<li>OpenTracing是以Dapper，Zipkin等分布式追踪系统为依据, 建立了统一的标准。</li>
<li>Opentracing跟踪每一个请求，记录请求所经过的每一个微服务，以链条的方式串联起来，对分析微服务的性能瓶颈至关重要。</li>
<li>使用opentracing框架，但是在输出时转换成zipkin格式。 因为大多数分布式追踪系统考虑到性能问题，都是使用的thrift进行通信的，本着简单，Restful风格的精神，没有使用RPC通信。以日志的方式输出, 可以使用fluentd, logstash等日志收集再输入到Zipkin。Zipkin是支持HTTP输入的。</li>
<li>生成的span先无阻塞的放入queue中，在task中消费队列的span。后期可以添加上采样频率。</li>
<li>对于DB，Client都加上了tracing</li>
</ul>
<h4 id="相关连接-4"><a href="#相关连接-4" class="headerlink" title="相关连接"></a>相关连接</h4><p><a href="https://github.com/opentracing/opentracing-python" target="_blank" rel="external">opentracing</a><br><a href="https://github.com/openzipkin/zipkin" target="_blank" rel="external">zipkin</a><br><a href="https://uber.github.io/jaeger/" target="_blank" rel="external">jaeger</a></p>
<h1 id="API接口"><a href="#API接口" class="headerlink" title="API接口"></a>API接口</h1><blockquote>
<p>api文档使用swagger标准。</p>
</blockquote>
<p>Example:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line">from sanic_ms import doc</div><div class="line"></div><div class="line">@user_bp.post('/')</div><div class="line">@doc.summary('create user')</div><div class="line">@doc.description('create user info')</div><div class="line">@doc.consumes(Users)</div><div class="line">@doc.produces({'id': int})</div><div class="line">async def create_user(request):</div><div class="line"> data = request['data']</div><div class="line"> async with request.app.db.transaction(request) as cur:</div><div class="line"> record = await cur.fetchrow(</div><div class="line"> """ INSERT INTO users(name, age, city_id, role_id)</div><div class="line"> VALUES($1, $2, $3, $4, $5)</div><div class="line"> RETURNING id</div><div class="line"> """, data['name'], data['age'], data['city_id'], data['role_id']</div><div class="line"> )</div><div class="line"> return {'id': record['id']}</div></pre></td></tr></table></figure>
<ul>
<li>summary: api概要</li>
<li>description: 详细描述</li>
<li>consumes: request的body数据</li>
<li>produces: response的返回数据</li>
<li>tag: API标签</li>
<li>在consumes和produces中传入的参数可以是peewee的model,会解析model生成API数据, 在field字段的help_text参数来表示引用对象</li>
<li><a href="http://host:ip/openapi/spec.json" target="_blank" rel="external">http://host:ip/openapi/spec.json</a> 获取生成的json数据</li>
</ul>
<h4 id="相关连接-5"><a href="#相关连接-5" class="headerlink" title="相关连接"></a>相关连接</h4><p><a href="https://swagger.io/" target="_blank" rel="external">swagger</a></p>
<h1 id="Response-数据"><a href="#Response-数据" class="headerlink" title="Response 数据"></a>Response 数据</h1><p>在返回时，不要返回sanic的response，直接返回原始数据，会在Middleware中对返回的数据进行处理，返回统一的格式，具体的格式可以[查看]</p>
<h1 id="单元测试"><a href="#单元测试" class="headerlink" title="单元测试"></a>单元测试</h1><blockquote>
<p>单元测试使用unittest。 mock是自己创建了MockClient，因为unittest还没有asyncio的mock，并且sanic的测试接口也是发送request请求，所以比较麻烦. 后期可以使用pytest。</p>
</blockquote>
<p>Example:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div></pre></td><td class="code"><pre><div class="line">from sanic_ms.tests import APITestCase</div><div class="line">from server import app</div><div class="line"></div><div class="line">class TestCase(APITestCase):</div><div class="line"> _app = app</div><div class="line"> _blueprint = 'visit'</div><div class="line"></div><div class="line"> def setUp(self):</div><div class="line"> super(TestCase, self).setUp()</div><div class="line"> self._mock.get('/cities/1',</div><div class="line"> payload={'id': 1, 'name': 'shanghai'})</div><div class="line"> self._mock.get('/roles/1',</div><div class="line"> payload={'id': 1, 'name': 'shanghai'})</div><div class="line"></div><div class="line"> def test_create_user(self):</div><div class="line"> data = {</div><div class="line"> 'name': 'test',</div><div class="line"> 'age': 2,</div><div class="line"> 'city_id': 1,</div><div class="line"> 'role_id': 1,</div><div class="line"> }</div><div class="line"> res = self.client.create_user(data=data)</div><div class="line"> body = ujson.loads(res.text)</div><div class="line"> self.assertEqual(res.status, 200)</div></pre></td></tr></table></figure>
<ul>
<li>其中_blueprint为blueprint名称</li>
<li>在setUp函数中，使用_mock来注册mock信息, 这样就不会访问真实的服务器, payload为返回的body信息</li>
<li>使用client变量调用各个函数, data为body信息，params为路径的参数信息，其他参数是route的参数</li>
</ul>
<h3 id="代码覆盖"><a href="#代码覆盖" class="headerlink" title="代码覆盖"></a>代码覆盖</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">coverage erase</div><div class="line">coverage run --source . -m sanic_ms tests</div><div class="line">coverage xml -o reports/coverage.xml</div><div class="line">coverage2clover -i reports/coverage.xml -o reports/clover.xml</div><div class="line">coverage html -d reports</div></pre></td></tr></table></figure>
<ul>
<li>coverage2colver 是将coverage.xml 转换成 clover.xml，bamboo需要的格式是clover的。</li>
</ul>
<h4 id="相关连接-6"><a href="#相关连接-6" class="headerlink" title="相关连接"></a>相关连接</h4><p><a href="https://docs.python.org/3/library/unittest.html" target="_blank" rel="external">unittest</a><br><a href="https://coverage.readthedocs.io/en/coverage-4.4.1/" target="_blank" rel="external">coverage</a></p>
<h1 id="异常处理-1"><a href="#异常处理-1" class="headerlink" title="异常处理"></a>异常处理</h1><blockquote>
<p>使用 app.error_handler = CustomHander() 对抛出的异常进行处理</p>
</blockquote>
<p>Example:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line">from sanic_ms.exception import ServerError</div><div class="line"></div><div class="line">@visit_bp.delete('/users/<id:int>')</div><div class="line">async def del_user(request, id):</div><div class="line"> raise ServerError(error='内部错误',code=10500, message="msg")</div></pre></td></tr></table></figure>
<ul>
<li>code: 错误码，无异常时为0，其余值都为异常</li>
<li>message: 状态码信息</li>
<li>error: 自定义错误信息</li>
<li>status_code: http状态码，使用标准的http状态码</li>
</ul>
]]></content>
<summary type="html">
<p>基于sanic的微服务基础架构</p>
<h1 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h1><p>使用python做web开发面临的一个最大的问题就是性能，在解决C10K问题上显的有点吃力。有
</summary>
<category term="asyncio" scheme="https://www.songcser.com/categories/asyncio/"/>
<category term="sanic" scheme="https://www.songcser.com/categories/asyncio/sanic/"/>
<category term="asyncio" scheme="https://www.songcser.com/tags/asyncio/"/>
<category term="sanic" scheme="https://www.songcser.com/tags/sanic/"/>
<category term="aiohttp" scheme="https://www.songcser.com/tags/aiohttp/"/>
<category term="peewee" scheme="https://www.songcser.com/tags/peewee/"/>
<category term="opentracing" scheme="https://www.songcser.com/tags/opentracing/"/>
</entry>
<entry>
<title>Hello World</title>
<link href="https://www.songcser.com/2017/09/04/hello-world/"/>
<id>https://www.songcser.com/2017/09/04/hello-world/</id>
<published>2017-09-04T01:57:01.000Z</published>
<updated>2017-09-04T01:57:01.000Z</updated>
<content type="html"><![CDATA[<p>Welcome to <a href="https://hexo.io/" target="_blank" rel="external">Hexo</a>! This is your very first post. Check <a href="https://hexo.io/docs/" target="_blank" rel="external">documentation</a> for more info. If you get any problems when using Hexo, you can find the answer in <a href="https://hexo.io/docs/troubleshooting.html" target="_blank" rel="external">troubleshooting</a> or you can ask me on <a href="https://github.com/hexojs/hexo/issues" target="_blank" rel="external">GitHub</a>.</p>
<h2 id="Quick-Start"><a href="#Quick-Start" class="headerlink" title="Quick Start"></a>Quick Start</h2><h3 id="Create-a-new-post"><a href="#Create-a-new-post" class="headerlink" title="Create a new post"></a>Create a new post</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ hexo new <span class="string">"My New Post"</span></div></pre></td></tr></table></figure>
<p>More info: <a href="https://hexo.io/docs/writing.html" target="_blank" rel="external">Writing</a></p>
<h3 id="Run-server"><a href="#Run-server" class="headerlink" title="Run server"></a>Run server</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ hexo server</div></pre></td></tr></table></figure>
<p>More info: <a href="https://hexo.io/docs/server.html" target="_blank" rel="external">Server</a></p>
<h3 id="Generate-static-files"><a href="#Generate-static-files" class="headerlink" title="Generate static files"></a>Generate static files</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ hexo generate</div></pre></td></tr></table></figure>
<p>More info: <a href="https://hexo.io/docs/generating.html" target="_blank" rel="external">Generating</a></p>
<h3 id="Deploy-to-remote-sites"><a href="#Deploy-to-remote-sites" class="headerlink" title="Deploy to remote sites"></a>Deploy to remote sites</h3><figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">$ hexo deploy</div></pre></td></tr></table></figure>
<p>More info: <a href="https://hexo.io/docs/deployment.html" target="_blank" rel="external">Deployment</a></p>
]]></content>
<summary type="html">
<p>Welcome to <a href="https://hexo.io/" target="_blank" rel="external">Hexo</a>! This is your very first post. Check <a href="https://hexo.
</summary>
</entry>
</feed>