benchmarks and performance

[allen-munsch]

In regards to the performance characteristics.

I just started looking into this.

• https://github.com/allen-munsch/benchmark-django-fastapi/blob/main/slowasgi/asgi.py#L25-L41
• https://github.com/allen-munsch/benchmark-django-fastapi/blob/main/load_tests/k6_test.js#L133-L162

I had pulled whatever was out on the pypi, when i ran with those tests. I need to segment the variations a bit before re-running this.

@rednaks thanks for the quick response with the merge of the PR #3

I'll report back with my findings.

edit:

• https://github.com/allen-munsch/benchmark-django-fastapi/blob/main/slowasgi/asgi.py#L43-L50
• https://github.com/allen-munsch/benchmark-django-fastapi/blob/main/testdjango/wsgi_with_slow_api_mounted.py
• https://github.com/allen-munsch/benchmark-django-fastapi/blob/main/docker/web/docker-gunicorn-eventlet-slowapi-wsgi-w2-entrypoint.sh

Looks like it works really nicely inside of django.

SERVED_BY=fastapi-with-django-mounted-asgi
PYTHON_DEP_VARIATION=_django-async-orm.0.1.8

05:25:12 jmunsch@pop-os benchmark-django-fastapi ±|main ✗|→ ab -n 100 -c 10 127.0.0.1:8000/api/clowncollege/with-djang-async-orm/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:        uvicorn
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /api/clowncollege/with-djang-async-orm/
Document Length:        828 bytes

Concurrency Level:      10
Time taken for tests:   1.405 seconds
Complete requests:      100
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Non-2xx responses:      1
Total transferred:      260517 bytes
HTML transferred:       239288 bytes
Requests per second:    71.19 [#/sec] (mean)
Time per request:       140.461 [ms] (mean)
Time per request:       14.046 [ms] (mean, across all concurrent requests)
Transfer rate:          181.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:    18  139  24.3    134     305
Waiting:       17  138  24.3    134     304
Total:         18  139  24.3    134     305

Percentage of the requests served within a certain time (ms)
  50%    134
  66%    135
  75%    135
  80%    141
  90%    177
  95%    178
  98%    178
  99%    305
 100%    305 (longest request)

Trying to mount fastapi into wsgi, and re-using the django orm with async is ... uh ... i dunno

PYTHON_DEP_VARIATION="_django-async-orm.0.1.8"
SERVED_BY=gunicorn-eventlet-slowapi-wsgi-w2

ab -v all -n 10 -c 10 127.0.0.1:8000/slowapi/slowapi/sleep/ 

04:22:18 jmunsch@pop-os benchmark-django-fastapi ±|main ✗|→ ab -v all -n 10 -c 10 127.0.0.1:8000/slowapi/slowapi/sleep/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:        gunicorn
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /slowapi/slowapi/sleep/
Document Length:        143 bytes

Concurrency Level:      10
Time taken for tests:   6.064 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      2890 bytes
HTML transferred:       1430 bytes
Requests per second:    1.65 [#/sec] (mean)
Time per request:       6063.948 [ms] (mean)
Time per request:       606.395 [ms] (mean, across all concurrent requests)
Transfer rate:          0.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:  1008 4248 1243.8   5056    5056
Waiting:     1007 4247 1243.5   5054    5055
Total:       1008 4248 1243.9   5056    5057

Percentage of the requests served within a certain time (ms)
  50%   5056
  66%   5056
  75%   5056
  80%   5056
  90%   5057
  95%   5057
  98%   5057
  99%   5057
 100%   5057 (longest request)

[allen-munsch]

[rednaks]

Oh wow, thank you for the numbers ! indeed it looks like there is no much difference and closure is ~0.01 slower.

I have question though, the first test with version 0.1.4 shouldn't be failing ?

[allen-munsch]

these were run with 512mb shared_buffers and 200 max connections on postgres, might have been the reason for the couple of failed requests.

[allen-munsch]

I just deleted the comment, the numbers were a bit funky.

[allen-munsch]

[rednaks]

just to be sure, the number we're looking at is iteration_during right ? having an avg of ~800ms is quite a lot right ?

[allen-munsch]

@rednaks yeah i think there is a bottle neck somewhere. The number that I was looking at though was http_reqs 22718 371.016748/s

The way the numbers get made here. 300 VUs means 300 simultaneous (V)irtual (U)sers running as many loops as each can go through. Doing this over and over : https://github.com/allen-munsch/benchmark-django-fastapi/blob/main/load_tests/k6_test.js#L133-L162

I think the reason why the iteration_duration is higher is because of having a single postgres instance, on the same laptop that was running the test. And the timer that k6 uses for the iteration duration. Also, I was only using 1 gunicorn worker.

[allen-munsch]

150 VUs
postgres 8GB shared_buffer, max_connections=32000
10 gunicorn workers

All of the failed requests are from the postgres connections maxing out.

web_1         | DEBUG:root:could not connect to server: Cannot assign requested address
web_1         | 	Is the server running on host "postgres" (172.18.0.2) and accepting
web_1         | 	TCP/IP connections on port 5432?

running (1m00.2s), 000/150 VUs, 41013 complete and 0 interrupted iterations
default ✓ [======================================] 000/150 VUs  1m0s
INFO[0061] {
  "test_user_password": "password",
  "test_review_app_url": "",
  "PYTHON_VERSION": "3.8.10",
  "TERM": "xterm",
  "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "test_with_slowapi": "",
  "test_try_python_async": "",
  "SERVED_BY": "gunicorn-gevent-slowapi-wsgi-psycogreen",
  "PYTHON_DEP_VARIATION": "_django-async-orm.with-gevent",
  "test_scenario_option": "easy",
  "HOSTNAME": "89b24bee68d0",
  "HOME": "/home/k6",
  "test_email": "a0@b",
  "test_only_slowapi": "honk"
}  source=console
INFO[0061] SERVED_BY: gunicorn-gevent-slowapi-wsgi-psycogreen  source=console
INFO[0061] PYTHONV_VERSION: 3.8.10                       source=console
INFO[0061] options:                                      source=console
INFO[0061] {
  "stages": [
    {
      "duration": "1m",
      "target": 150
    }
  ]
}  source=console
     ✓ http://web:8000/slowapi/slowapi/: status is 200
     ✓ Honk Honk
     ✗ http://web:8000/slowapi/slowapi/django-async-orm/closure/: status is 200
      ↳  69% — ✓ 28367 / ✗ 12646

     checks.........................: 89.72% ✓ 110393      ✗ 12646
     data_received..................: 16 MB  269 kB/s
     data_sent......................: 8.4 MB 140 kB/s
     http_req_blocked...............: avg=4.79µs   min=933ns    med=2.61µs  max=49.81ms  p(90)=3.92µs   p(95)=4.45µs  
     http_req_connecting............: avg=1.68µs   min=0s       med=0s      max=49.75ms  p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=54.83ms  min=589.86µs med=27.82ms max=637.74ms p(90)=150.94ms p(95)=211.92ms
       { expected_response:true }...: avg=45.02ms  min=589.86µs med=20.42ms max=637.74ms p(90)=127.2ms  p(95)=180.71ms
     http_req_failed................: 15.41% ✓ 12646       ✗ 69380
     http_req_receiving.............: avg=268.26µs min=13.33µs  med=64.21µs max=40.58ms  p(90)=187.42µs p(95)=564.05µs
     http_req_sending...............: avg=17.45µs  min=4.96µs   med=13.95µs max=17.93ms  p(90)=21.02µs  p(95)=24.68µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s      max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=54.55ms  min=538.1µs  med=27.56ms max=637.7ms  p(90)=150.6ms  p(95)=211.43ms
     http_reqs......................: 82026  1363.081532/s
     iteration_duration.............: avg=109.99ms min=5.31ms   med=57.26ms max=801.14ms p(90)=315.95ms p(95)=393.52ms
     iterations.....................: 41013  681.540766/s
     vus............................: 149    min=3         max=149
     vus_max........................: 150    min=150       max=150
     ```

[allen-munsch]

deleted comment the numbers seemed funky