Skip to content
This repository has been archived by the owner on Mar 4, 2020. It is now read-only.
/ asyncurl Public archive

🤖 Python Asynchronous cURL Requests

License

Notifications You must be signed in to change notification settings

hidekuma/asyncurl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

79 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

asyncurl Build Status

Asynchronous cURL Requests using python, which is inspired by this benchmark(KR, EN).

asyncurl-logo


Support python versions

python >= 3.6

Dependencies

AsyncURL project consists of the following packages:

Package Version Description
asyncio >=3.4.3 for Asynchronous
requests >=2.22.0 pycurl substitutes
uvloop >=0.12.2 for event loop policy

Installation

You can download asyncurl executable and binary distributions from PyPI.

Using pip

pip install asyncurl

Usage

Import AsyncURL.

from asyncurl.fetch import AsyncURLFetch

ac_fetch = AsyncURLFetch()

Default worker's count is 2. you can change it if you want.

ac_fetch.worker = 3

and you can put urls to <AsyncURL.queue>.

for x in range(2):
    ac_fetch.queue.put_nowait('http://localhost')

ac_fetch.parallel()

Then call parallel(). The fucntion fetch urls using <requests>(HTTP library for Python).

and AsyncURL can change <requests>'s method and else properties.

from asyncurl.session import AsyncURLSession
from asyncurl.fetch import AsyncURLFetch

ac_fetch = AsyncURLFetch()

for x in range(2):
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
    session = AsyncURLSession('GET', 'http://localhost', headers=headers)
    ac_fetch.queue.put_nowait(session)

ac_fetch.parallel()

AsyncURLSession is inheritance of <requests.Session>.

parallel() will return <AsncURLFetch>, and it can show results to you.

Show results:

ac_fetch.parallel().results

The order of result is nonsequential. and it will return list of <requests.Response>.

AsyncURLSession's properties

equals to <requests.Request>

  • params=None
  • data=None
  • headers=None
  • cookies=None
  • files=None
  • auth=None
  • timeout=None
  • allow_redirects=True
  • proxies=None
  • hooks=None
  • stream=None
  • verify=None
  • cert=None
  • json=None

Examples

for x in range(3):
    session = AsyncURLSession('GET', 'http://localhost')
    ac_fetch.queue.put_nowait(session)
    
# case.1) with callback
print('[with callback]')
ac_fetch.parallel(callback=lambda x: print('with callback : {0}'.format(x)))

# case.2) return results
print('[return results]')
print(ac_fetch.parallel().results)

>>>
[with callback]
with callback : <Future finished result=<Response [403]>>
with callback : <Future finished result=<Response [403]>>
with callback : <Future finished result=<Response [403]>>

[return results]
[<Response [403]>, <Response [403]>, <Response [403]>]

The MIT License (MIT)

Copyright (c) 2019 Hidden function by hidekuma