os-scrapy-uvicorn

This project provide a extension to start a ASGI http server(Uvicorn) along with Scrapy in the same process.

You can use the ASGI framework(recommend FastAPI) to create app to communicate with Scrapy.

Require: Python 3.6+, Scrapy 2.0+

Install

pip install os-scrapy-uvicorn

You can run example spider directly in the project root path

scrapy crawl example

Settings

• use asyncio reactor

  TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
  

  or you can use os-scrapy(installed with this project) to start crawling with -r command line option

  os-scrapy crawl -r asyncio example
  

• enable extension

  EXTENSIONS = {
      "os_scrapy_uvicorn.Uvicron": 1,
  }
  

• app path, you can use ASGI app frameworks to create your app, FastAPI is recommended

  UVICORN_APP = "app_module:app"
  

• uvicorn server settings, supported settings

  UVICORN_CONFIG = {"host": "0.0.0.0", "port": 5000}
  

• when the server started, Scrapy crawler instance is attached to the app, it is the entrypoint to commnicate with Scrapy

Unit Tests

sh scripts/test.sh

License

MIT licensed.