-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨ Add Core #40
✨ Add Core #40
Conversation
|
||
|
||
class AioHttpEngine: | ||
default_request_limit = 1 | ||
request_cls = Request | ||
response_cls = Response |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
where use the request_cls, response_cls?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice catch, forgot to use them in code. Will make another commit now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.response_cls
this is already used. when translating response.
cond="every 1 second", | ||
controller=ProcessController(app=RocketryApplication()), | ||
) | ||
await process.add_spiders() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add_spiders() -> empty?!
where are the spiders to add?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we define them in the chain
async def main():
process = Process(
spider=MySpider() >> MySpiderTwo() >> MySpiderThree(),
cond="every 1 second",
controller=ProcessController(app=RocketryApplication()),
)
await process.add_spiders()
the method add_spiders
also adds them to the controller, if there's any defined.
it is not meant to be called like this, usually this happen underhood in FastCrawler App. but if a user want to initiate and run a process manually, that's the way to go.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need Some Changes:
- add_spiders ?
- request_cls using for?
closes #16