Should not visit pages that have already been visited

[abineetds]

How can I make it not visit the same page multiple times?
How can I make it so that it doesn't visit any pages outside of the domain?

[abineetds]

Also when I ran it with a memo, I got an error eventually

.../gems/ruby-3.1.2/gems/ferrum-0.11/lib/ferrum/browser/web_socket.rb:19:in `initialize': Too many open files - socket(2) for "127.0.0.1" port 65073 (Errno::EMFILE)

[route]

Also when I ran it with a memo, I got an error eventually

.../gems/ruby-3.1.2/gems/ferrum-0.11/lib/ferrum/browser/web_socket.rb:19:in `initialize': Too many open files - socket(2) for "127.0.0.1" port 65073 (Errno::EMFILE)

I think you should tune your OS for example for Linux

[route]

As for the issue I have a plan to intro an option for request but unfortunately it won't work for all the websites. So it's going to be very optional.

[abineetds]

Also when I ran it with a memo, I got an error eventually

.../gems/ruby-3.1.2/gems/ferrum-0.11/lib/ferrum/browser/web_socket.rb:19:in `initialize': Too many open files - socket(2) for "127.0.0.1" port 65073 (Errno::EMFILE)

I think you should tune your OS for example for Linux

What is the root cause for this? It seems to me that while opening a TCP Socket connection, ferrum opens a file but never closes it? Shouldn't this not happen since the number of pages being processed at once is at most the number of processors (unless overridden).

[route]

Ferrum opens only one connection per page and closes it when page is processed releasing the page and connection. So something is wrong with the crawler most likely