Mongo Connection Timeouts

[jmrichardson]

Hi,

I am using the Mongo crawlDataStoreFactory:

      <crawlDataStoreFactory
        class="com.norconex.collector.fs.data.store.impl.mongo.MongoCrawlDataStoreFactory">
        <host>localhost</host>
        <port>27017</port>
        <dbname>clients</dbname>
      </crawlDataStoreFactory>

Unfortunately, the connections to mongo sometimes are inactive (for some reason) which causes the crawler to stop and wait (indefinitely). This appears to happen at random times and I am not sure the root cause. If I restart Mongo, it closes the connections but the crawler also crashes as well. I am wondering if there is a way to put the maxIdleTimeout parameter into the Mongo connection string to avoid the hang:

https://docs.mongodb.com/manual/reference/connection-string/#maxIdleTimeMS

I am still investigating what is causing the issue. I am lowering the thread count to see if that helps as well. Thanks for your help

[essiembre]

I am marking as a feature request. In the meantime, the filesystem collector should be able to resume where it stopped when you restart it (using the "resume" flag when starting it). You could use this to make sure you do not have to restart from the beginning.