Error with Default run

[ameygat]

I have installed the fresh onioningestor from git and on first run it gave following error:
(it also gave error about monitoring.txt not there but I created a file with single onion link)

$ python -m onioningestor --config onioningestor.yml
[23 Jul 2020 18:38:05] - INFO - Starting OnionScraper
[23 Jul 2020 18:38:05] - INFO - Loading config file
[23 Jul 2020 18:38:05] - INFO - Creating Elasticsearch mapping
[23 Jul 2020 18:38:05] - INFO - Onions will be saved synchronously
[23 Jul 2020 18:38:05] - INFO - Initializing simple-html
[23 Jul 2020 18:38:05] - INFO - Initializing onionscan-go
[23 Jul 2020 18:38:05] - ERROR - __init__() takes 2 positional arguments but 4 were given
Traceback (most recent call last):
  File "/home/user1/OnionIngestor/onioningestor/__init__.py", line 66, in __init__
    for name, operator, kwargs in self.config.operators()}
  File "/home/user1/OnionIngestor/onioningestor/__init__.py", line 66, in <dictcomp>
    for name, operator, kwargs in self.config.operators()}
TypeError: __init__() takes 2 positional arguments but 4 were given

Current onioningestor.yml:

# This is an example ThreatIngestor config file with some preconfigured RSS
# sources, feeding extracted artifacts into a CSV file.

general:
    # Run forever, check feeds once an hour.
    daemon: True
    sleep: 10
    onion_validation: ([a-z2-7]{16,56}\.onion)
    blacklist: porn,cvv
    interestingKeywords: leak,deface,hack,ddos,exploit
    save-thread: no         # Use a separate thread to save onions
    TorController:
        port: 9051
        password: mysecreatetorpass

monitor:
    filename: monitoring.txt

sources:
    # A few threat intel blogs to get you started!
    - name: simple-text-file
      module: simplefile
      filename: onion_master_list.txt

    - name: source-gist
      module: gist
      url: https://gist.github.com/search?l=Text&q=.onion

    - name: source-reddit
      module: reddit
      url: https://api.pushshift.io/reddit/search/comment/?subreddit=onions&limit=1000000
      feed_type: messy

    - name: pastebin
      module: pastebin-account
      url: https://gist.github.com/search?l=Text&q=.onion
      feed_type: messy

    - name: hunchly-report
      module: gmail-hunchly
      url: https://gist.github.com/search?l=Text&q=.onion
      feed_type: messy

    - name: onionland-search
      module: collect-onions
      url: http://3bbaaaccczcbdddz.onion/discover
      feed_type: messy
 
    - name: torch
      module: collect-onions
      url: http://xmh57jrzrnw6insl.onion
      feed_type: messy
	  
operators:
   - name: simple-html
     module: html
     timeout: 300
     retries: 2
     interestingKeywords: leak,deface,exploit,hack
     socks5:
         http: 'socks5h://127.0.0.1:9050'
         https: 'socks5h://127.0.0.1:9050'

   - name: onionscan-go
     module: onionscan
     binpath: /home/user1/go/bin/onionscan


   - name: simple-screenshot
     module: screenshot
     screenshots_path: /home/user1/screenshots/


#  - name: yara-rule
#    module: yara
#    filename: categories.yar
#    base_score: 50


database_Engines:
    # Simple telegram notifier
    - name: telegram-notifer
      module: telegram
      chat_id: YOUR-TELEGRAM-CHAT
      token: YOUR-TELEGRAM-TOKEN

    - name: elasticsearch
      module: elasticsearch
      index: onioningest
      port : 9200
      host : 127.1.1.1

#  - name: email
#    module: send_email
#    alert: no             # Enable/disable email alerts
#    from: [email protected]
#    to: [email protected]
#    server: 127.0.0.1     # Address of the server (hostname or IP)
#    port: 25              # Outgoing SMTP port: 25, 587, ...
#    tls: no               # Enable/disable tls support
#    username: ''          # (optional) Username for authentication. Leave blank for no authentication.
#    password: ''          # (optional) Password for authentication. Leave blank for no authentication.
#    subject: '[onioningestor] - {subject}'
#    size-limit: 1048576   # Size limit for pastie, above it's sent as attachement

[danieleperera]

Hi,

The error is thrown because only the simple-text-file module is currently tested and supported!
I'm still working on other sources at the moment. To fix the issue I suggest commenting out all under-development sources in the config file and remember to create a list of onion links and name the file onion_master_list.txt as in the config file.

onion_master_list.txt file should look like this:

onionlinkaddress1.onion
onionlinkaddress2.onion
...
etc

For operators simple-screenshot is still under development so please comment it also in the config file. The onionscan-go module works fine. Have you already installed it? Before using it as a module I suggest running a test scan using onionscan.
onionscan --webport=0 --jsonReport --simpleReport=false YOUR-onionlinkadress.onion

There is an issue on the go binary. I haven't created documentation for this module yet but you can try it out yourself. The issue is that onionscan's link validation regex does not support V3 .onion address that is 56 char long. To fix this check out their issues

Finally, If you are not using telegram-notifer comment it out and the collected items should be indexed only on elasticsearch.

Hope this helps.