Export logs from tpot to SIEM

[vasokolov]

Hello everyone!
I am a newbee in ELK and docker, so i hope somebody will help me to find the way.
I have been install a tpot in my infrastructure, it was hive installation.
So it looks like everything works fine. But now i want to send events from tpot sensors (i mean honeypots logs) to my SIEM system.
I know that every honeypot save all his logs in ~tpotce/data in his folder.
But, as i see, all of this logs going to logstash, so maybe the simplest way for exporting logs on external siem is reconfiguring logstash with some file? but i have no idea how to do it. Also, i think that i don't want to replace a "target point" for logstash, because it will remove data from kibana, i want to add a second, so can i send data from logstash for 2 ways?

[vasokolov]

hi again! so, if you want to export honeypots data to another destination, such as SIEM, you can use logstash container from tpot instance.
First step you should add a syslog output plugin to logstash, (command for this action already include in logstash Dockerfile, but you need to uncomment this).
After that you should build you own image of logstash from this Dockerfile and use this image for logstash on tpot (replace logstash image path in main docker-compose.yml)
And finally, in main docker-compose.yml add a logstash configuration file to a logstash container via volumes of contaier.

I hope this short answer will help somebody with similar situation. Sorry for my english, this is not my mother tongue=)

[akajhon]

Helped me a lot @vasokolov ! Thanks!

[vasokolov]

so, finally i meet a new trouble with logstash output syslog plugin. It does not work very well. logstash-plugins/logstash-output-syslog#51 (comment)
and this

I preffer to use http logstash output plugin, which are already included in TPOTCE logstash instance.
If you want to do this, u should put something like this inside your logstash config. Finaly, you will get a structured similar json events from TPOT inside your SIEM.
http { url => 'http://ip:port/input' # input is only in my case! http_method => post retry_non_idempotent => true format => json }

[Sagarshar06]

Do I have to add environment variables for forwarding syslog to remote ip?

[vasokolov]

sorry, but i don't get what u mean. if u want to forward some event to another destination via rsyslog, or syslog ng, you dont need to add some variables to environment. but it may depence of your personal case. maybe u will find a full answer in rsyslog documentation

[Sagarshar06]

Apologies for not elaborating it. Here is what I want to do! I want to send all the honeytrap logs to my SIEM so that I can make SIEM use cases for monitoring the events. In order to achieve this should I forward all the container logs to Syslog or just the TPOT's syslog?
I tried sending TPOT's syslog to SIEM and It was successful, but I am not getting the event which is required to make use case. The only logs I am getting is related to tpot service whether it is failed or started here is reference log "tpot.service: Scheduled restart job, restart counter is at 4315."

[vasokolov]

if u want catch only honeypot logs, u can go for my way with using logstash. this is work fine, and this is strucrureg log format (JSON), so it is very nice for parsing events, making a rules and cases for SIEM.
If u want to see a honeypot platform log (OS log) and control system.d services, like tpot.service or docker service or anything else, u should connect it on to SIEM like typical linux OS.

[Sagarshar06]

Can you elaborate your way like how did you achieved this what changes did you made in the logstash configuration? Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: vasokolov ***@***.***> Sent: Monday, August 12, 2024 1:04:04 PM To: telekom-security/tpotce ***@***.***> Cc: Sagar Sharma ***@***.***>; Comment ***@***.***> Subject: Re: [telekom-security/tpotce] Export logs from tpot to SIEM (Discussion #1617) CAUTION: This email originated from outside of Milliman. Do not click links or open attachments unless you recognize the sender and know the content is safe. if u want catch only honeypot logs, u can go for my way with using logstash. this is work fine, and this is strucrureg log format (JSON), so it is very nice for parsing events, making a rules and cases for SIEM. If u want to see a honeypot platform log (OS log) and control system.d services, like tpot.service or docker service or anything else, u should connect it on to SIEM like typical linux OS. — Reply to this email directly, view it on GitHub<#1617 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BEK3DEM5KAE333JHMQNUQQ3ZRBQOZAVCNFSM6AAAAABKW4CALSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZRGA4DEMA>. You are receiving this because you commented.Message ID: ***@***.***>

________________________________ Milliman values and protects your personal information. You can find information on how we use your personal information and how to exercise your rights here<https://us.milliman.com/en/global-privacy-policy>.

[dmille6]

i have done something similar. i'm not sure its the best approach, but it does work.. its currently more a proof of concept.. but it is effective.

Tpot Hive --> ElasticSearch --> OpenCTI --> <Stix/Taxi Feed> --> SIEM

I modified my tpot hive http_input.conf to send the data to the Hive install AND a separate elasticsearch instance.. i didn't want to mess with the hive elasticsearch instance.. so I just send the data to another instance while keeping the hive intact.

#http_input.conf file:

# Output section
output {
  elasticsearch {. #hive elasticserach instance
    hosts => ["elasticsearch:9200"]
    # With templates now being legacy we need to set the daily index with its template manually. Otherwise a new index might be created with differents settings configured through Kibana.
    index => "logstash-%{+YYYY.MM.dd}"
    template => "/etc/logstash/tpot-template.json"
    template_overwrite => "true"
  }

  elasticsearch { #seperate instance 
    hosts => ["http://<URL>:9200"]
    index => "logstash-hive-1"
    user => "<username>"
    password => "<password>"
    ssl => false
    action => "create"	# used for es data streams
    #template => "/etc/logstash/tpot-template.json"
    #template_overwrite => "true"
  }

}

I use a python script run once an hour that queries ES and pulls the tpot data I want and dumps it into OpenCTI: https://github.com/OpenCTI-Platform/opencti

OpenCTI has an option to create a STIX/TAXII feed based on criteria.. so I have a feed.. and this feed is pulled in by various infrastructure. i wish the OpenCTI API was a little more documented.. but there is a good community behind the project, and its a great open source project.

its been very effective and works well.

i hope this makes sense.

[dmille6]

OpenCTI has "connectors" and i'd love to write a connector that pulls in tpot data directly.. i just haven't had time to play with that aspect of the project yet.

[akajhon]

Hello, @dmille6 !

I took a similar approach for shipping the logs to a SIEM.... Modifying the http_input.conf file seems to be the best approach. I Wrote a little script for pulling the data from the TPOT Hive ES and sending them directly via syslog.

Also, loved your approach for creating feeds from the TPOT data via OpenCTI. Would like to recommend the 'GreedyBear' project for this: GreedyBear

Hope it helps!

[dmille6]

thanks a lot, i'll take a look at greedybear.

[ab-sits]

So as far as i understand the actual conversion from json to stix is done by opencti? Would you mind sharing your python script as well as your opencti mappings?

[dmille6]

@ab-sits I dont send everything from tpot over.. doing a direct 1 to 1 mapping from tpot to openCTI would take a HUGE amount of space and not really relate to actionable good data.. so at this point I'm just pulling the following bits of info:

• ipv4 and ipv6 malicious IP's : src_ip
• usernames and passwords : username + password
• bash commands run : input
• file hashes : shasum

thats what I stuff in.. and I add the geo location info

I just create a stix bundle for all the info, and relationships.. then push the stix bundle into openCTI. I add a bunch of tags to keep things organized.

in openCTI, I create a stix/taxii feed based on different tags

my script is just a cronjob that runs every 15 min... it works.. okay (not great).. I really need to build a openCTI plugin.. this is a far better approach.. because it adds the stix bundles to a queue for the workers to process when they have time and available resources.

its a side project.. so I'll get to it eventually.

[t3chn0m4g3]

The Elastic Stack (Logstash) now includes the Syslog Output Plugin, images have been deployed a few minutes ago.

Logstash Configuration Customization (recommended only on Hive)

The Logstash configurations must be customized using volumes. To do this, stop T-Pot (sudo systemctl stop tpot) and then modify both logstash.conf and http_input.conf located in tpotce/docker/elk/logstash/dist for example:

  syslog {
    host => "192.168.1.100"
    port => 514
    protocol => tcp
    appname => "logstash-logs"
    severity => "6"
  }

In the docker-compose.yml file located at tpotce/docker-compose.yml, add two volumes to the configuration of the Logstash service:

Logstash service

  logstash:
    container_name: logstash
    restart: always
    depends_on:
      elasticsearch:
        condition: service_healthy
    networks:
     - nginx_local
    environment:
     - LS_JAVA_OPTS=-Xms1024m -Xmx1024m
     - TPOT_TYPE=${TPOT_TYPE:-HIVE}
     - TPOT_HIVE_USER=${TPOT_HIVE_USER}
     - TPOT_HIVE_IP=${TPOT_HIVE_IP}
     - LS_SSL_VERIFICATION=${LS_SSL_VERIFICATION:-full}
    ports:
     - "127.0.0.1:64305:64305"
    mem_limit: 2g
    image: ${TPOT_REPO}/logstash:${TPOT_VERSION}
    pull_policy: ${TPOT_PULL_POLICY}
    volumes:
     - ${TPOT_DATA_PATH}:/data
     - modified_logstash.conf:/etc/logstash/logstash.conf
     - modified_http_input.conf:/etc/logstash/http_input.conf

Then start T-Pot (sudo systemctl start tpot) again.

Monitor the logs with (docker compose logs; run this command directly in the tpotce folder) in case something doesn't work as expected.

Important Notes:

• modified_logstash.conf and modified_http_input.conf are the names of the files you will create with your custom configurations. You'll need to create these files locally before running docker-compose.
• This setup mounts those local configuration files into the container, overriding the default ones.