Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] how to setup tlog to log to elasticsearch #360

Open
haiwu opened this issue Apr 5, 2023 · 14 comments
Open

[Question] how to setup tlog to log to elasticsearch #360

haiwu opened this issue Apr 5, 2023 · 14 comments

Comments

@haiwu
Copy link

haiwu commented Apr 5, 2023

The current documentation is not clear to me.

Not going to use sssd, only to use tlog-rec-session.

By default /etc/tlog/tlog-rec-session.conf uses journal. If switching to use file, then the whole logging to elasticsearch from rsyslog would not work.

After switching /etc/tlog/tlog-rec-session.conf to use syslog, I could see /var/log/tlog.log sometimes would have tlog entries, most time it would log NO MATCH every second, if there's no tlog activity. Is this expected?

Also it seems rsyslog auto created an index in elasticsearch, but there's no docs in this index, so apparently it is not working at all. How to get this to work?

@gstorme
Copy link

gstorme commented Apr 11, 2023

You could use the file writer and use logstash to ship the file to elasticsearch.
The basic config below works well:

input {
  file {
    path => "/var/log/tlog/tlog.log"
    codec => "json"
  }
}

output {
  elasticsearch {
    hosts => [ "" ]
    index => ""
    user => ""
    password => ""
  }
}

@haiwu
Copy link
Author

haiwu commented Apr 11, 2023

Thanks @gstorme!

Does the above require creating index mapping and creating new index on Elasticsearch side first?
Could the same be done via Filebeat -> Elasticsearch?

My understanding is that Logstash might need more compute resources on client side than Filebeat..

@gstorme
Copy link

gstorme commented Apr 11, 2023

If you grant the required permissions to the configured user within elasticsearch, logstash will create the index.
And filebeat -> ES should also work, it's indeed more lightweight compared to logstash.

@haiwu
Copy link
Author

haiwu commented Apr 11, 2023

@gstorme: Since I have not got this to work yet, I am still wondering about index mappings that Logstash/Filebeat/Rsyslog could auto create by default. In order for tlog-play -r es CLI to work with tlog-rsyslog index, what kind of index mappings are to be expected? Will it create each field from each log line in /var/log/tlog/tlog.log file, such as ver, term, in_bin, out_txt?

@gstorme
Copy link

gstorme commented Apr 12, 2023

I don't know how rsyslog handles this.
With logstash you don't need to create a mapping yourself. It will import the JSON fields from the logfile automatically.
I assume filebeat can also do this, if your input config is correct.

@haiwu
Copy link
Author

haiwu commented Apr 12, 2023

@gstorme: I hit the same issue as this one with Filebeat: #257 (no solution listed in this issue), where Filebeat would ensure almost the same as this one:
"host": {
"name": "my"
},

I managed to use some Filebeat directives to update the above to be:
"host": "my"

But Elasticsearch keeps refusing to create index due to the above host not in its expected "host": { "name": format.

What does your "host" field look like?

@SowAbdoul
Copy link

Hello @gstorme @haiwu, Help me please!
How to configure Tlog to submit sessions to elasticsearch (SSL and Xpack are enabled). Can it send directly, or should I use Filebeat or Logstash? Is it possible to view sessions via the elasticsearch machine?

@macgahe
Copy link

macgahe commented May 6, 2024

Hello @gstorme @haiwu, Help me please! How to configure Tlog to submit sessions to elasticsearch (SSL and Xpack are enabled). Can it send directly, or should I use Filebeat or Logstash? Is it possible to view sessions via the elasticsearch machine?

usually I capture de logs from Tlog via any log agent, if you are using ES, I'd use Filebeat which encrypts the traffic till your ES Cluster. Regards your 2nd question you can retrieve the data from ES cluster using tlog and reproduce that session, but TLOG is the one which does the work, ES is just purely the storage to retrieve the data..

@SowAbdoul
Copy link

SowAbdoul commented May 6, 2024

@macgahe, Thank you so much!, I used tlog --> rsyslog --> elasticsearch
How to see all sessions from the ES machine (in one place)

@macgahe
Copy link

macgahe commented May 6, 2024

@macgahe, Thank you so much!, I used tlog --> rsyslog --> elasticsearch How to see all sessions from the ES machine (in one place)

even using rsyslog you can encrypt the traffic with certificates..

to play the session and retrieve it from ES.. I'd suggest you to look into tlog-play command

@SowAbdoul
Copy link

Thank you for your attention, @macgahe

@macgahe
Copy link

macgahe commented May 6, 2024

@macgahe, Thank you so much!, I used tlog --> rsyslog --> elasticsearch How to see all sessions from the ES machine (in one place)

if you allow me the suggestion I would change rsyslog to filebeat ( ElasticSearch Native Agent ) mostly because parsing the data will be much more easier and more productive.. unless you move the Rsyslog into another server and there you process the logs through filebeat or elastic agent depending on your ES version

@SowAbdoul
Copy link

SowAbdoul commented May 6, 2024

I considered configuring tlog-rec-session to record sessions in a file and then using Filebeat to send them to Elasticsearch. However, a concern is that the files might be deleted before the data is sent to Elasticsearch or there could be a longer wait time.
In comparison, with Rsyslog, I use UDP and sessions are logged as journal files.

@SowAbdoul
Copy link

Hello @macgahe, please. In addition to having numerous recordings per session, I have this error.

Message ID is out of order
Failed reading the source at message #1

tlog-rec-session.conf

{
    "shell": "/bin/bash",
    "notice": "",
    "latency": 10,
    "payload": 16384,
    "log": {
        "input": true,
        "output": true,
        "window": false
    },
    "limit": {
        "rate": 16384,
        "burst": 32768,
        "action": "pass"
    },
    "file": {
        "path": ""
    },
    "syslog": {
        "facility": "authpriv",
        "priority": "info"
    },
    "journal": {
        "priority": "info",
        "augment": true
    },
    "writer": "journal"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants