New feature: alert-elastic command

[YamatoSecurity]

New alert-elastic (sub)command:

Usage:
hayabusa.exe alert <INPUT> --sent-alerts <FILE> --host <HOST> --index <INDEX_NAME> --password <PASSWORD> [OPTIONS]

Help:
Input:
  -d, --directory <DIR>  Directory of multiple .evtx files
  -f, --file <FILE>      File path to one .evtx file
  -l, --live-analysis    Analyze the local C:\Windows\System32\winevt\Logs folder

Output:
  -s, --sent-alerts <FILE>       Save a log of sent alerts in order to not resend them (ex: sent-alerts.csv)
  -p, --profile <PROFILE>       Specify output profile

Display Settings:
      --no-color            Disable color output
  -q, --quiet               Quiet mode: do not display the launch banner
  -v, --verbose             Output verbose information

Elastic Settings:
      --strict               strict mode: do not only warn, but abort if an error occurs
  -i, --index <INDEX_NAME>   name of the elasticsearch index
  -H, --host <HOST>          server name or IP address of elasticsearch server
  -P, --port <PORT>          API port number of elasticsearch server [default: 9200]
      --proto <PROTOCOL>     protocol to be used to connect to elasticsearch [default: https] [possible values: http, https]
  -k, --insecure             omit certificate validation
  -U, --username <USERNAME>  username for elasticsearch server [default: elastic]
  -W, --password <PASSWORD>  password for authenticating at elasticsearch 

Filtering:
  -E, --EID-filter                Scan only common EIDs for faster speed (./rules/config/target_event_IDs.txt)
      --enable-deprecated-rules   Enable rules marked as deprecated
  -n, --enable-noisy-rules        Enable rules marked as noisy (./rules/config/noisy_rules.txt)
      --enable-unsupported-rules  Enable rules marked as unsupported
  -e, --exact-level <LEVEL>       Scan for only specific levels (informational, low, medium, high, critical)
      --exclude-status <STATUS>   Ignore rules according to status (ex: experimental) (ex: stable,test)
  -m, --min-level <LEVEL>         Minimum level for rules (default: informational)
      --timeline-end <DATE>       End time of the event logs to load (ex: "2022-02-22 23:59:59 +09:00")
      --timeline-start <DATE>     Start time of the event logs to load (ex: "2020-02-22 00:00:00 +09:00")

General Options:
  -Q, --quiet-errors                     Quiet errors mode: do not save error logs
  -r, --rules <DIR/FILE>                 Specify a custom rule directory or file (default: ./rules)
  -c, --rules-config <DIR>               Specify custom rule config directory (default: ./rules/config)
  -t, --threads <NUMBER>                 Number of threads (default: optimal number for performance)

Time Format:
      --European-time     Output timestamp in European time format (ex: 22-02-2022 22:00:00.123 +02:00)
      --ISO-8601          Output timestamp in ISO-8601 format (ex: 2022-02-22T10:10:10.1234567Z) (Always UTC)
      --RFC-2822          Output timestamp in RFC 2822 format (ex: Fri, 22 Feb 2022 22:00:00 -0600)
      --RFC-3339          Output timestamp in RFC 3339 format (ex: 2022-02-22 22:00:00.123456-06:00)
      --US-military-time  Output timestamp in US military time format (ex: 02-22-2022 22:00:00.123 -06:00)
      --US-time           Output timestamp in US time format (ex: 02-22-2022 10:00:00.123 PM -06:00)
  -U, --UTC               Output time in UTC format (default: local time)

When using alert-xxxx, hayabusa will not display alerts to the screen like with csv-timeline. Instead, it will display messages about sending the logs. Example:

Sending 300 new alerts to https://www.elastic.co/hogehoge:4040
Error: could not connect

0 alerts were already sent.
Sending 300 new alerts to https://www.elastic.co/hogehoge:4040
Alerts sent successfully

200 alerts were already sent.
Sending 100 new alerts to https://www.elastic.co/hogehoge:4040
Alerts sent successfully

etc...

If the alerts are sent successfully then the following information is written to the sent-alerts.csv file. (If the file already exists, then the file is updated)
Timestamp (Original ISO-8601 timestamp), Rule ID, Rule Title
Before Hayabusa sends the alerts, it checks the Timestamp and Rule ID in this file. If the timestamp and rule ID match, then we assume that the alert was previously sent and the alert does not get sent again.
The Rule Title is not needed for checking as the title may be changed over time, however, I want to save it to help out when debugging.

The following rust crates look good to use:

• https://github.com/elastic/elasticsearch-rs The official Elasticsearch rust client
• es4forensics https://github.com/janstarke/es4forensics This uses the official elasticsearch crate and implements mostly what we need so may be useful or as a reference. It is updated at the moment but may not be maintained over time so best to just use the official elasticsearch client only if possible.

@hitenkoku Do you think you can do this issue?

[hitenkoku]

@YamatoSecurity Thank you for raising the issue.
I basically don't see the problem, but I am concerned that the size of the csv file is getting bigger and bigger each time I do it.

For example, how about writing only the latest timestamp in the File and deeming anything before that time to have been sent?

[YamatoSecurity]

@hitenkoku Thank you for your comment! Indeed, the CSV file will get bigger and bigger over time but should not double the log size.
What about keeping the file compressed? That will reduce the file by 10x or more. Maybe we can use something like rust-brotli or brotlic ?

The problem with just writing the latest timestamp and scanning recent logs is that Hayabusa won't be able to find past incidents (perform threat hunting) which is the main goal. It would just become an inferior host IDS which I couldn't really recommend to use.

[YamatoSecurity]

How about we also not include the Rule Title in the CSV file which will save space. I thought it might be useful for debugging but now that I think about it we probably do not need it. If it is just timestamps and rule IDs, the file should not get too big. And if we compress it, it should become pretty small.

[hitenkoku]

Thanks for the comment.

I think Timestamp and Rule ID are sufficient for the contents of the csv.

I will try to create one.

[hitenkoku]

@YamatoSecurity Would it be correct to sort the following config in alphabetical order of the long option?

Elastic Settings:
      --strict               strict mode: do not only warn, but abort if an error occurs
  -i, --index <INDEX_NAME>   name of the elasticsearch index
  -H, --host <HOST>          server name or IP address of elasticsearch server
  -P, --port <PORT>          API port number of elasticsearch server [default: 9200]
      --proto <PROTOCOL>     protocol to be used to connect to elasticsearch [default: https] [possible values: http, https]
  -k, --insecure             omit certificate validation
  -U, --username <USERNAME>  username for elasticsearch server [default: elastic]
  -W, --password <PASSWORD>  password for authenticating at elasticsearch 

[YamatoSecurity]

@hitenkoku
Yes, let's organize this in alphabetical order of the long options:

Elastic Settings:
  -H, --host <HOST>          server name or IP address of elasticsearch server
  -i, --index <INDEX_NAME>   name of the elasticsearch index
  -k, --insecure             omit certificate validation
  -W, --password <PASSWORD>  password for authenticating at elasticsearch 
  -P, --port <PORT>          API port number of elasticsearch server [default: 9200]
      --proto <PROTOCOL>     protocol to be used to connect to elasticsearch [default: https] [possible values: http, https]
      --strict               strict mode: do not only warn, but abort if an error occurs
  -U, --username <USERNAME>  username for elasticsearch server [default: elastic]

[YamatoSecurity]

This is going to take a while to test so I changed the milestone to 2.4.0. Maybe release next month in April?