fixed MitreTactics, MitreTags, OtherTags do not output in json timeline output #1062

hitenkoku · 2023-05-28T15:18:56Z

What Changed

fixed no output of MitreTactics, MitreTags, OtherTags in json-timeline
added test to json output Enhancement: raise code coverage percentage in other #864

I would appreciate it if you could review when you have time

#1061

codecov · 2023-05-28T15:20:05Z

Codecov Report

Patch coverage: 98.91% and project coverage change: +1.70 🎉

Comparison is base (949c7b6) 74.00% compared to head (2970c7a) 75.70%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1062      +/-   ##
==========================================
+ Coverage   74.00%   75.70%   +1.70%     
==========================================
  Files          24       24              
  Lines       18186    18636     +450     
==========================================
+ Hits        13459    14109     +650     
+ Misses       4727     4527     -200

Impacted Files	Coverage Δ
src/main.rs	`26.83% <0.00%> (ø)`
src/afterfact.rs	`63.14% <99.12%> (+17.73%)`	⬆️
src/detections/message.rs	`91.11% <100.00%> (ø)`

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

hitenkoku · 2023-05-28T15:57:06Z

Evidence

Case1 (hayabusa-sample-evtx)

main

refs: #1061 "Step to Reproduce"

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2023-01-24 20:54:42.899 +09:00

Events with hits / Total events: 19,663 / 47,472 (Data reduction: 27,809 events (58.58%))

Total | Unique detections: 32,109 | 622
Total | Unique critical detections: 55 (0.17%) | 21 (3.38%)
Total | Unique high detections: 6,047 (18.83%) | 277 (44.53%)
Total | Unique medium detections: 1,874 (5.84%) | 201 (32.32%)
Total | Unique low detections: 6,002 (18.69%) | 70 (11.25%)
Total | Unique informational detections: 18,131 (56.47%) | 53 (8.52%)
...
Saved file: main.csv (38.9 MB)
Elapsed time: 00:00:23.684
Rule Parse Processing Time: 00:00:02.930
Analysis Processing Time: 00:00:19.111
Output Processing Time: 00:00:01.639

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:   947.8 MiB     2.9 GiB     2.4 GiB   484.8 MiB
     reset:     0
    purged:     1.8 GiB
   touched:    64.2 KiB     3.4 MiB     3.6 GiB    -3.6 GiB                          ok
  segments:    16          55          47           8                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0          33.8 Ki    -33.8 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:    10.3 Ki
    resets:     0
    purges:   623
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:    23.697 s
   process: user: 50.562 s, system: 2.437 s, faults: 343596, rss: 688.2 MiB, commit: 795.2 MiB

json output

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2023-01-24 20:54:42.899 +09:00

Events with hits / Total events: 19,663 / 47,472 (Data reduction: 27,809 events (58.58%))

Total | Unique detections: 32,109 | 622
Total | Unique critical detections: 55 (0.17%) | 21 (3.38%)
Total | Unique high detections: 6,047 (18.83%) | 277 (44.53%)
Total | Unique medium detections: 1,874 (5.84%) | 201 (32.32%)
Total | Unique low detections: 6,002 (18.69%) | 70 (11.25%)
Total | Unique informational detections: 18,131 (56.47%) | 53 (8.52%)
...
Saved file: main.json (54.8 MB)
Elapsed time: 00:00:30.644
Rule Parse Processing Time: 00:00:02.975
Analysis Processing Time: 00:00:23.670
Output Processing Time: 00:00:03.969

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:     1.0 GiB     3.5 GiB     3.1 GiB   422.5 MiB
     reset:     0
    purged:     2.4 GiB
   touched:    64.2 KiB     3.6 MiB     3.7 GiB    -3.7 GiB                          ok
  segments:    16          59          51           8                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0          34.7 Ki    -34.7 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:    12.0 Ki
    resets:     0
    purges:     1.0 Ki
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:    30.818 s
   process: user: 53.859 s, system: 2.531 s, faults: 389136, rss: 692.1 MiB, commit: 799.1 MiB

this PR
- csv output

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2023-01-24 20:54:42.899 +09:00

Events with hits / Total events: 19,663 / 47,472 (Data reduction: 27,809 events (58.58%))

Total | Unique detections: 32,109 | 622
Total | Unique critical detections: 55 (0.17%) | 21 (3.38%)
Total | Unique high detections: 6,047 (18.83%) | 277 (44.53%)
Total | Unique medium detections: 1,874 (5.84%) | 201 (32.32%)
Total | Unique low detections: 6,002 (18.69%) | 70 (11.25%)
Total | Unique informational detections: 18,131 (56.47%) | 53 (8.52%)
...

Saved file: 1061.csv (38.9 MB)
Elapsed time: 00:00:23.469
Rule Parse Processing Time: 00:00:02.657
Analysis Processing Time: 00:00:18.976
Output Processing Time: 00:00:01.835

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:   955.3 MiB     3.1 GiB     2.6 GiB   457.0 MiB
     reset:     0
    purged:     2.0 GiB
   touched:    64.2 KiB     3.5 MiB     3.6 GiB    -3.6 GiB                          ok
  segments:    16          57          49           8                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0          33.8 Ki    -33.8 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:    10.4 Ki
    resets:     0
    purges:   659
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:    23.523 s
   process: user: 49.218 s, system: 2.203 s, faults: 346209, rss: 693.3 MiB, commit: 803.4 MiB

json output

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2023-01-24 20:54:42.899 +09:00

Events with hits / Total events: 19,663 / 47,472 (Data reduction: 27,809 events (58.58%))

Total | Unique detections: 32,109 | 622
Total | Unique critical detections: 55 (0.17%) | 21 (3.38%)
Total | Unique high detections: 6,047 (18.83%) | 277 (44.53%)
Total | Unique medium detections: 1,874 (5.84%) | 201 (32.32%)
Total | Unique low detections: 6,002 (18.69%) | 70 (11.25%)
Total | Unique informational detections: 18,131 (56.47%) | 53 (8.52%)

...
Saved file: 1061.json (56.1 MB)
Elapsed time: 00:00:25.166
Rule Parse Processing Time: 00:00:02.734
Analysis Processing Time: 00:00:19.292
Output Processing Time: 00:00:03.130

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:  1003.9 MiB     3.1 GiB     2.7 GiB   426.9 MiB
     reset:     0
    purged:     2.1 GiB
   touched:    64.2 KiB     3.5 MiB     3.7 GiB    -3.7 GiB                          ok
  segments:    16          57          49           8                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0          34.7 Ki    -34.7 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:    10.2 Ki
    resets:     0
    purges:   684
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:    25.220 s
   process: user: 51.515 s, system: 2.125 s, faults: 350702, rss: 694.3 MiB, commit: 803.4 MiB

checked #1061 reproduce data in 1061.json

{
    "Timestamp": "2013-10-24 01:17:29.468 +09:00",
    "Computer": "37L4247D28-05",
    "Channel": "Sys",
    "EventID": 7045,
    "Level": "info",
    "RuleTitle": "Svc Installed",
    "RuleAuthor": "Zach Mathis",
    "RuleModifiedDate": "2022/06/21",
    "Status": "stable",
    "RecordID": 202,
    "Details": {
        "Svc": "Hyper-V Heartbeat Service",
        "Path": "%SystemRoot%\\system32\\vmicsvc.exe -feature Heartbeat",
        "Acct": "NT AUTHORITY\\NetworkService",
        "StartType": "auto start"
    },
    "ExtraFieldInfo": {
        "ServiceType": "user mode service"
    },
    "MitreTactics": [
        "Persis"
    ],
    "Provider": "SCM",
    "RuleCreationDate": "2022/02/06",
    "RuleFile": "Sys_7045_Info_SvcInstalled.yml",
    "EvtxFile": "..\\hayabusa-sample-evtx\\DeepBlueCLI\\many-events-system.evtx"
}

hitenkoku · 2023-05-28T16:38:52Z

Test2(all-evtx.tgz(6.1GB))

main
- csv output

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2022-09-18 23:37:13.088 +09:00

Events with hits / Total events: 1,594,166 / 4,817,181 (Data reduction: 3,223,015 events (66.91%))

Total | Unique detections: 1,627,328 | 156
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 11,634 (0.71%) | 17 (10.90%)
Total | Unique medium detections: 11,016 (0.68%) | 43 (27.56%)
Total | Unique low detections: 1,054,591 (64.81%) | 46 (29.49%)
Total | Unique informational detections: 550,087 (33.80%) | 50 (32.05%)
...

Saved file: main2.csv (1.6 GB)
Elapsed time: 00:10:09.931
Rule Parse Processing Time: 00:00:03.008
Analysis Processing Time: 00:09:39.876
Output Processing Time: 00:00:27.045

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     8.0 GiB     8.0 GiB     0           8.0 GiB
 committed:    10.9 GiB    87.7 GiB    80.7 GiB     6.9 GiB
     reset:     0
    purged:    46.8 GiB
   touched:    64.2 KiB    43.6 MiB    54.4 GiB   -54.4 GiB                          ok
  segments:    16         698         689           9                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0         735.6 Ki   -735.6 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   710.2 Ki
    resets:     0
    purges:    24.6 Ki
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   609.957 s
   process: user: 3004.765 s, system: 39.750 s, faults: 14532525, rss: 6.9 GiB, commit: 7.1 GiB

json output

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2022-09-18 23:37:13.088 +09:00

Events with hits / Total events: 1,594,166 / 4,817,181 (Data reduction: 3,223,015 events (66.91%))

Total | Unique detections: 1,627,328 | 156
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 11,634 (0.71%) | 17 (10.90%)
Total | Unique medium detections: 11,016 (0.68%) | 43 (27.56%)
Total | Unique low detections: 1,054,591 (64.81%) | 46 (29.49%)
Total | Unique informational detections: 550,087 (33.80%) | 50 (32.05%)
..
Saved file: main2.json (2.4 GB)
Elapsed time: 00:05:01.963
Rule Parse Processing Time: 00:00:01.086
Analysis Processing Time: 00:04:20.104
Output Processing Time: 00:00:40.771

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     8.0 GiB     8.0 GiB     0           8.0 GiB
 committed:    11.4 GiB    53.6 GiB    46.8 GiB     6.8 GiB
     reset:     0
    purged:    23.1 GiB
   touched:    64.2 KiB    43.6 MiB    54.5 GiB   -54.5 GiB                          ok
  segments:    16         699         690           9                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0         747.8 Ki   -747.8 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   324.8 Ki
    resets:     0
    purges:    14.8 Ki
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   301.976 s
   process: user: 2418.421 s, system: 14.750 s, faults: 7927120, rss: 6.9 GiB, commit: 7.1 GiB

this PR
- csv output

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2022-09-18 23:37:13.088 +09:00

Events with hits / Total events: 1,594,166 / 4,817,181 (Data reduction: 3,223,015 events (66.91%))

Total | Unique detections: 1,627,328 | 156
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 11,634 (0.71%) | 17 (10.90%)
Total | Unique medium detections: 11,016 (0.68%) | 43 (27.56%)
Total | Unique low detections: 1,054,591 (64.81%) | 46 (29.49%)
Total | Unique informational detections: 550,087 (33.80%) | 50 (32.05%)
...
Saved file: 1061-2.csv (1.6 GB)
Elapsed time: 00:05:42.408
Rule Parse Processing Time: 00:00:01.708
Analysis Processing Time: 00:05:14.527
Output Processing Time: 00:00:26.171

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     8.0 GiB     8.0 GiB     0           8.0 GiB
 committed:    11.2 GiB    55.9 GiB    49.1 GiB     6.8 GiB
     reset:     0
    purged:    24.2 GiB
   touched:    64.2 KiB    43.7 MiB    54.5 GiB   -54.4 GiB                          ok
  segments:    16         700         691           9                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0         735.6 Ki   -735.6 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   342.9 Ki
    resets:     0
    purges:    14.6 Ki
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   342.425 s
   process: user: 2412.578 s, system: 13.343 s, faults: 8273225, rss: 6.9 GiB, commit: 7.1 GiB

json output

Results Summary:

First Timestamp: 2009-07-14 13:56:45.074 +09:00
Last Timestamp: 2022-09-18 23:37:13.088 +09:00

Events with hits / Total events: 1,594,166 / 4,817,181 (Data reduction: 3,223,015 events (66.91%))

Total | Unique detections: 1,627,328 | 156
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 11,634 (0.71%) | 17 (10.90%)
Total | Unique medium detections: 11,016 (0.68%) | 43 (27.56%)
Total | Unique low detections: 1,054,591 (64.81%) | 46 (29.49%)
Total | Unique informational detections: 550,087 (33.80%) | 50 (32.05%)
...
Saved file: 1061-2.json (2.4 GB)
Elapsed time: 00:06:01.766
Rule Parse Processing Time: 00:00:02.896
Analysis Processing Time: 00:05:16.287
Output Processing Time: 00:00:42.154

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     8.0 GiB     8.0 GiB     0           8.0 GiB
 committed:    11.2 GiB    56.2 GiB    49.4 GiB     6.8 GiB
     reset:     0
    purged:    24.7 GiB
   touched:    64.2 KiB    43.6 MiB    54.5 GiB   -54.5 GiB                          ok
  segments:    16         698         689           9                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0         748.6 Ki   -748.6 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:   347.3 Ki
    resets:     0
    purges:    15.5 Ki
   threads:    32          32           0          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   361.805 s
   process: user: 2433.750 s, system: 15.000 s, faults: 8462021, rss: 6.9 GiB, commit: 7.0 GiB

fukusuket

I confirmed that Tags are output in json-timeline :)

I think it's expected behavior, but the result of "EventID": 4104 has the following difference.

json-timeline -d hayabusa-sample-evtx -o out.json

This PR

{
    "Timestamp": "2021-04-22 19:04:37.081 +09:00",
    "Computer": "win10-02.offsec.lan",
    "Channel": "PwSh",
    "EventID": 4104,
    "Level": "info",
    "RecordID": 135,
    "RuleTitle": "PwSh Scriptblock",
    "Details": {
        "ScriptBlock": "Write-Host 'Final | 1';"
    }
}

main

{
    "Timestamp": "2021-04-22 19:04:37.081 +09:00",
    "Computer": "win10-02.offsec.lan",
    "Channel": "PwSh",
    "EventID": 4104,
    "Level": "info",
    "RecordID": 135,
    "RuleTitle": "PwSh Scriptblock",
    "Details": {
        "ScriptBlock": "Write-Host 'Final",
        "result": "1';"
    }
}

Is the above diff the expected behavior?(If it's OK, it's LGTM!🚀)

hitenkoku · 2023-05-29T07:54:28Z

@fukusuket Thank you for your super fast review.

Is the above diff the expected behavior?(If it's OK, it's LGTM!🚀)

I think that indication is correct for main branch result. I will fix it.

hitenkoku · 2023-05-29T15:28:23Z

@fukusuket
I apologize for the delay in responding to your request.
In f2b8a89, I have made the corrections you pointed out at #1062 (review)
Sorry for the inconvenience. Would you review it?

fukusuket

Thank you for the quick fix!
I have verified that the default profile result has no diffs with the v2.5.1 :) LGTM!!🚀

hitenkoku · 2023-05-29T16:21:41Z

@fukusuket Thank you for your kindness review!

YamatoSecurity

@hitenkoku Thank you so much!
%ExtraFieldInfo% is also being outputted now as well.
LGTM

hitenkoku added 7 commits May 29, 2023 00:01

feat: changed MESSAGEKEYS type to avoid poisoning when test

ba4db94

fix(afterfact): fixed no output of MitreTactics etc... in json-timeline

4499f36

#1061

style(main): fixed clippy warn

b243ed2

chore(afterfact): fixed typo

ee3a7f9

test(afterfact): added output_json test #1061 #864

25e5d93

test(afterfact): fixed typo

9373202

test(afterfact): added jsonl output test #1061 #864

1710be0

hitenkoku added the bug Something isn't working label May 28, 2023

hitenkoku requested review from fukusuket and YamatoSecurity May 28, 2023 15:18

hitenkoku self-assigned this May 28, 2023

hitenkoku linked an issue May 28, 2023 that may be closed by this pull request

[bug] MitreTactics,MitreTags,OtherTags do not output in json-timeline output #1061

Closed

hitenkoku marked this pull request as ready for review May 28, 2023 16:39

fukusuket reviewed May 29, 2023

View reviewed changes

fix(afterfact): fixed wrong details devide in #1061

f2b8a89

docs(CHANGELOG): updated changelog #1061

c7def49

hitenkoku requested a review from fukusuket May 29, 2023 15:35

fukusuket approved these changes May 29, 2023

View reviewed changes

update changelog

2970c7a

YamatoSecurity approved these changes May 30, 2023

View reviewed changes

hitenkoku merged commit 70c1c76 into main May 31, 2023

hitenkoku deleted the 1061-bug-mitretacticsmitretagsothertags-do-not-output-in-json-timeline-output branch May 31, 2023 07:52

fukusuket mentioned this pull request Sep 30, 2023

update readme #1185

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixed MitreTactics, MitreTags, OtherTags do not output in json timeline output #1062

fixed MitreTactics, MitreTags, OtherTags do not output in json timeline output #1062

hitenkoku commented May 28, 2023 •

edited

Loading

codecov bot commented May 28, 2023 •

edited

Loading

hitenkoku commented May 28, 2023

hitenkoku commented May 28, 2023

fukusuket left a comment

hitenkoku commented May 29, 2023 •

edited

Loading

hitenkoku commented May 29, 2023

fukusuket left a comment

hitenkoku commented May 29, 2023

YamatoSecurity left a comment

fixed MitreTactics, MitreTags, OtherTags do not output in json timeline output #1062

fixed MitreTactics, MitreTags, OtherTags do not output in json timeline output #1062

Conversation

hitenkoku commented May 28, 2023 • edited Loading

What Changed

codecov bot commented May 28, 2023 • edited Loading

Codecov Report

hitenkoku commented May 28, 2023

Evidence

Case1 (hayabusa-sample-evtx)

hitenkoku commented May 28, 2023

Test2(all-evtx.tgz(6.1GB))

fukusuket left a comment

Choose a reason for hiding this comment

hitenkoku commented May 29, 2023 • edited Loading

hitenkoku commented May 29, 2023

fukusuket left a comment

Choose a reason for hiding this comment

hitenkoku commented May 29, 2023

YamatoSecurity left a comment

Choose a reason for hiding this comment

hitenkoku commented May 28, 2023 •

edited

Loading

codecov bot commented May 28, 2023 •

edited

Loading

hitenkoku commented May 29, 2023 •

edited

Loading