-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathCHANGES
25629 lines (16128 loc) · 897 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
4.0.0 | 2021-03-01 12:14:19 -0800
* Release 4.0.0.
4.0.0-rc3 | 2021-02-17 09:49:12 -0800
* Release 4.0.0-rc3.
* Fix `major_subsys_version` field in `pe_optional_header` event (Jon Siwek, Corelight)
It was incorrectly set the same as the `minor_subsys_version` field
of the `PE::OptionalHeader` record.
4.0.0-rc2.7 | 2021-02-12 14:42:17 -0800
* Fix CentOS 8 CI Dockerfile (Jon Siwek, Corelight)
The "PowerTools" repoid changed to "powertools":
https://bugs.centos.org/view.php?id=17920
4.0.0-rc2.6 | 2021-02-12 12:39:45 -0800
* Bump macOS jobs in CI from 4 to 12 cores (Christian Kreibich, Corelight)
* Update Cirrus CI naming update for MacOS images (Christian Kreibich, Corelight)
The docs on https://cirrus-ci.org/guide/macOS/ changed from
"osx_instance" to "macos_instance", so let's reflect that.
4.0.0-rc2.4 | 2021-02-12 11:53:15 -0800
* GH-1398: Fix buffer overread in ascii formatter (Johanna Amann, Corelight)
When a text with an (escaped) zero byte was passed to ParseValue, only
the part of the string up to the zero byte was copied, but the length of
the full string was passed to the input framework.
This leads to the input manager reading over the end of the buffer.
4.0.0-rc2 | 2021-01-25 12:50:34 -0800
* Release 4.0.0-rc2.
4.0.0-rc1.15 | 2021-01-25 12:49:48 -0800
* Update submodules: zkg 2.7.0, broker 2.0.0-rc2, CAF 0.18.0 (Jon Siwek, Corelight)
4.0.0-rc1.14 | 2021-01-21 17:10:26 -0800
* GH-1352: Added flag to stop processing SMTP headers in attached messages (Jon Oakley)
Previously, header values from sub-entities could overwrite the top-level
MIME entity values, causing the SMTP log to contain misleading information.
* Update COPYING to 2021 (Johanna Amann, Corelight)
4.0.0-rc1.11 | 2021-01-21 15:51:17 -0800
* Use xcode 11 on the Catalina CI image (Tim Wojtulewicz, Corelight)
4.0.0-rc1.10 | 2021-01-21 13:40:45 -0800
* Install BTest with Zeek (Christian Kreibich, Corelight)
Configuring with --disable-btest skips the installation. The main
reason for doing this is that many Zeek packages come with testsuites
that nearly always require btest, so providing btest out of the box
makes sense. Note that zkg's recent additions to its PATH environment
variable management mean that this btest instance is automatically
found also when the Zeek installation's binary folder isn't in the
path.
4.0.0-rc1.6 | 2020-12-23 10:40:23 -0800
* Fix subtle race on data store initialization (Dominik Charousset, Corelight)
4.0.0-rc1.5 | 2020-12-22 11:06:58 -0800
* Add new TLS 1.3 constants. (Johanna Amann)
* Fix TLS 1.3 session resumption detection. (Johanna Amann)
Now we detect TLS 1.3 session resumption by looking if both sides have
the PSK extension set, which is much more exact than the previous
approach.
* Introduce ssl_probable_encrypted_handshake_message event (Johanna Amann)
This event is raised for messages that (probably) are TLS 1.3 handshake
messages, including finished. This allows scripts to examine handshake
messages without having to handle all encrypted messages.
* SSL Analyzer: ignore CCS for TLS 1.3 (Johanna Amann)
In TLS 1.3, the ChangeCipherSpec message is meaningless; it only is
included to convince middleboxes that the devices actually are speaking
TLS 1.2. Nowadays some TLS 1.3 implementations also just don't send the
packet.
In a push to unify our handling of TLS 1.3 connections - ignore CCS and
always go with application data packet counting to determine if
connections are or are not encrypted.
* GH-1323: TLS analyzer: change logic to track TLS 1.3 connection establishment (Johanna Amann)
This commit changes the logic that is used to tracks connection
establishment - and moves it from scriptland into the core.
TLS 1.3 connection establishment is much more finnicky for us than the
establishment of earlier versions - since we cannot rely on the CCS
message anymore (which is meaningless and not sent in a lot of cases).
With this commit, the ssl_encrypted_data message gets raised for
encrypted TLS 1.3 handshake messages - which is much more correct than
the behavior before that just interpreted them as plaintext messages.
4.0.0-rc1.4 | 2020-12-21 11:02:55 -0800
* Add MacOS Big Sur to CI (Christian Kreibich, Corelight)
4.0.0-rc1.2 | 2020-12-16 15:27:21 -0800
* logging/ascii: Atomically create .shadow files using rename() (Arne Welzel, Corelight)
A logger process being terminated/killed while in the process of creating
a new .shadow file may leave an empty (invalid) one around. This in turn
causes the logger to error and exit during startup.
4.0.0-rc1 | 2020-12-14 22:22:29 -0800
* Update local.zeek compatibility test (Jon Siwek, Corelight)
* Add test case to cover weird EDNS ECS parsing situations (Jon Siwek, Corelight)
* Fix EDNS ECS option parsing bugs (Jon Siwek, Corelight)
* The parsing of IPv6 addresses tried to fill a stack-buffer with as
much data as supplied in the Option even if it was in excess of the
desired prefix or maximum IPv6 address size. This could result in an
overflow of that stack-buffer.
* The parsing of IPv4 addresses would overwrite the storage used for
that address as many times as there were bytes in the Option in excess
of the desired prefix length or maximum IPv4 address size. This could
cause the resulting IPv4 address to be derived from the incorrect
data.
* Upon encountering unexpected/excessive option-length or source-prefix
parameters, the data pointer used for parsing was also not always
advanced to the start of the next alleged option's data. Assuming all
other parsing code correctly guards against invalid input, there's no
further harm from that other than the subsequent parsing being more
likely to encounter unexpected values and emitting more Weirds.
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28336
(Link to details becomes public 30 days after patch release)
3.3.0-dev.664 | 2020-12-14 21:23:01 -0800
* Update submodules (Jon Siwek, Corelight)
3.3.0-dev.663 | 2020-12-14 14:27:45 -0800
* Rename a 'do_net_run' variable to 'do_run_loop'
For clarity, since the net_run() function was renamed to run_loop(). (Jon Siwek, Corelight)
* GH-1329: call Zeek's cleanup function from standalone fuzzer driver (Jon Siwek, Corelight)
Otherwise, the global Broker manager object containing CAF/threading
logic is never destructed and can result in a heap-use-after-free if it
tries to access other global objects after they're cleaned up from
__cxa_finalize().
3.3.0-dev.660 | 2020-12-14 10:55:15 -0800
* Fix typo in table iterator invalidation test comment (Tim Wojtulewicz, Corelight)
* GH-1328: Improve behavior of Dictionary iterator invalidation (Jon Siwek, Corelight)
Previously, an assertion was triggered in debug builds upon any attempt
to insert or remove a Dictionary entry while any iteration of that
Dictionary is underway and also even in cases where Dictionary membership
was not actually modified (and thus invalidates a loop).
Now, it emits run-time warnings regardless of build-type and only when
insert/remove operations truly change the Dictionary membership. In the
context of a Zeek script causing an invalidation, the warning message
also now helps pinpoint the exact expression that causes it.
* Add a note about aligning .cirrus.yml to platform support matrix wiki page (Johanna Amann, Corelight)
3.3.0-dev.650 | 2020-12-12 20:24:19 -0800
* Install zkg as part of the Zeek distribution. (Christian Kreibich, Corelight)
- Add auxil/package-manager submodule as an optional build
source. When the submodule is present, zkg gets installed into the
Zeek installation's bin directory, its config file into etc/zkg, and
its state into var/lib/zkg. Like zeekctl, zkg finds its own module
independently of any PYTHONPATH. Installation via pip remains
supported. You can skip zkg explicitly via --disable-zkg. See the
NEWS update for details.
- Establish a "zeek/python" subdirectory under libdir as the common place
for Python modules in the Zeek distribution. This now separates out
the Broker Python bindings, ZeekControl, and zkg's Python module.
- Add configure flags to allow customizing this Python folder, in
three ways: --python-dir, --python-prefix, and --python-home. These
differ in the logic they automatically add to the path, and build on
the logic already used in Broker.
- Include a (commented-out) @load for zkg's packages folder in local.zeek.
3.3.0-dev.648 | 2020-12-12 13:39:28 -0800
* Update Coverity Scan GitHub Action (Jon Siwek, Corelight)
* Remove old Python 2 vs. 3 workaround
* Build CAF separately to prevent internal errors in Coverity tools
3.3.0-dev.645 | 2020-12-10 14:14:12 -0800
* Fix message ordering of Broker messages (Dominik Charousset, Corelight)
Using two separate Broker subscribers for status events and regular
messages introduces a race on the two objects. Even if Broker sends all
messages in a particular (deterministic) order, Zeek may still process
them in a different order as a result. Since several tests rely on a
strict ordering of Broker events, these tests could fail sporadically.
Using only a single subscriber for all Broker messages makes sure that
Zeek observes all messages in the same order as Broker emits them.
* Fix UB in shutdown of Broker manager (Dominik Charousset, Corelight)
* Migrate to CAF 0.18 (Dominik Charousset, Corelight)
3.3.0-dev.638 | 2020-12-09 17:29:03 -0800
* Update Mozilla Root Store (Johanna Amann, Corelight)
3.3.0-dev.634 | 2020-12-09 10:51:49 -0800
* GH-1321: Prevent compounding of `connection_status_update` event timers (Jon Siwek, Corelight)
Particularly for ICMP connections, a new timer got added every time a
`connection` record was updated even if there was still a pending timer
for that connection.
* Update Debug.cc to use setsignal() wrapper (Jon Siwek, Corelight)
So as to prefer sigaction() over either sigset() or signal(), which are
less favorable due to underspecification issues and some treating them
as obsolete/deprecated.
3.3.0-dev.630 | 2020-12-07 16:52:52 -0800
* Fix narrowing conversion compiler error in SMB error-response logic (Jon Siwek, Corelight)
3.3.0-dev.629 | 2020-12-07 16:03:25 -0800
* Add tests for new SMB3 multichannel support (Vlad Grigorescu)
* Fix SMB2 response status parsing. Fixes #1286 (Vlad Grigorescu)
3.3.0-dev.624 | 2020-12-07 18:17:34 +0000
* Switch test baselines to canonified baselines
All baselines are now using the new btest feature to save baselines in the canonified
form (after the canonifiers are run). This should, hopefully, lead to less changes in
the future - that are easier to diff.
In addition, this cleans up a few small problems of the canonifiers which were over-eager
in identifying some numerical and path values.
We also use the new btest-diff --binary flag to identify binary files on which no
canonifier (not even the default canonifier) is applied.
(Christian Kreibich, Corelight with tiny additions by Johanna Amann, Corelight)
3.3.0-dev.611 | 2020-12-07 09:42:37 -0800
* Fix misassociation of Zeekygen-style comments within function scopes (Jon Siwek, Corelight)
All Zeekygen-style comments relate to entities at global scope, so those
found within functions are now ignored instead of misassociated.
3.3.0-dev.609 | 2020-12-04 19:20:39 -0800
* Remove an extraneous Zeekygen-style comment (Jon Siwek, Corelight)
3.3.0-dev.607 | 2020-12-04 11:16:09 -0800
* Fix the CMake 'dist' target of Zeek plugins to only run when outdated (Benjamin Bannier, Corelight)
3.3.0-dev.604 | 2020-12-04 18:40:03 +0000
* Sumstats: allow users to manage epoch manually
This change allows users to specify an epoch length of 0, which means
that the user manually has to finish the epochs. A new next_epoch
function is introduced to allow users to manually end epochs.
Addresses GH-348 (Johanna Amann, Corelight)
* Sumstats: epoch_finished was not called under certain circumstances
In non-clustered mode, epoch_finished was not called when there was no
data during the epoch.
This behavior does not fit the documentation, and also is different in
cluster-mode, where epoch_finished is, indeed, called after every epoch.
This small change fixes this behavior. (Johanna Amann, Corelight)
3.3.0-dev.600 | 2020-12-03 18:02:22 -0800
* Add a CI task for compiling with static broker/binpac (Johanna Amann, Corelight)
3.3.0-dev.596 | 2020-12-03 09:35:42 -0700
* Fix a couple of life-time issues when plugin loading fails.
Reported by Coverity.
Follow-up to #1179. (Robin Sommer, Corelight)
3.3.0-dev.593 | 2020-12-02 12:53:04 -0800
* Add `count_to_double` and `int_to_double` bif functions (Yacin Nadji, Corelight)
3.3.0-dev.590 | 2020-12-02 11:11:26 -0800
* Update minimum required CMake to 3.5 (Jon Siwek, Corelight)
Also now uses CMake's ENABLE_EXPORTS target property for the zeek
executable to ensure symbols are visible to plugins. Prior to CMake
3.4, the policy was to export symbols by default for certain platforms,
but later versions need either the explicit target property or policy.
3.3.0-dev.587 | 2020-12-01 10:17:42 -0700
* GH-1184: Add 'source' field to weird log denoting where the weird was reported (Tim Wojtulewicz, Corelight)
3.3.0-dev.585 | 2020-12-01 14:42:54 +0000
* Retry loading plugins on failure to resolve to dependencies.
Closes #1179. (Robin Sommer, Corelight)
3.3.0-dev.580 | 2020-11-30 14:07:39 -0700
* Find correct zeek namespace in debug logger macros.
These macros forward to functionality in `zeek::detail::debug_logger`
and are not intended for customization. This patch fixes the macros to
always use `::zeek::detail::debug_logger` as without the leading `::`
lookup could happen in any potentially local namespace `zeek` which does
not need to provide this symbol.
This closes zeek/spicy#597. (Benjamin Bannier, Corelight)
3.3.0-dev.576 | 2020-11-26 18:16:07 +0000
* Remove Python2 compatibility logic. We now require at least Python 3.5.
This includes script changes, improves the cmake logic to find python3,
makes scripts explicitly call python3 and documentation updates.
(Jon Siwek, Corelight)
* Remove Fedora 31 (EOL) from CI (Jon Siwek, Corelight)
3.3.0-dev.564 | 2020-11-24 15:23:50 -0800
* Improve support for custom libdir locations (Christian Kreibich, Corelight)
- Remove hardwiring of $ZEEK_ROOT/lib throughout the three and
defaults the name of Zeek's library directory to the default on the
given platform (e.g. lib64), via GNUInstallDirs.
- Consistently use that lib directory, instead of two lib folders
resulting when using a custom libdir.
- Remove the old lib directory in the installation prefix, if one exists
- Add --lib_dir to zeek-config (and sort its options a bit).
3.3.0-dev.561 | 2020-11-23 21:50:19 -0800
* Move implementation of internal_{type,var,etc} methods back into global namespace.
(Tim Wojtulewicz, Corelight)
This fixes an unknown symbol error if using those methods. They're defined
as extern in the global namespace in Var.h, but Var.cc had their
implementations defined in the zeek::detail namespace.
3.3.0-dev.559 | 2020-11-23 21:39:29 -0800
* Simplify Debian/Ubuntu CI dependencies and setup (Dominik Charousset, Corelight)
* Update .gitignore to ignore pyenv .python-version (Otto Fowler)
3.3.0-dev.554 | 2020-11-19 18:09:01 -0800
* Reverts the SMTP regex change in dead3226a545e264072ced40284f86ac41528ba8. (Tim Wojtulewicz, Corelight)
The regex change broke some of the external tests. I added some more cases
to the regular email btest to hopefully cover all of the cases better.
3.3.0-dev.551 | 2020-11-17 15:01:04 -0700
* Added unit tests for regex fix (christina23)
* Improved regex for SMTP parsing (christina23)
3.3.0-dev.548 | 2020-11-17 13:32:57 -0800
* GH-1033: BrokerStore <-> Zeek Tables: support complex indices (Johanna Amann, Corelight)
This change adds support for complex indexes for sets and tables. With
this change, sets with indexes like:
set[string, count, count]
will function. Before this change, Zeek raised an error message in these
cases.
3.3.0-dev.546 | 2020-11-17 11:50:47 +0000
* Fix memory leak in deprecated Analyzer::ConnectionEvent() (Jon Siwek, Corelight)
3.3.0-dev.543 | 2020-11-16 11:50:49 -0800
* GH-352: Improve HTTP::match_sql_injection_uri regex (Jon Siwek, Corelight)
Changes \x00-\x37 ranges to \x00-\x1f with assumption that the former
was attempting to match ASCII control characters, but mistook an octal
range for hex. This change reduces some false positives.
3.3.0-dev.541 | 2020-11-16 11:22:00 -0800
* Simplify ssh/main.zeek by using "ssh_server_host_key" for fingerprinting (Jon Siwek, Corelight)
* Deprecate "ssh1_server_host_key" parameters *e* and *p* (Jon Siwek, Corelight)
They are named such that *e* is actually the modulus, not the exponent.
The replacement parameters are named *exponent* and *modulus* for
clarity.
* GH-1264: Implement "ssh_server_host_key" event (Jon Siwek, Corelight)
This event provides host key fingerprints for both SSH1 and SSH2.
3.3.0-dev.537 | 2020-11-16 11:03:05 +0000
* [SSH] Handle SSH version 1.99
SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
That means the client/server is compatible with SSHv1 and SSHv2.
So the version choice depends of the both side.
1.99 : 1.99 => 2.0
1.99 : 1.x => 1.x
1.99 : 2.0 => 2.O
(see "Compatibility With Old SSH Versions" in RFC 4253) (Brevet Vivien)
3.3.0-dev.534 | 2020-11-12 14:31:10 -0800
* Move UnknownProtocol options to init-bare.zeek (Jon Siwek, Corelight)
Otherwise the `unknown_protocol` event cannot be used independently
from `policy/mic/unknown-protocols.zeek`.
* Coverity 1436183: Initialize packet_analysis::Manager fields (Jon Siwek, Corelight)
* GH-1273: Change SizeExpr to yield "any" type when operating on "any" (Jon Siwek, Corelight)
* Add enum_names() BIF to return names of an enum type's values (Jon Siwek, Corelight)
* Add type_aliases() BIF for introspecting type-names of types/values (Jon Siwek, Corelight)
* Change Type::type_aliases map to store IntrusivePtr (Jon Siwek, Corelight)
And deprecate Type::GetAliases() and Type::AddAlias() since they
took raw pointers. Now replaced with Type::Aliases() and
Type::RegisterAlias().
* Fix lookup_ID() BIF to return enum values (Jon Siwek, Corelight)
Looking up an enum value from a string equal to its name previously
returned "<no ID value>".
3.3.0-dev.524 | 2020-11-12 12:16:00 -0700
* Move 'using namespace' declaration after other includes (Tim Wojtulewicz, Corelight)
* Use binpac.h as if it was an external dependency (Tim Wojtulewicz, Corelight)
* Fix includes of bif.h and _pac.h files to use full paths inside build directory (Tim Wojtulewicz, Corelight)
* Remove unnecessary include of NetVar.h from packet analysis plugins (Tim Wojtulewicz, Corelight)
* GH-1079: Use full paths starting with zeek/ when including files (Tim Wojtulewicz, Corelight)
* Fix type clash fatal error with log filters that use $ext_func and $include/$exclude
The logging manager's Manager::TraverseRecord(), called when adding a
log filter to a stream, skipped any fields intoduced by a filter's
$ext_func when such fields weren't mentioned in a $include restriction
or mentioned in an $exclude restriction. This was inconsistent with
Manager::RecordToFilterVals, used when actually writing log entries,
which does include those values.
The result was that the record indices descent in Manager::RecordToFilterVals
expects to find only record values, when in fact only the record
provided by ext_func is present. This leads to type mismatches and
hard Zeek exits like this one:
1300475173.475401 fatal error in zeek/share/zeek//base/init-bare.zeek, line 4810: Val::CONVERTER (string/record) (zeek)
The fix makes ext_func's field additions decisive, meaning the
filter's include/exclude lists don't apply to it. If a user really
wants to override this, they can reset the filter's ext_func back to
our no-op default.
The included btest produces the above error when the fix is not present. (Christian Kreibich, Corelight)
3.3.0-dev.516 | 2020-11-11 12:53:02 -0700
* Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] (Fatema BW)
3.3.0-dev.514 | 2020-11-10 14:05:51 -0700
* GH-1269: Fix LogAscii::enable_leftover_log_rotation crash in bad dirs
Running with that option enabled inside a bad directory (e.g. lack of
permissions) crashed due to not checking for failure of opendir(). (Jon Siwek, Corelight)
3.3.0-dev.512 | 2020-11-10 13:55:33 -0700
* Change Debian 10 CI config to use GCC (Jon Siwek, Corelight)
* Update external testing repo hashes for canonification updates (Tim Wojtulewicz, Corelight)
3.3.0-dev.509 | 2020-11-10 08:41:54 -0700
* GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures (Tim Wojtulewicz, Corelight)
* Remove default_analyzer for Ethernet packet analzyer (Tim Wojtulewicz, Corelight)
3.3.0-dev.506 | 2020-11-09 16:27:04 -0800
* Add CI support for Debian 10, Fedora 31 & 33, FreeBSD 11, Ubuntu 20.04 (Christian Kreibich, Corelight)
This also adds two environment variables to the Dockerfiles for the
Debian/Ubuntu flavors to avoid stalling on user input when configuring
tzdata.
3.3.0-dev.504 | 2020-11-09 10:56:09 -0700
* Remove now-unused Packet::l2_valid field (Tim Wojtulewicz, Corelight)
* GH-1186: Remove Packet::hdr_size and uses of it.
This change also removes Packet::IP(), since Packet now contains an ip_hdr member
that points at the IP header if it exists. (Tim Wojtulewicz, Corelight)
3.3.0-dev.501 | 2020-11-04 10:41:56 -0800
* GH-1252: rename files with colons for Windows compatibility (Jon Siwek, Corelight)
A short-term measure so that the Zeek source tree can simply exist on a
Windows filesystem. For true support, the logic that decides/generates
the filename format will need to change.
* Use std::function instead of a function pointer in packet_analysis::Component (Tim Wojtulewicz, Corelight)
3.3.0-dev.495 | 2020-11-02 12:03:52 -0700
* GH-1215: Remove dispatch_map from packet analysis, replace with BIF methods for registering dispatches (Tim Wojtulewicz, Corelight)
3.3.0-dev.493 | 2020-11-02 09:51:03 -0800
* GH-1256: Write out strerror when writing errno during safe_write (Tim Wojtulewicz, Corelight)
3.3.0-dev.490 | 2020-10-27 13:41:27 -0700
* updated to mention notice_alarm.log by name. That has taken the place of the
file formerly named alarm.log (Duffy O'Craven)
3.3.0-dev.488 | 2020-10-26 11:03:51 -0700
* Improve how Zeekygen generated record/enum redefinition docs
It now provides a summary of the new fields/enums added by any given
redefinition along with associated commentary. (Jon Siwek, Corelight)
3.3.0-dev.486 | 2020-10-26 10:41:48 -0700
* GH-1245: require TLD of hostname literals to start with a letter (Jon Siwek, Corelight)
3.3.0-dev.484 | 2020-10-23 14:47:44 -0700
* Add coveralls badge to README.md (Tim Wojtulewicz)
* Remove the entire auxil directory from the results (Tim Wojtulewicz)
* Prevent non-master builds, add cirrus information to upload (Tim Wojtulewicz)
* Add support for sending data to coveralls with lcov_html.sh via Cirrus (Tim Wojtulewicz, Corelight)
3.3.0-dev.479 | 2020-10-23 12:25:15 -0700
* GH-251 (revert): remove coercion-to-signed-integer for |x| expressions (Jon Siwek, Corelight)
For `|x|`, where `x` is an expression with an integral result, an
implicit coercion of that result into signed `int` type no longer takes
place.
This was actually the behavior before Zeek 3.0 as well, but the attempt
to prevent mistakes that easily result from integer literals in Zeek
being unsigned like `|5 - 9|` causing an overflow/wraparound and
yielding a very large number is not generally consistent since overflows
are still generally able to happen in other ways and also in other
contexts besides just absolute-values. So the preference was to revert
to a behavior that favors consistency. For reference, see
https://github.com/zeek/zeek/pull/251#issuecomment-713956976
3.3.0-dev.476 | 2020-10-22 15:59:56 -0400
* Add an option to ignore packets sourced from particular subnets.
It's implemented with a new set[subnet] option named ignore_checksums_nets.
If you populate this set with subnets, any packet with a src address within
that set of subnets will not have it's checksum validated. (Seth Hall, Corelight)
* Update submodule(s) [nomail] (Jon Siwek, Corelight)
3.3.0-dev.467 | 2020-10-21 11:06:18 -0700
* Fix a couple of Coverity findings (1433618, 1433619) (Tim Wojtulewicz, Corelight)
3.3.0-dev.465 | 2020-10-20 13:34:20 +0000
* Bump BTest to 0.64.
3.3.0-dev.464 | 2020-10-20 08:43:40 +0000
* Update BTest to 0.63. New test baselines will now be store
canonified. (Robin Sommer, Corelight)
3.3.0-dev.463 | 2020-10-19 18:57:00 -0700
* Add length checks for IP header values before parsing TCP/UDP (Tim Wojtulewicz, Corelight)
3.3.0-dev.461 | 2020-10-19 11:54:23 -0700
* Fix deprecation warning in POP3 fuzzer (Jon Siwek, Corelight)
* Add header length check to GRE packet analyzer (Tim Wojtulewicz, Corelight)
3.3.0-dev.456 | 2020-10-16 12:41:58 -0700
* Avoid passing null pointers to memcmp in ZeekString comparison methods (Jon Siwek, Corelight)
3.3.0-dev.454 | 2020-10-16 10:34:53 -0700
* Change ICMP Neighbor Discovery option length storage to a uint16 (Vlad Grigorescu)
This fixes an overflow in the calculation of option lengths in
ICMP Neighbor Discovery messages.
3.3.0-dev.451 | 2020-10-16 07:09:43 +0000
* Make event ordering deterministic
NetControl::init and filter_change_tracking could basically be raised in
random order. (Johanna Amann, Corelight)
* Introduce generate_all_events bif and add option to misc/dump-events
generate_all_events causes all events to be raised internally; this
makes it possible for dump_events to really capture all events (and not
just those that were handled).
Addresses GH-169 (Johanna Amann, Corelight)
3.3.0-dev.444 | 2020-10-15 13:25:12 -0700
* Rework Sessions::Weird (Tim Wojtulewicz, Corelight)
* Prevent String::Set from crashing if passed a nullptr (Tim Wojtulewicz, Corelight)
* Store packet's ip header as unique_ptr (Tim Wojtulewicz, Corelight)
* Remove some unused includes from Packet.h (Tim Wojtulewicz, Corelight)
* Use shared_ptr for encapsulation data instead of raw pointer (Tim Wojtulewicz, Corelight)
* Review cleanup (Tim Wojtulewicz, Corelight)
* Update external testing repo hashes (Tim Wojtulewicz, Corelight)
* Reorder Packet member variables slightly for better packing (Tim Wojtulewicz, Corelight)
* Store the ip header in the packet after processing, reuse other places (Tim Wojtulewicz, Corelight)
* Change to store data in packet directly instead of keystore (Tim Wojtulewicz, Corelight)
* Don't always insert data into keystore for tunnels (Tim Wojtulewicz, Corelight)
* Add comment about packet header size and session analysis (Tim Wojtulewicz, Corelight)
* Set data to ip header's payload instead of advancing the pointer (Tim Wojtulewicz, Corelight)
* Move packet dumping to packet_mgr (Tim Wojtulewicz, Corelight)
* Make Sessions::NextPacket call packet_mgr, fix fuzzer code to do the same (Tim Wojtulewicz, Corelight)
* Move IP and IP tunnel code from Sessions into packet analyzers (Tim Wojtulewicz, Corelight)
* Prep work for IP changes
- Move all of the time handling code out of PktSrc into RunState
- Call packet_mgr->ProcessPacket() from various places to setup layer 2 data in packets (Tim Wojtulewicz, Corelight)
3.3.0-dev.426 | 2020-10-14 10:47:37 -0700
* GH-1211: Improve error message for already-defined functions (Jon Siwek, Corelight)
3.3.0-dev.422 | 2020-10-13 16:26:24 -0700
* GH-1208: Use Dictionary validity assertions only during CI (Jon Siwek, Corelight)
3.3.0-dev.420 | 2020-10-13 15:34:02 -0700
* Change "Cluster::Node$p" field to use "&default=0/unknown" (Seth Hall, Corelight)
This effectively makes defining the listening port optional in the cluster
node layout config, with "0/unknown" meaning the node is not pre-configured
to listen for incoming connections from other cluster nodes.
3.3.0-dev.414 | 2020-10-13 13:49:05 -0700
* Update cmake submodule to pull in fix for building libkqueue (Tim Wojtulewicz, Corelight)
* GH-1063: Update libkqueue to fix pf_ring-zc failures (Tim Wojtulewicz, Corelight)
3.3.0-dev.411 | 2020-10-13 13:38:38 -0700
* Remove unused LoginConn type and variable in Conn.h (Tim Wojtulewicz, Corelight)
3.3.0-dev.409 | 2020-10-13 13:32:18 -0700
* Make it possible to pass command line options through to scripts. (Seth Hall, Corelight)
A new ``zeek_script_args`` variable contains a list of arguments passed
to a script. E.g. either when explicitly executing Zeek like
``zeek -- myscript.zeek -arg1 -arg2``, or when using Zeek to interpret
executable scripts that contain a hashbang line at the top like::
#!/usr/local/zeek/bin/zeek --
3.3.0-dev.403 | 2020-10-13 10:50:12 -0700
* Add new Pcap::findalldevs() BIF (Seth Hall, Corelight)
* Remove superfluous RuleCondition destructors (Jon Siwek, Corelight)
* Silence Clang's warning about ignoring GCC's maybe-uninitialized warning (Jon Siwek, Corelight)
* Add reference to network_time_init from zeek_init docs (Jon Siwek, Corelight)
3.3.0-dev.390 | 2020-10-12 17:43:15 -0700
* Improve documentation for zeek_init event scheduling pitfalls (Jon Siwek, Corelight)
3.3.0-dev.388 | 2020-10-12 17:02:20 -0700
* Add CaptureLoss::Too_Little_Traffic notice (Vlad Grigorescu)
* Add CaptureLoss::initial_watch_interval for a quick read on cluster health after startup. (Vlad Grigorescu)
* Improve capture-loss.zeek documentation. (Vlad Grigorescu)
* Fix whitespace in capture-loss.zek (Vlad Grigorescu)
3.3.0-dev.381 | 2020-10-12 11:15:29 -0700
* GH-779: Add "udp-state" signature condition (Jon Siwek, Corelight)
It accepts "originator" or "responder" states as a way to enforce that
the signature only matches packets in the associated direction.
The "established" state is rejected as an error since it doesn't
have a useful meaning like it does for the "tcp-state" condition.
* Rename RuleConditionTCPState::TCPState enum values (Jon Siwek, Corelight)
* Rename "tcp-state" signature parser tokens to not be TCP-specific (Jon Siwek, Corelight)
3.3.0-dev.377 | 2020-10-08 12:07:42 -0700
* GH-1200: ignore a maybe-uninitialized warning (Jon Siwek, Corelight)
3.3.0-dev.374 | 2020-10-07 10:44:11 -0700
* Fix multipart MIME leak of sub-part found after closing-boundary (Jon Siwek, Corelight)
After detecting a closing-boundary for a given multipart MIME entity, it
enters into an "end of data" state, however any subsequent boundary
delimiter could still cause the allocation of a sub-entity object that
is never released due to cleanup logic being bypassed upon finding the
"end of data" state already reached.
This change prevents allocation/processing of sub-entities after the
"end of data" state is reached (e.g. from detecting a multipart
closing-boundary). This new behavior still aligns with RFC 2046
expectations:
"There appears to be room for additional information prior to the first
boundary delimiter line and following the final boundary delimiter line.
These areas should generally be left blank, and implementations must
ignore anything that appears before the first boundary delimiter line or
after the last one."
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26027
(Link to details becomes public 30 days after patch release)
* Btests for log filter policy hooks (Christian Kreibich, Corelight)
* Btest baseline updates to reflect new logging policy hooks (Christian Kreibich, Corelight)
* Migrate existing use of filter predicates to policy hooks (Christian Kreibich, Corelight)
* Support for log filter policy hooks (Christian Kreibich, Corelight)
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:
hook(rec: any, id: Log::ID, filter: Log::Filter);
The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.
The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
{
if ( some_veto_reason(rec) )
break;
}
By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.
The existing predicates are deprecated for removal in 4.1 but continue
to work.
3.3.0-dev.365 | 2020-10-06 12:19:49 -0700
* GH-425: Avoid temporary value while coercing records already of the right type. (Robin Sommer, Corelight)
The combination of this commit with the previous one now lets the examples
in GH-425 all execute with the same performance.
* GH-425: Optimize record constructor expression. (Robin Sommer, Corelight)
We remove the inheritance from UnaryExpr because we know the type of the
operand precisely and can skip a temporary when evaluating the expression.
* Unify type comparisions for records. (Robin Sommer, Corelight)
For records, same_type(r1, r2) would not check if the fields'
attributes match as well. That seems like an oversight, and some
callers of same_type() did indeed add that check on their end. This
commit moves the check into same_type() itself. That generally doesn't
seem make any differences except for a couple of places validating
code, which we update a bit. That in turn leans to slightly different
(better?) error messages for a couple of test cases.
3.3.0-dev.361 | 2020-10-06 10:13:37 -0700
* logging/ascii: Support leftover log rotation in non-supervisor setups (Arne Welzel, Corelight)
Allow enabling leftover log rotation through
LogAscii::enable_leftover_log_rotation and redef this for the
logger node in a supervisor setup individually.
3.3.0-dev.359 | 2020-10-02 10:49:35 -0700
* GH-1180: Add external dependency include paths to `zeek-config --include_dir` (Jon Siwek, Corelight)
This helps ensure successful compilation of a plugin that includes a
Zeek header which includes an external dependency header even if the
plugin itself doesn't necessarily depend on that external library.
3.3.0-dev.357 | 2020-10-02 10:26:11 -0700
* Switch one's complement checksum implementation (Jon Siwek, Corelight)
Borrows the `in_cksum` code from tcpdump, which borrowed from FreeBSD.
It handles unaligned data better and also unrolls the inner loop to
process 16 two-byte values at a time versus 2 one-byte values at a time
in the previous version. Generally measured as ~1.5x faster in a
release build. The new API should generally be more amenable to any
future optimization explorations since all relevant data blocks are
available within a single call rather than spread across multiple.
3.3.0-dev.355 | 2020-09-29 17:08:44 -0700
* Improve broker/ssl_auth_failure.zeek test (Jon Siwek, Corelight)
There was a race that previously may cause it to not truly test
whether the connecting-process exited due to SSL authentication
failure or just because the listening-process wasn't available yet
(e.g. due to process scheduling variability).
3.3.0-dev.352 | 2020-09-25 15:09:29 -0700
* GH-160: change find-filtered-trace to ignore pcaps with any non-TCP traffic (Robin Sommer, Corelight)
3.3.0-dev.350 | 2020-09-25 14:37:58 -0700
* Add dce_rpc_request_stub and dce_rpc_response_stub events (Yacin Nadji, Corelight)
* Fix namespace warning in fuzzer code (Tim Wojtulewicz, Corelight)
3.3.0-dev.346 | 2020-09-24 16:03:28 -0700
* Fix a Sphinx warning about misformatted packet analyzer comment (Jon Siwek, Corelight)
* Add Zeekygen documentation support for packet analyzers (Jon Siwek, Corelight)
* Move packet_mgr to the zeek namespace (Tim Wojtulewicz, Corelight)
3.3.0-dev.341 | 2020-09-24 08:16:45 -0700
* Fix negative-value-left-shift undefined behavior in patricia trie (Jon Siwek, Corelight)
* Improve negation of ConstExpr (Jon Siwek, Corelight)
* Instead of creating a NegExpr for negation of a literal/constant,
a ConstExpr is now created directly.
* For negation of integer literals, there's now an additional check
for whether the integer would be outside the range of possible 'int'
values. This can also help prevent the undefined behavior due to
overflow as a result of trying to represent the minimum 'int' value of
-9223372036854775808 as a literal in a script -- the unsigned value is
cast to signed yielding INT64_MIN, then INT64_MIN is negated.
* Avoid signed integer overflow when combining SMB header PID bits (Jon Siwek, Corelight)
Such an overflow invokes undefined behavior.
* Avoid unary negation of INT64_MIN in modp_litoa10 (Jon Siwek, Corelight)
Overlow can occur in that case, which is undefined behavior.
* Avoid double-to-int conversion overflows in modp_dtoa functions (Jon Siwek, Corelight)
Those methods already had a fallback to use sprintf() for large values
except:
* The check-for-large-value was unnecessarily done after many
operations that aren't relevant to the check and those operations can
result in a conversion overflow (undefined behavior).
* The check-for-large-value was using the literal value for a
32-bit INT_MAX instead of just using INT_MAX. For a platform where
`int` is less than 32-bits, the same conversion overflow from the
previous point could still occur (undefined behavior).
* The check-for-large-value was not inclusive of INT_MAX.
In a case where the conversion of INT_MAX itself to a double
can't be represented exactly, it's implementation-defined whether
the closest higher or closest lower representable-value is selected.
If the higher value is selected, then a `double` value comparing equal
to INT_MAX-as-converted-to-double would cause an overflow of an `int`
upon conversion (undefined behavior).
* Fix divide-by-zero in Entropy analyzer (Jon Siwek, Corelight)
* Fix divide-by-zero in stats/profiling memory usage calculation (Jon Siwek, Corelight)
* Fix uninitialized field in POP3 fuzzer (Jon Siwek, Corelight)