-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Big Ideas
Christian Kreibich edited this page Mar 1, 2024
·
10 revisions
This document is a list of various ideas collected over many years by the Zeek team for features, changes, research, and other tasks. Sometimes, ideas from this list end up as new issues and/or pull requests on Github. We do our best to link to those where possible. Completed tasks are listed separately, below.
NOTE: This shouldn’t be considered an implementation roadmap, but a collection of such ideas to track over time.
See the discussion at https://github.com/zeek/zeek/discussions/2850 for more.
- Infrastructure & Resources
- Retire the machine behind old.zeek.org
- New Package website (Tim)
- Website revamp — minimize to resolve:
- Lots of outdated information
- 170+ broken links
- Blogging in limbo between Discourse and Wordpress
- Documentation
- Complete push to cover packet capture more comprehensively
- Write in-depth plugin concepts guide
- Explain the full set of environment variables and what they do --
zeek --help
is partial and arguably not the right place
- Code Base
- Subsystems
- Logger-less clusters, in settings where export already naturally talks many-to-one and loggers are unnatural bottlenecks
- Reconfigurable flow tuples (https://github.com/zeek/zeek/issues/284)
- Clusterized PCAP ingestion (Jan)
- Plugin SDK
- Management Framework (Christian)
- Replace regex engine (on hold due to lack of options)
- Customizable session semantics
- Broker
- Multi-hop routing (Dominik)
- New serialization format (Dominik)
- Persistent in-network state
- Hook up Broker’s internal logging to proper Zeek log
-
Spicy & HILTI (Robin/Benjamin)
- New analyzers (ongoing)
- Replace existing analyzers (Benjamin/Johanna)
- Speed up compilation (Benjamin)
- Containerized packages
- EuroZeek Presentation
- Libraries of potential interest
- Cap’n Proto
- Libdart
- LAVA’s tbl (low-level data structure)
- Shm_container
- concurrent_queue
- Zeek Agent v2 (Robin)
- Provide native HTTP request functionality (replace current ActiveHTTP — but see JavaScript ideas below)
- Change time and interval to store nanosecond resolution
- Code quality and maintenance
- Migrate zeek-archiver into zeek-aux and consider dropping stale zeek-aux content
- Security audit
- Code Modernization
- Add structured binding support to List and Dictionary
- Use std::string_view to standardize how we handle strings (#708)
- Fix Reporter so that it can take std::string directly as arguments instead of needing to call c_str() first (using libfmt) (Tim)
- CMake overhaul (Dominik)
- Analyzers
- Multipath TCP (recognize at least)
- SCTCP
- Better VLAN visibility
- Scripting language
- async statement, or more generally: coroutines?
- Event dependencies
- String extraction
- Review & overhaul handling of text vs binary data (includes UTF-8 work from #596)
- String interpolation
- Overhaul name spacing
- LSP server (Benjamin)
- New types
- nullable/optional
- result/error
- Provide functionality to operate on JSON data
- JSON reader
- Script formatter (Christian)
- Script linter (need a place to collect ideas for this)
- Disable event handlers (or groups of event handler) on a per connection basis
- If there's no handler anymore, stop processing connection
- Scripts, Frameworks, Packages
- Package curation for commonly desired infrastructure
- List recommended packages prominently, e.g. for data export options
- Also curate most-wanted packages/plugins
- General direction
- Focus core scripts on infrastructure
- Push analysis & detection into packages
- Move policy scripts into packages
- Logging
- Zeek log schemas as a release-level deliverable:
- standardize a JSON-level rendering of what Zeek logs look like
- Produce such as log schema with each release, to allow tracking changes in one place and via tooling
- Possibly expand tooling to capture how packages adapt this log schema by adding columns, logs, etc.
- Delayed logging, with a generalized notion of additional log annotations
- Universal log format alterations (based on a predicate, say — anything that contains a conn_id changes in a particular way).
- Could relate to delayed logging, as a generalized “log alteration framework”
- Zeek log schemas as a release-level deliverable:
- New frameworks
- High-level cluster state sharing framework
- Persistence / data sharing framework
- SQLite-backed datastores are the only out-of-the-box persistence option in Zeek right now, and are not designed to address any and all use cases
- Could initially explore alternatives via dedicated packages and low-level BiFs, to shape into a framework at a later stage
- Contenders: generic SQL backend, Redis, memcached
- Authentication framework
- Track host/IP/user mapping
- LDAP/AD integration
- Alerting framework (Replace notice framework with something more generic than interfaces to Slack etc.)
- Asset database framework (aggregate information from different sources: network, zeek-agent, Suricata)
- Port Sumstats to new Broker ALM functionality [ALM is on hold]
- Higher-level logs: create narratives that can be understood without detailed protocol knowledge
- Provide extended reflection capabilities
- TODO: need specific use cases
- Zeek profiles: create packages that configure Zeek for.a specific use-case, e.g.
- Minimum log volume
- Comprehensive logging, record whatever you can
- Configuration wizard
- Select basic profile, tune common options
- Package curation for commonly desired infrastructure
- Performance
- Script compiler (Vern)
- ZAM
- C++
- CPU & memory profiling (Tim)
- Scripts
- Interpreter
- Event engine
- Redundancy of file analysis
- Timer buckets (Tim)
- Script compiler (Vern)
- Build system & deployment
- New install/RPM structure (after supervisor)
- "Easy install": One-click package installation
- Rewrite Cirrus config to use Starlark build system
- Subsystems
Kept largely for reference since we often need to recall the decision-making around past work, so this is mostly useful for items that link to past discussions, tickets, etc.
- Infrastructure & Resources
- Move docs to RTD (Jon)
- Move code to GitHub (Jon)
- Move JIRA tickets to GitHub (Jon)
- Website redesign (Amber)
- Testlab setup for performance benchmarking (Tim)
- Move mailing lists to Discourse (Johanna)
- Documentation
- New User Manual (Richard)
- Code Base
- Install headers (Jon)
- Bro-to-Zeek rename
- README redesign and modernization (Zeke)
- Subsystems
- DPDify transport-layer (Jan, Tim)
- Support for non-IP protocols (Tim)
- New I/O loop (Tim)
- Broker
- Port cluster framework & BroControl (Jon)
- New/real network protocol (Dominik)
- Windows support (Dominik)
- Synchronize data stores with script tables (Johanna)
- Add Websocket API for clients to easily subscribe/publish (Dominik)
-
Spicy & HILTI (Robin/Benjamin)
- 1.0 release
- Error recovery et al (Benjamin)
- Move plugin over into Zeek proper (Robin)
- Code quality and maintenance
- Define & implement coding conventions (Tim)
- Remove 2.x legacy functionality (Johanna)
- Unit testing using doctest
- Code Modernization
- Convert internal containers types to C++ templates
- Add proper iterator support
- Remove loop_over_list/loop_over_queue methods
- Extend iterator support to support more std algorithms like std::sort
- Add support for standard for loops to Dict to allow looping over std::pair for the keys and values
- Convert the manual ref counting to use IntrusivePtr<> (Jon)
- Convert internal containers types to C++ templates
- Analyzers
- QUIC (Jon)
- MQTT (Seth)
- Scripting language
- Paraglob (Zeke)
- Scripts, Frameworks, Packages
- Logging
- Configuration framework (Johanna)
- Port existing options
- Integrate with BroControl/Broker
- Telemetry on Zeek's operation (Arne/Dominik)
- Community ID (Christian)
- JavaScript
- Include ZeekJS as an external builtin plugin that is activated when libnode-dev is installed. Starting with Ubuntu 22.10 and Debian Bookworm, this package is an apt-get install away. Fedora has one since much longer
- Basic motivation: Open access to a vast and active ecosystem of third-party libraries and tools. Possibly attract JavaScript developers. "Lets not build a replacement for ActiveHTTP".