Malcolm v25.06.0

[mmguero]

Malcolm v25.06.0 includes a some new and oft-requested features, bug fixes, and component version bumps.

v25.05.0...v25.06.0

NOTE: As this Malcolm release enables the OpenSearch Security Plugin as described below, even inter-container access to OpenSearch must now be authenticated when using Malcolm's embedded OpenSearch instance. To accomplish this, an internal-use-only account and password is used for connecting to OpenSearch by Malcolm's other components as needed. This credential (saved in .opensearch.primary.curlrc in the Malcolm installation directory) needs to be generated before Malcolm starts up the first time after upgrading. To do so, please run ./scripts/auth_setup and select (Re)generate internal passwords for local primary OpenSearch instance. This credential is only used internally for OpenSearch and cannot be used to remotely access Malcolm.

• ✨ Features and enhancements
  • This release adds role-based access control (RBAC) to Malcolm (Role-Based Access Control (RBAC) #460).
    • Malcolm's RBAC feature is based on Keycloak realm roles and is implemented in to layers:
      1. Whenever possible, Malcolm's backend Keycloak realm roles are mapped to the roles/groups/permissions features provided by the components that make up Malcolm, for example:
         • OpenSearch (provided by the OpenSearch Security plugin)
           • Configuring the Security backend
           • Modifying the YAML files
           • Predefined roles
         • Arkime
           • Note: As per @awick, until Arkime v6.0.0 is out, not all of the Arkime permissions can be set on roles. For now, then, Malcolm's Arkime roles are going to be handled purely based on URI path in the NGINX stuff as described below.
         • NetBox
           • Authentication & Permissions
           • Object-Based Permissions
           • Mastering NetBox User Access with Permission Constraints
      2. For other Malcolm components that don't implement their own permission management systems, Malcolm handles the enforcement roles based on request URIs in its NGINX proxy layer.
    • This is an optional feature. RBAC is only available when the authentication method is keycloak or keycloak_remote. With other authentication methods such as HTTP basic or LDAP, or when RBAC is disabled, all Malcolm users effectively have administrator privileges.
    • Because the OpenSearch Security Plugin requires TLS even internally, Malcolm's internal connections to the embedded OpenSearch instance, when used, are now all performed over HTTPS. However, this is all handled internally and should not behave or appear different to the user than it did in previous versions.
    • See the role-based access control documentation for more information on this feature.
  • Malcolm's embedded KeyCloak instance now automatically creates and configures the default client by ID, if specified in ./config/keycloak.env.
  • Allow user to specify subnet filters for NetBox autopopulation (allow user to specify subnet filters for NetBox autopopulation #634)
    • This feature is especially useful for excluding dynamic address ranges such as those used by DHCP, which should generally not trigger autopopulation in NetBox. Since these addresses can change frequently and aren't tied to specific devices, including them could result in inaccurate or noisy inventory data. By fine-tuning which private subnets are included or excluded, users can ensure that only meaningful, typically static assignments are autopopulated.
  • Expose init arguments for Arkime's db.pl and also use them for Malcolm's creation of its own index templates (Expose init arguments for Arkime's db.pl #692)
  • Extend Zeek's intel.log with additional fields using corelight/ExtendIntel (part 1) (extend intel.log with additional fields using corelight/ExtendIntel (part 1) #502)
    • This integrates the corelight/ExtendIntel plugin into Malcolm internally but does not significantly change how Malcolm presents intel.log to the user. Further work to do so will be continued in extend intel.log with additional fields using corelight/ExtendIntel (part 2) #695.
  • Some internal tweaks to the PCAP processing pipeline that are going to be leveraged by the Malcolm-Helm project (Upstream file changes to get PCAP processor to work. idaholab/Malcolm#630)
  • Handle a fix in the ICSNPP OPCUA-Binary plugin that adds a new sec_token_id field (Fixing sec_token_id not being properly logged icsnpp-opcua-binary#101)
  • Moved the configuration for Zeek's use of the zeek-kafka plugin to its own file (kafka.zeek) to make it easier to override in Docker using a volume bind mount or in K8s using a configMap.
  • Changed some internal objects used for NetBox enrichment caching from Ruby's Concurrent::Hash to Concurrent::Map for better performance
  • Minor improvements to the icons, shortcuts, and convenience bash functions in the ISO-installed Malcolm desktop environment
  • NGINX now generates a robots.txt file to avoid web crawlers
• ✅ Component version updates
  • Alpine base Docker image to v3.22.0
  • Arkime to v5.7.0
  • capa to v9.2.1
  • flask-cors Python library to v6.0.0 to address CVE-2024-6839, CVE-2024-6844, and CVE-2024-6866
  • OpenSearch and OpenSearch Dashboards to v3.0.0
    • What To Expect
    • Announcing OpenSearch 3.0
    • Performance Improvements
  • opensearch-py Python library to v3.0.0
  • osd_transform_vis Dashboards visualization library to v3.0.0
  • requests Python library to v2.32.4 to address CVE-2024-47081
  • YARA to v4.5.3
  • Zeek to v7.2.1
• 🐛 Bug fixes
  • NetBox autodiscovery no longer populating host name from DNS, DHCP, NTLM (regression, NetBox autodiscovery no longer seems to populate host name from DNS, DHCP, etc. #699)
  • documentation served at /readme is trying to pull fonts from use.fontawesome.com (documentation served at /readme is trying to pull fonts from use.fontawesome.com #694)
  • support fractional gigabytes correctly when generating Arkime's config.ini setting maxFileSizeG from PCAP_ROTATE_MEGABYTES
  • Improved logstash filters that calculate unique hashes used as document IDs for Zeek and Suricata logs to better prevent duplicate logs from being written to the document store
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux. The Malcolm control script (e.g., ./scripts/status, ./scripts/start, etc.) should take care of creating new variables and migrating existing ones as needed based on the rules in ./config/env-var-actions.yml.
  • arkime.env's OPENSEARCH_MAX_SHARDS_PER_NODE has been moved to opensearch.env and renamed to CLUSTER_MAX_SHARDS_PER_NODE
  • auth-common.env's NGINX_LDAP_TLS_… variables have been moved to nginx.env
  • auth-common.env's has many new ROLE… variables that support the new RBAC feature (see Features and enhancements above)
  • netbox.env has some new variables and some changed default variable values for supporting RBAC
  • added NETBOX_AUTO_POPULATE_SUBNETS to netbox-common.env for the "subnet filters for NetBox autopopulation" feature (see Features and enhancements above)
  • opensearch.env's OPENSEARCH_URL now defaults to https://opensearch:9200 when using Malcolm's embedded OpenSearch instance
  • opensearch.env's added ARKIME_INIT_… variables that are used, if specified, during the setup of index patterns and templates (see Expose init arguments… in Features and enhancements above)
• 🧹 Code and project maintenance
  • Tweaked some code comments and documentation to bring the cisagov and idaholab repos into harmony.
  • Documentation improvements
  • Removed some unused files and outdated comments

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

---

This discussion was created from the release Malcolm v25.06.0.