Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enrichment support #21

Open
4 of 11 tasks
Samrose-Ahmed opened this issue Nov 5, 2022 · 0 comments
Open
4 of 11 tasks

Enrichment support #21

Samrose-Ahmed opened this issue Nov 5, 2022 · 0 comments
Labels
enhancement New feature or request planned

Comments

@Samrose-Ahmed
Copy link
Contributor

Samrose-Ahmed commented Nov 5, 2022 •
edited by shaeqahmed
Loading

Tracking issue for enrichment support

Goal

Provide enrichment through enrichment tables in Matano

  • Enrichment table as Iceberg table
  • Enrichment tables for lookup in Python detections
  • Matching against IOCs
  • Dynamic & static enrichment tables (ingesting)
  • Ingesting threat intelligence feed data (IOC’s)

Managed Integrations

  • AbuseCH (Malware Bazaar, Threatfox, URLHaus)
  • GeoIP (Maxmind)
  • Greynoise intelligence (RIOT, Noise)

Forward looking
@Samrose-Ahmed Samrose-Ahmed added enhancement New feature or request planned labels Nov 5, 2022
@Samrose-Ahmed Samrose-Ahmed moved this to Researching in Matano Public Roadmap Nov 13, 2022
@Samrose-Ahmed Samrose-Ahmed moved this from Researching to In Progress in Matano Public Roadmap Nov 13, 2022
@shaeqahmed shaeqahmed mentioned this issue Mar 8, 2023
Merged
@shaeqahmed shaeqahmed linked a pull request Mar 8, 2023 that will close this issue
🔥 Realtime Data Enrichment - add get_enrichment_table_record fn to VRL log transform pipeline #111
Merged
@shaeqahmed shaeqahmed removed a link to a pull request Mar 8, 2023
🔥 Realtime Data Enrichment - add get_enrichment_table_record fn to VRL log transform pipeline #111
Merged
shaeqahmed added a commit that referenced this issue Mar 8, 2023
@shaeqahmed
🔥 Realtime Data Enrichment - add get_enrichment_table_record fn to VR…
ee4654a 
…L log transform pipeline (#111)

### Summary

This pull request introduces support for real-time data enrichment in
Matano during ingest, addressing #99 and #21. The new
`get_enrichment_table_record` function has been added to the VRL log
transform pipeline, enabling retrieval of enrichment data and adding it
to the incoming data stream in real-time, before the detection / lake
writing steps.

For many use cases, this feature means users no longer need to perform
manual JOINS in their queries or do manual lookups in their detection
rules and improves downstream analytics performance by providing
pre-joined/enriched records in the data lake and detection engine.

<img width="820" alt="Screenshot 2023-03-07 at 11 42 46 PM"
src="https://user-images.githubusercontent.com/13088492/223651670-702b7191-d844-418c-a0dc-6a360d869e05.png">

### Up next
Next step, will be to add extend support to GeoIP enrichment tables
(MaxMind), which will require special handling logic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request planned
Projects
Matano Public Roadmap
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Samrose-Ahmed