-
Notifications
You must be signed in to change notification settings - Fork 1.7k
feat(azure_logs_ingestion sink): Initial azure_logs_ingestion
sink
#22912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
feat(azure_logs_ingestion sink): Initial azure_logs_ingestion
sink
#22912
Conversation
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
check-spelling run (pull_request_target) for feature-azure_logs_ingestion Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @jlaundry, thanks for this contribution!
It looks good. Two things:
- We will need some documentation files. See an example here (all files under
website
). Note thatbase/
is generated bymake generate-component-docs
. - Is the intention to complete replace the
azure_monitor_logs
sink? If that's the case maybe we can mark the existing one as deprecated in favor of this new sink.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your contribution! Approving with one very minor suggestion.
website/cue/reference/components/sinks/base/azure_logs_ingestion.cue
Outdated
Show resolved
Hide resolved
Apologies, I thought this page was auto-generated as well - added now.
Good point, added 🙂 |
""" | ||
The upstream Data Collector API [has been deprecated](urls.azure_monitor_data_collector_deprecation), | ||
and will stop working in September 2026. Consider migrating to the `azure_logs_ingestion` sink, which | ||
requires additional Azure resources. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What extra resources does the new sink require? Worth explaining this a bit better for users of the old sink.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done (bb71403)
website/cue/reference/components/sinks/base/azure_logs_ingestion.cue
Outdated
Show resolved
Hide resolved
…on.cue Co-authored-by: Rosa Trieu <[email protected]>
Signed-off-by: Jed Laundry <[email protected]>
azure_logs_ingestion
sinkazure_logs_ingestion
sink
Signed-off-by: Jed Laundry <[email protected]>
From memory, there is a minor refactor required in https://github.com/jlaundry/vector/blob/02562be6447af36404d8b5668434e317c87a45b2/src/sinks/azure_logs_ingestion/config.rs#L139 to change it back to Probably easiest if you rollback the package first, and then I'll rebase, retest, and push. |
FYI - Reverted deps #23039 |
FYI, I started preparing the rebase, but I've seen that the Azure Rust team have recently decided to change how authentication works with the Depending on what they decide, we may need to explicitly configure credentials, either via the vector config file or environment variables. The So instead of releasing an initial sink, and then requiring another config/environment change, I'll wait until there's stability in the SDK before moving forward with this PR. (I'm not giving up!!!) |
Hi @jlaundry, By the way, I see that the discussion here is closed. Does it mean you will go forward with merging your PR? Thanks a lot! |
Summary
The current
azure_monitor_logs
sink uses the Data Collector API, which has been deprecated and will be removed in September 2026.This sink uses the replacement Logs Ingestion API.
While I did consider making this a drop-in replacement for the existing sink, users need to make numerous breaking infrastructure changes, including:
_CL
custom tables.Change Type
Is this a breaking change?
How did you test this PR?
AZURE_TENANT_ID
,AZURE_CLIENT_ID
, andAZURE_CLIENT_SECRET
environment variables from the App Registrationvector.yaml
:Does this PR include user facing changes?
References
azure_blob
, by using the Azure Identity crate, this sink supports passwordless credentials.