Skip to content

OP Remap VRL processor #2464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: v2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
356 changes: 356 additions & 0 deletions .generator/schemas/v2/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25153,6 +25153,7 @@ components:
- $ref: '#/components/schemas/ObservabilityPipelineSentinelOneDestination'
- $ref: '#/components/schemas/ObservabilityPipelineOpenSearchDestination'
- $ref: '#/components/schemas/ObservabilityPipelineAmazonOpenSearchDestination'
- $ref: '#/components/schemas/ObservabilityPipelineSocketDestination'
ObservabilityPipelineConfigProcessorItem:
description: A processor for the pipeline.
oneOf:
Expand All @@ -25172,6 +25173,7 @@ components:
- $ref: '#/components/schemas/ObservabilityPipelineEnrichmentTableProcessor'
- $ref: '#/components/schemas/ObservabilityPipelineReduceProcessor'
- $ref: '#/components/schemas/ObservabilityPipelineThrottleProcessor'
- $ref: '#/components/schemas/ObservabilityPipelineRemapVrlProcessor'
ObservabilityPipelineConfigSourceItem:
description: A data source for the pipeline.
oneOf:
Expand All @@ -25190,6 +25192,7 @@ components:
- $ref: '#/components/schemas/ObservabilityPipelineGooglePubSubSource'
- $ref: '#/components/schemas/ObservabilityPipelineHttpClientSource'
- $ref: '#/components/schemas/ObservabilityPipelineLogstashSource'
- $ref: '#/components/schemas/ObservabilityPipelineSocketSource'
ObservabilityPipelineData:
description: "Contains the pipeline\u2019s ID, type, and configuration attributes."
properties:
Expand Down Expand Up @@ -26835,6 +26838,87 @@ components:
type: string
x-enum-varnames:
- REDUCE
ObservabilityPipelineRemapVrlProcessor:
description: The `remap_vrl` processor (also called "Custom processor") transforms
events using [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl/)
scripts with advanced filtering capabilities.
properties:
id:
description: The unique identifier for this processor.
example: remap-vrl-processor
type: string
include:
description: A Datadog search query used to determine which logs this processor
targets. This field should always be set to `*` for the remap_vrl processor.
example: '*'
type: string
inputs:
description: A list of component IDs whose output is used as the input for
this processor.
example:
- datadog-agent-source
items:
type: string
type: array
remaps:
description: Array of VRL remap rules.
items:
$ref: '#/components/schemas/ObservabilityPipelineRemapVrlProcessorRemap'
maxItems: 15
minItems: 1
type: array
type:
$ref: '#/components/schemas/ObservabilityPipelineRemapVrlProcessorType'
required:
- id
- type
- include
- remaps
- inputs
type: object
ObservabilityPipelineRemapVrlProcessorRemap:
description: Defines a single VRL remap rule with its own filtering and transformation
logic.
properties:
dropOnError:
default: false
description: Whether to drop events that caused errors during processing.
example: false
type: boolean
enabled:
default: true
description: Whether this remap rule is enabled.
example: true
type: boolean
include:
description: A Datadog search query used to filter events for this specific
remap rule.
example: service:web
type: string
name:
description: A descriptive name for this remap rule.
example: Parse JSON from message field
type: string
source:
description: The VRL script source code that defines the processing logic.
Must not exceed 3000 characters and cannot contain forbidden functions.
example: . = parse_json!(.message)
maxLength: 3000
type: string
required:
- include
- name
- source
type: object
ObservabilityPipelineRemapVrlProcessorType:
default: remap_vrl
description: The processor type. The value should always be `remap_vrl`.
enum:
- remap_vrl
example: remap_vrl
type: string
x-enum-varnames:
- REMAP_VRL
ObservabilityPipelineRemoveFieldsProcessor:
description: The `remove_fields` processor deletes specified fields from logs.
properties:
Expand Down Expand Up @@ -27473,6 +27557,278 @@ components:
type: string
x-enum-varnames:
- SENTINEL_ONE
ObservabilityPipelineSocketDestination:
description: 'The `socket` destination sends logs over TCP or UDP to a remote
server.

'
properties:
encoding:
$ref: '#/components/schemas/ObservabilityPipelineSocketDestinationEncoding'
framing:
$ref: '#/components/schemas/ObservabilityPipelineSocketDestinationFraming'
id:
description: The unique identifier for this component.
example: socket-destination
type: string
inputs:
description: A list of component IDs whose output is used as the `input`
for this component.
example:
- filter-processor
items:
type: string
type: array
mode:
$ref: '#/components/schemas/ObservabilityPipelineSocketDestinationMode'
tls:
$ref: '#/components/schemas/ObservabilityPipelineTls'
description: TLS configuration. Relevant only when `mode` is `tcp`.
type:
$ref: '#/components/schemas/ObservabilityPipelineSocketDestinationType'
required:
- id
- type
- inputs
- encoding
- framing
- mode
type: object
ObservabilityPipelineSocketDestinationEncoding:
description: Encoding format for log events.
enum:
- json
- raw_message
example: json
type: string
x-enum-varnames:
- JSON
- RAW_MESSAGE
ObservabilityPipelineSocketDestinationFraming:
description: Framing method configuration.
oneOf:
- $ref: '#/components/schemas/ObservabilityPipelineSocketDestinationFramingNewlineDelimited'
- $ref: '#/components/schemas/ObservabilityPipelineSocketDestinationFramingBytes'
- $ref: '#/components/schemas/ObservabilityPipelineSocketDestinationFramingCharacterDelimited'
ObservabilityPipelineSocketDestinationFramingBytes:
description: Event data is not delimited at all.
properties:
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketDestinationFramingBytesMethod'
required:
- method
type: object
ObservabilityPipelineSocketDestinationFramingBytesMethod:
description: The definition of `ObservabilityPipelineSocketDestinationFramingBytesMethod`
object.
enum:
- bytes
example: bytes
type: string
x-enum-varnames:
- BYTES
ObservabilityPipelineSocketDestinationFramingCharacterDelimited:
description: Each log event is separated using the specified delimiter character.
properties:
delimiter:
description: A single ASCII character used as a delimiter.
example: '|'
maxLength: 1
minLength: 1
type: string
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod'
required:
- method
- delimiter
type: object
ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod:
description: The definition of `ObservabilityPipelineSocketDestinationFramingCharacterDelimitedMethod`
object.
enum:
- character_delimited
example: character_delimited
type: string
x-enum-varnames:
- CHARACTER_DELIMITED
ObservabilityPipelineSocketDestinationFramingNewlineDelimited:
description: Each log event is delimited by a newline character.
properties:
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod'
required:
- method
type: object
ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod:
description: The definition of `ObservabilityPipelineSocketDestinationFramingNewlineDelimitedMethod`
object.
enum:
- newline_delimited
example: newline_delimited
type: string
x-enum-varnames:
- NEWLINE_DELIMITED
ObservabilityPipelineSocketDestinationMode:
description: Protocol used to send logs.
enum:
- tcp
- udp
example: tcp
type: string
x-enum-varnames:
- TCP
- UDP
ObservabilityPipelineSocketDestinationType:
default: socket
description: The destination type. The value should always be `socket`.
enum:
- socket
example: socket
type: string
x-enum-varnames:
- SOCKET
ObservabilityPipelineSocketSource:
description: 'The `socket` source ingests logs over TCP or UDP.

'
properties:
framing:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceFraming'
id:
description: The unique identifier for this component. Used to reference
this component in other parts of the pipeline (e.g., as input to downstream
components).
example: socket-source
type: string
mode:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceMode'
tls:
$ref: '#/components/schemas/ObservabilityPipelineTls'
description: TLS configuration. Relevant only when `mode` is `tcp`.
type:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceType'
required:
- id
- type
- mode
- framing
type: object
ObservabilityPipelineSocketSourceFraming:
description: Framing method configuration for the socket source.
oneOf:
- $ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingNewlineDelimited'
- $ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingBytes'
- $ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingCharacterDelimited'
- $ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingOctetCounting'
- $ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingChunkedGelf'
ObservabilityPipelineSocketSourceFramingBytes:
description: Byte frames are passed through as-is according to the underlying
I/O boundaries (for example, split between messages or stream segments).
properties:
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingBytesMethod'
required:
- method
type: object
ObservabilityPipelineSocketSourceFramingBytesMethod:
description: Byte frames are passed through as-is according to the underlying
I/O boundaries (for example, split between messages or stream segments).
enum:
- bytes
example: bytes
type: string
x-enum-varnames:
- BYTES
ObservabilityPipelineSocketSourceFramingCharacterDelimited:
description: Byte frames which are delimited by a chosen character.
properties:
delimiter:
description: A single ASCII character used to delimit events.
example: '|'
maxLength: 1
minLength: 1
type: string
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingCharacterDelimitedMethod'
required:
- method
- delimiter
type: object
ObservabilityPipelineSocketSourceFramingCharacterDelimitedMethod:
description: Byte frames which are delimited by a chosen character.
enum:
- character_delimited
example: character_delimited
type: string
x-enum-varnames:
- CHARACTER_DELIMITED
ObservabilityPipelineSocketSourceFramingChunkedGelf:
description: Byte frames which are chunked GELF messages.
properties:
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingChunkedGelfMethod'
required:
- method
type: object
ObservabilityPipelineSocketSourceFramingChunkedGelfMethod:
description: Byte frames which are chunked GELF messages.
enum:
- chunked_gelf
example: chunked_gelf
type: string
x-enum-varnames:
- CHUNKED_GELF
ObservabilityPipelineSocketSourceFramingNewlineDelimited:
description: Byte frames which are delimited by a newline character.
properties:
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingNewlineDelimitedMethod'
required:
- method
type: object
ObservabilityPipelineSocketSourceFramingNewlineDelimitedMethod:
description: Byte frames which are delimited by a newline character.
enum:
- newline_delimited
example: newline_delimited
type: string
x-enum-varnames:
- NEWLINE_DELIMITED
ObservabilityPipelineSocketSourceFramingOctetCounting:
description: Byte frames according to the octet counting format as per RFC6587.
properties:
method:
$ref: '#/components/schemas/ObservabilityPipelineSocketSourceFramingOctetCountingMethod'
required:
- method
type: object
ObservabilityPipelineSocketSourceFramingOctetCountingMethod:
description: Byte frames according to the octet counting format as per RFC6587.
enum:
- octet_counting
example: octet_counting
type: string
x-enum-varnames:
- OCTET_COUNTING
ObservabilityPipelineSocketSourceMode:
description: Protocol used to receive logs.
enum:
- tcp
- udp
example: tcp
type: string
x-enum-varnames:
- TCP
- UDP
ObservabilityPipelineSocketSourceType:
default: socket
description: The source type. The value should always be `socket`.
enum:
- socket
example: socket
type: string
x-enum-varnames:
- SOCKET
ObservabilityPipelineSpec:
description: Input schema representing an observability pipeline configuration.
Used in create and validate requests.
Expand Down
Loading