Skip to content

File source no longer can decompress Gzip #24280

@FishNumber7

Description

@FishNumber7

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am currently pulling gzip compressed files from S3 to be parsed and shipped. Last time this was used was before v0.51.0 and it worked fine. Now, it cannot handle Gzip files, though in the documentation it states it can automatically: https://vector.dev/docs/reference/configuration/sources/file/#compressed-files. The output message cannot be parsed, and the output events from the file source seem to still be compressed. When I decompress the file through the command line and have vector watch that file instead, it works perfectly fine.

Configuration

sources:
  file_source:
    type: file
    include:
      - /var/log/s3/**/*.log
    multiline:
      start_pattern: "^[{]"
      condition_pattern: "^[}]"
      mode: halt_with
      timeout_ms: 10000
    fingerprint:
      strategy: device_and_inode

Version

v0.51.1

Debug Output


Example Data

No response

Additional Context

I am using log namespacing.

References

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugA code related bug.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions