Unable to send logs to cloudwatch_logs plugin

[yuzhengchua]

Which image/versions are related to this issue/feature request?

image used are

fluent-bit:latest
fluent-bit:latest-dev
aws-for-fluent-bit:latest

Issue/Feature description

The chainguard images for fluent-bit are unable to send logs to a cloudwatch log group.
Given the following fluent-bit.conf file

[SERVICE]
    # Flush
    # =====
    # set an interval of seconds before to flush records to a destination
    flush        1

    # Daemon
    # ======
    # instruct Fluent Bit to run in foreground or background mode.
    daemon       Off

    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    #
    # - error
    # - warning
    # - info
    # - debug
    # - trace
    #
    # by default 'info' is set, that means it includes 'error' and 'warning'.
    log_level    debug

    # Parsers File
    # ============
    # specify an optional 'Parsers' configuration file
    parsers_file parsers.conf

    # Plugins File
    # ============
    # specify an optional 'Plugins' configuration file to load external plugins.
    plugins_file plugins.conf

    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020

[INPUT]
    name cpu
    tag  cpu.local

    # Read interval (sec) Default: 1
    interval_sec 1

[OUTPUT]
    name cloudwatch_logs
    region   ap-southeast-1
    log_group_name test
    log_stream_prefix myprefix
    auto_create_group true

When running the image using podman
podman run -d -v /tmp/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf -e AWS_ACCESS_KEY_ID=<MYID> -e AWS_SECRET_ACCESS_KEY=<MYKEY> cgr.dev/chainguard/fluent-bit:latest

I will encounter the DNS error
[ warn] [net] getaddrinfo(host='logs.ap-southeast-1.amazonaws.com', err=12): Timeout while contacting DNS servers
[2024/01/25 19:10:48] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=10 assigned to thread #0
[2024/01/25 19:10:48] [debug] [upstream] connection #-1 failed to logs.ap-southeast-1.amazonaws.com:443

If I add the "endpoint" parameter to the "OUTPUT" section with the public IP Address of the logs.ap-southeast-1.amazonaws.com url, I will then encounter the "unexpected EOF" error
[2024/01/25 19:13:15] [debug] [upstream] connection #43 failed to 13.212.3.116:443
[2024/01/25 19:13:15] [error] [aws_client] connection initialization error
[2024/01/25 19:13:15] [debug] [aws_client] auto-retrying
[2024/01/25 19:13:15] [error] [tls] error: unexpected EOF
[2024/01/25 19:13:15] [debug] [upstream] connection #43 failed to 13.212.3.116:443

Any help to troubleshoot would be appreciated.
When using the official AWS fluent-bit image, I am not encountering such issues. I am currently testing the chainguard images locally on my machine because of the low vulnerabilities scores and really hope they can be incorporated to my workloads.

Thank you.

[NeckBeardPrince]

I'm also having this issue.