Blob flush request failed - code: 400

[metalbahamut]

Hi,

We are receiving the following while running the plugin through the image "docker.io/fluent/fluentd-aggregator":

failed to flush the buffer. retry_times=8 next_retry_time=2023-03-21 08:31:03 +0000 chunk="5f76494f6b6576c7e4ee0a4895fd8e38" error_class=Fluent::Plugin::BlobOperationError error="Blob flush request failed - code: 400, body: {\"error\":{\"code\":\"InvalidFlushPosition\",\"message\":\"The uploaded data is not contiguous or the position query parameter value is not equal to the length of the file after appending the uploaded data.\\nRequestId:96cd2c94-201f-0041-2ece-5b4681000000\\nTime:2023-03-21T08:26:54.6092036Z\"}}, headers: {\"Content-Length\"=>\"284\", \"Content-Type\"=>\"application/json;charset=utf-8\", \"Server\"=>\"Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0\", \"x-ms-error-code\"=>\"InvalidFlushPosition\", \"x-ms-request-id\"=>\"96cd2c94-201f-0041-2ece-5b4681000000\", \"x-ms-version\"=>\"2018-11-09\", \"Date\"=>\"Tue, 21 Mar 2023 08:26:54 GMT\

This is the fluentd configuration:

<match **>
  @type azurestorage_gen2
  azure_storage_account ocpuateuwlogging
  azure_oauth_tenant_id "#{ENV['AZURE_TENANT_ID']}"
  azure_oauth_app_id "#{ENV['AZURE_CLIENT_ID']}"
  azure_oauth_secret "#{ENV['AZURE_CLIENT_SECRET']}"
  azure_oauth_refresh_interval 3000
  azure_container application
  url_storage_resource https://xxx.blob.core.windows.net
  auto_create_container true
  enable_retry true
  time_slice_format %Y%m%d-%H
  azure_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  store_as gzip
  utc true
  http_timeout_seconds 300
  path ${tag}/%Y/%m/%d/
  <buffer tag,time>
    @type file
    flush_thread_count 1
    path /var/log/fluent/azurestorage_gen2
    overflow_action drop_oldest_chunk
    timekey 3600
    timekey_wait 10m
    timekey_use_utc true
    chunk_limit_size 256m
    flush_mode interval
    flush_interval 600s
    flush_at_shutdown true
    retry_max_times 10
  </buffer>
</match>

Do you know if anything in our fluentd configuration is wrong or not supported?

Regards,

[oleewere]

hi @metalbahamut ,
thanks for the feedback - will require some time in this week to review the behaviour
the plugin mostly are used with MSI on azure environments, any chance you can use MSI and see the same behaviour is happening? or alternatively use / install azure cli on the image and set "azure_oauth_use_azure_cli" to "true" ?

although error tells that something wrong in the data input itself, as azure api call is versioned - azure side should not have affect on this (ideally), btw, is this error continuously happening for every request or some data has been flushed properly?

[metalbahamut]

Hi,
We are running fluentd in a Kubernetes (non AKS) so MSI is not suitable for us. We can try installing azure cli (but not me preferable choice).
The error happens randomly (not always), so some data has been flushed properly.
We have 2 fluentd (with different sources) and the error is only happening in one of them, so probably it's something about the data but I need to understand why data source may affect if we are pushing .gz files...
Regards,

[oleewere]

@metalbahamut , i see, thanks for the feedback. I can also try to finish the azure_storage_access_key, but it would not have any impact on the data itself ... so would be better to reproduce the same issue by myself or maybe what could help as well if you could set debug log level and when the error happens you can provide the logs around the issue, (meanwhile I will work on to reproduce the issue - could be related with too large files - but would I notice that error earlier, could be a miscalculation if the data parts are splitted)

[metalbahamut]

Hi,

With the following configuration is working fine (no log errors in the past 3 days), so I suppose it's a matter of sizing (as you mentioned). Let me know if something else can support you.

<match **>
  @type azurestorage_gen2
  @log_level info
  azure_storage_account xxx
  azure_oauth_tenant_id "#{ENV['AZURE_TENANT_ID']}"
  azure_oauth_app_id "#{ENV['AZURE_CLIENT_ID']}"
  azure_oauth_secret "#{ENV['AZURE_CLIENT_SECRET']}"
  azure_oauth_refresh_interval 3000
  azure_container audit
  url_storage_resource https://xxx.blob.core.windows.net
  auto_create_container true
  enable_retry true
  time_slice_format %Y%m%d-%H
  azure_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  store_as gzip
  utc true
  http_timeout_seconds 30
  path ${tag}/%Y/%m/%d/
  <buffer tag, time>
    @type file
    flush_thread_count 1
    path /var/log/fluent/azurestorage_gen2
    timekey 5m
    timekey_wait 0s
    timekey_use_utc true
    chunk_limit_size 32MB
    queued_chunks_limit_size 32
    total_limit_size 8GB
    flush_mode interval
    flush_interval 60s
    flush_at_shutdown true
    retry_max_times 10
    overflow_action drop_oldest_chunk
  </buffer>
</match>