SIGABRT and Segfault in _S_create_c_locale

[netantho]

I'm not sure yet if it's reproducible.

Tenzir version: ~4.2 (I don't find a command or flag to get the exact version)

zstdcat 2022-01-14_1642149858466556000_tshark_4.0.8-1.ndjson.zst | /usr/bin/docker run --network=host --mount type=bind,source=`pwd`,target=/mnt -i 27183269cf
09 --config=/mnt/tenzir.yaml --dump-metrics "read json --ndjson | discard"
[19:31:44.061] loaded configuration file: "/mnt/tenzir.yaml"
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
tenzir-v4.2.0: Error: signal 6 (Aborted)
0x7f77e117e0cc: (fatal_handler+0x3c)
0x7f77ddd5dfd0: (__sigaction+0x40)
0x7f77dddacd3c: (pthread_key_delete+0x14c)
0x7f77ddd5df32: (gsignal+0x12)
0x7f77ddd48472: (abort+0xd3)
0x7f77de09f919: (__cxa_throw_bad_array_new_length+0x553)
0x7f77de0aae1a: (std::rethrow_exception(std::__exception_ptr::exception_ptr)+0x7a)
0x7f77de0aae85: (std::terminate()+0x15)
0x7f77de0ab0d8: (__cxa_throw+0x48)
0x7f77de0a22bd: (std::__throw_runtime_error(char const*)+0x40)
0x7f77de0cc318: (std::locale::facet::_S_create_c_locale(__locale_struct*&, char const*, __locale_struct*)+0x28)
0x7f77de0be538: (std::locale::_Impl::_Impl(char const*, unsigned long)+0x48)
0x7f77de0bf0dd: (std::locale::locale(char const*)+0x12d)
0x56498cbca0d3: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsi
gned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_c
xx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615u
l> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x32563)
0x56498cbcc608: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsi
gned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_c
xx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615u
l> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x34a98)
0x56498cbccb77: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsi
gned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_c
xx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615u
l> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x35007)
0x7f77e102764d: (tenzir::run(tenzir::invocation const&, caf::actor_system&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<caf::message (tenzir::inv
ocation const&, caf::actor_system&)>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits
<char>, std::allocator<char> > const, std::function<caf::message (tenzir::invocation const&, caf::actor_system&)> > > > const&)+0x22d)
0x56498caaf41e: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>::invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x6606e
)
0x7f77ddd491ca: (__libc_init_first+0x8a)
0x7f77ddd49285: (__libc_start_main+0x85)
0x56498caafbd1: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>::invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x66821
)
tenzir-v4.2.0: Error: signal 11 (Segmentation fault)
0x7f77e117e0cc: (fatal_handler+0x3c)
0x7f77ddd5dfd0: (__sigaction+0x40)
0x7f77ddd4850f: (abort+0x170)
0x7f77de09f919: (__cxa_throw_bad_array_new_length+0x553)
0x7f77de0aae1a: (std::rethrow_exception(std::__exception_ptr::exception_ptr)+0x7a)
0x7f77de0aae85: (std::terminate()+0x15)
0x7f77de0ab0d8: (__cxa_throw+0x48)
0x7f77de0a22bd: (std::__throw_runtime_error(char const*)+0x40)
0x7f77de0cc318: (std::locale::facet::_S_create_c_locale(__locale_struct*&, char const*, __locale_struct*)+0x28)
0x7f77de0be538: (std::locale::_Impl::_Impl(char const*, unsigned long)+0x48)
0x7f77de0bf0dd: (std::locale::locale(char const*)+0x12d)
0x56498cbca0d3: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsi
gned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_c
xx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615u
l> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x32563)
0x56498cbcc608: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsi
gned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_c
xx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615u
l> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x34a98)
0x56498cbccb77: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsi
gned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_c
xx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615u
l> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x35007)
0x7f77e102764d: (tenzir::run(tenzir::invocation const&, caf::actor_system&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::function<caf::message (tenzir::inv
ocation const&, caf::actor_system&)>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits
<char>, std::allocator<char> > const, std::function<caf::message (tenzir::invocation const&, caf::actor_system&)> > > > const&)+0x22d)
0x56498caaf41e: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>::invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x6606e
)
0x7f77ddd491ca: (__libc_init_first+0x8a)
0x7f77ddd49285: (__libc_start_main+0x85)
0x56498caafbd1: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>::invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x66821
)

[dominiklohmann]

I don't find a command or flag to get the exact version

You can do this with tenzir 'show version', although it does look like you're using a custom-built Docker image, and for that you either need to pass in the exact version from the outside when building, or all you get is the last release version. This is because git-describe cannot be used in the Docker context.

---

Let's take a look at the stacktrace. There isn't much to see here because it's from an optimized binary, so let me try to explain bottom-up what can be seen:

0x56498caafbd1: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>::invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x66821
)

We use fu2::function with the signature auto () noecxcept -> void as deletion handlers for chunk.

This deletion handler is marked noexcept, and we appear to throw an exception in it, which causes the program to terminate by calling std::terminate, which causes the program to exit with SIGABRT.

The exception seems to be coming from a call to fmt::format:

0x56498cbca0d3: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsi
gned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_c
xx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615u
l> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x32563)

Here it looks like we are formatting the output of fmt::join over a range of const unsigned char, so likely a string or a byte range of some sort. It's not entirely clear to me what this is. As far as I can tell, the {fmt} library never throws std::runtime_error internally and neither do we ourselves, so this is likely to come from some interaction with the standard library.

My best guess is that we have some call to chunk::make in which we pass a deleter that does something it shouldn't be doing when formatting a log message. However, finding that just from this stacktrace is a lot of work.

If you can reproduce this (somewhat) reliably then it should be easy to find the source of this runtime error and to fix it.

[netantho]

Version (sorry for the limited research previously):

commit 7500759f6830828b873ee7e33d79d9441ec4dbb9 (HEAD -> main, origin/main, origin/HEAD)
Merge: 4734c0cf42 164f48bea3
Author: Daniel Kostuj <[email protected]>
Date:   Fri Sep 29 08:38:07 2023 +0200

    Add the `/pipeline/launch` endpoint to the `tenzir_features()` (#3543)

    This change adds the `/pipeline/launch` endpoint to the
    `tenzir_features()` to help the app discern whether unifying the "run &
    deploy" workflow in the app is possible.

I managed to reproduce reliably, will reach out via Discord for the file (but it's not urgent problem on my side).

[dominiklohmann]

I got the file you shared with me—thanks!—and used this build, which is current main as of writing this message:

$ tenzir 'show version'
{
  "version": "v4.3.0-84-g75485c5b8d"
}

$ # for the release build
$ tenzir 'show build'
{
  "type": "Release",
  "tree_hash": "7a150694e0d6ba989ddbda9221ba1526",
  "assertions": false,
  "sanitizers": {
    "address": false,
    "undefined_behavior": false
  }
}

$ # for the debug build
$ tenzir 'show build'
{
  "type": "Debug",
  "tree_hash": "7a150694e0d6ba989ddbda9221ba1526",
  "assertions": true,
  "sanitizers": {
    "address": true,
    "undefined_behavior": false
  }
}

I used this pipeline:

zstdcat file.ndjson.zst | tenzir --dump-metrics 'read json --ndjson | discard'

The release build ran through; since we also discussed performance relating to this input in Discord, here's the relevant portion of the metrics:

operator #2 (read)
  total: 29.92s
  scheduled: 28.58s (95.51%)
  processing: 26.18s (87.50%)
  runs: 44281 (99.98% processing / 99.25% input / 99.95% output)
  inbound:
    bytes: 767,232,783 at a rate of 25642884.48/s
    batches: 1,215 (631467.31 bytes/batch)
  outbound:
    events: 496,402 at a rate of 16591.03/s
    bytes: 35,646,782 at a rate of 1191406.75/s (estimate)
    batches: 42,733 (11.62 events/batch)

The performance for this is oddly low despite the batching being near optimal. We definitely need to look into that further as 16k EPS is a lot lower than what we see for other events, and it's not even of a particularly complex shape.

Since that had not crashed for me I tried again with the debug build. That was a lot slower (as expected), but it did not crash for me either.

I then tried to build 7500759 specifically, but that did also not crash for either a release or a debug build on my end.

I will try again tomorrow with a Docker build of this specific commit.

[tobim]

@netantho the exception message in your backtrace indicates an issue with the locale, this is hard to investigate without the specific Dockerfile and build invocation that you used.

There may still be a potential workaround that you could try: Add --env LC_ALL=C as an argument to docker run, so that the complete call is:

zstdcat 2022-01-14_1642149858466556000_tshark_4.0.8-1.ndjson.zst | /usr/bin/docker run --env LC_ALL=C --network=host --mount type=bind,source=`pwd`,target=/mnt -i 27183269cf

[netantho]

zstdcat 2022-01-14_1642149858466556000_tshark_4.0.8-1.ndjson.zst | /usr/bin/docker run --env LC_ALL=C --network=host --mount type=bind,source=`pwd`,target=/mnt -i 27183269cf09 --config=/mnt/tenzir.yaml "show build"
[20:15:24.785] loaded configuration file: "/mnt/tenzir.yaml"
{
  "type": "Release",
  "tree_hash": "01d2a8011fc21ca4794f3ddaea31547c",
  "assertions": false,
  "sanitizers": {
    "address": false,
    "undefined_behavior": false
  }
}

# cat tenzir.yaml
tenzir:
  import:
    blocking: true

zstdcat 2022-01-14_1642149858466556000_tshark_4.0.8-1.ndjson.zst | /usr/bin/docker run --env LC_ALL=C --network=host --mount type=bind,source=`pwd`,target=/mnt -i 27183269cf09 --config=/mnt/tenzir.yaml --dump-metrics "read json --ndjson | discard"
[20:17:32.318] loaded configuration file: "/mnt/tenzir.yaml"
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
tenzir-v4.2.0: Error: signal 6 (Aborted)
0x7f16a88ab0cc: (fatal_handler+0x3c)                                                                                        0x7f16a548afd0: (__sigaction+0x40)                                                                                          0x7f16a54d9d3c: (pthread_key_delete+0x14c)                                                                                  0x7f16a548af32: (gsignal+0x12)                                                                                              0x7f16a5475472: (abort+0xd3)                                                                                                0x7f16a57cc919: (__cxa_throw_bad_array_new_length+0x553)                                                                    0x7f16a57d7e1a: (std::rethrow_exception(std::__exception_ptr::exception_ptr)+0x7a)
0x7f16a57d7e85: (std::terminate()+0x15)                                                                                     0x7f16a57d80d8: (__cxa_throw+0x48)                                                                                          0x7f16a57cf2bd: (std::__throw_runtime_error(char const*)+0x40)                                                              0x7f16a57f9318: (std::locale::facet::_S_create_c_locale(__locale_struct*&, char const*, __locale_struct*)+0x28)             0x7f16a57eb538: (std::locale::_Impl::_Impl(char const*, unsigned long)+0x48)                                                0x7f16a57ec0dd: (std::locale::locale(char const*)+0x12d)                                                                    0x561284d620d3: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt
::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&)+0x32563)                                                                              0x561284d64608: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt
::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9::formatter<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_form
at_context<fmt::v9::appender, char>&)+0x34a98)                                                                              0x561284d64b77: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt
::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9
::formatter<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_form
at_context<fmt::v9::appender, char>&)+0x35007)
0x7f16a875464d: (tenzir::run(tenzir::invocation const&, caf::actor_system&, std::map<std::__cxx11::basic_string<char, std::c
har_traits<char>, std::allocator<char> >, std::function<caf::message (tenzir::invocation const&, caf::actor_system&)>, std::
less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx1
1::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<caf::message (tenzir::invocation c
onst&, caf::actor_system&)> > > > const&)+0x22d)
0x561284c4741e: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>:
:invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x6606e)
0x7f16a54761ca: (__libc_init_first+0x8a)
0x7f16a5476285: (__libc_start_main+0x85)
0x561284c47bd1: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>:
:invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x66821)
tenzir-v4.2.0: Error: signal 11 (Segmentation fault)
0x7f16a88ab0cc: (fatal_handler+0x3c)
0x7f16a548afd0: (__sigaction+0x40)
0x7f16a547550f: (abort+0x170)
0x7f16a57cc919: (__cxa_throw_bad_array_new_length+0x553)
0x7f16a57d7e1a: (std::rethrow_exception(std::__exception_ptr::exception_ptr)+0x7a)
0x7f16a57d7e85: (std::terminate()+0x15)
0x7f16a57d80d8: (__cxa_throw+0x48)
0x7f16a57cf2bd: (std::__throw_runtime_error(char const*)+0x40)
0x7f16a57f9318: (std::locale::facet::_S_create_c_locale(__locale_struct*&, char const*, __locale_struct*)+0x28)
0x7f16a57eb538: (std::locale::_Impl::_Impl(char const*, unsigned long)+0x48)
0x7f16a57ec0dd: (std::locale::locale(char const*)+0x12d)
0x561284d620d3: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt
::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9
::formatter<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073
709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_form
at_context<fmt::v9::appender, char>&)+0x32563)
0x561284d64608: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt
::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9
::formatter<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073
709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_form
at_context<fmt::v9::appender, char>&)+0x34a98)
0x561284d64b77: (void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<fmt
::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >, char>, fmt::v9
::formatter<fmt::v9::join_view<__gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073
709551615ul> >, __gnu_cxx::__normal_iterator<unsigned char const*, std::span<unsigned char const, 18446744073709551615ul> >,
 char>, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_form
at_context<fmt::v9::appender, char>&)+0x35007)
0x7f16a875464d: (tenzir::run(tenzir::invocation const&, caf::actor_system&, std::map<std::__cxx11::basic_string<char, std::c
har_traits<char>, std::allocator<char> >, std::function<caf::message (tenzir::invocation const&, caf::actor_system&)>, std::
less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx1
1::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::function<caf::message (tenzir::invocation c
onst&, caf::actor_system&)> > > > const&)+0x22d)
0x561284c4741e: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>:
:invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x6606e)
0x7f16a54761ca: (__libc_init_first+0x8a)
0x7f16a5476285: (__libc_start_main+0x85)
0x561284c47bd1: (tenzir::detail::fu2::type_erasure::invocation_table::function_trait<void () noexcept>::empty_invoker<true>:
:invoke(tenzir::detail::fu2::type_erasure::data_accessor*, unsigned long)+0x66821)

I get the same error after rebooting. I'm spinning up a new cloud instance to check if I can reproduce there as well.

[netantho]

After more thorough testing:

• The bugs manifests itself only when using docker. It's consistent across various cloud instances (I'm using Ubuntu as base image on these instances).
• Without using docker, with the current head, the pipeline is stall. With Add blob type for arbitrary binary data #3581, it returns, similar to Dominik's result
• Add blob type for arbitrary binary data #3581 still produces the error when using docker.

I've been unsuccessful with a docker container. Here's what I've tried so far:

• Many combinations of locales including C, POSIX, en_US.UTF-8 and C.UTF-8
• Changing where the zstd decompression is done (host machine vs in tenzir pipeline)
• Same locale on the host machine and container
• Loading the file from the pipeline itself.

[tobim]

I took a closer look just now. This is simply a bug in --dump-metrics where a hard-coded locale is used.

[tobim]

I created a fix for the problem at #3593.

[netantho]

This solves the issue on my setup, thanks @tobim!