Child Panic Wrong turn at cache/cache_main.c:341 errno = 12 (Cannot allocate memory)

[lrowe]

Seen this a handful of time today, this one with libvmod-tinykvm b32e490

Varnish recovers and successfully starts another child immediately afterwards.

I'd be surprised if it really was out of memory, top showed 32GB available when I checked after seeing this.

$ podman run --rm -p 127.0.0.1:8080:8080 --device /dev/kvm --group-add keep-groups --name deno-varnish -p 127.0.0.1:2159:2159 -v .:/mnt deno-varnish
Debug: Version: varnish-7.7.0 revision bb7a918611ee85d17aefd07115fb8c66015151b5
Debug: Platform: Linux,6.11.0-19-generic,x86_64,-jnone,-smalloc,-sdefault,-hcritbit
Debug: Child (15) Started
Child launched OK
Info: Child (15) said Child starts
Error: Child (15) died signal=6
Error: Child (15) Panic at: Fri, 02 May 2025 04:18:08 GMT
Wrong turn at cache/cache_main.c:341:
Signal 11 (Segmentation fault) received at 0x7c5218400040 si_code 1
version = varnish-7.7.0 revision bb7a918611ee85d17aefd07115fb8c66015151b5, vrt api = 21.0
ident = Linux,6.11.0-19-generic,x86_64,-jnone,-smalloc,-sdefault,-hcritbit,epoll
now = 4091919.769615 (mono), 1746159488.549368 (real)
Backtrace:
  0x56691cdc5772: varnishd(VBT_format+0x72) [0x56691cdc5772]
  0x56691cd43199: varnishd(+0x63199) [0x56691cd43199]
  0x56691cdc4d45: varnishd(VAS_Fail+0x45) [0x56691cdc4d45]
  0x56691cd3d966: varnishd(+0x5d966) [0x56691cd3d966]
  0x7c82cd97b050: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7c82cd97b050]
  0x7c82cdaacb81: /lib/x86_64-linux-gnu/libc.so.6(+0x16db81) [0x7c82cdaacb81]
  0x7c82c01b59e3: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZN7tinykvm7Machine10elf_loaderESt17basic_string_viewIcSt11char_traitsIcEERKNS_14MachineOptionsE+0x1a3) [0x7c82c01b59e3]
  0x7c82c0166922: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZN7tinykvm7MachineC1ESt17basic_string_viewIcSt11char_traitsIcEERKNS_14MachineOptionsE+0x158) [0x7c82c0166922]
  0x7c82c0182dad: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZN3kvm15MachineInstanceC2ERKSt6vectorIhSaIhEEPK7vrt_ctxPKNS_14TenantInstanceEPNS_15ProgramInstanceEbb+0x1ed) [0x7c82c0182dad]
  0x7c82c018971a: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZN3kvm15ProgramInstance20begin_initializationEPK7vrt_ctxPNS_14TenantInstanceEb+0x13a) [0x7c82c018971a]
  0x7c82c0189f9b: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(+0x58f9b) [0x7c82c0189f9b]
  0x7c82c018a619: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_DeleterEEvENS1_12_Task_setterIS0_INS1_7_ResultIlEES3_EZNS1_11_Task_stateISt8functionIFlvEESaIiESC_E6_M_runEvEUlvE_lEEE9_M_invokeERKSt9_Any_data+0x29) [0x7c82c018a619]
  0x7c82c0170f6b: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZNSt13__future_base13_State_baseV29_M_do_setEPSt8functionIFSt10unique_ptrINS_12_Result_baseENS3_8_DeleterEEvEEPb+0x1b) [0x7c82c0170f6b]
  0x7c82cd9cd057: /lib/x86_64-linux-gnu/libc.so.6(+0x8e057) [0x7c82cd9cd057]
  0x7c82c018afc7: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZNSt13__future_base11_Task_stateISt8functionIFlvEESaIiES2_E6_M_runEv+0xe7) [0x7c82c018afc7]
  0x7c82c018e35a: ./vmod_cache/_vmod_tinykvm.73a4a91010514ccff9d7fdf3e67e1e9a6e132dc1a525c855b44769b14fbaae62(_ZZN7tinykvm10ThreadTaskISt8functionIFlvEEE12start_workerERKSt11unique_lockISt5mutexEENKUlvE_clEv+0x11a) [0x7c82c018e35a]
  0x7c82cd6d44a3: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3) [0x7c82cd6d44a3]
  0x7c82cd9c81f5: /lib/x86_64-linux-gnu/libc.so.6(+0x891f5) [0x7c82cd9c81f5]
  0x7c82cda4889c: /lib/x86_64-linux-gnu/libc.so.6(+0x10989c) [0x7c82cda4889c]
errno = 12 (Cannot allocate memory)
argv = {
  [0] = \"varnishd\",
  [1] = \"-F\",
  [2] = \"-f\",
  [3] = \"/mnt/default.vcl\",
  [4] = \"-a\",
  [5] = \"http=:8080,HTTP\",
  [6] = \"-a\",
  [7] = \"proxy=:8443,PROXY\",
  [8] = \"-p\",
  [9] = \"feature=+http2\",
  [10] = \"-s\",
  [11] = \"malloc,100M\",
}
pthread.self = 0x7c62333ff6c0
pthread.attr = {
  guard = 4096,
  stack_bottom = 0x7c6232c00000,
  stack_top = 0x7c6233400000,
  stack_size = 8388608,
}
thr.req = NULL
thr.busyobj = NULL
thr.worker = NULL
vmods = {
  tinykvm = {p=0x7c82cd22c800, abi=\"Varnish 7.7.0 bb7a918611ee85d17aefd07115fb8c66015151b5\", vrt=0.0,
    vcs=\"b32e4900c22c263794eae699f0bb05ef157ea9a0\", version=\"\\n\"
\"\"},
  std = {p=0x7c82cd22c880, abi=\"Varnish 7.7.0 bb7a918611ee85d17aefd07115fb8c66015151b5\", vrt=0.0,
    vcs=\"bb7a918611ee85d17aefd07115fb8c66015151b5\", version=\"Varnish 7.7.0\"},
},
pools = {
  pool = 0x7c82c6000000 {
    nidle = 96,
    nthr = 100,
    lqueue = 0
  },
  pool = 0x7c82c6000640 {
    nidle = 96,
    nthr = 100,
    lqueue = 0
  },
},


Debug: Child cleanup complete

[dridi]

Maybe you ran out of memory maps.