[CI] pytorch-rocm-llms / run-summarizer failed on stx (linux)

[github-actions]

This issue was opened automatically by the Test Playbooks workflow after the test run-summarizer failed on the main branch.

Failure scope

• Playbook: pytorch-rocm-llms
• Test id: run-summarizer
• Device: stx
• Operating system: linux
• Runner labels: self-hosted, Linux, stx
• Runner name: xsj-aimlab-stxp-01
• Commit: 36cd64215e5662a8880a9ae25b85e2f66edec256
• Workflow run: https://github.com/amd/playbooks/actions/runs/28559316323

Hardware / OS to use to reproduce

Run the failing test on a machine that matches the runner labels above (OS = linux, device = stx). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.

How to dispatch the same test from CI

Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:

gh workflow run test-playbooks.yml --repo amd/playbooks -f playbook_id=pytorch-rocm-llms

The workflow's matrix narrows down to this (device, platform) combination automatically based on the playbook's tested_platforms.

How to run just this test locally

python .github/scripts/run_playbook_tests.py --playbook pytorch-rocm-llms --platform linux --device stx

The runner extracts test blocks from playbooks/*/pytorch-rocm-llms/README.md (the failing block starts around line 460).

Failing test (verbatim from the README)

• Setup: source pytorch-env/bin/activate
• Timeout: 1000s

python summarizer.py --model Qwen/Qwen3.5-4B

Result

• Exit code: -6

stderr (last lines)

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Fetching 2 files:   0%|          | 0/2 [00:00<?, ?it/s]
Fetching 2 files: 100%|██████████| 2/2 [00:00<00:00, 2102.94it/s]

Loading weights:   0%|          | 0/426 [00:00<?, ?it/s]
Loading weights:   0%|          | 1/426 [00:00<02:18,  3.07it/s]
Loading weights: 100%|██████████| 426/426 [00:00<00:00, 1282.76it/s]
Some parameters are on the meta device because they were offloaded to the cpu.
MIOpen(HIP): Warning [ParseAndLoadDb] File is unreadable: "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx1150/share/miopen/db/gfx1150_8.HIP.fdb.txt"
/home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/transformers/integrations/sdpa_attention.py:92: UserWarning: Flash Efficient attention on Current AMD GPU is still experimental. Enable it with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1. (Triggered internally at /__w/rockrel/rockrel/external-builds/pytorch/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:323.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
/home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/transformers/integrations/sdpa_attention.py:92: UserWarning: Mem Efficient attention on Current AMD GPU is still experimental. Enable it with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1. (Triggered internally at /__w/rockrel/rockrel/external-builds/pytorch/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:383.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
:0:rocdevice.cpp            :3956: 104312010192 us:  Callback: Queue 0x7e0629600000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016
terminate called after throwing an instance of 'c10::AcceleratorError'
  what():  CUDA error: unspecified launch failure
Search for `hipErrorLaunchFailure' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Device-side assertion tracking was not enabled by user.
Exception raised from SetDevice at /__w/rockrel/rockrel/external-builds/pytorch/pytorch/c10/hip/HIPFunctions.cpp:334 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x7e08a52b605d in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x1293e (0x7e07974a493e in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libc10_hip.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x51 (0x7e07974e23b1 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libc10_hip.so)
frame #3: <unknown function> + 0x2ca43 (0x7e07974bea43 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libc10_hip.so)
frame #4: <unknown function> + 0xc63288 (0x7e0790063288 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_hip.so)
frame #5: at::_ops::_local_scalar_dense::redispatch(c10::DispatchKeySet, at::Tensor const&) + 0xde (0x7e0799ea35be in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x51d3f36 (0x7e079c7d3f36 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #7: at::_ops::_local_scalar_dense::call(at::Tensor const&) + 0x175 (0x7e0799f1ab65 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #8: at::native::item(at::Tensor const&) + 0x10a (0x7e079947dcca in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x30d9aec (0x7e079a6d9aec in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::_ops::item::call(at::Tensor const&) + 0x175 (0x7e0799d4a155 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #11: at::native::is_nonzero(at::Tensor const&) + 0x47 (0x7e079964d557 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::_ops::is_nonzero::call(at::Tensor const&) + 0x168 (0x7e0799f0d0d8 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x4650b5 (0x7e07a64650b5 in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #30: <unknown function> + 0x2a1ca (0x7e08dcc2a1ca in /lib/x86_64-linux-gnu/libc.so.6)
frame #31: __libc_start_main + 0x8b (0x7e08dcc2a28b in /lib/x86_64-linux-gnu/libc.so.6)

stdout (last lines)

Loading Qwen/Qwen3.5-4B...
[ERROR] `loss` is part of Qwen3_5CausalLMOutputWithPast.__init__'s signature, but not documented. Make sure to add it to the docstring of the function in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py.
[ERROR] `logits` is part of Qwen3_5CausalLMOutputWithPast.__init__'s signature, but not documented. Make sure to add it to the docstring of the function in /home/user/actions-runner/_work/playbooks/playbooks/playbooks/core/pytorch-rocm-llms/assets/pytorch-env/lib/python3.13/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py.
[OK] Model ready!

Using example document...

Generating summary...
Kernel Name: _ZN2at6native25_assert_async_cuda_kernelIbEEvPKT_NS0_3MsgE
VGPU=0x5b6177b2ab30 SWq=0x7e08a8910000, HWq=0x7e0629600000, id=1
	Dispatch Header =0xb02 (type=2, barrier=1, acquire=1, release=1), setup=0
	grid=[1, 1, 1], workgroup=[1, 1, 1]
	private_seg_size=0, group_seg_size=0
	kernel_obj=0x7e0601084a00, kernarg_address=0x0x7e0628a62480
	completion_signal=0x0, correlation_id=0
	rptr=294182, wptr=294212
 

---

This issue is opened and deduplicated by .github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.