[do not merge][CB] Reduce wastage in prefill compute and pad blocks in homogeneous continuous batching

[yannicks1]

solves #255

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[yannicks1]

great news: This runs on Spyre 🎉

I just ran cb_spyre_inference.py which (with the parameters on this branch) exploits all functionality:

• prefill seq 1 of size 128 (left pads)
• prefill seq 2 of size 64 (left pads) (see [CB] Reduce wastage in prefill compute and pad blocks in homogeneous continuous batching #255)
• decode of batch 2
• strips fully padded blocks (see [CB] strip repeated left padding on batch level #131 )
• prefill seq 3 by left padding to 66 (align tkv) and right padding to 128 (pad to block boundary)
• decode of batch 2
• decode of batch 1 (see [CB] add min batch size of 2 in decode #182)

cc: @tdoublep @JRosenkranz @joerunde @nikolaospapandreou @sducouedic

[yannicks1]

bot:test
TEST_FILE=tests/e2e/test_spyre_cb.py MARKERS="spyre"

[yannicks1]

bot:test
TEST_FILE=tests/e2e/test_spyre_cb.py MARKERS="spyre"

[yannicks1]

bot:test
TEST_FILE=tests/e2e/test_spyre_cb.py MARKERS="spyre"

[yannicks1]

6/7 tests passed on the Spyre card! looks like the failure is a known issue unrelated to this PR. 🥳