ci: bench: fix master not schedule, fix commit status failed on external repo #6365

phymbert · 2024-03-28T10:14:02Z

Context

master not scheduled: https://github.com/ggerganov/llama.cpp/actions/runs/8464726825
fail add commit status on external repo: https://github.com/ggerganov/llama.cpp/actions/runs/8465507726

…nal repo

github-actions · 2024-03-28T10:28:55Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3: 499 iterations 🚀

Concurrent users: 8, duration: 10m
HTTP request : avg=9386.25ms p(90)=25860.82ms fails=0, finish reason: stop=499 truncated=0
Prompt processing (pp): avg=243.1tk/s p(90)=734.8tk/s total=198.68tk/s
Token generation (tg): avg=98.44tk/s p(90)=265.24tk/s total=129.61tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=hp/hotfix/ci/bench commit=166b807a15cb2907742fe7a486f608e3ffa73700

Time series

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 499 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1711621104 --> 1711621728
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 368.39, 368.39, 368.39, 368.39, 368.39, 652.82, 652.82, 652.82, 652.82, 652.82, 653.21, 653.21, 653.21, 653.21, 653.21, 658.82, 658.82, 658.82, 658.82, 658.82, 698.81, 698.81, 698.81, 698.81, 698.81, 697.24, 697.24, 697.24, 697.24, 697.24, 699.28, 699.28, 699.28, 699.28, 699.28, 714.25, 714.25, 714.25, 714.25, 714.25, 713.3, 713.3, 713.3, 713.3, 713.3, 718.67, 718.67, 718.67, 718.67, 718.67, 717.21, 717.21, 717.21, 717.21, 717.21, 721.73, 721.73, 721.73, 721.73, 721.73, 724.52, 724.52, 724.52, 724.52, 724.52, 742.79, 742.79, 742.79, 742.79, 742.79, 761.24, 761.24, 761.24, 761.24, 761.24, 708.71, 708.71, 708.71, 708.71, 708.71, 715.86, 715.86, 715.86, 715.86, 715.86, 714.18, 714.18, 714.18, 714.18, 714.18, 712.12, 712.12, 712.12, 712.12, 712.12, 726.6, 726.6, 726.6, 726.6, 726.6, 725.04, 725.04, 725.04, 725.04, 725.04, 723.17, 723.17, 723.17, 723.17, 723.17, 724.86, 724.86, 724.86, 724.86, 724.86, 724.75, 724.75, 724.75, 724.75, 724.75, 726.86, 726.86, 726.86, 726.86, 726.86, 738.94, 738.94, 738.94, 738.94, 738.94, 738.6, 738.6, 738.6, 738.6, 738.6, 739.53, 739.53, 739.53, 739.53, 739.53, 745.3, 745.3, 745.3, 745.3, 745.3, 743.98, 743.98, 743.98, 743.98, 743.98, 743.82, 743.82, 743.82, 743.82, 743.82, 746.04, 746.04, 746.04, 746.04, 746.04, 746.4, 746.4, 746.4, 746.4, 746.4, 744.7, 744.7, 744.7, 744.7, 744.7, 747.06, 747.06, 747.06, 747.06, 747.06, 750.53, 750.53, 750.53, 750.53, 750.53, 755.68, 755.68, 755.68, 755.68, 755.68, 755.99, 755.99, 755.99, 755.99, 755.99, 741.31, 741.31, 741.31, 741.31, 741.31, 739.53, 739.53, 739.53, 739.53, 739.53, 737.94, 737.94, 737.94, 737.94, 737.94, 740.23, 740.23, 740.23, 740.23, 740.23, 739.88, 739.88, 739.88, 739.88, 739.88, 743.05, 743.05, 743.05, 743.05, 743.05, 741.18, 741.18, 741.18, 741.18, 741.18, 736.12, 736.12, 736.12, 736.12, 736.12, 732.98, 732.98, 732.98, 732.98, 732.98, 729.97, 729.97, 729.97, 729.97, 729.97, 728.37, 728.37, 728.37, 728.37, 728.37, 725.54, 725.54, 725.54, 725.54, 725.54, 727.79, 727.79, 727.79, 727.79, 727.79, 729.98, 729.98, 729.98, 729.98, 729.98, 728.56, 728.56, 728.56, 728.56, 728.56, 732.2, 732.2, 732.2, 732.2, 732.2, 734.41, 734.41, 734.41, 734.41, 734.41, 735.35, 735.35, 735.35, 735.35, 735.35, 726.34, 726.34, 726.34, 726.34, 726.34, 726.37, 726.37, 726.37, 726.37, 726.37, 729.15, 729.15, 729.15, 729.15, 729.15, 730.08, 730.08, 730.08, 730.08, 730.08, 729.92, 729.92, 729.92]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 499 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1711621104 --> 1711621728
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 29.64, 29.64, 29.64, 29.64, 29.64, 20.72, 20.72, 20.72, 20.72, 20.72, 17.04, 17.04, 17.04, 17.04, 17.04, 17.27, 17.27, 17.27, 17.27, 17.27, 17.16, 17.16, 17.16, 17.16, 17.16, 17.44, 17.44, 17.44, 17.44, 17.44, 18.56, 18.56, 18.56, 18.56, 18.56, 18.86, 18.86, 18.86, 18.86, 18.86, 19.03, 19.03, 19.03, 19.03, 19.03, 19.23, 19.23, 19.23, 19.23, 19.23, 19.25, 19.25, 19.25, 19.25, 19.25, 19.17, 19.17, 19.17, 19.17, 19.17, 18.95, 18.95, 18.95, 18.95, 18.95, 18.9, 18.9, 18.9, 18.9, 18.9, 18.6, 18.6, 18.6, 18.6, 18.6, 18.16, 18.16, 18.16, 18.16, 18.16, 17.85, 17.85, 17.85, 17.85, 17.85, 17.86, 17.86, 17.86, 17.86, 17.86, 18.0, 18.0, 18.0, 18.0, 18.0, 18.04, 18.04, 18.04, 18.04, 18.04, 17.97, 17.97, 17.97, 17.97, 17.97, 17.88, 17.88, 17.88, 17.88, 17.88, 17.82, 17.82, 17.82, 17.82, 17.82, 17.87, 17.87, 17.87, 17.87, 17.87, 17.99, 17.99, 17.99, 17.99, 17.99, 17.92, 17.92, 17.92, 17.92, 17.92, 17.98, 17.98, 17.98, 17.98, 17.98, 18.06, 18.06, 18.06, 18.06, 18.06, 18.11, 18.11, 18.11, 18.11, 18.11, 18.07, 18.07, 18.07, 18.07, 18.07, 18.13, 18.13, 18.13, 18.13, 18.13, 18.25, 18.25, 18.25, 18.25, 18.25, 18.3, 18.3, 18.3, 18.3, 18.3, 18.43, 18.43, 18.43, 18.43, 18.43, 18.51, 18.51, 18.51, 18.51, 18.51, 18.57, 18.57, 18.57, 18.57, 18.57, 18.5, 18.5, 18.5, 18.5, 18.5, 18.49, 18.49, 18.49, 18.49, 18.49, 18.36, 18.36, 18.36, 18.36, 18.36, 18.34, 18.34, 18.34, 18.34, 18.34, 18.36, 18.36, 18.36, 18.36, 18.36, 18.4, 18.4, 18.4, 18.4, 18.4, 18.44, 18.44, 18.44, 18.44, 18.44, 18.38, 18.38, 18.38, 18.38, 18.38, 18.35, 18.35, 18.35, 18.35, 18.35, 18.24, 18.24, 18.24, 18.24, 18.24, 17.93, 17.93, 17.93, 17.93, 17.93, 17.88, 17.88, 17.88, 17.88, 17.88, 17.84, 17.84, 17.84, 17.84, 17.84, 17.41, 17.41, 17.41, 17.41, 17.41, 17.23, 17.23, 17.23, 17.23, 17.23, 17.22, 17.22, 17.22, 17.22, 17.22, 17.31, 17.31, 17.31, 17.31, 17.31, 17.34, 17.34, 17.34, 17.34, 17.34, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.43, 17.44, 17.44, 17.44, 17.44, 17.44, 17.46, 17.46, 17.46, 17.46, 17.46, 17.46, 17.46, 17.46, 17.46, 17.46, 17.45, 17.45, 17.45, 17.45, 17.45, 17.51, 17.51, 17.51]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 499 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1711621104 --> 1711621728
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.07, 0.07, 0.07, 0.07, 0.07, 0.28, 0.28, 0.28, 0.28, 0.28, 0.21, 0.21, 0.21, 0.21, 0.21, 0.23, 0.23, 0.23, 0.23, 0.23, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.2, 0.2, 0.2, 0.2, 0.2, 0.21, 0.21, 0.21, 0.21, 0.21, 0.24, 0.24, 0.24, 0.24, 0.24, 0.27, 0.27, 0.27, 0.27, 0.27, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.22, 0.22, 0.22, 0.22, 0.22, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.25, 0.25, 0.25, 0.25, 0.25, 0.29, 0.29, 0.29, 0.29, 0.29, 0.22, 0.22, 0.22, 0.22, 0.22, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.27, 0.27, 0.27, 0.27, 0.27, 0.11, 0.11, 0.11, 0.11, 0.11, 0.17, 0.17, 0.17, 0.17, 0.17, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.19, 0.19, 0.19, 0.19, 0.19, 0.22, 0.22, 0.22, 0.22, 0.22, 0.18, 0.18, 0.18, 0.18, 0.18, 0.13, 0.13, 0.13, 0.13, 0.13, 0.17, 0.17, 0.17, 0.17, 0.17, 0.22, 0.22, 0.22, 0.22, 0.22, 0.1, 0.1, 0.1, 0.1, 0.1, 0.13, 0.13, 0.13, 0.13, 0.13, 0.4, 0.4, 0.4, 0.4, 0.4, 0.45, 0.45, 0.45, 0.45, 0.45, 0.55, 0.55, 0.55, 0.55, 0.55, 0.48, 0.48, 0.48, 0.48, 0.48, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.52, 0.26, 0.26, 0.26, 0.26, 0.26, 0.13, 0.13, 0.13, 0.13, 0.13, 0.16, 0.16, 0.16, 0.16, 0.16, 0.1, 0.1, 0.1, 0.1, 0.1, 0.11, 0.11, 0.11, 0.11, 0.11, 0.17, 0.17, 0.17, 0.17, 0.17, 0.25, 0.25, 0.25, 0.25, 0.25, 0.18, 0.18, 0.18, 0.18, 0.18, 0.2, 0.2, 0.2, 0.2, 0.2, 0.1, 0.1, 0.1, 0.1, 0.1, 0.16, 0.16, 0.16, 0.16, 0.16, 0.18, 0.18, 0.18]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 499 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1711621104 --> 1711621728
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0]

…nal repo (ggerganov#6365)

phymbert requested a review from ggerganov March 28, 2024 10:14

ci: bench: fix master not schedule, fix commit status failed on exter…

166b807

…nal repo

phymbert force-pushed the hp/hotfix/ci/bench branch from 7824ff4 to 166b807 Compare March 28, 2024 10:14

ggerganov approved these changes Mar 28, 2024

View reviewed changes

phymbert merged commit 28cb9a0 into master Mar 28, 2024
23 of 24 checks passed

phymbert deleted the hp/hotfix/ci/bench branch March 28, 2024 10:28

hodlen pushed a commit to hodlen/llama.cpp that referenced this pull request Apr 1, 2024

ci: bench: fix master not schedule, fix commit status failed on exter…

8dc38b1

…nal repo (ggerganov#6365)

hodlen pushed a commit to hodlen/llama.cpp that referenced this pull request Apr 3, 2024

ci: bench: fix master not schedule, fix commit status failed on exter…

c61f3cc

…nal repo (ggerganov#6365)

tybalex pushed a commit to tybalex/function.cpp that referenced this pull request Apr 17, 2024

ci: bench: fix master not schedule, fix commit status failed on exter…

7acd485

…nal repo (ggerganov#6365)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: bench: fix master not schedule, fix commit status failed on external repo #6365

ci: bench: fix master not schedule, fix commit status failed on external repo #6365

phymbert commented Mar 28, 2024

github-actions bot commented Mar 28, 2024

ci: bench: fix master not schedule, fix commit status failed on external repo #6365

ci: bench: fix master not schedule, fix commit status failed on external repo #6365

Conversation

phymbert commented Mar 28, 2024

Context

github-actions bot commented Mar 28, 2024