server : return error on too large embedding input

[ggerganov]

fix #7277

The input for embeddings must fit inside the physical batch size. If it does not, we now return an error:

$ ▶ curl -X POST "http://localhost:8080/embedding" --data '{"content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam rhoncus mauris eget magna semper, ut varius arcu eleifend. Vestibulum quis justo eget ex pretium sollicitudin. Nam euismod orci vulputate erat sagittis, sed pulvinar ante varius. Proin in dui non eros sodales tempus. Proin et mi scelerisque tellus eleifend auctor. Sed sagittis erat sapien, in porttitor augue bibendum nec. Nam ut mi accumsan lorem volutpat tempus. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. In ac nulla tempor, pharetra felis id, venenatis tortor. Donec felis turpis, egestas non ligula at, eleifend fringilla est. Fusce elit mi, fermentum a sapien eleifend, rutrum scelerisque eros. Sed et vestibulum orci. Quisque ut magna vel nibh accumsan dictum eget eu urna. Duis rhoncus, lacus in imperdiet tincidunt, turpis turpis vestibulum ante, at mollis nisi massa et purus. Phasellus sed ante eros. Aenean consequat nisi non massa eleifend finibus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce ultrices libero id metus consequat semper. Nam venenatis, est quis interdum commodo, nisl ex placerat diam, sed fringilla ex nisi sed sem. Pellentesque luctus orci id tellus dictum tristique. Integer molestie varius risus quis maximus. In id feugiat nulla, at scelerisque massa. Nulla neque diam, consequat ac orci laoreet, venenatis pharetra enim.Aenean rhoncus dapibus augue ac volutpat. Nullam laoreet, lorem quis fermentum scelerisque,"}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1613  100   120  100  1493  32795   398k --:--:-- --:--:-- --:--:--  525k
{
  "error": {
    "code": 500,
    "message": "input is too large to process. increase the physical batch size",
    "type": "server_error"
  }
}

[JohannesGaessler]

Definitely an improvement over master.

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 533 iterations 🚀

Expand details for performance related PR only

• Concurrent users: 8, duration: 10m
• HTTP request : avg=8754.75ms p(95)=20897.88ms fails=, finish reason: stop=477 truncated=56
• Prompt processing (pp): avg=99.93tk/s p(95)=440.6tk/s
• Token generation (tg): avg=51.45tk/s p(95)=48.05tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/server-embd-err commit=85137247b2de79aeede8cf0a8dbd8be9ddc0bafb

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 304.83, 304.83, 304.83, 304.83, 304.83, 730.64, 730.64, 730.64, 730.64, 730.64, 728.45, 728.45, 728.45, 728.45, 728.45, 788.91, 788.91, 788.91, 788.91, 788.91, 848.64, 848.64, 848.64, 848.64, 848.64, 840.83, 840.83, 840.83, 840.83, 840.83, 842.84, 842.84, 842.84, 842.84, 842.84, 843.04, 843.04, 843.04, 843.04, 843.04, 856.74, 856.74, 856.74, 856.74, 856.74, 856.72, 856.72, 856.72, 856.72, 856.72, 874.51, 874.51, 874.51, 874.51, 874.51, 857.16, 857.16, 857.16, 857.16, 857.16, 851.66, 851.66, 851.66, 851.66, 851.66, 874.92, 874.92, 874.92, 874.92, 874.92, 882.05, 882.05, 882.05, 882.05, 882.05, 888.67, 888.67, 888.67, 888.67, 888.67, 885.46, 885.46, 885.46, 885.46, 885.46, 891.95, 891.95, 891.95, 891.95, 891.95, 899.53, 899.53, 899.53, 899.53, 899.53, 899.66, 899.66, 899.66, 899.66, 899.66, 903.84, 903.84, 903.84, 903.84, 903.84, 905.83, 905.83, 905.83, 905.83, 905.83, 903.78, 903.78, 903.78, 903.78, 903.78, 845.62, 845.62, 845.62, 845.62, 845.62, 847.34, 847.34, 847.34, 847.34, 847.34, 846.87, 846.87, 846.87, 846.87, 846.87, 840.47, 840.47, 840.47, 840.47, 840.47, 835.94, 835.94, 835.94, 835.94, 835.94, 836.87, 836.87, 836.87, 836.87, 836.87, 837.41, 837.41, 837.41, 837.41, 837.41, 841.62, 841.62, 841.62, 841.62, 841.62, 843.04, 843.04, 843.04, 843.04, 843.04, 840.4, 840.4, 840.4, 840.4, 840.4, 844.78, 844.78, 844.78, 844.78, 844.78, 857.29, 857.29, 857.29, 857.29, 857.29, 862.24, 862.24, 862.24, 862.24, 862.24, 862.47, 862.47, 862.47, 862.47, 862.47, 856.11, 856.11, 856.11, 856.11, 856.11, 854.2, 854.2, 854.2, 854.2, 854.2, 853.46, 853.46, 853.46, 853.46, 853.46, 857.91, 857.91, 857.91, 857.91, 857.91, 857.74, 857.74, 857.74, 857.74, 857.74, 851.36, 851.36, 851.36, 851.36, 851.36, 820.63, 820.63, 820.63, 820.63, 820.63, 821.75, 821.75, 821.75, 821.75, 821.75, 821.1, 821.1, 821.1, 821.1, 821.1, 818.13, 818.13, 818.13, 818.13, 818.13, 823.1, 823.1, 823.1, 823.1, 823.1, 825.71, 825.71, 825.71, 825.71, 825.71, 825.13, 825.13, 825.13, 825.13, 825.13, 824.86, 824.86, 824.86, 824.86, 824.86, 828.22, 828.22, 828.22, 828.22, 828.22, 826.37, 826.37, 826.37, 826.37, 826.37, 833.99, 833.99, 833.99, 833.99, 833.99, 832.5, 832.5, 832.5, 832.5, 832.5, 837.94, 837.94, 837.94, 837.94, 837.94, 841.42, 841.42, 841.42, 841.42, 841.42, 841.15, 841.15, 841.15, 841.15, 841.15, 842.28, 842.28, 842.28, 842.28, 842.28, 842.32, 842.32, 842.32, 842.32, 842.32, 843.18]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 41.67, 41.67, 41.67, 41.67, 41.67, 25.86, 25.86, 25.86, 25.86, 25.86, 27.65, 27.65, 27.65, 27.65, 27.65, 32.44, 32.44, 32.44, 32.44, 32.44, 31.54, 31.54, 31.54, 31.54, 31.54, 31.93, 31.93, 31.93, 31.93, 31.93, 33.11, 33.11, 33.11, 33.11, 33.11, 33.61, 33.61, 33.61, 33.61, 33.61, 34.12, 34.12, 34.12, 34.12, 34.12, 34.11, 34.11, 34.11, 34.11, 34.11, 33.51, 33.51, 33.51, 33.51, 33.51, 33.5, 33.5, 33.5, 33.5, 33.5, 33.35, 33.35, 33.35, 33.35, 33.35, 33.17, 33.17, 33.17, 33.17, 33.17, 30.75, 30.75, 30.75, 30.75, 30.75, 30.35, 30.35, 30.35, 30.35, 30.35, 30.44, 30.44, 30.44, 30.44, 30.44, 30.7, 30.7, 30.7, 30.7, 30.7, 30.1, 30.1, 30.1, 30.1, 30.1, 29.96, 29.96, 29.96, 29.96, 29.96, 30.06, 30.06, 30.06, 30.06, 30.06, 30.09, 30.09, 30.09, 30.09, 30.09, 30.04, 30.04, 30.04, 30.04, 30.04, 30.02, 30.02, 30.02, 30.02, 30.02, 30.05, 30.05, 30.05, 30.05, 30.05, 30.17, 30.17, 30.17, 30.17, 30.17, 30.39, 30.39, 30.39, 30.39, 30.39, 30.22, 30.22, 30.22, 30.22, 30.22, 30.3, 30.3, 30.3, 30.3, 30.3, 30.48, 30.48, 30.48, 30.48, 30.48, 30.64, 30.64, 30.64, 30.64, 30.64, 30.71, 30.71, 30.71, 30.71, 30.71, 30.8, 30.8, 30.8, 30.8, 30.8, 30.95, 30.95, 30.95, 30.95, 30.95, 31.02, 31.02, 31.02, 31.02, 31.02, 30.85, 30.85, 30.85, 30.85, 30.85, 30.59, 30.59, 30.59, 30.59, 30.59, 30.29, 30.29, 30.29, 30.29, 30.29, 30.39, 30.39, 30.39, 30.39, 30.39, 30.52, 30.52, 30.52, 30.52, 30.52, 30.7, 30.7, 30.7, 30.7, 30.7, 30.72, 30.72, 30.72, 30.72, 30.72, 30.92, 30.92, 30.92, 30.92, 30.92, 30.89, 30.89, 30.89, 30.89, 30.89, 30.65, 30.65, 30.65, 30.65, 30.65, 30.15, 30.15, 30.15, 30.15, 30.15, 29.05, 29.05, 29.05, 29.05, 29.05, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 29.02, 28.95, 28.95, 28.95, 28.95, 28.95, 28.91, 28.91, 28.91, 28.91, 28.91, 28.9, 28.9, 28.9, 28.9, 28.9, 29.03, 29.03, 29.03, 29.03, 29.03, 29.04, 29.04, 29.04, 29.04, 29.04, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.95, 28.94, 28.94, 28.94, 28.94, 28.94, 29.09, 29.09, 29.09, 29.09, 29.09, 29.23, 29.23, 29.23, 29.23, 29.23, 29.25, 29.25, 29.25, 29.25, 29.25, 29.36]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.11, 0.11, 0.11, 0.11, 0.11, 0.39, 0.39, 0.39, 0.39, 0.39, 0.09, 0.09, 0.09, 0.09, 0.09, 0.13, 0.13, 0.13, 0.13, 0.13, 0.24, 0.24, 0.24, 0.24, 0.24, 0.22, 0.22, 0.22, 0.22, 0.22, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.32, 0.32, 0.32, 0.32, 0.32, 0.31, 0.31, 0.31, 0.31, 0.31, 0.25, 0.25, 0.25, 0.25, 0.25, 0.37, 0.37, 0.37, 0.37, 0.37, 0.24, 0.24, 0.24, 0.24, 0.24, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.27, 0.27, 0.27, 0.27, 0.27, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 0.24, 0.24, 0.24, 0.24, 0.24, 0.21, 0.21, 0.21, 0.21, 0.21, 0.29, 0.29, 0.29, 0.29, 0.29, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.34, 0.34, 0.34, 0.34, 0.34, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.17, 0.17, 0.17, 0.17, 0.17, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.29, 0.29, 0.29, 0.29, 0.29, 0.19, 0.19, 0.19, 0.19, 0.19, 0.24, 0.24, 0.24, 0.24, 0.24, 0.23, 0.23, 0.23, 0.23, 0.23, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.31, 0.31, 0.31, 0.31, 0.31, 0.52, 0.52, 0.52, 0.52, 0.52, 0.53, 0.53, 0.53, 0.53, 0.53, 0.51, 0.51, 0.51, 0.51, 0.51, 0.23, 0.23, 0.23, 0.23, 0.23, 0.16, 0.16, 0.16, 0.16, 0.16, 0.24, 0.24, 0.24, 0.24, 0.24, 0.25, 0.25, 0.25, 0.25, 0.25, 0.12, 0.12, 0.12, 0.12, 0.12, 0.19, 0.19, 0.19, 0.19, 0.19, 0.12, 0.12, 0.12, 0.12, 0.12, 0.36, 0.36, 0.36, 0.36, 0.36, 0.19, 0.19, 0.19, 0.19, 0.19, 0.2, 0.2, 0.2, 0.2, 0.2, 0.12, 0.12, 0.12, 0.12, 0.12, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.21]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 533 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1716156379 --> 1716156999
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0]
                    

Loading