check for nans in imatrix and quantize

[slaren]

Checks for nans during imatrix generation with imatrix, and also verifies the imatrix before quantization with quantize.

This doesn't fix the underlying issue, but it should reduce the amount of time wasted creating or using bad imatrices.

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 535 iterations 🚀

Expand details for performance related PR only

• Concurrent users: 8, duration: 10m
• HTTP request : avg=8772.52ms p(95)=21022.14ms fails=, finish reason: stop=467 truncated=68
• Prompt processing (pp): avg=98.04tk/s p(95)=464.9tk/s
• Token generation (tg): avg=32.15tk/s p(95)=47.52tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=sl/detect-imatrix-nan commit=d857e5192ec34b6b6b4c98e24baa738309faf97b

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 535 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717708658 --> 1717709290
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 302.38, 302.38, 302.38, 302.38, 302.38, 853.87, 853.87, 853.87, 853.87, 853.87, 865.43, 865.43, 865.43, 865.43, 865.43, 897.57, 897.57, 897.57, 897.57, 897.57, 950.87, 950.87, 950.87, 950.87, 950.87, 955.43, 955.43, 955.43, 955.43, 955.43, 945.98, 945.98, 945.98, 945.98, 945.98, 942.05, 942.05, 942.05, 942.05, 942.05, 944.85, 944.85, 944.85, 944.85, 944.85, 941.93, 941.93, 941.93, 941.93, 941.93, 958.57, 958.57, 958.57, 958.57, 958.57, 989.45, 989.45, 989.45, 989.45, 989.45, 983.26, 983.26, 983.26, 983.26, 983.26, 988.7, 988.7, 988.7, 988.7, 988.7, 985.41, 985.41, 985.41, 985.41, 985.41, 982.87, 982.87, 982.87, 982.87, 982.87, 980.36, 980.36, 980.36, 980.36, 980.36, 970.26, 970.26, 970.26, 970.26, 970.26, 966.32, 966.32, 966.32, 966.32, 966.32, 983.02, 983.02, 983.02, 983.02, 983.02, 983.32, 983.32, 983.32, 983.32, 983.32, 984.16, 984.16, 984.16, 984.16, 984.16, 982.59, 982.59, 982.59, 982.59, 982.59, 980.07, 980.07, 980.07, 980.07, 980.07, 988.04, 988.04, 988.04, 988.04, 988.04, 983.12, 983.12, 983.12, 983.12, 983.12, 982.58, 982.58, 982.58, 982.58, 982.58, 945.31, 945.31, 945.31, 945.31, 945.31, 941.8, 941.8, 941.8, 941.8, 941.8, 940.28, 940.28, 940.28, 940.28, 940.28, 938.26, 938.26, 938.26, 938.26, 938.26, 938.42, 938.42, 938.42, 938.42, 938.42, 934.53, 934.53, 934.53, 934.53, 934.53, 937.61, 937.61, 937.61, 937.61, 937.61, 937.35, 937.35, 937.35, 937.35, 937.35, 940.14, 940.14, 940.14, 940.14, 940.14, 942.2, 942.2, 942.2, 942.2, 942.2, 914.24, 914.24, 914.24, 914.24, 914.24, 911.86, 911.86, 911.86, 911.86, 911.86, 907.73, 907.73, 907.73, 907.73, 907.73, 909.96, 909.96, 909.96, 909.96, 909.96, 909.68, 909.68, 909.68, 909.68, 909.68, 909.38, 909.38, 909.38, 909.38, 909.38, 887.39, 887.39, 887.39, 887.39, 887.39, 886.38, 886.38, 886.38, 886.38, 886.38, 884.93, 884.93, 884.93, 884.93, 884.93, 881.35, 881.35, 881.35, 881.35, 881.35, 879.38, 879.38, 879.38, 879.38, 879.38, 868.01, 868.01, 868.01, 868.01, 868.01, 867.25, 867.25, 867.25, 867.25, 867.25, 869.88, 869.88, 869.88, 869.88, 869.88, 870.07, 870.07, 870.07, 870.07, 870.07, 870.64, 870.64, 870.64, 870.64, 870.64, 866.34, 866.34, 866.34, 866.34, 866.34, 873.96, 873.96, 873.96, 873.96, 873.96, 875.61, 875.61, 875.61, 875.61, 875.61, 875.04, 875.04, 875.04, 875.04, 875.04, 875.14, 875.14, 875.14, 875.14, 875.14, 874.62, 874.62, 874.62, 874.62, 874.62, 875.72, 875.72, 875.72, 875.72, 875.72, 875.49, 875.49, 875.49, 875.49, 875.49, 875.49, 875.49]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 535 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717708658 --> 1717709290
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 42.35, 42.35, 42.35, 42.35, 42.35, 29.03, 29.03, 29.03, 29.03, 29.03, 26.65, 26.65, 26.65, 26.65, 26.65, 29.33, 29.33, 29.33, 29.33, 29.33, 32.06, 32.06, 32.06, 32.06, 32.06, 32.02, 32.02, 32.02, 32.02, 32.02, 33.38, 33.38, 33.38, 33.38, 33.38, 34.06, 34.06, 34.06, 34.06, 34.06, 34.71, 34.71, 34.71, 34.71, 34.71, 35.09, 35.09, 35.09, 35.09, 35.09, 35.1, 35.1, 35.1, 35.1, 35.1, 34.3, 34.3, 34.3, 34.3, 34.3, 33.69, 33.69, 33.69, 33.69, 33.69, 33.63, 33.63, 33.63, 33.63, 33.63, 31.62, 31.62, 31.62, 31.62, 31.62, 30.99, 30.99, 30.99, 30.99, 30.99, 28.98, 28.98, 28.98, 28.98, 28.98, 28.94, 28.94, 28.94, 28.94, 28.94, 29.08, 29.08, 29.08, 29.08, 29.08, 29.08, 29.08, 29.08, 29.08, 29.08, 29.19, 29.19, 29.19, 29.19, 29.19, 29.43, 29.43, 29.43, 29.43, 29.43, 29.46, 29.46, 29.46, 29.46, 29.46, 29.71, 29.71, 29.71, 29.71, 29.71, 29.7, 29.7, 29.7, 29.7, 29.7, 29.6, 29.6, 29.6, 29.6, 29.6, 29.74, 29.74, 29.74, 29.74, 29.74, 29.92, 29.92, 29.92, 29.92, 29.92, 29.98, 29.98, 29.98, 29.98, 29.98, 30.07, 30.07, 30.07, 30.07, 30.07, 30.45, 30.45, 30.45, 30.45, 30.45, 30.52, 30.52, 30.52, 30.52, 30.52, 30.68, 30.68, 30.68, 30.68, 30.68, 30.83, 30.83, 30.83, 30.83, 30.83, 30.85, 30.85, 30.85, 30.85, 30.85, 30.7, 30.7, 30.7, 30.7, 30.7, 30.46, 30.46, 30.46, 30.46, 30.46, 30.21, 30.21, 30.21, 30.21, 30.21, 30.21, 30.21, 30.21, 30.21, 30.21, 30.29, 30.29, 30.29, 30.29, 30.29, 30.5, 30.5, 30.5, 30.5, 30.5, 30.58, 30.58, 30.58, 30.58, 30.58, 30.71, 30.71, 30.71, 30.71, 30.71, 30.55, 30.55, 30.55, 30.55, 30.55, 30.18, 30.18, 30.18, 30.18, 30.18, 30.16, 30.16, 30.16, 30.16, 30.16, 28.92, 28.92, 28.92, 28.92, 28.92, 28.89, 28.89, 28.89, 28.89, 28.89, 28.89, 28.89, 28.89, 28.89, 28.89, 28.91, 28.91, 28.91, 28.91, 28.91, 28.94, 28.94, 28.94, 28.94, 28.94, 28.95, 28.95, 28.95, 28.95, 28.95, 29.03, 29.03, 29.03, 29.03, 29.03, 28.93, 28.93, 28.93, 28.93, 28.93, 28.95, 28.95, 28.95, 28.95, 28.95, 28.92, 28.92, 28.92, 28.92, 28.92, 29.01, 29.01, 29.01, 29.01, 29.01, 29.11, 29.11, 29.11, 29.11, 29.11, 29.21, 29.21, 29.21, 29.21, 29.21, 29.27, 29.27, 29.27, 29.27, 29.27, 29.3, 29.3, 29.3, 29.3, 29.3, 29.37, 29.37]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 535 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717708658 --> 1717709290
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.13, 0.13, 0.13, 0.13, 0.13, 0.38, 0.38, 0.38, 0.38, 0.38, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.2, 0.2, 0.2, 0.2, 0.2, 0.19, 0.19, 0.19, 0.19, 0.19, 0.1, 0.1, 0.1, 0.1, 0.1, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.24, 0.24, 0.24, 0.24, 0.24, 0.34, 0.34, 0.34, 0.34, 0.34, 0.3, 0.3, 0.3, 0.3, 0.3, 0.47, 0.47, 0.47, 0.47, 0.47, 0.39, 0.39, 0.39, 0.39, 0.39, 0.44, 0.44, 0.44, 0.44, 0.44, 0.27, 0.27, 0.27, 0.27, 0.27, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.26, 0.26, 0.26, 0.26, 0.26, 0.14, 0.14, 0.14, 0.14, 0.14, 0.23, 0.23, 0.23, 0.23, 0.23, 0.17, 0.17, 0.17, 0.17, 0.17, 0.11, 0.11, 0.11, 0.11, 0.11, 0.34, 0.34, 0.34, 0.34, 0.34, 0.17, 0.17, 0.17, 0.17, 0.17, 0.14, 0.14, 0.14, 0.14, 0.14, 0.27, 0.27, 0.27, 0.27, 0.27, 0.16, 0.16, 0.16, 0.16, 0.16, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.29, 0.29, 0.29, 0.29, 0.29, 0.26, 0.26, 0.26, 0.26, 0.26, 0.33, 0.33, 0.33, 0.33, 0.33, 0.26, 0.26, 0.26, 0.26, 0.26, 0.2, 0.2, 0.2, 0.2, 0.2, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.29, 0.29, 0.29, 0.29, 0.29, 0.54, 0.54, 0.54, 0.54, 0.54, 0.49, 0.49, 0.49, 0.49, 0.49, 0.58, 0.58, 0.58, 0.58, 0.58, 0.15, 0.15, 0.15, 0.15, 0.15, 0.21, 0.21, 0.21, 0.21, 0.21, 0.26, 0.26, 0.26, 0.26, 0.26, 0.24, 0.24, 0.24, 0.24, 0.24, 0.16, 0.16, 0.16, 0.16, 0.16, 0.22, 0.22, 0.22, 0.22, 0.22, 0.21, 0.21, 0.21, 0.21, 0.21, 0.18, 0.18, 0.18, 0.18, 0.18, 0.26, 0.26, 0.26, 0.26, 0.26, 0.12, 0.12, 0.12, 0.12, 0.12, 0.19, 0.19, 0.19, 0.19, 0.19, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.2, 0.2, 0.2, 0.2, 0.2, 0.21, 0.21, 0.21, 0.21, 0.21, 0.29, 0.29]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 535 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717708658 --> 1717709290
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 1.0, 1.0, 1.0, 1.0, 1.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0, 1.0]
                    

Loading