Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MiniCPM Support lm_head #7664

Merged
merged 3 commits into from
Jun 3, 2024
Merged

MiniCPM Support lm_head #7664

merged 3 commits into from
Jun 3, 2024

Conversation

zkh2016
Copy link
Contributor

@zkh2016 zkh2016 commented May 31, 2024

No description provided.

@github-actions github-actions bot added the python python script changes label May 31, 2024
@mofosyne mofosyne added the Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix label May 31, 2024
@zkh2016 zkh2016 changed the title support lm_head MiniCPM Support lm_head May 31, 2024
jpohhhh
jpohhhh reviewed May 31, 2024
View reviewed changes
llama.cpp Outdated
@@ -4941,7 +4941,8 @@ static bool llm_load_tensors(
// output
{
model.output_norm = ml.create_tensor(ctx_output, tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd});
if (model.arch != LLM_ARCH_MINICPM){
//if (model.arch != LLM_ARCH_MINICPM){
{
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be commented out?
If so, should the code block removed? (it now has no if)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this if is not needed.

@ggerganov ggerganov merged commit 6f28a33 into ggerganov:master Jun 3, 2024
5 of 9 checks passed
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 524 iterations 🚀

Expand details for performance related PR only
  • Concurrent users: 8, duration: 10m
  • HTTP request : avg=8933.87ms p(95)=22254.64ms fails=, finish reason: stop=464 truncated=60
  • Prompt processing (pp): avg=105.11tk/s p(95)=482.81tk/s
  • Token generation (tg): avg=34.36tk/s p(95)=48.61tk/s
  • ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=support_lm_head commit=5d19aa8250bdeec260a412fe8c3ad976c83b656d

prompt_tokens_seconds

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 524 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1717442609 --> 1717443231
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 463.12, 463.12, 463.12, 463.12, 463.12, 878.4, 878.4, 878.4, 878.4, 878.4, 796.09, 796.09, 796.09, 796.09, 796.09, 842.86, 842.86, 842.86, 842.86, 842.86, 903.9, 903.9, 903.9, 903.9, 903.9, 896.85, 896.85, 896.85, 896.85, 896.85, 904.25, 904.25, 904.25, 904.25, 904.25, 915.21, 915.21, 915.21, 915.21, 915.21, 907.46, 907.46, 907.46, 907.46, 907.46, 913.91, 913.91, 913.91, 913.91, 913.91, 931.61, 931.61, 931.61, 931.61, 931.61, 925.59, 925.59, 925.59, 925.59, 925.59, 911.11, 911.11, 911.11, 911.11, 911.11, 845.96, 845.96, 845.96, 845.96, 845.96, 851.08, 851.08, 851.08, 851.08, 851.08, 851.83, 851.83, 851.83, 851.83, 851.83, 854.01, 854.01, 854.01, 854.01, 854.01, 836.42, 836.42, 836.42, 836.42, 836.42, 835.45, 835.45, 835.45, 835.45, 835.45, 838.56, 838.56, 838.56, 838.56, 838.56, 821.75, 821.75, 821.75, 821.75, 821.75, 820.36, 820.36, 820.36, 820.36, 820.36, 827.01, 827.01, 827.01, 827.01, 827.01, 822.75, 822.75, 822.75, 822.75, 822.75, 823.35, 823.35, 823.35, 823.35, 823.35, 822.44, 822.44, 822.44, 822.44, 822.44, 825.16, 825.16, 825.16, 825.16, 825.16, 840.15, 840.15, 840.15, 840.15, 840.15, 832.88, 832.88, 832.88, 832.88, 832.88, 832.2, 832.2, 832.2, 832.2, 832.2, 832.02, 832.02, 832.02, 832.02, 832.02, 837.37, 837.37, 837.37, 837.37, 837.37, 837.95, 837.95, 837.95, 837.95, 837.95, 835.17, 835.17, 835.17, 835.17, 835.17, 839.93, 839.93, 839.93, 839.93, 839.93, 831.48, 831.48, 831.48, 831.48, 831.48, 832.97, 832.97, 832.97, 832.97, 832.97, 836.61, 836.61, 836.61, 836.61, 836.61, 835.85, 835.85, 835.85, 835.85, 835.85, 832.61, 832.61, 832.61, 832.61, 832.61, 835.66, 835.66, 835.66, 835.66, 835.66, 837.56, 837.56, 837.56, 837.56, 837.56, 837.61, 837.61, 837.61, 837.61, 837.61, 820.57, 820.57, 820.57, 820.57, 820.57, 812.87, 812.87, 812.87, 812.87, 812.87, 812.45, 812.45, 812.45, 812.45, 812.45, 811.65, 811.65, 811.65, 811.65, 811.65, 809.6, 809.6, 809.6, 809.6, 809.6, 807.86, 807.86, 807.86, 807.86, 807.86, 809.62, 809.62, 809.62, 809.62, 809.62, 809.87, 809.87, 809.87, 809.87, 809.87, 815.13, 815.13, 815.13, 815.13, 815.13, 814.46, 814.46, 814.46, 814.46, 814.46, 816.66, 816.66, 816.66, 816.66, 816.66, 820.46, 820.46, 820.46, 820.46, 820.46, 820.12, 820.12, 820.12, 820.12, 820.12, 817.08, 817.08, 817.08, 817.08, 817.08, 816.83, 816.83, 816.83, 816.83, 816.83, 817.59, 817.59, 817.59, 817.59, 817.59, 819.0, 819.0, 819.0, 819.0, 819.0, 819.48, 819.48]
Loading
predicted_tokens_seconds
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 524 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1717442609 --> 1717443231
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 40.21, 40.21, 40.21, 40.21, 40.21, 41.23, 41.23, 41.23, 41.23, 41.23, 29.7, 29.7, 29.7, 29.7, 29.7, 32.78, 32.78, 32.78, 32.78, 32.78, 33.26, 33.26, 33.26, 33.26, 33.26, 34.7, 34.7, 34.7, 34.7, 34.7, 34.87, 34.87, 34.87, 34.87, 34.87, 35.16, 35.16, 35.16, 35.16, 35.16, 35.29, 35.29, 35.29, 35.29, 35.29, 35.12, 35.12, 35.12, 35.12, 35.12, 35.36, 35.36, 35.36, 35.36, 35.36, 35.32, 35.32, 35.32, 35.32, 35.32, 34.48, 34.48, 34.48, 34.48, 34.48, 34.18, 34.18, 34.18, 34.18, 34.18, 32.86, 32.86, 32.86, 32.86, 32.86, 31.98, 31.98, 31.98, 31.98, 31.98, 29.46, 29.46, 29.46, 29.46, 29.46, 29.34, 29.34, 29.34, 29.34, 29.34, 29.43, 29.43, 29.43, 29.43, 29.43, 29.72, 29.72, 29.72, 29.72, 29.72, 29.45, 29.45, 29.45, 29.45, 29.45, 29.36, 29.36, 29.36, 29.36, 29.36, 29.4, 29.4, 29.4, 29.4, 29.4, 29.57, 29.57, 29.57, 29.57, 29.57, 29.67, 29.67, 29.67, 29.67, 29.67, 29.47, 29.47, 29.47, 29.47, 29.47, 29.73, 29.73, 29.73, 29.73, 29.73, 29.96, 29.96, 29.96, 29.96, 29.96, 29.88, 29.88, 29.88, 29.88, 29.88, 29.71, 29.71, 29.71, 29.71, 29.71, 29.74, 29.74, 29.74, 29.74, 29.74, 29.89, 29.89, 29.89, 29.89, 29.89, 29.94, 29.94, 29.94, 29.94, 29.94, 30.12, 30.12, 30.12, 30.12, 30.12, 30.26, 30.26, 30.26, 30.26, 30.26, 30.35, 30.35, 30.35, 30.35, 30.35, 30.31, 30.31, 30.31, 30.31, 30.31, 30.23, 30.23, 30.23, 30.23, 30.23, 29.78, 29.78, 29.78, 29.78, 29.78, 29.53, 29.53, 29.53, 29.53, 29.53, 29.66, 29.66, 29.66, 29.66, 29.66, 29.84, 29.84, 29.84, 29.84, 29.84, 30.07, 30.07, 30.07, 30.07, 30.07, 30.1, 30.1, 30.1, 30.1, 30.1, 30.0, 30.0, 30.0, 30.0, 30.0, 29.81, 29.81, 29.81, 29.81, 29.81, 29.52, 29.52, 29.52, 29.52, 29.52, 28.44, 28.44, 28.44, 28.44, 28.44, 28.43, 28.43, 28.43, 28.43, 28.43, 28.43, 28.43, 28.43, 28.43, 28.43, 28.51, 28.51, 28.51, 28.51, 28.51, 28.61, 28.61, 28.61, 28.61, 28.61, 28.62, 28.62, 28.62, 28.62, 28.62, 28.66, 28.66, 28.66, 28.66, 28.66, 28.6, 28.6, 28.6, 28.6, 28.6, 28.62, 28.62, 28.62, 28.62, 28.62, 28.56, 28.56, 28.56, 28.56, 28.56, 28.57, 28.57, 28.57, 28.57, 28.57, 28.68, 28.68, 28.68, 28.68, 28.68, 28.79, 28.79, 28.79, 28.79, 28.79, 28.88, 28.88]
Loading

Details

kv_cache_usage_ratio

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 524 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1717442609 --> 1717443231
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.13, 0.13, 0.13, 0.13, 0.13, 0.4, 0.4, 0.4, 0.4, 0.4, 0.11, 0.11, 0.11, 0.11, 0.11, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 0.11, 0.11, 0.11, 0.11, 0.11, 0.21, 0.21, 0.21, 0.21, 0.21, 0.33, 0.33, 0.33, 0.33, 0.33, 0.23, 0.23, 0.23, 0.23, 0.23, 0.47, 0.47, 0.47, 0.47, 0.47, 0.44, 0.44, 0.44, 0.44, 0.44, 0.46, 0.46, 0.46, 0.46, 0.46, 0.25, 0.25, 0.25, 0.25, 0.25, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.14, 0.14, 0.14, 0.14, 0.29, 0.29, 0.29, 0.29, 0.29, 0.28, 0.28, 0.28, 0.28, 0.28, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.34, 0.34, 0.34, 0.34, 0.34, 0.08, 0.08, 0.08, 0.08, 0.08, 0.12, 0.12, 0.12, 0.12, 0.12, 0.38, 0.38, 0.38, 0.38, 0.38, 0.29, 0.29, 0.29, 0.29, 0.29, 0.25, 0.25, 0.25, 0.25, 0.25, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.21, 0.21, 0.21, 0.21, 0.21, 0.13, 0.13, 0.13, 0.13, 0.13, 0.17, 0.17, 0.17, 0.17, 0.17, 0.27, 0.27, 0.27, 0.27, 0.27, 0.28, 0.28, 0.28, 0.28, 0.28, 0.42, 0.42, 0.42, 0.42, 0.42, 0.34, 0.34, 0.34, 0.34, 0.34, 0.18, 0.18, 0.18, 0.18, 0.18, 0.13, 0.13, 0.13, 0.13, 0.13, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.44, 0.44, 0.44, 0.44, 0.44, 0.61, 0.61, 0.61, 0.61, 0.61, 0.51, 0.51, 0.51, 0.51, 0.51, 0.46, 0.46, 0.46, 0.46, 0.46, 0.24, 0.24, 0.24, 0.24, 0.24, 0.21, 0.21, 0.21, 0.21, 0.21, 0.23, 0.23, 0.23, 0.23, 0.23, 0.15, 0.15, 0.15, 0.15, 0.15, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.2, 0.2, 0.2, 0.2, 0.2, 0.27, 0.27, 0.27, 0.27, 0.27, 0.14, 0.14, 0.14, 0.14, 0.14, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.16, 0.16, 0.16, 0.16, 0.16, 0.1, 0.1, 0.1, 0.1, 0.1, 0.16, 0.16]
Loading
requests_processing
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 524 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1717442609 --> 1717443231
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 1.0, 1.0, 1.0, 1.0, 1.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0]
Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jpohhhh jpohhhh jpohhhh left review comments

@ggerganov ggerganov ggerganov approved these changes

Assignees
No one assigned
Labels
python python script changes Review Complexity : Low Trivial changes to code that most beginner devs (or those who want a break) can tackle. e.g. UI fix
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@zkh2016 @ggerganov @jpohhhh @mofosyne