Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable rpc for server #7292

Closed
steampunque opened this issue May 15, 2024 · 0 comments
Closed

enable rpc for server #7292

steampunque opened this issue May 15, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@steampunque
Copy link

steampunque commented May 15, 2024

I made a quick patch to server to test RPC running phi-3 fully offloaded onto a remote GPU with the server and all seemed OK, timings:

pp: 258.19 tokens per second
tg: 48.41 tokens per second

Run locally on the same GPU as the remote machine gives:

pp: 563.30 tokens per second
tg: 92.00 tokens per second

Possible Implementation

If you have an idea as to how it can be implemented, please write a detailed description. Feel free to give links to external sources or share visuals that might be helpful to understand the details better.

Patches are trivial:

printf("  --port PORT               port to listen (default  (default: %d)\n", sparams.port);

+ printf(" --rpc SERVERS comma separated list of RPC servers\n");

    } else if (arg == "--host") {
        if (++i >= argc) {
            invalid_param = true;
            break;
        }
        sparams.hostname = argv[i];

+ } else if (arg == "--rpc") {
+ if (++i >= argc) {
+ invalid_param = true;
+ break;
+ }
+ params.rpc_servers = argv[i];

@steampunque steampunque added the enhancement New feature or request label May 15, 2024
@rgerganov rgerganov self-assigned this May 15, 2024
rgerganov added a commit to rgerganov/llama.cpp that referenced this issue May 15, 2024
rgerganov added a commit to rgerganov/llama.cpp that referenced this issue May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants