Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Having plugins installed slows even basic llm --help from 1.3s to 10.7s #732

Open
mcint opened this issue Feb 2, 2025 · 1 comment
Open

Comments

@mcint
Copy link

mcint commented Feb 2, 2025

I would like to make use of the extensive plugin offering to try various models available with ease and without unnecessary overhead. Latency is everything for UIs, and CLIs are generally great because they're fast and responsive, no memory and cpu hungry custom rendering to support text entry -- .5 s is noticable, 1.3 is checking my pulse, 10s is why am I using this tool. Could plugins be architecture to work in still more minimal ways until needed? Declare more of their offerings up front, defer initialization, or cache state, perhaps with modified time cache keys.

For subcommands that don't rely on loading accessory plugins, llm logs -t -n1, setting LLM_LOAD_PLUGINS='' speeds up access from 6.2s to 0.9s.

@mcint
Copy link
Author

mcint commented Feb 2, 2025

Plugins blocked with env var
Plugins blocked

With plugins -- seems domainted by sentence-transformers embedding plugin, especially its importing of torch modules.
Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant