Support for gpt4all

[SeanMcTex]

gpt4all makes it pretty easy to download and run LLMs on your own machine. It also has an OpenAI-compatible API.

It would be great to be able to point Goose to a local LLM this way, thereby saving costs and mitigating concerns about what's getting sent to one's LLM provider.

[michaelneale]

@SeanMcTex yeah - can currently use ollama for that (which is really openai API to it - and it works ok as long as can run a suitable model). Can gpt4all run large enough models to be worthwhile? (you need fairly hefty ones with some local GPU support to be worth it for goose from my experience). I have used llamafile and llama.cpp as well (antthing that offers a openai like http interface, and has to have tool calling).

Would ollama be interesting?

https://github.com/square/exchange/blob/main/src/exchange/providers/ollama.py

[SeanMcTex]

Oh, very interesting! Quen's super-slow here as well, and the results still seemed less useful than those OpenAI is giving. But I'm sure all of these models will continue to become better and more efficient over time.

The multimodel approach is an interesting one, for sure! Will continue to experiment and watch how these tools develop with interest. Thanks very much!

[michaelneale]

@SeanMcTex yes I can imagine todays frontier models are next years run on laptop models!

[wwilson83]

@SeanMcTex Any chance you can share a guide or configuration changes you made for connecting to your local LLM instance? I'm currently using LM Studio but a switch to ollama wouldnt be hard.

[SeanMcTex]

@wwilson83 Here are a couple of configs I've tried for Goose, pointing to ollama. Hope that's some help!

ollama:
  provider: ollama
  processor: llama3.1:70b
  accelerator: llama3.1:70b
  moderator: passive
  toolkits:
  - name: developer
    requires: {}
qwen:
  provider: ollama
  processor: qwen2:72b
  accelerator: qwen2:72b
  moderator: passive
  toolkits:
  - name: developer
    requires: {}

[wwilson83]

@SeanMcTex Thank you.