Execution code behavior and models

[aborruso]

Hi @ErikBjare ,
if I use an ANTHROPIC_API_KEY, gtpme runs python code for me (without asking to execute it).

If I set a gemini key, gptme propose me the code in shell, without executing it.

How do I get the same behavior in gemini (or other models)?

Thank you

[aborruso]

Another simple example: What is 2 + 2?

[aborruso]

A detail: if I start a new session, and I write What is 2 + 2?, it executes the python code.

If, on the other hand, I first start with in the pnrr.db duckdb file there are references to projects and locations. How could I make a query that would return the number of projects by region?, it gives me the code on the screen without executing it.
If after this I type What is 2 + 2? again, it no longer executes it, but shows it to me on the screen.

[ErikBjare]

The model you are using is hallucinating an incorrect code use format (gptme has no tool_code). Use a better model.

Even better models sometimes suffer from #327, you can try the --tool-format flag to change between the formats markdown, XML, and "tools" API. The "tools" API may even work for smaller models trained to work with it, but for small models it can be very hit/miss.

[aborruso]

The model you are using is hallucinating an incorrect code use format (gptme has no tool_code). Use a better model.

I'm using gemini-1.5-flash. Is it not a good model, because it's small?
For gemini models, do I must use gemini-1.5-pro?

Even better models sometimes suffer from #327, you can try the --tool-format flag to change between the formats markdown, XML, and "tools" API. The "tools" API may even work for smaller models trained to work with it, but for small models it can be very hit/miss.

I have tried using markdown, XML, and "tools" API, and it does not work.

Thank you

[aborruso]

Hi @ErikBjare and if I use llama3.1:latest via ollama, I have another way to get a reply: it does not execute code, it does not write to me the code to run, it calculates it for me and it writes the answer me by dialoguing.

User: What is 2 + 2?
Assistant: The answer is 4.

And I have the same using qwen2.5-coder:7b.

[ErikBjare]

I'm using gemini-1.5-flash. Is it not a good model, because it's small?

Yes, too small.

For gemini models, do I must use gemini-1.5-pro?

Or better, I would say. I haven't found their models very reliable with tooluse.

Hi @ErikBjare and if I use llama3.1:latest via ollama, I have another way to get a reply: it does not execute code, it does not write to me the code to run, it calculates it for me and it writes the answer me by dialoguing.

That's a small 7B Q4_K model, it will not perform well.

And I have the same using qwen2.5-coder:7b.

Too small to use tools reliably, or at least not good enough to understand how to reliably use the specific tools in gptme. Also quantized.

Local models are still really bad at tool use, things start to get good at Llama 3.1 70B unquantized, and pretty great with Llama 405B, but both are ofc still infeasible to actually run locally in their unquantized forms (which is necessary imo).

[aborruso]

Thank you very much @ErikBjare I am learning so much