LiteLLM ollama bugs Update

[RolandJAAI]

Hi @merveenoyan as requested in #406 here is the current status with ollama along with code to reproduce.
TL;DR:
If people have trouble using ollama, pls try ollama/modelname instead of ollama_chat/modelname (yes, LiteLLM recommends otherwise) and prefer CodeAgent over ToolCallingAgent (which you should do anyway if possible)

• there is different behavior for ollama_chat/model and ollama/model
• there is different behaviour of models rated either as "tool-compatible" by ollama or "not tool-compatible", and in 1.7.0 only the later worked 🙈
• in 1.8.0 ollama/model now works for all models I tested and for both ToolCallingAgent and CodeAgent; vision does not work since we hardcorded flatten_images_as_text=True in Allow flatten_messages_as_text to be pushed via LiteLLMModel to fix Ollama integration #406 . It would work in theory, but since currently only Llava is supported for ollama in LiteLLM not a big issue imo
• in 1.8.0 ollama_chat works for the CodeAgent with all models, but the ToolCallingAgent only works for models marked as "tool-compatible" and seems rather unstable compared to ollama/model

Details:

v1.8.0
ollama_chat/llama3.3 (tool-compatible):

• ToolCallingAgent: ✅ (thought in all my runs it throws an error in the first 2-3 steps before adapting something in it's answer: Error in generating tool call with model: 'NoneType' object is not iterable. In Step3 or 4 and afterwards it gets it right

ollama_chat/phi4:latest (also deepseek-r1:70b both NOT classified as tools-compatible by ollama):

• ToolCallingAgent: ❌ always throws Error in generating tool call with model: 'NoneType' object is not iterable. After max_steps it gives a final answer like "The current weather in Paris cannot be provided due to technical difficulties accessing real-time data."
• CodeAgent: ✅

ollama/llama3.3 AND phi4:latest & deepseek-r1:70b both NOT classified as tools-compatible by ollama):

• ToolCallingAgent: ✅ (though deepseek-r1:70b really has trouble with this, often needs multiple steps to get the tool call right, phi4 and llama3.3 no problem)
• CodeAgent: ✅

ollama/llava & ollama_chat/llava

• Vision+CodeAgent: ❌ Error in generating model output: Cannot use images with flatten_messages_as_text=True (to be expected due to the hack in Allow flatten_messages_as_text to be pushed via LiteLLMModel to fix Ollama integration #406)

v1.7.0
ollama_chat/all_models:

• ToolCallingAgent: ❌ Error in generating tool call with model: litellm.APIConnectionError: Ollama_chatException - Client error '400 Bad Request' for url 'http://localhost:11434/api/chat'
• CodeAgent: ❌ Error in generating tool call with model: litellm.APIConnectionError: Ollama_chatException - Client error '400 Bad Request' for url 'http://localhost:11434/api/chat'
• Vision: ❌ Error in generating model output:
  litellm.APIConnectionError: Ollama_chatException - Client error '400 Bad Request' for url 'http://localhost:11434/api/chat'

ollama/phi4 (or deepseek-r1:70b both NOT classified as tools-compatible by ollama):

• ToolCallingAgent: ✅ 🙃
• CodeAgent: ✅

ollama/llama3.3 which is compatible for tool calling according to ollama's list of tool-compatible models

• ToolCallingAgent: ❌ Error in generating tool call with model: litellm.APIConnectionError: 'arguments'
  Traceback (most recent call last):
  File "/Users/roland/venvs/sa_1_7/lib/python3.12/site-packages/litellm/main.py", line 2808, in
  completion
  response = base_llm_http_handler.completion(
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
  "/Users/roland/venvs/sa_1_7/lib/python3.12/site-packages/litellm/llms/custom_httpx/llm_http_handler.
  py", line 370, in completion
  return provider_config.transform_response(
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
  "/Users/roland/venvs/sa_1_7/lib/python3.12/site-packages/litellm/llms/ollama/completion/transformati
  on.py", line 264, in transform_response
  "arguments": json.dumps(function_call["arguments"]),
  ~~~~~~~~~~~~~^^^^^^^^^^^^^
  KeyError: 'arguments'

• CodeAgent: ✅

ollama/llava

• Vision+CodeAgent: ✅ (!) There is other bugs in the example code which caused it to not work for me earlier, but after fixing those, in 1.7.0. it actually works. Though LiteLLM currently only supports Llava with ollama, which is a very weak model and doesn't really get stuff done, but there are PRs pending to support better models

Code to reproduce the error
Python3.12.9
smolagents v1.7.0 / v1.8.0 fresh .venv's
Test for ToolCallingAgent and CodeAgent:
examples/agent_from_any_llm.py
with setting chosen_inference = "ollama" and
model_id="ollama_chat/modelname" or
model_id="ollama/modelname" respectively

Vision:

model = LiteLLMModel(
     api_key="",
     api_base="http://localhost:11434",
     model_id="ollama/llava",
     num_ctx=16384
 )

Example code from the blogpost
(There are some other bugs in this example code, will fix those separately, mainly a missing driver = helium.get_driver() in a couple of places and a missing hint to start the browser first in the helium prompt for weaker models)

@aymeric-roucher , @sysradium fyi

[sysradium]

@RolandJAAI Thanks very much for your input. I can see what can be done about message flattening so it does work both with vlms and llms.

[sysradium]

An attempt to tackle this here #553

[sysradium]

@RolandJAAI

By the way, is the issue you were facing with the vision_web_browser.py similar to the one mentioned in the issue #570

It appears that the observations are being set before the tool is called. I had it with llava which wasn't producing any code on the first step.

I was able to resolve this issue by making a change to #571, but I’m not sure if the callback function there should have been more sophisticated. Have you made any modifications to the vision_web_browser?

[aymeric-roucher]

@albertvillanova for visibility on this fantastic reporting by @RolandJAAI !