webagent and browser script is broken

[merveenoyan]

I have a bit of a backlog to melt until I get to this but changes in memory has broken the browser and thus webagent.

easy repro (this used to work)

webagent --prompt "go to asos.com/women, get to sale section, click the first clothing item you see. get the product details, and the price, return them. note that I'm shopping from France"

what I get (easy fix, perhaps we should also get them from env vars):

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/bin/webagent", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/mervenoyan/Desktop/smolagents/smolagents/src/smolagents/vlm_web_browser.py", line 190, in main
    model = load_model(args.model_type, args.model_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: load_model() missing 2 required positional arguments: 'api_base' and 'api_key'

when I fixed this I started getting following one, I was working on releases during memory changes so I don't have a visibility over them, will take a look and fix.

Captured a browser screenshot: (1000, 1158) pixels
[Step 2: Duration 0.00 seconds]
Traceback (most recent call last):
  File "/home/merve/anaconda3/envs/py311/bin/webagent", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/merve/smolagents/smolagents/src/smolagents/vision_web_browser.py", line 205, in main
    agent.run(args.prompt + helium_instructions)
  File "/home/merve/smolagents/smolagents/src/smolagents/agents.py", line 322, in run
    return deque(self._run(task=self.task, max_steps=max_steps, images=images), maxlen=1)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/smolagents/smolagents/src/smolagents/agents.py", line 333, in _run
    final_answer = self._execute_step(task, memory_step)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/smolagents/smolagents/src/smolagents/agents.py", line 353, in _execute_step
    final_answer = self.step(memory_step)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/smolagents/smolagents/src/smolagents/agents.py", line 1212, in step
    memory_messages = self.write_memory_to_messages()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/smolagents/smolagents/src/smolagents/agents.py", line 531, in write_memory_to_messages
    messages.extend(memory_step.to_messages(summary_mode=summary_mode))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/merve/smolagents/smolagents/src/smolagents/memory.py", line 109, in to_messages
    "text": f"Call id: {self.tool_calls[0].id}\nObservation:\n{self.observations}",
                        ~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

[albertvillanova]

Thanks for reporting, @merveenoyan!

We should have tests to detect this and avoid future breakings of the CLI.

I'm having a look at it, as we are planning to make a release.

[sysradium]

@albertvillanova The second problem has been fixed here: #571

@merveenoyan Maybe the origins of your problems are a bit different. But I personally came around the situation when a callback in vision_browser was adding observations, but the model itself was failing to produce correct output.