feat: working instrumentation with smolagents #1184

aymeric-roucher · 2025-01-09T20:20:14Z

Fixes #1182
cc @harrisonchu I've copied the work you did for crewAI, ended up working really well!

I've put a test.py file at the root to let you try out the instrumentation.

Beware that I've not adapted the tests yet.

github-actions · 2025-01-09T20:20:31Z

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

axiomofjoy · 2025-01-09T21:03:09Z

Excited to dig into this @aymeric-roucher!

aymeric-roucher · 2025-01-09T21:05:51Z

...ference-instrumentation-smolagents/src/openinference/instrumentation/smolagents/_wrappers.py

+                    agent.model.last_input_token_count + agent.model.last_output_token_count
+                )
+                span.set_attribute("Observations", step_log.observations)
+                # if step_log.error is not None:


@axiomofjoy As you can see I've commented out these 3 lines.
It is because currently their visualization in a platform like Arize Phoenix is unsatisfactory: having an error in one step shows the whole run as failing, when indeed I can have an error at one step but then the multi-step agent recovers in the next steps to successfully solve the task. Is there a way to display an error without upwards transmission of the error to the whole run?

Interesting, can you give me a code snippet to reproduce this behavior?

This is the code in script test.py

Will take a look and get back to you!

I think this might be a bug in Phoenix. I have filed an issue here.

aymeric-roucher · 2025-01-09T21:06:28Z

Nice tomeet you @axiomofjoy! 🤗 This is ready for review. Final things to do are:

how to not propagate errors upwards (see above)
remove the test.py that's here only for testing
make tests work

axiomofjoy · 2025-01-09T21:27:06Z

Great to meet you @aymeric-roucher and thanks so much for this contribution! I'm digging into the PR now. Let me know how I can help you get it over the line, e.g., if you need help writing tests. Our team would also love to dogfood the instrumentation once it's in a good state!

python/instrumentation/openinference-instrumentation-smolagents/pyproject.toml

aymeric-roucher · 2025-01-09T22:08:25Z

@axiomofjoy sorry for the issue above, I had been working on a specific branch of smolagents to make it compatible: now installing from main should work!

aymeric-roucher · 2025-01-09T22:18:42Z

@axiomofjoy one more problem that I just detected: detecting calls to Model.get_tool_call from within ToolCallingAgent does not work. I think this is because I actually use HfApiModel as my LLM: HfApiModel inherits from Model but since it overrides its parent's get_tool_call method, maybe the wrapper targeting Model.get_tool_call() does not work for HfApiModel.

Is there a way to create a wrapper that works for Subclass.get_tool_call for any subclass of Model?

Method __call__ works for standard LLM calls (LLM calls that are just "please generate some text", not "please generate a tool call") because it's defined in Model and not overridden. But it would seem like a hack to build such a wrapper method for Model.get_tool_call, so I'd prefer the universal wrapper described above.

aymeric-roucher · 2025-01-09T22:24:36Z

I have read the CLA Document and I hereby sign the CLA

aymeric-roucher · 2025-01-09T22:27:39Z

Also wondering how to make the nice LLM messages format with system/user shown in this screenshot:

…s/pyproject.toml Co-authored-by: Xander Song <axiomofjoy@gmail.com>

axiomofjoy · 2025-01-09T22:53:50Z

@axiomofjoy sorry for the issue above, I had been working on a specific branch of smolagents to make it compatible: now installing from main should work!

Got it, no worries! I just got it running from dev 😄

axiomofjoy · 2025-01-09T22:56:43Z

Also wondering how to make the nice LLM messages format with system/user shown in this screenshot:

You need to our LLM message semantic conventions. These can be a bit tricky due to constraints on OTel attribute value types. You can see an example here.

axiomofjoy · 2025-01-09T23:00:38Z

@axiomofjoy one more problem that I just detected: detecting calls to Model.get_tool_call from within ToolCallingAgent does not work. I think this is because I actually use HfApiModel as my LLM: HfApiModel inherits from Model but since it overrides its parent's get_tool_call method, maybe the wrapper targeting Model.get_tool_call() does not work for HfApiModel.

Is there a way to create a wrapper that works for Subclass.get_tool_call for any subclass of Model?

Method __call__ works for standard LLM calls (LLM calls that are just "please generate some text", not "please generate a tool call") because it's defined in Model and not overridden. But it would seem like a hack to build such a wrapper method for Model.get_tool_call, so I'd prefer the universal wrapper described above.

Good catch! My first thought is that you might try instrumenting each subclass individually in addition to the base class. This can be accomplished by iterating over subclasses. We do something similar in DSPy here.

axiomofjoy · 2025-01-10T00:33:57Z

@aymeric-roucher It's looking really promising so far! A few findings from my initial testing.

A few span kinds are missing inputs and outputs (agent spans are missing input, LLM spans are missing both). We typically add these values for all OpenInference span kinds to show in the spans and traces table.

Tool attributes are missing. These attributes are the tools conventions in this table.

Retriever tools might best be instrumented as retriever span kind rather than a tool to give us rich UI for retrieved documents, scores, etc.

If you don't mind, I'll open a PR against your branch including some of the examples I used!

aymeric-roucher · 2025-01-10T11:52:12Z

I've applied your suggestions @axiomofjoy and implemented a wrapper over all subclasses of Model, + done many other changes to mimick dspy implementation! Here's what my dashboard now looks like.
Addressing your points:

Should be solved (cf screenshot)
I think tool attributes are now input correctly. Is there any attribute you still see missing?
We have no specific Retriever tool (or the community might build some but we can't know in advance how they'll work, they could return only a big string report instead of individual docs), so this does not apply here!

axiomofjoy · 2025-01-11T00:56:07Z

Hey @aymeric-roucher, awesome progress! I'm excited to test it out.

I opened a PR to your fork that sets up CI and adds examples here. This enables the following commands:

tox run -e mypy-smolagents (run type checks)
tox run -e ruff-smolagents (run formatters and linters)
tox run -e test-smolagents (run tests, my PR nixes the CrewAI tests since they are out of date with our current patterns)
tox run -e smolagents (run everything)

You can read about how to get set up with tox here.

I'll start dogfooding your changes and add some tests in a subsequent PR if you don't mind!

Working instrumentation with smolagents

c2a22e4

aymeric-roucher requested a review from a team as a code owner January 9, 2025 20:20

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Jan 9, 2025

aymeric-roucher mentioned this pull request Jan 9, 2025

[support new package] Hugging Face smolagents #1182

Open

Remove changelog

569338f

axiomofjoy self-requested a review January 9, 2025 21:02

Support multiagent

fcd55df

aymeric-roucher commented Jan 9, 2025

View reviewed changes

Support ToolCallingAgent

9457ab6

axiomofjoy reviewed Jan 9, 2025

View reviewed changes

python/instrumentation/openinference-instrumentation-smolagents/pyproject.toml Outdated Show resolved Hide resolved

aymeric-roucher mentioned this pull request Jan 9, 2025

Support OpenTelemetry huggingface/smolagents#136

Merged

github-actions bot added a commit that referenced this pull request Jan 9, 2025

@aymeric-roucher has signed the CLA in #1184

2dc4c2a

Update python/instrumentation/openinference-instrumentation-smolagent…

d47b4e5

…s/pyproject.toml Co-authored-by: Xander Song <axiomofjoy@gmail.com>

aymeric-roucher mentioned this pull request Jan 9, 2025

[enhancement] Observability huggingface/smolagents#103

Open

axiomofjoy changed the title ~~Working instrumentation with smolagents~~ feat: working instrumentation with smolagents Jan 9, 2025

Simplify wrappers following refacto in smolagents

62aac6d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: working instrumentation with smolagents #1184

feat: working instrumentation with smolagents #1184

aymeric-roucher commented Jan 9, 2025

github-actions bot commented Jan 9, 2025 •

edited

Loading

axiomofjoy commented Jan 9, 2025

aymeric-roucher Jan 9, 2025 •

edited

Loading

axiomofjoy Jan 9, 2025

aymeric-roucher Jan 9, 2025

axiomofjoy Jan 9, 2025

axiomofjoy Jan 10, 2025

aymeric-roucher commented Jan 9, 2025 •

edited

Loading

axiomofjoy commented Jan 9, 2025

aymeric-roucher commented Jan 9, 2025

aymeric-roucher commented Jan 9, 2025 •

edited

Loading

aymeric-roucher commented Jan 9, 2025

aymeric-roucher commented Jan 9, 2025

axiomofjoy commented Jan 9, 2025

axiomofjoy commented Jan 9, 2025

axiomofjoy commented Jan 9, 2025

axiomofjoy commented Jan 10, 2025 •

edited

Loading

aymeric-roucher commented Jan 10, 2025 •

edited

Loading

axiomofjoy commented Jan 11, 2025

feat: working instrumentation with smolagents #1184

Are you sure you want to change the base?

feat: working instrumentation with smolagents #1184

Conversation

aymeric-roucher commented Jan 9, 2025

github-actions bot commented Jan 9, 2025 • edited Loading

axiomofjoy commented Jan 9, 2025

aymeric-roucher Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

axiomofjoy Jan 9, 2025

Choose a reason for hiding this comment

aymeric-roucher Jan 9, 2025

Choose a reason for hiding this comment

axiomofjoy Jan 9, 2025

Choose a reason for hiding this comment

axiomofjoy Jan 10, 2025

Choose a reason for hiding this comment

aymeric-roucher commented Jan 9, 2025 • edited Loading

axiomofjoy commented Jan 9, 2025

aymeric-roucher commented Jan 9, 2025

aymeric-roucher commented Jan 9, 2025 • edited Loading

aymeric-roucher commented Jan 9, 2025

aymeric-roucher commented Jan 9, 2025

axiomofjoy commented Jan 9, 2025

axiomofjoy commented Jan 9, 2025

axiomofjoy commented Jan 9, 2025

axiomofjoy commented Jan 10, 2025 • edited Loading

aymeric-roucher commented Jan 10, 2025 • edited Loading

axiomofjoy commented Jan 11, 2025

github-actions bot commented Jan 9, 2025 •

edited

Loading

aymeric-roucher Jan 9, 2025 •

edited

Loading

aymeric-roucher commented Jan 9, 2025 •

edited

Loading

aymeric-roucher commented Jan 9, 2025 •

edited

Loading

axiomofjoy commented Jan 10, 2025 •

edited

Loading

aymeric-roucher commented Jan 10, 2025 •

edited

Loading