Expose raw openai client in OpenAIChatCompletionClient, etc.

[afourney]

What feature would you like to be added?

Advanced agents are increasingly needing more direct access to the underlying clients. For example, Gemini models are able to both accept AND produce images, but there is no way to express this generally with ChatCompletionClient. Likewise R1 and other reasoning models expose reasoning tokens. Anthropic clients allow prefix completion etc. etc. etc. We could add these things to the general superclass interface, but then every client will need to handle every feature from every provider (even if just to throw an NotImplementedError)

To access these features, we would benefit from direct client access. As an example, use of Guidance all but requires the underlying client.

This feature proposal is to expose a property in ChatCompletionClient, or its subclasses, that returns the underlying client -- whatever type or class it is.

Why is this needed?

See above.

[victordibia]

I like this and generally agree.
Do you see code in the library consuming these raw clients or it will mostly occur in custom agents?
Can we update this with a complete example usecase?

---

My shallow thinking on this is that

• We standardize on clients (and all their assumptions) because we want a generic interface that plays well with components in the library e.g., AssistantAgent etc
• In cases where there is behaviour beyond what AssistantAgent is accomondating, we either extend the client to fit this, or developers create custom agents (in which case they can implement thier own clients) - and lose guarantees that these agents will fit in perfectly with GroupChat et al?

[afourney]

@victordibia This is why I wanted to bring it up. I of course could just make a specialized agent that takes an OpenAI client rather than an OpenAIChatCompletion client, and just go to town with this....

BUT!

Then I lose other neat features like the component spec and ability to serialize or store/load configs. Right now our abstractions both give us a generic model interface AND the ability to fit with AGStudio plus other neat stuff that comes from a declarative spec... but in many cases, its only the later I want.

[victordibia]

One thing I am getting from this is that we probably need more COMPLETE examples of custom agents showing:

• How to extend it to use custom model clients and participate in a groupchat
• How to implement load/save state for serialization
• How to implement component config such that it can work in AGS

We can do all of these today, we just need to show examples. our current custom agents example falls short in these things.

What other useful things should the example show?

[ekzhu]

I agree. I think custom agent should be moved outside of tutorial, and deserve a longer version with what you described. Created issue: #5450 too kepp track of the changes.

[gagb]

What other useful things should the example show?

Custom agents that use custom message types.

An example of that could also be useful. But compatibility w/ in built agents would be tricky.

[ekzhu]

We can close this as it is resolved by providing example of custom client implementation.