Are images in tool responses and embeddings supported?

[selalipop]

For example, Claude Sonnet on Bedrock allows tool calls to return images, is there some way to return a value that's passed in its raw form to the provider?

And similarly, I see embeddings use a generic type for value, do any providers support an image value?

[lgrammel]

a) image results from llms are currently not supported
b) it's architected to support embeddings of non-text content, but there are no such providers atm

[rohanneilkapoor]

@lgrammel do you have any guidance on how to work around this, if it's not supported?

I'm working on an "image search" RAG app. It needs to take a user's query, call a tool to grab the relevant images, and then have the LLM "look" at the images to do a final ranking of them. I can't seem to figure out how to get the LLM to actually "look" at the images that get returned by the tool. Curious if I'm just missing something?

[lgrammel]

one solution: instead of using a tool, do the rag step and add the images as a message to the prompt

[rohanneilkapoor]

this worked! thank you