How to send multiple images to OpenAI using the Vercel AI SDK

[diasmashikov]

Hello everyone!

I want to send multiple images to the ChatGPT API, but I am not sure what a reliable way is; do we simply create an array of image urls and pass it to "image"? This is my current code that can send only one image.

`
import OpenAI from "openai";
import { convertToCoreMessages, streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { CoreMessage } from "ai";

// Create an OpenAI API client (that's edge friendly!)
const openaiObject = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});

export const dynamic = "force-dynamic";

export async function POST(req: Request) {
const { messages, data } = await req.json();
const initialMessages = messages.slice(0, -1);
const currentMessage = messages[messages.length - 1];

let userMessage: CoreMessage;

console.log("Image URL is being sent", data.imageUrl);
userMessage = {
role: "user",
content: [
{ type: "text", text: currentMessage.content },
{ type: "image", image: new URL(data.imageUrl) },
],
};

// Call the language model
const result = await streamText({
model: openai("gpt-4o"),
messages: [...convertToCoreMessages(initialMessages), userMessage],
});

return result.toAIStreamResponse();
}
`

[lgrammel]

We have added a "attachments" feature to use chat that you can use in combination with convertToCoreMessages. You can find an example here: https://sdk.vercel.ai/docs/ai-sdk-ui/chatbot#attachments-experimental

Btw you are mixing the OpenAI SDK and the Vercel AI SDK with an OpenAI provider, which is confusing and can lead to issues.

[nroy20]

@lgrammel do you have an estimate as to when this feature can be safely used in a production environment? Looks cool!

[lgrammel]

This is what experimental means:

Experimental features let you use the latest AI SDK functionality as soon as possible.
However, they can change in patch versions (or even be fully removed).
Please pin the patch version if you decide to use experimental features.

So you could use it. Otherwise I expect it to be ready in the next major release, maybe earlier.

[MarufRayhan]

@lgrammel How does the backend system handle multiple PDF uploads, and what internal process is used to analyze and process these files, such as converting them to embeddings or using another method?

[lgrammel]

@MarufRayhan it does nothing special - the underlying provider needs to support PDFs on the LLM level (currently on Google GenAI and Google Vertex).

[rockywangxiaolei]

@MarufRayhan it does nothing special - the underlying provider needs to support PDFs on the LLM level (currently on Google GenAI and Google Vertex).

from what i learned, looks like the experimental attachment feature can help to:

• 1 send the image (url or base64) to model directly(subject to model image input support or not), here we can check the model image input features:https://sdk.vercel.ai/docs/foundations/providers-and-models

• 2 send file(text or pdf,binary or url) to model diretcly (subject to model file input support or not),

• 3 send audio to model diretcly (subject to model audio input support or not),
  for 2&3,we have to check the model features before we use this feature.

for thoese who wanna leverage experimental attachment for RAG, it looks like you have to build your own attachment handle logic, i am not sure whether we can handle it by extract from messages at the backend API endpoint.