use-chat-attachments-url: Uploaded files are not recognized correctly by the model

[lesenelir]

I've been working with the example provided in the following link:

https://github.com/vercel/ai/blob/main/examples/next-openai/app/use-chat-attachments-url/page.tsx

In the example, when I upload a .txt file, the model seems unable to recognize the uploaded file. However, if I upload an image, it is recognized perfectly by the model. I have tested this, and it appears that only image files are recognized, while .pdf, .txt, and other formats are not.

Is there any way to resolve this issue?

I tried reading the file URL in api/route.ts, then fetching the content and converting it to base64, but this doesn't seem to be a good solution.🧐

[lesenelir]

convertToCoreMessages

attachmentsToParts

When calling convertToCoreMessages, attachments are converted to base64. So if a URL is from the database, it also needs to be converted to base64 in /api/chat/route.ts.

A possible solution:

// It might cause the pending time to be longer.
// PERFORMANCE: JUST PROCESS THE LAST MESSAGE ATTACHMENTS, Trade-off between performance and message completeness.
const processedMessagesPromises = messages.map(
  async (message: ResponseMessage) => {
    if (message.experimental_attachments) {
      const processedAttachments = await Promise.all(
        message.experimental_attachments.map(async (attachment) => {
          switch (attachment.contentType) {
            case 'text/plain':
            case 'text/csv':
              const resText = await fetch(attachment.url)
              const text = await resText.text()
              const base64 = Buffer.from(text).toString('base64')
              attachment.url = `data:${attachment.contentType};base64,${base64}`
              break
          }
          return attachment
        })
      )
      return {
        ...message,
        experimental_attachments: processedAttachments
      }
    }
    return message
  }
)
const processedMessages = await Promise.all(processedMessagesPromises)

// ....
await streamText(
  messages: convertToCoreMessages(processedMessages),
)

[xsayedmahmud]

@lesenelir I'm encountering the same issue. Based on the documentation, I assumed text files would be handled, if not other file types. Since the AI recognizes images, I expected it would also recognize text files.

My current workflow:

1. Convert all incoming files (PDF, EPUB, etc.) to text format upon upload.
2. Save the text content in the database.
3. Use the URL of this saved text as an attachment.

Despite this approach, the AI still fails to recognize the text files.

Have you received an official response from the team regarding the solution you proposed? Is it the recommended method, or is there a more effective way to handle this situation?

[lesenelir]

Sorry, I haven't received any official response on this solution, but my solution can work in this limited scenario.

Please make sure that the attachment URL regarding the text is in base64 format. It might be that the URL in the third step of your workflow should be a base64 formatted URL.

switch (attachment.contentType) {
	case 'text/plain':
	case 'text/csv': {
		const response = await fetch(attachment.url)
		const text = await response.text()
		const base64 = Buffer.from(text).toString('base64')
		attachment.url = `data:${attachment.contentType};base64,${base64}`
		break
	}
	case 'application/pdf': {
		const response = await fetch(attachment.url)
		const arrayBuffer = await response.arrayBuffer()
		const text = await parsePDF(arrayBuffer)
		const base64 = Buffer.from(text).toString('base64')
		attachment.url = `data:text/plain;base64,${base64}`
		break
	}
	// more case ...
}

[xsayedmahmud]

Your solution should work, and I would have chosen the approach you provided. However, I recently discovered a pull request that added the feature to convert attachments to file parts, which, according to the release notes, should function from version 3.4.7. I updated to version 3.4.9, and now I'm encountering a different type of error with file attachments than I did previously. I’ve reported this issue here #3146 and am currently waiting for a response.

[tgonzales]

for files, I tested with pdf, it worked with google generative ui, gpt image only, I understood it as a limitation of the model. The option to use base64 can overload the sum of input tokens, it would be the same as sending raw text as context, I imagine that these feature parts should solve this problem. But I confess that I haven't been able to look into this topic yet.