Increased latency when using structuredOutputs on first

[viktorlarsson]

Not sure this is related to ai-sdk, but my thoughts are that this is on open ai, but I am finding it hard to find where to be able to discuss this.

I've been utilizing OpenAI's Structured Outputs feature with the gpt-4o-2024-08-06 model, alongside Langfuse for tracing. While this feature ensures that model-generated outputs conform precisely to specified JSON schemas, I've encountered notable latency issues when using tools.

Key Observations:

Initial Schema Processing Latency:

The first time I use a new schema, it seems that open ai might be running it as a context-free grammar (CFG) process, which introduces a latency of approximately 2 to 60 seconds, depending on the schema's complexity. But even on simple schemas the latency is happening.

Subsequent Requests

After the initial processing, the schema is cached, allowing subsequent requests to proceed at regular inference speeds. However, I've observed that even repeat prompts experience increased latency compared to using plain JSON outputs.

Scope and Duration of Caching

The exact scope (e.g., API key-specific or global) and duration of schema caching remain unclear.

What have I tried

1. I have cached the data using kv, added a console.log in the start and then in the end of execute. All of the responses are within 0.1-0.2s. The actual rendering of the tool call could take anywhere from 2s to 30s. This for example is not even cached and quite simple.

dateTool
{
  "dateRangeStart": "2024-01-01",
  "dateRangeEnd": "2024-12-31",
  "confidenceScore": 0.8
}

2. I have tried to return the result as string, and the latency goes down completely. But the issue here is that the subsequent agents zod params are hallucinating some of the data, sometimes.

I will try to make a repro case, but I just wanted to check if anyone else has experienced this?

Example of a insane latency that occured.

[lgrammel]

This is a known OpenAI limitation that they announced when they released structured outputs.

See https://platform.openai.com/docs/guides/structured-outputs#how-to-use

Note: the first request you make with any schema will have additional latency as our API processes the schema, but subsequent requests with the same schema will not have additional latency.

[viktorlarsson]

But does the tool params tool({ schema: }) some how use structuredOutputs? Would you say making better describe on this schema would takeaway some of the potential hallucinations?

[lgrammel]

structured outputs apply to both tool calls and json schema answers. we switch tool calling to strict mode (structured) when structured outputs are enabled. if you want tool calls without strict mode, switch off structured outputs