Support user input, message history, structured output, and tool calls in prompt experiments

[marcklingen]

Describe the feature or potential improvement

Currently, all variables need to be inserted via prompt variables. However, there are some standard variables that I might now want to track in Langfuse prompt management explicitly, but rather insert implicitly when starting a prompt experiment.

• input (final user message)
• message history, see related thread: https://github.com/orgs/langfuse/discussions/2210
• tool calls
• structured output

Additional information

No response

[simonwh]

+1, much needed especially when working with agents. When adding a production/live trace to a dataset, you want to include all the chat history that led to this output, and preferably without having to modify the dataset entry before saving.

[marcklingen]

Additional feedback that I received on this

I was hoping to test different prompt versions directly within the Langfuse prompt management UI. Our prompts incorporate advanced features such as JSON schema, structured output, or function calls. While the Langfuse documentation suggests placing these details into the config, I noticed it’s not currently possible to set up experiments and/or dataset to use this config.

It would be fantastic if this capability could be added, as it would greatly enhance the flexibility and usability of the platform.

config currently serves as an escape hatch for various prompt-related features, but its structure lacks opinions (it is a plain json object). This highlights the need for more native support for tools and structured outputs in Langfuse prompts, which can then be relied upon in prompt experiments.