Multi-turn / session experiments in datasets

[bartsimo]

Hi,

with the Datasets feature, is the creation of multi-turn items supported, e.g. in a RAG case:

Turn 1:
Human: Hi, can you please give me data on X?
AI: < Gives data about X >

Turn 2:
Human: Please focus on the years 2020-2021
AI: <Gives filtered data from Turn 1>

For up to n turns.

From what I can see in the documentation and the cookbook, only single-turn is shown.
https://langfuse.com/docs/datasets/python-cookbook

https://langfuse.com/docs/datasets/overview

Can experiments that contain multi-turn interactions be automated with Datasets?

[jannikmaierhoefer]

Hi @bartsimo, this is not yet supported. I, therefore, converted this post into an "idea" post.

[marcklingen]

Quick addition:

As langfuse dataset runs can be linked to traces, you can run an integration test against a multi-step RAG application or agent. Each dataset item will then be linked to a single execution trace of the whole system. You can add evaluators for every step of your application and add them as scores to langfuse.

Alternatively, we often see teams splitting these chains into unit tests which they test separately with different datasets. Thereby they keep all other variables constant and can optimize the different steps independently. This also applies to testing single turns within a conversation separately to make the tests more robust.

Sessions (multiple-invocations) for each dataset item are currently not supported natively. You can however do them all within one trace if you want. Linking sessions to datasets would be interesting, thank you for raising this.

[bartsimo]

Thank you both very much for your quick replies!

Our main focus is on test automation. In the tests, we are not so much interested in asserting expected output nor scores. We are interested in the interactive part of the RAG chatbot, which is currently assessed qualitatively.
90% of what we are interested in are multi-turn interactions:

• How well is context remembered
• How fluent is the conversation
• Are guardrails honored mid-turn
• ...

In our case, my example
Turn 1:
Human: Hi, can you please give me data on X?
AI: < Gives data about X >

Turn 2:
Human: Please focus on the years 2020-2021
AI: <Gives filtered data from Turn 1>

Would be ONE test case in which we do not want to split up the turns in any way since we are interested in the joined qualitative performance.

@marcklingen
If I cannot create multi-turn items, I do not know how I should automate such testing in Langfuse?
Do your additions apply, if this is our testing objective?
I could not quite map your addition to this.

Thank you guys so much!
The traces are awesome!

[marcklingen]

Have you tried running all turns within a single trace when in development? Thereby you can map one multi-turn interaction to a single dataset item and then evaluate the interaction

[Dehde]

I was looking for the same thing in the docs - may I suggest to specifically point this out in your docs? Maybe one page on the topic of multi-turn conversations, and that the UI does not natively support it? Would have saved me a lot of time, until I finally found this. I'm likely not alone.
Thanks for langfuse btw, it's pretty nice!