Enable to use variable of prompt on evaluator.

[takryo00]

Describe the feature or potential improvement

Currently the evaluator can use the whole input and output of the generation.
I'd like to pass the variable/context that is part of prompt.

For example, if we have a prompt like this,

# Goal
Summarize the input document.
Think step by step and output it first,ant then output final result following output format.

# OutputFormat
{ "summary" : $result_of_summary }

# Input
{{document}}

I want to pass the document variable without whole other prompt.

Why

The evaluator is aware of not only the target input and processing result, but also the entire prompt including instructions, and the output other than the desired output including CoT.
This makes it difficult to function as an evaluator that focuses only on the input and result.

Additional information

No response

[marcklingen]

Great idea, thanks for sharing!