Unable to run Ragas metrics, evaluate ends with TypeError: capture_metric_type() missing 1 required positional argument: 'async_mode'

[pawelppi]

Dears,
Using RAGAS metrics for evaluation, error appears: TypeError: capture_metric_type() missing 1 required positional argument: 'async_mode'

Steps to reproduce:

1. Get example from: https://docs.confident-ai.com/docs/metrics-ragas#example
2. Install Deepeval, Ragas, set OPENAI_API_KEY

pip install ragas
...
deepeval==2.4.1
ragas==0.2.13

3. Run example:

from deepeval import evaluate
from deepeval.metrics.ragas import RagasMetric
from deepeval.test_case import LLMTestCase

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the expected output from your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = RagasMetric(threshold=0.5, model="gpt-3.5-turbo")
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

metric.measure(test_case)
print(metric.score)

# or evaluate test cases in bulk
evaluate([test_case], [metric])

Whole Output:

_.py:53: UserWarning: You are using deepeval version 2.4.1, however version 2.4.2 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
  warnings.warn(
Traceback (most recent call last):
  File "/Volumes/Store/perpetual/deepeval_trys/ragas_aync_error.py", line 22, in <module>
    metric.measure(test_case)
  File "/Volumes/Store/perpetual/deepeval_trys/venv/lib/python3.11/site-packages/deepeval/metrics/ragas.py", line 503, in measure
    with capture_metric_type(self.__name__):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 289, in helper
    return _GeneratorContextManager(func, args, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 105, in __init__
    self.gen = func(*args, **kwds)
               ^^^^^^^^^^^^^^^^^^^
TypeError: capture_metric_type() missing 1 required positional argument: 'async_mode'

Could you give an instruction how to deal with this?

[fred-lav]

Adding async_mode=False as a parameter to every call of capture_metric_type in deepeval/metrics/ragas.py should fix it, I think?

[penguine-ip]

believe it has been solved, closing this now