update framework

docs/docs/framework.md

@@ -0,0 +1,39 @@
+# About Our Framework
+
+We segmented LLMs into 4 components - where we are interested in testing the relationship between each component.
+
+- Query (what the user asks the LLM to do)
+- Output (What the LLM returns)
+- Context (The context required for the LLM to return a correct response)
+- Ethical Considerations (bias, toxicity, etc.)
+
+:::note
+
+Over time, we may be updating this framework as we learn more about this field but we are currently presenting this to help new users navigate the large number of metrics that will get added over time.
+
+:::
+
+## Why is testing for LLMs so hard?
+
+LLMs present a paradigm shift in how applications work. They present unique challenges because:
+
+- Long-tailed nature of queries can mean the answers and queries can be anything
+- The answers are stochastic and can be different each time
+
+DeepEval provides an opinionated way to test these LLM outputs using ML models combined with traditional metrics. While there is inherent circularity in using deep learning to test deep learning model outputs, we believe that this type of approach can provide better guides than other methods into which areas to firstly investigate.
+
+Below, we share our framework for large language models.
+
+![Evaluation Framework](../assets/llm-evaluation-framework.png)
+
+## Why not use GPT?
+
+GPT itself is an auto-regressive model that can provide evaluation of the model. GPT aims to predict the next best token and can provide a score based on that. However, imagine the following situation:
+
+GPT predicts 0.2.  
+It then needs to predict the next number based on the given output.
+GPT then predicts 0.23
+It then needs to predict the next number based on the given output.
+GPT then predicts 0.235
+
+The task of predicting the next number does not seem to provide a strong enough reason to get this working.

docs/docs/quickstart/custom-metrics.md

@@ -2,6 +2,12 @@
 
 You can define a custom metric by defining the `measure` and `is_successful` functions and inheriting the base `Metric` class. An example is provided below.
 
+:::note
+
+As of right now, we do not currently support custom metrics for our dashboard but this will be supported in an upcoming version - we apologise for the wait! For any requests of additional metrics, please feel free to e-mail jacky@confident-ai.com
+
+:::
+
 ```python
 import asyncio
 from deepeval.metrics.metric import Metric
@@ -30,9 +36,3 @@ class LengthMetric(Metric):
 
 metric = LengthMetric()
 ```
-
-:::note
-
-As of right now, we do not currently support custom metrics for our dashboard but this will be supported in an upcoming version - we apologise for the wait! For any requests of additional metrics, please feel free to e-mail jacky@confident-ai.com
-
-:::

docs/sidebars.js

@@ -20,13 +20,14 @@ const sidebars = {
   tutorialSidebar: [
     'index',
     'install',
+    'framework',
     {
       type: 'category',
       label: 'QuickStart',
       items: [
         'quickstart/quickstart',
         'quickstart/dataset',
-        // 'quickstart/custom-metrics',
+        'quickstart/custom-metrics',
         'quickstart/synthetic-data-creation',
         'quickstart/dashboard-app'
       ],