Feature/add conversation logger merged (#1385)

* Add KG tests (#1351) * cli tests * add sdk tests * typo fix * change workflow ordering * add collection integration tests (#1352) * bump pkg * remove workflows * fix sdk test port * fix delete collection return check * Fix document info serialization (#1353) * Update integration-test-workflow-debian.yml * pre-commit * slightly modify * up * up * smaller file * up * typo, change order * up * up * change order --------- Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com> Co-authored-by: emrgnt-cmplxty <owen@algofi.org> Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com> * add graphrag docs (#1362) * add documentation * up * Update js/sdk/src/models.tsx Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * pre-commit --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Concurrent index creation, allow -1 for paginated entries (#1363) * update webdev-template for current next.js and r2r-js sdk (#1218) Co-authored-by: Simeon <simeon@theobald.nz> * Feature/extend integration tests rebased (#1361) * cleanups * add back overzealous edits * extend workflows * fix full setup * simplify cli * add ymls * rename to light * try again * start light * add cli tests * fix * fix * testing.. * trying complete matrix testflow * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * up * up * up * All actions * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * try offic pgvec formula * sudo make * sudo make * push and pray * push and pray * add new actions * add new actions * docker push & pray * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * setup docker * setup docker * fix default * fix default * Feature/rebase to r2r vars (#1364) * cleanups * add back overzealous edits * extend workflows * fix full setup * simplify cli * add ymls * rename to light * try again * start light * add cli tests * fix * fix * testing.. * trying complete matrix testflow * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * up * up * up * All actions * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * try offic pgvec formula * sudo make * sudo make * push and pray * push and pray * add new actions * add new actions * docker push & pray * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * setup docker * setup docker * fix default * fix default * make changes * update the windows workflow * update the windows workflow * remove extra workflows for now * bump pkg * push and pray * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive tests * revive tests * revive tests * revive tests * update tests * fix typos (#1366) * update tests * up * up * up * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * Add ingestion concurrency limit (#1367) * up * up * up --------- Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com> * tweaks and fixes * Fix Ollama Tool Calling (#1372) * Update graphrag.mdx * Fix Ollama tool calling --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * Clean up Docker Compose (#1368) * Fix hatchet, dockerfile * Update compose * point to correct docker image * Fix bug in deletion, better validation error handling (#1374) * Update graphrag.mdx * Fix bug in deletion, better validation error handling --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * vec index creation endpoint (#1373) * Update graphrag.mdx * upload files * create vector index endpoint * add to fastapi background task * pre-commit * move logging * add api spec, support for all vecs * pre-commit * add workflow * Modify KG Endpoints and update API spec (#1369) * Update graphrag.mdx * modify API endpoints and update documentation * Update ingestion_router.py * try different docker setup (#1371) * try different docker setup * action * add login * add full * update action * cleanup upload script * cleanup upload script * tweak action * tweak action * tweak action * tweak action * tweak action * tweak action * Nolan/ingest chunks js (#1375) * Update graphrag.mdx * Clean up ingest chunks, add to JS SDK * Update JS docs --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * up (#1376) * Bump JS package (#1378) * add conversation * checkin progress * checkin progress * Fix Create Graph (#1379) * up * up * modify assertion * up * up * increase entity limit * changing aristotle back to v2 * pre-commit * typos * add test_ingest_sample_file_2_sdk * Update server.py * checkin progress * up * update * Graphrag docs (#1382) * add docs and refine code * add python SDK documentation * up * update * checkin * up * cleanup * working sync logging * test conversation history * fix runner tests, rename `CHUNKS` to `chunks` --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: FutureProofTechOps <operations@theobald.nz> Co-authored-by: Simeon <simeon@theobald.nz> Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>
SciPhi-AI · Oct 13, 2024 · 975a405 · 975a405
1 parent 8ae04c5
commit 975a405
Show file tree

Hide file tree

Showing 41 changed files with 1,115 additions and 547 deletions.
diff --git a/.github/actions/run-sdk-retrieval-tests/action.yml b/.github/actions/run-sdk-retrieval-tests/action.yml
@@ -23,3 +23,9 @@ runs:
       working-directory: ./py
       shell: bash
       run: poetry run python tests/integration/runner_sdk.py test_rag_response_sample_file_sdk
+
+
+    - name: Agent response sample file (SDK)
+      working-directory: ./py
+      shell: bash
+      run: poetry run python tests/integration/runner_sdk.py test_conversation_history_sdk
diff --git a/docs/api-reference/openapi.json b/docs/api-reference/openapi.json
diff --git a/docs/documentation/python-sdk/observability.mdx b/docs/documentation/python-sdk/observability.mdx
@@ -6,30 +6,6 @@ description: 'Manage and observe your R2R system.'
 ## Analytics and Observability
 R2R provides various tools for analytics and observability to help you monitor and improve the performance of your RAG system.
 
-### Completion Scores
-Scores act as an observability metric and are attached to RAG completions in R2R.
-
-Scores are bound to values between -1 (a strong negative association) and 1 (a strong positive association), where:
-- A score close to 1 may indicate a highly relevant and accurate completion
-- A score close to 0 may indicate a neutral or uncertain quality
-- A score close to -1 may indicate a potentially irrelevant or incorrect completion
-
-```python
-response = client.score_completion("message_id_here", 0.75)
-```
-
-
-<AccordionGroup>
-  <Accordion title="Response">
-    <ResponseField name="response" type="dict">
-      The response from the R2R system.
-      ```python
-      { 'results': 'ok' }
-      ```
-    </ResponseField>
-  </Accordion>
-</AccordionGroup>
-
 
 <ParamField path="message_id" type="uuid.UUID" default="None" required>
 The unique identifier of the completion message to be scored. Found in logs for a message.

diff --git a/docs/mint.json b/docs/mint.json
@@ -314,7 +314,8 @@
       "pages": [
         "api-reference/endpoint/search",
         "api-reference/endpoint/rag",
-        "api-reference/endpoint/agent"
+        "api-reference/endpoint/agent",
+        "api-reference/endpoint/completion"
       ]
     },
     {
@@ -375,8 +376,7 @@
     {
       "group": "Analytics and Observability",
       "pages": [
-        "api-reference/endpoint/analytics",
-        "api-reference/endpoint/score_completion"
+        "api-reference/endpoint/analytics"
       ]
     },
     {

diff --git a/py/Dockerfile b/py/Dockerfile
@@ -47,4 +47,8 @@ COPY r2r.toml /app/r2r.toml
 COPY pyproject.toml /app/pyproject.toml
 
 # Run the application
+<<<<<<< HEAD
+CMD ["sh", "-c", "uvicorn core.main.app_entry:app --host $HOST --port $R2R_PORT"]
+=======
 CMD ["sh", "-c", "uvicorn core.main.app_entry:app --host $R2R_HOST --port $R2R_PORT"]
+>>>>>>> 8ae04c5bfdbeab77073b6ae1169c5bff1b32489b
diff --git a/py/cli/commands/management.py b/py/cli/commands/management.py
@@ -31,9 +31,6 @@ def app_settings(client):
     click.echo(response)
 
 
-# TODO: Implement score_completion
-
-
 @cli.command()
 @click.option("--user-ids", multiple=True, help="User IDs to overview")
 @click.option(

diff --git a/py/core/__init__.py b/py/core/__init__.py
@@ -112,8 +112,6 @@
     # Logging Providers
     "LocalRunLoggingProvider",
     "LoggingConfig",
-    "PostgresLoggingConfig",
-    "PostgresRunLoggingProvider",
     "RunLoggingSingleton",
     # Run Manager
     "RunManager",

diff --git a/py/core/agent/base.py b/py/core/agent/base.py
@@ -1,4 +1,5 @@
 import asyncio
+import logging
 from abc import ABCMeta
 from typing import AsyncGenerator, Generator, Optional
 
@@ -10,7 +11,6 @@
     syncable,
 )
 from core.base.agent import Agent, Conversation
-import logging
 
 logger = logging.getLogger(__name__)
 
@@ -48,11 +48,12 @@ def _reset(self):
     @syncable
     async def arun(
         self,
+        messages: list[Message],
         system_instruction: Optional[str] = None,
-        messages: Optional[list[Message]] = None,
         *args,
         **kwargs,
     ) -> list[dict]:
+        # TODO - Make this method return a list of messages.
         self._reset()
         await self._setup(system_instruction)
 
@@ -69,7 +70,22 @@ async def arun(
             )
             await self.process_llm_response(response, *args, **kwargs)
 
-        return await self.conversation.get_messages()
+        # Get the output messages
+        all_messages: list[dict] = await self.conversation.get_messages()
+        all_messages.reverse()
+
+        output_messages = []
+        for message_2 in all_messages:
+            if (
+                message_2.get("content")
+                and message_2.get("content") != messages[-1].content
+            ):
+                output_messages.append(message_2)
+            else:
+                break
+        output_messages.reverse()
+
+        return output_messages
 
     async def process_llm_response(
         self, response: LLMChatCompletion, *args, **kwargs

diff --git a/py/core/base/__init__.py b/py/core/base/__init__.py
@@ -85,8 +85,6 @@
     # Logging Providers
     "LocalRunLoggingProvider",
     "LoggingConfig",
-    "PostgresLoggingConfig",
-    "PostgresRunLoggingProvider",
     "RunLoggingSingleton",
     # Run Manager
     "RunManager",

diff --git a/py/core/base/agent/agent.py b/py/core/base/agent/agent.py
@@ -1,5 +1,6 @@
 import asyncio
 import json
+import logging
 from abc import ABC, abstractmethod
 from typing import Any, AsyncGenerator, Dict, List, Optional, Type, Union
 
@@ -14,7 +15,6 @@
 from core.base.providers import CompletionProvider, PromptProvider
 
 from .base import Tool, ToolResult
-import logging
 
 logger = logging.getLogger(__name__)
 
@@ -45,7 +45,7 @@ async def add_message(self, message):
         async with self._lock:
             self.messages.append(message)
 
-    async def get_messages(self) -> List[Dict[str, Any]]:
+    async def get_messages(self) -> list[dict[str, Any]]:
         async with self._lock:
             return [
                 {**msg.model_dump(exclude_none=True), "role": str(msg.role)}

diff --git a/py/core/base/api/models/__init__.py b/py/core/base/api/models/__init__.py
@@ -7,20 +7,20 @@
     WrappedUserResponse,
 )
 from shared.api.models.ingestion.responses import (
-    IngestionResponse,
     CreateVectorIndexResponse,
+    IngestionResponse,
+    WrappedCreateVectorIndexResponse,
     WrappedIngestionResponse,
     WrappedUpdateResponse,
-    WrappedCreateVectorIndexResponse,
 )
 from shared.api.models.kg.responses import (
     KGCreationResponse,
     KGEnrichmentResponse,
+    WrappedKGCommunitiesResponse,
     WrappedKGCreationResponse,
     WrappedKGEnrichmentResponse,
     WrappedKGEntitiesResponse,
     WrappedKGTriplesResponse,
-    WrappedKGCommunitiesResponse,
 )
 from shared.api.models.management.responses import (
     AnalyticsResponse,
@@ -41,14 +41,14 @@
     WrappedCollectionListResponse,
     WrappedCollectionOverviewResponse,
     WrappedCollectionResponse,
+    WrappedConversationResponse,
     WrappedDeleteResponse,
     WrappedDocumentChunkResponse,
     WrappedDocumentOverviewResponse,
     WrappedGetPromptsResponse,
     WrappedKnowledgeGraphResponse,
     WrappedLogResponse,
     WrappedPromptMessageResponse,
-    WrappedScoreCompletionResponse,
     WrappedServerStatsResponse,
     WrappedUserCollectionResponse,
     WrappedUserOverviewResponse,
@@ -101,8 +101,8 @@
     "WrappedLogResponse",
     "WrappedAnalyticsResponse",
     "WrappedAppSettingsResponse",
-    "WrappedScoreCompletionResponse",
     "WrappedUserOverviewResponse",
+    "WrappedConversationResponse",
     "WrappedDocumentChunkResponse",
     "WrappedDocumentOverviewResponse",
     "WrappedDocumentChunkResponse",

diff --git a/py/core/base/logging/__init__.py b/py/core/base/logging/__init__.py
@@ -9,8 +9,6 @@
 from .run_logger import (
     LocalRunLoggingProvider,
     LoggingConfig,
-    PostgresLoggingConfig,
-    PostgresRunLoggingProvider,
     RunLoggingSingleton,
 )
 from .run_manager import RunManager, manage_run
@@ -26,8 +24,6 @@
     # Logging Providers
     "LocalRunLoggingProvider",
     "LoggingConfig",
-    "PostgresLoggingConfig",
-    "PostgresRunLoggingProvider",
     "RunLoggingSingleton",
     # Run Manager
     "RunManager",