Feature/add zerox to unstructured rebased (#1406)

* Feature/add zerox parser (#1396) * Add KG tests (#1351) * cli tests * add sdk tests * typo fix * change workflow ordering * add collection integration tests (#1352) * bump pkg * remove workflows * fix sdk test port * fix delete collection return check * Fix document info serialization (#1353) * Update integration-test-workflow-debian.yml * pre-commit * slightly modify * up * up * smaller file * up * typo, change order * up * up * change order --------- Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com> Co-authored-by: emrgnt-cmplxty <owen@algofi.org> Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com> * add graphrag docs (#1362) * add documentation * up * Update js/sdk/src/models.tsx Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * pre-commit --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Concurrent index creation, allow -1 for paginated entries (#1363) * update webdev-template for current next.js and r2r-js sdk (#1218) Co-authored-by: Simeon <simeon@theobald.nz> * Feature/extend integration tests rebased (#1361) * cleanups * add back overzealous edits * extend workflows * fix full setup * simplify cli * add ymls * rename to light * try again * start light * add cli tests * fix * fix * testing.. * trying complete matrix testflow * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * up * up * up * All actions * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * try offic pgvec formula * sudo make * sudo make * push and pray * push and pray * add new actions * add new actions * docker push & pray * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * setup docker * setup docker * fix default * fix default * Feature/rebase to r2r vars (#1364) * cleanups * add back overzealous edits * extend workflows * fix full setup * simplify cli * add ymls * rename to light * try again * start light * add cli tests * fix * fix * testing.. * trying complete matrix testflow * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * up * up * up * All actions * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * try offic pgvec formula * sudo make * sudo make * push and pray * push and pray * add new actions * add new actions * docker push & pray * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * setup docker * setup docker * fix default * fix default * make changes * update the windows workflow * update the windows workflow * remove extra workflows for now * bump pkg * push and pray * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive tests * revive tests * revive tests * revive tests * update tests * fix typos (#1366) * update tests * up * up * up * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * Add ingestion concurrency limit (#1367) * up * up * up --------- Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com> * tweaks and fixes * Fix Ollama Tool Calling (#1372) * Update graphrag.mdx * Fix Ollama tool calling --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * Clean up Docker Compose (#1368) * Fix hatchet, dockerfile * Update compose * point to correct docker image * Fix bug in deletion, better validation error handling (#1374) * Update graphrag.mdx * Fix bug in deletion, better validation error handling --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * vec index creation endpoint (#1373) * Update graphrag.mdx * upload files * create vector index endpoint * add to fastapi background task * pre-commit * move logging * add api spec, support for all vecs * pre-commit * add workflow * Modify KG Endpoints and update API spec (#1369) * Update graphrag.mdx * modify API endpoints and update documentation * Update ingestion_router.py * try different docker setup (#1371) * try different docker setup * action * add login * add full * update action * cleanup upload script * cleanup upload script * tweak action * tweak action * tweak action * tweak action * tweak action * tweak action * Nolan/ingest chunks js (#1375) * Update graphrag.mdx * Clean up ingest chunks, add to JS SDK * Update JS docs --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * up (#1376) * Bump JS package (#1378) * add conversation * checkin progress * checkin progress * Fix Create Graph (#1379) * up * up * modify assertion * up * up * increase entity limit * changing aristotle back to v2 * pre-commit * typos * add test_ingest_sample_file_2_sdk * Update server.py * checkin progress * up * update * Graphrag docs (#1382) * add docs and refine code * add python SDK documentation * up * update * checkin * up * cleanup * working sync logging * test conversation history * fix runner tests, rename `CHUNKS` to `chunks` * adding zerox parser --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: FutureProofTechOps <operations@theobald.nz> Co-authored-by: Simeon <simeon@theobald.nz> Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com> * Nolan/update hatchet (#1397) * Move Hatchet to latest * Update js package-lock --------- Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com> * Add hatchet logging (#1391) * Add KG tests (#1351) * cli tests * add sdk tests * typo fix * change workflow ordering * add collection integration tests (#1352) * bump pkg * remove workflows * fix sdk test port * fix delete collection return check * Fix document info serialization (#1353) * Update integration-test-workflow-debian.yml * pre-commit * slightly modify * up * up * smaller file * up * typo, change order * up * up * change order --------- Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com> Co-authored-by: emrgnt-cmplxty <owen@algofi.org> Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com> * add graphrag docs (#1362) * add documentation * up * Update js/sdk/src/models.tsx Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * pre-commit --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Concurrent index creation, allow -1 for paginated entries (#1363) * update webdev-template for current next.js and r2r-js sdk (#1218) Co-authored-by: Simeon <simeon@theobald.nz> * Feature/extend integration tests rebased (#1361) * cleanups * add back overzealous edits * extend workflows * fix full setup * simplify cli * add ymls * rename to light * try again * start light * add cli tests * fix * fix * testing.. * trying complete matrix testflow * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * up * up * up * All actions * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * try offic pgvec formula * sudo make * sudo make * push and pray * push and pray * add new actions * add new actions * docker push & pray * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * setup docker * setup docker * fix default * fix default * Feature/rebase to r2r vars (#1364) * cleanups * add back overzealous edits * extend workflows * fix full setup * simplify cli * add ymls * rename to light * try again * start light * add cli tests * fix * fix * testing.. * trying complete matrix testflow * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * cleanup matrix logic * up * up * up * All actions * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * rename to runner * try offic pgvec formula * sudo make * sudo make * push and pray * push and pray * add new actions * add new actions * docker push & pray * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * inspect manifests during launch * setup docker * setup docker * fix default * fix default * make changes * update the windows workflow * update the windows workflow * remove extra workflows for now * bump pkg * push and pray * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive full workflow * revive tests * revive tests * revive tests * revive tests * update tests * fix typos (#1366) * update tests * up * up * up * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * bump max connections * Add ingestion concurrency limit (#1367) * up * up * up --------- Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com> * tweaks and fixes * Fix Ollama Tool Calling (#1372) * Update graphrag.mdx * Fix Ollama tool calling --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * Clean up Docker Compose (#1368) * Fix hatchet, dockerfile * Update compose * point to correct docker image * Fix bug in deletion, better validation error handling (#1374) * Update graphrag.mdx * Fix bug in deletion, better validation error handling --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * vec index creation endpoint (#1373) * Update graphrag.mdx * upload files * create vector index endpoint * add to fastapi background task * pre-commit * move logging * add api spec, support for all vecs * pre-commit * add workflow * Modify KG Endpoints and update API spec (#1369) * Update graphrag.mdx * modify API endpoints and update documentation * Update ingestion_router.py * try different docker setup (#1371) * try different docker setup * action * add login * add full * update action * cleanup upload script * cleanup upload script * tweak action * tweak action * tweak action * tweak action * tweak action * tweak action * Nolan/ingest chunks js (#1375) * Update graphrag.mdx * Clean up ingest chunks, add to JS SDK * Update JS docs --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> * up (#1376) * Bump JS package (#1378) * Fix Create Graph (#1379) * up * up * modify assertion * up * up * increase entity limit * changing aristotle back to v2 * pre-commit * typos * add test_ingest_sample_file_2_sdk * Update server.py * add docs and refine code * add python SDK documentation * up * add logs * clean * rm vq * rm conflicts * pre-commit * up * add logging * update logs * up * up * Update kg_service.py --------- Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com> Co-authored-by: emrgnt-cmplxty <owen@algofi.org> Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: FutureProofTechOps <operations@theobald.nz> Co-authored-by: Simeon <simeon@theobald.nz> Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com> * feat: Add delete_node_via_document_id method to KGProvider (#1387) Co-authored-by: shou.hsu <shou.hsu@utonia.com> * Finishing `zerox` integration * Finishing `zerox` integration * fix zerox, add cli docs * mention runtime config --------- Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com> Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: FutureProofTechOps <operations@theobald.nz> Co-authored-by: Simeon <simeon@theobald.nz> Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com> Co-authored-by: Shou-Hsu <148736054+Shou-Hsu@users.noreply.github.com> Co-authored-by: shou.hsu <shou.hsu@utonia.com>
SciPhi-AI · Oct 15, 2024 · 1c16af0 · 1c16af0
1 parent f58032d
commit 1c16af0
Show file tree

Hide file tree

Showing 64 changed files with 1,862 additions and 316 deletions.
diff --git a/.github/actions/setup-python-full/action.yml b/.github/actions/setup-python-full/action.yml
@@ -10,7 +10,7 @@ runs:
     - name: Set up Python
       uses: actions/setup-python@v4
       with:
-        python-version: '3.10'
+        python-version: '3.11'
 
     - name: Install R2R CLI & Python SDK
       shell: bash

diff --git a/.github/actions/setup-python-light/action.yml b/.github/actions/setup-python-light/action.yml
@@ -10,7 +10,7 @@ runs:
     - name: Set up Python environment
       uses: actions/setup-python@v4
       with:
-        python-version: '3.10'
+        python-version: '3.11'
 
     - name: Install Poetry and dependencies on Ubuntu and macOS
       if: inputs.os == 'ubuntu-latest' || inputs.os == 'macos-latest'

diff --git a/.github/workflows/build-r2r-docker.yml b/.github/workflows/build-r2r-docker.yml
@@ -30,7 +30,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.11'
 
       - name: Install toml package
         run: pip install toml

diff --git a/.github/workflows/build-unst-docker.yml b/.github/workflows/build-unst-docker.yml
@@ -16,7 +16,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.11'
 
       - name: Install toml package
         run: pip install toml

diff --git a/.github/workflows/publish-to-pypi.yml b/.github/workflows/publish-to-pypi.yml
@@ -17,7 +17,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.11'
 
       - name: Install poetry
         working-directory: ./py

diff --git a/docs/documentation/cli/graph.mdx b/docs/documentation/cli/graph.mdx
@@ -0,0 +1,127 @@
+---
+title: 'Knowledge Graph'
+description: 'Managing the knowledge graph with the R2R CLI.'
+---
+
+<Note>
+Occasionally this CLI documentation falls out of date, cross-check with the automatically generated <a href="/api-reference/introduction"> API Reference documentation </a> for the latest parameters.
+</Note>
+
+## Knowledge Graph Management
+
+### Create Graph
+
+Create a knowledge graph for a collection using the `create-graph` command:
+
+````bash
+r2r create-graph --collection-id my-collection --run --kg-creation-settings '{"key": "value"}' --force-kg-creation
+````
+
+<AccordionGroup>
+  <Accordion title="Arguments">
+    <ParamField path="--collection-id" type="str">
+      Collection ID to create graph for. Default is "".
+    </ParamField>
+
+    <ParamField path="--run" type="flag">
+      Run the graph creation process.
+    </ParamField>
+
+    <ParamField path="--kg-creation-settings" type="str">
+      Settings for the graph creation process as a JSON string.
+    </ParamField>
+
+    <ParamField path="--force-kg-creation" type="flag">
+      Force the graph creation process.
+    </ParamField>
+  </Accordion>
+</AccordionGroup>
+
+### Enrich Graph
+
+Enrich an existing knowledge graph using the `enrich-graph` command:
+
+````bash
+r2r enrich-graph --collection-id my-collection --run --force-kg-enrichment --kg-enrichment-settings '{"key": "value"}'
+````
+
+<AccordionGroup>
+  <Accordion title="Arguments">
+    <ParamField path="--collection-id" type="str">
+      Collection ID to enrich graph for. Default is "".
+    </ParamField>
+
+    <ParamField path="--run" type="flag">
+      Run the graph enrichment process.
+    </ParamField>
+
+    <ParamField path="--force-kg-enrichment" type="flag">
+      Force the graph enrichment process.
+    </ParamField>
+
+    <ParamField path="--kg-enrichment-settings" type="str">
+      Settings for the graph enrichment process as a JSON string.
+    </ParamField>
+  </Accordion>
+</AccordionGroup>
+
+### Get Entities
+
+Retrieve entities from the knowledge graph using the `get-entities` command:
+
+````bash
+r2r get-entities --collection-id my-collection --offset 0 --limit 10 --entity-ids entity1 entity2
+````
+
+<AccordionGroup>
+  <Accordion title="Arguments">
+    <ParamField path="--collection-id" type="str" required>
+      Collection ID to retrieve entities from.
+    </ParamField>
+
+    <ParamField path="--offset" type="int">
+      Offset for pagination. Default is 0.
+    </ParamField>
+
+    <ParamField path="--limit" type="int">
+      Limit for pagination. Default is 100.
+    </ParamField>
+
+    <ParamField path="--entity-ids" type="list[str]">
+      Entity IDs to filter by.
+    </ParamField>
+  </Accordion>
+</AccordionGroup>
+
+### Get Triples
+
+Retrieve triples from the knowledge graph using the `get-triples` command:
+
+````bash
+r2r get-triples --collection-id my-collection --offset 0 --limit 10 --triple-ids triple1 triple2 --entity-names entity1 entity2
+````
+
+<AccordionGroup>
+  <Accordion title="Arguments">
+    <ParamField path="--collection-id" type="str" required>
+      Collection ID to retrieve triples from.
+    </ParamField>
+
+    <ParamField path="--offset" type="int">
+      Offset for pagination. Default is 0.
+    </ParamField>
+
+    <ParamField path="--limit" type="int">
+      Limit for pagination. Default is 100.
+    </ParamField>
+
+    <ParamField path="--triple-ids" type="list[str]">
+      Triple IDs to filter by.
+    </ParamField>
+
+    <ParamField path="--entity-names" type="list[str]">
+      Entity names to filter by.
+    </ParamField>
+  </Accordion>
+</AccordionGroup>
+````
diff --git a/docs/documentation/cli/ingestion.mdx b/docs/documentation/cli/ingestion.mdx
@@ -0,0 +1,105 @@
+---
+title: 'Ingestion'
+description: 'Ingesting files with the R2R CLI.'
+---
+
+<Note>
+Occasionally this CLI documentation falls out of date, cross-check with the automatcially generated <a href="/api-reference/introduction"> API Reference documentation </a> for the latest parameters.
+</Note>
+
+## Document Ingestion and Management
+
+### Ingest Files
+
+Ingest files or directories into your R2R system using the `ingest-files` command:
+
+```bash
+r2r ingest-files path/to/file1.txt path/to/file2.txt \
+  --document-ids 9fbe403b-c11c-5aae-8ade-ef22980c3ad1 \
+  --metadatas '{"key1": "value1"}'
+```
+
+<AccordionGroup>
+  <Accordion title="Arguments">
+    <ParamField path="file_paths" type="list[str]" required>
+      The paths to the files to ingest.
+    </ParamField>
+
+    <ParamField path="--document-ids" type="list[str]">
+      Optional document IDs to assign to the ingested files. If not provided, new document IDs will be generated.
+    </ParamField>
+
+    <ParamField path="--metadatas" type="str">
+      Optional metadata to attach to the ingested files, provided as a JSON string. If ingesting multiple files, the metadata will be applied to all files.
+    </ParamField>
+  </Accordion>
+</AccordionGroup>
+
+### Retry Failed Ingestions
+
+Retry ingestion for documents that previously failed using the `retry-ingest-files` command:
+
+```bash
+r2r retry-ingest-files 9fbe403b-c11c-5aae-8ade-ef22980c3ad1
+```
+
+<AccordionGroup>
+  <Accordion title="Arguments">
+    <ParamField path="document_ids" type="list[str]" required>
+      The IDs of the documents to retry ingestion for.
+    </ParamField>
+  </Accordion>
+</AccordionGroup>
+
+### Update Files
+
+Update existing documents using the `update-files` command:
+
+```bash
+r2r update-files path/to/file1_v2.txt \
+  --document-ids 9fbe403b-c11c-5aae-8ade-ef22980c3ad1 \
+  --metadatas '{"key1": "value2"}'
+```
+
+<AccordionGroup>
+  <Accordion title="Arguments">
+    <ParamField path="file_paths" type="list[str]" required>
+      The paths to the updated files.
+    </ParamField>
+
+    <ParamField path="--document-ids" type="str" required>
+      The IDs of the documents to update, provided as a comma-separated string.
+    </ParamField>
+
+    <ParamField path="--metadatas" type="str">
+      Optional updated metadata to attach to the documents, provided as a JSON string. If updating multiple files, the metadata will be applied to all files.
+    </ParamField>
+  </Accordion>
+</AccordionGroup>
+
+### Ingest Sample Files
+
+Ingest one or more sample files from the R2R GitHub repository using the `ingest-sample-file` or `ingest-sample-files` commands:
+
+```bash
+# Ingest a single sample file
+r2r ingest-sample-file
+
+# Ingest a smaller version of the sample file
+r2r ingest-sample-file --v2
+
+# Ingest multiple sample files
+r2r ingest-sample-files
+```
+
+These commands have no additional arguments. The `--v2` flag for `ingest-sample-file` ingests a smaller version of the sample Aristotle text file.
+
+### Ingest Local Sample Files
+
+Ingest the local sample files in the `core/examples/data_unstructured` directory using the `ingest-sample-files-from-unstructured` command:
+
+```bash
+r2r ingest-sample-files-from-unstructured
+```
+
+This command has no additional arguments. It will ingest all files found in the `data_unstructured` directory.
diff --git a/docs/documentation/cli/introduction.mdx b/docs/documentation/cli/introduction.mdx
@@ -0,0 +1,62 @@
+---
+title: 'Introduction'
+description: 'Build, scale, and manage user-facing Retrieval-Augmented Generation applications with the R2R CLI.'
+---
+
+# R2R CLI Documentation
+
+## Installation
+
+Before starting, make sure you have completed the [R2R installation](/documentation/installation).
+
+The R2R CLI is automatically installed as part of the R2R Python package. No additional installation steps are required.
+
+## Getting Started
+
+1. Ensure R2R is running correctly by checking the health status:
+
+```bash
+r2r health
+```
+
+You should see output similar to:
+```
+{"status":"ok"}
+```
+
+2. Ingest a sample file to test the ingestion process:
+
+```bash
+r2r ingest-sample-file
+```
+
+This will download and ingest a sample text file from the R2R GitHub repository. You can verify the ingestion was successful by checking the documents overview:
+
+```bash
+r2r documents-overview
+```
+
+3. Perform a search on the ingested document:
+
+```bash
+r2r search --query "What did Aristotle contribute to philosophy?"
+```
+
+This will search the ingested document for relevant information to answer the query.
+
+4. Generate a response using RAG (Retrieval-Augmented Generation):
+
+```bash
+r2r rag --query "What were Aristotle's main contributions to logic?"
+```
+
+This will perform a search on the ingested document and use the retrieved information to generate a complete response to the query.
+
+## Additional Documentation
+
+For more detailed information on specific functionalities of the R2R CLI, please refer to the following documentation:
+
+- [Document Ingestion](/documentation/cli/ingestion): Learn how to add, retrieve, and manage documents using the CLI.
+- [Search & RAG](/documentation/cli/retrieval): Explore various querying techniques and Retrieval-Augmented Generation capabilities.
+- [Knowledge Graphs](/documentation/cli/graphrag): Learn how to create and enrich knowledge graphs, and perform GraphRAG.
+- [Server Management](/documentation/cli/server): Manage your R2R server, including health checks, logs, and updates.