Release 3.2.11 (#1365)

• Added Knowledge Graph (KG) tests (#1351)
  - CLI tests
  - SDK tests
  - Workflow ordering changes

• Added collection integration tests (#1352)
  - Fixed SDK test port
  - Fixed delete collection return check

• Fixed document info serialization (#1353)

• Added GraphRAG documentation (#1362)

• Implemented concurrent index creation (#1363)
  - Allow -1 for paginated entries

• Updated webdev-template for current Next.js and r2r-js SDK (#1218)

• Extended integration tests (#1361)
  - Cleaned up and simplified CLI
  - Added new workflows and actions
  - Implemented matrix test flow
  - Docker setup and configuration

• Rebased to r2r vars (#1364)
  - Updated Windows workflow
  - Revived full workflow and tests

• Fixed typos (#1366)

• Added ingestion concurrency limit (#1367)

• Fixed Ollama Tool Calling (#1372)

• Cleaned up Docker Compose (#1368)

• Fixed bug in deletion, improved validation error handling (#1374)

• Added vector index creation endpoint (#1373)

• Modified KG Endpoints and updated API spec (#1369)

• Implemented new Docker setup (#1371)
  - Updated actions and scripts

• Cleaned up ingest chunks, added to JS SDK (#1375)

• Various updates and fixes (#1376, #1378, #1379)
  - Increased entity limit
  - Changed Aristotle back to v2
  - Added test_ingest_sample_file_2_sdk

• Added GraphRAG documentation and refined code (#1382)
  - Added Python SDK documentation

• Added R2R_ prefix to env vars (#1383)
  - Cleaned up docker compose
  - Bumped Python and JS package versions
  - Updated form-data

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

.env.example

@@ -6,9 +6,9 @@ export OPENAI_API_KEY=sk-...
 # Add other provider keys as needed
 
 # Environment variables for the Postgres database
-export POSTGRES_USER=your_user
-export POSTGRES_PASSWORD=your_password
-export POSTGRES_HOST=your_host
-export POSTGRES_PORT=your_port
-export POSTGRES_DBNAME=your_db
+export R2R_POSTGRES_USER=your_user
+export R2R_POSTGRES_PASSWORD=your_password
+export R2R_POSTGRES_HOST=your_host
+export R2R_POSTGRES_PORT=your_port
+export R2R_POSTGRES_DBNAME=your_db
 export R2R_PROJECT_NAME=your_project_name

.github/actions/login-docker/action.yml

@@ -0,0 +1,17 @@
+name: 'Login Docker'
+description: 'Sets up Docker for running R2R'
+inputs:
+  docker_username:
+    description: 'Docker Hub username'
+    required: true
+  docker_password:
+    description: 'Docker Hub password or token'
+    required: true
+runs:
+  using: "composite"
+  steps:
+    - name: Login to Docker Hub
+      uses: docker/login-action@v2
+      with:
+        username: ${{ inputs.docker_username }}
+        password: ${{ inputs.docker_password }}

.github/actions/run-cli-graphrag/action.yml → .github/actions/run-cli-graphrag-tests/action.yml

@@ -1,24 +1,24 @@
-name: 'Run SDK Ingestion Tests'
-description: 'Runs SDK ingestion tests for R2R'
+name: 'Run CLI Graphrag Tests'
+description: 'Runs CLI graphrag tests for R2R'
 runs:
   using: "composite"
   steps:
     - name: Ingest sample file (SDK)
       working-directory: ./py
       shell: bash
-      run: poetry run python tests/integration/runner_sdk.py test_remove_all_files_and_ingest_sample_file_cli
+      run: poetry run python tests/integration/runner_cli.py test_ingest_sample_file_2_cli
 
     - name: Create the graph (SDK)
       working-directory: ./py
       shell: bash
-      run: poetry run python tests/integration/runner_sdk.py test_kg_create_graph_sample_file_cli
+      run: poetry run python tests/integration/runner_cli.py test_kg_create_graph_sample_file_cli
 
     - name: Enrich the graph (SDK)
       working-directory: ./py
       shell: bash
-      run: poetry run python tests/integration/runner_sdk.py test_kg_enrich_graph_sample_file_cli
+      run: poetry run python tests/integration/runner_cli.py test_kg_enrich_graph_sample_file_cli
 
     - name: Search over the graph (SDK)
       working-directory: ./py
       shell: bash
-      run: poetry run python tests/integration/runner_sdk.py test_kg_search_sample_file_cli
+      run: poetry run python tests/integration/runner_cli.py test_kg_search_sample_file_cli

.github/actions/run-sdk-graphrag/action.yml → .github/actions/run-sdk-graphrag-tests/action.yml

@@ -1,12 +1,12 @@
-name: 'Run SDK Ingestion Tests'
-description: 'Runs SDK ingestion tests for R2R'
+name: 'Run SDK Graphrag Tests'
+description: 'Runs SDK graphrag tests for R2R'
 runs:
   using: "composite"
   steps:
     - name: Ingest sample file (SDK)
       working-directory: ./py
       shell: bash
-      run: poetry run python tests/integration/runner_sdk.py test_remove_all_files_and_ingest_sample_file_sdk
+      run: poetry run python tests/integration/runner_sdk.py test_ingest_sample_file_2_sdk
 
     - name: Create the graph (SDK)
       working-directory: ./py

.github/actions/setup-docker/action.yml

@@ -1,15 +1,5 @@
 name: 'Setup Docker'
 description: 'Sets up Docker for running R2R'
-inputs:
-  os:
-    description: 'Operating system'
-    required: true
-  docker_username:
-    description: 'Docker Hub username'
-    required: true
-  docker_password:
-    description: 'Docker Hub password or token'
-    required: true
 runs:
   using: "composite"
   steps:
@@ -18,10 +8,6 @@ runs:
       with:
         docker_version: 20.10
         docker_buildx: true
-        docker_nightly: true
 
-    - name: Login to Docker Hub
-      uses: docker/login-action@v2
-      with:
-        username: ${{ inputs.docker_username }}
-        password: ${{ inputs.docker_password }}
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v2

.github/actions/setup-postgres-ext/action.yml

@@ -26,6 +26,10 @@ runs:
         sudo -u postgres /usr/lib/postgresql/15/bin/psql -c "ALTER USER postgres PASSWORD 'postgres';"
         sudo -u postgres /usr/lib/postgresql/15/bin/psql -c "CREATE EXTENSION vector;"
 
+        # Set max_connections to 1024
+        echo "max_connections = 1024" | sudo tee -a /etc/postgresql/15/main/postgresql.conf
+        sudo systemctl reload postgresql@15-main
+
     - name: Setup PostgreSQL on Windows
       if: inputs.os == 'windows-latest'
       shell: pwsh
@@ -58,6 +62,10 @@ runs:
 
         psql -U postgres -c "CREATE EXTENSION vector;"
 
+        # Set max_connections to 1024
+        Add-Content -Path "C:\Program Files\PostgreSQL\15\data\postgresql.conf" -Value "max_connections = 1024"
+        Restart-Service postgresql-x64-15
+
     - name: Setup PostgreSQL on macOS
       if: inputs.os == 'macos-latest'
       shell: bash
@@ -76,3 +84,7 @@ runs:
         export PG_CONFIG=/opt/homebrew/opt/postgresql@15/bin/pg_config
         make
         make install # may need sudo
+
+        # Set max_connections to 1024
+        echo "max_connections = 1024" | sudo tee -a /opt/homebrew/var/postgresql@15/postgresql.conf
+        brew services restart postgresql@15

.github/actions/start-r2r-full/action.yml

@@ -12,25 +12,4 @@ runs:
     shell: bash
     run: |
       cd py
-      poetry run r2r serve --docker --full --config-name=full &
-
-  - name: Wait for R2R Server to be ready
-    shell: bash
-    run: |
-      echo "Waiting for R2R server to be ready..."
-
-      attempt_counter=0
-      max_attempts=20
-
-      until $(curl --output /dev/null --silent --head --fail http://localhost:7272/v2/health); do
-          if [ ${attempt_counter} -eq ${max_attempts} ]; then
-            echo "Max attempts reached"
-            exit 1
-          fi
-
-          printf '.'
-          attempt_counter=$(($attempt_counter+1))
-          sleep 5
-      done
-
-      echo "R2R server is up and running"
+      poetry run r2r serve --docker --full --config-name=full

.github/workflows/r2r-full-py-integration-tests.yml

@@ -1,8 +1,14 @@
-# yaml-language-server: $schema=https://json.schemastore.org/github-workflow.json
-
 name: R2R Full Python Integration Test
 
 on:
+  push:
+    branches:
+      - dev
+      - dev-minor
+  pull_request:
+    branches:
+      - dev
+      - dev-minor
   workflow_dispatch:
 
 jobs:
@@ -11,8 +17,9 @@ jobs:
 
     strategy:
       matrix:
-#        os: [ubuntu-latest, windows-latest, macos-latest]
-        os: [ubuntu-latest]
+        # os: [ubuntu-latest, windows-latest, macos-latest]
+        os: [ubuntu-latest, macos-latest]
+        # os: [ubuntu-latest]
         test_category:
           - cli-ingestion
           - cli-retrieval
@@ -38,8 +45,10 @@ jobs:
 
       - name: Setup and start Docker
         uses: ./.github/actions/setup-docker
+
+      - name: Login Docker
+        uses: ./.github/actions/login-docker
         with:
-          os: ${{ matrix.os }}
           docker_username: ${{ secrets.RAGTORICHES_DOCKER_UNAME }}
           docker_password: ${{ secrets.RAGTORICHES_DOCKER_TOKEN }}
 

.github/workflows/r2r-light-py-integration-tests.yml

@@ -32,11 +32,11 @@ jobs:
     env:
       OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
       TELEMETRY_ENABLED: 'false'
-      POSTGRES_HOST: localhost
-      POSTGRES_DBNAME: postgres
-      POSTGRES_PORT: '5432'
-      POSTGRES_PASSWORD: postgres
-      POSTGRES_USER: postgres
+      R2R_POSTGRES_HOST: localhost
+      R2R_POSTGRES_DBNAME: postgres
+      R2R_POSTGRES_PORT: '5432'
+      R2R_POSTGRES_PASSWORD: postgres
+      R2R_POSTGRES_USER: postgres
       R2R_PROJECT_NAME: r2r_default
 
     steps:

docs/api-reference/endpoint/communities.mdx

@@ -0,0 +1,4 @@
+---
+title: 'Communities'
+openapi: 'GET /v2/communities'
+---

docs/api-reference/endpoint/create_vector_index.mdx

@@ -0,0 +1,4 @@
+---
+title: Create Vector Index
+openapi: 'POST /v2/create_vector_index'
+---

docs/api-reference/endpoint/entities.mdx

@@ -0,0 +1,4 @@
+---
+title: 'Entities'
+openapi: 'GET /v2/entities'
+---

docs/api-reference/endpoint/triples.mdx

@@ -0,0 +1,4 @@
+---
+title: 'Triples'
+openapi: 'GET /v2/triples'
+---

docs/cookbooks/graphrag.mdx

@@ -218,7 +218,7 @@ This will run a cost estimation step to give you an estimate of the cost of the
 Time taken: 0.21 seconds
 {
   "results": {
-    "message": "These are estimated ranges, actual values may vary. To run the KG creation process, run `create-graph` with `--run` in the cli, or `run_mode=\"run\"` in the client.",
+    "message": "Ran Graph Creation Estimate (not the actual run). Note that these are estimated ranges, actual values may vary. To run the KG creation process, run `create-graph` with `--run` in the cli, or `run_type=\"run\"` in the client.",
     "document_count": 2,
     "number_of_jobs_created": 3,
     "total_chunks": 29,

docs/documentation/configuration/postgres.mdx

@@ -41,11 +41,11 @@ your_project_name = "your_project_name"
 2. Alternatively, you can set the following environment variables:
 
 ```bash
-export POSTGRES_USER=your_postgres_user
-export POSTGRES_PASSWORD=your_postgres_password
-export POSTGRES_HOST=your_postgres_host
-export POSTGRES_PORT=your_postgres_port
-export POSTGRES_DBNAME=your_database_name
+export R2R_POSTGRES_USER=your_postgres_user
+export R2R_POSTGRES_PASSWORD=your_postgres_password
+export R2R_POSTGRES_HOST=your_postgres_host
+export R2R_POSTGRES_PORT=your_postgres_port
+export R2R_POSTGRES_DBNAME=your_database_name
 export R2R_PROJECT_NAME=your_project_name
 ```
 

docs/documentation/deep-dive/providers/database.mdx

@@ -35,11 +35,11 @@ your_project_name = "your_project_name"
 Alternatively, you can set these values using environment variables:
 
 ```bash
-export POSTGRES_USER=your_postgres_user
-export POSTGRES_PASSWORD=your_postgres_password
-export POSTGRES_HOST=your_postgres_host
-export POSTGRES_PORT=your_postgres_port
-export POSTGRES_DBNAME=your_database_name
+export R2R_POSTGRES_USER=your_postgres_user
+export R2R_POSTGRES_PASSWORD=your_postgres_password
+export R2R_POSTGRES_HOST=your_postgres_host
+export R2R_POSTGRES_PORT=your_postgres_port
+export R2R_POSTGRES_DBNAME=your_database_name
 export R2R_PROJECT_NAME=your_project_name
 ```
 Environment variables take precedence over the config settings in case of conflicts. The R2R Docker includes configuration options that facilitate integration with a combined Postgres+pgvector database setup.

docs/documentation/deployment/troubleshooting/environment.mdx

@@ -14,9 +14,9 @@ Signs that you might be dealing with missing environment variables:
 
 Here are some critical environment variables for R2R:
 
-- Database credentials (e.g., `POSTGRES_USER`, `POSTGRES_PASSWORD`)
+- Database credentials (e.g., `R2R_POSTGRES_USER`, `R2R_POSTGRES_PASSWORD`)
 - API keys (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
-- Configuration settings (e.g., `CONFIG_NAME`, `CONFIG_PATH`)
+- Configuration settings (e.g., `R2R_CONFIG_NAME`, `R2R_CONFIG_PATH`)
 
 ## 3. Checking for Missing Variables
 
@@ -31,9 +31,9 @@ Example:
 services:
   r2r:
     environment:
-      - POSTGRES_USER=${POSTGRES_USER:-postgres}
+      - R2R_POSTGRES_USER=${R2R_POSTGRES_USER:-postgres}
       - OPENAI_API_KEY=${OPENAI_API_KEY:-}
-      - CONFIG_NAME=${CONFIG_NAME:-}
+      - R2R_CONFIG_NAME=${R2R_CONFIG_NAME:-}
       # ... other variables
 ```
 
@@ -44,9 +44,9 @@ services:
 
 Example `.env` file:
 ```
-POSTGRES_USER=myuser
+R2R_POSTGRES_USER=myuser
 OPENAI_API_KEY=sk-...
-CONFIG_NAME=default
+R2R_CONFIG_NAME=default
 ```
 
 ### 3.3 Verify Environment in Running Container
@@ -111,8 +111,8 @@ For enhanced security, consider using Docker secrets for sensitive data:
 
 Ensure these key R2R variables are set:
 
-- `CONFIG_NAME` or `CONFIG_PATH`: Specifies which configuration to use.
-- `POSTGRES_*`: Database connection details.
+- `R2R_CONFIG_NAME` or `R2R_CONFIG_PATH`: Specifies which configuration to use.
+- `R2R_POSTGRES_*`: Database connection details.
 - `OPENAI_API_KEY`: If using OpenAI services.
 - `ANTHROPIC_API_KEY`: If using Anthropic models.
 - `OLLAMA_API_BASE`: For local LLM integration.

docs/documentation/deployment/troubleshooting/incorrect_credentials.mdx

@@ -18,11 +18,11 @@ Database connectivity issues due to incorrect credentials are a common problem w
    - Ensure all database-related environment variables are set correctly
    - Common variables include:
      ```
-     POSTGRES_USER
-     POSTGRES_PASSWORD
-     POSTGRES_HOST
-     POSTGRES_PORT
-     POSTGRES_DBNAME
+     R2R_POSTGRES_USER
+     R2R_POSTGRES_PASSWORD
+     R2R_POSTGRES_HOST
+     R2R_POSTGRES_PORT
+     R2R_POSTGRES_DBNAME
      ```
 
 3. **Test Database Connection**

docs/documentation/deployment/troubleshooting/vector_store_issues.mdx

@@ -14,11 +14,11 @@ Vector storage is a crucial component in R2R (RAG to Riches) for efficient simil
 
 2. **Verify Environment Variables:**
    Ensure these are correctly set in your R2R configuration:
-   - `POSTGRES_USER`
-   - `POSTGRES_PASSWORD`
-   - `POSTGRES_HOST`
-   - `POSTGRES_PORT`
-   - `POSTGRES_DBNAME`
+   - `R2R_POSTGRES_USER`
+   - `R2R_POSTGRES_PASSWORD`
+   - `R2R_POSTGRES_HOST`
+   - `R2R_POSTGRES_PORT`
+   - `R2R_POSTGRES_DBNAME`
    - `R2R_PROJECT_NAME`
 
 3. **Check Docker Network:**

docs/documentation/glossary.mdx

@@ -18,11 +18,11 @@ icon: 'book'
 
 - `OPENAI_API_KEY`: API key for OpenAI's language models and embeddings.
 - `ANTHROPIC_API_KEY`: API key for Anthropic's language models and embeddings.
-- `POSTGRES_USER`: Username for the Postgres database.
-- `POSTGRES_PASSWORD`: Password for the Postgres database.
-- `POSTGRES_HOST`: Hostname or IP address of the Postgres database server.
-- `POSTGRES_PORT`: Port number for the Postgres database server.
-- `POSTGRES_DBNAME`: Name of the Postgres database to use for R2R.
+- `R2R_POSTGRES_USER`: Username for the Postgres database.
+- `R2R_POSTGRES_PASSWORD`: Password for the Postgres database.
+- `R2R_POSTGRES_HOST`: Hostname or IP address of the Postgres database server.
+- `R2R_POSTGRES_PORT`: Port number for the Postgres database server.
+- `R2R_POSTGRES_DBNAME`: Name of the Postgres database to use for R2R.
 - `R2R_PROJECT_NAME`: Defines the tables within the Postgres database where the selected R2R project resides.
 - `R2R_PORT`: Defines the port over which the R2R process is served.
 - `R2R_HOST`: Defines the host address over which the R2R process is served.