Add Oracle23ai as a vector datasource #5342

tmuife · 2024-06-18T03:37:27Z

Description

Add Oracle23ai as a new vector datasource

Oracle23ai Easily bring the power of similarity search to your business data without having to manage and integrate multiple databases. AI Vector Search enables you to search structured and unstructured data based on its semantics or meaning, in addition to its values. Native vector capabilities can help large language models (LLMs) deliver more-accurate and contextually relevant results with retrieval-augmented generation (RAG).

Change/Add List:

api/config.py
api/controllers/console/datasets/datasets.py
api/core/rag/datasource/vdb/vector_factory.py
api/core/rag/datasource/vdb/vector_type.py
api/requirements.txt
api/core/rag/datasource/vdb/oracle/oraclevector.py
docker/startupscripts/create_user.sql
api/tests/integration_tests/vdb/oracle/test_oraclevector.py
docker/docker-compose.yaml

Dependence:

oracledb==2.2.1

Type of Change

New feature (non-breaking change which adds functionality)
This change requires a documentation update, included: Dify Document
Dependency upgrade

How Has This Been Tested?

run docker build with a test tag, for example 'langgenius/dify-api:test'
change the images in docker-compose.yml to use the test image you just build( api and work services)
comment Weaviate vector store, and uncomment oracle vector store(don't forget the volumes)
change VECTOR_STORE value to oracle

Suggested Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
optional I have made corresponding changes to the documentation
optional I have added tests that prove my fix is effective or that my feature works
optional New and existing unit tests pass locally with my changes

takatost · 2024-06-19T04:48:07Z

there are some conflicts that need to be resolved, thx!

tmuife · 2024-06-19T05:57:16Z

conflicts fixed, thx!

takatost

Due to the current version being in the transition period from pip to poetry, we need to add oracledb = "~2.2.1" to api/pyproject.toml [tool.poetry.dependencies]. Thank you very much.

tmuife · 2024-06-19T14:42:19Z

Yeah, I added the oracledb dependency in pyproject.toml. By the way, I don’t have much experience with pull requests, so if there is anything I can do, please let me know and I will do my best to push this feature. Thanks!

tmuife · 2024-06-19T15:14:04Z

In "API Tests" when goes to "Set Up Vector Store" step, i can't see docker/docker-compose.oracle.yaml runs, what am i missing?

takatost · 2024-06-19T18:53:04Z

In "API Tests" when goes to "Set Up Vector Store" step, i can't see docker/docker-compose.oracle.yaml runs, what am i missing?

Oh, starting an oracle DB in GitHub action might be a disaster, don't you think? 😅

Proper unit testing is beneficial for us.

tmuife · 2024-06-20T00:55:01Z

In "API Tests" when goes to "Set Up Vector Store" step, i can't see docker/docker-compose.oracle.yaml runs, what am i missing?

Oh, starting an oracle DB in GitHub action might be a disaster, don't you think? 😅

Proper unit testing is beneficial for us.

Because the docker image is relatively large, it may require more resources, yeah 😅

takatost

Thanks!

takatost · 2024-06-20T03:05:25Z

Need to fix the issues with lint and testing.

tmuife · 2024-06-20T03:48:25Z

Need to fix the issues with lint and testing.

Pls check, Thanks.

takatost · 2024-06-20T03:52:13Z

Need to fix the issues with lint and testing.

Pls check, Thanks.

Ops, still having issues.

tmuife · 2024-06-20T04:02:09Z

Need to fix the issues with lint and testing.

Pls check, Thanks.

Ops, still having issues.

Sorry, a mistake, overwritten by conflict file.

tmuife · 2024-06-20T05:47:37Z

Need to fix the issues with lint and testing.

Pls check, Thanks.

Ops, still having issues.

Sorry, a mistake, overwritten by conflict file.

What can i do for this "Python Style Expected — Waiting for status to be reported"

takatost · 2024-06-20T08:35:37Z

Test failed, maybe you can add some mock logic 😅

tmuife · 2024-06-20T12:56:28Z

Test failed, maybe you can add some mock logic 😅

Thanks, i went into the details, and i think now i know how to do more test in my local env.

tmuife

add start oracle docker in api-tests.yaml

takatost · 2024-06-20T17:55:35Z

You can use dev/reformat under the root directory to fix lint issues. If it can't be fixed, you'll need to manually resolve it based on the error.

  Error: /workspace/docker/docker-compose.oracle.yaml:19:1: [error] too many blank lines (1 > 0) (empty-lines)
  Error: /workspace/docker/docker-compose.yaml:589:1: [error] too many blank lines (1 > 0) (empty-lines)

tmuife · 2024-06-21T00:39:11Z

You can use dev/reformat under the root directory to fix lint issues. If it can't be fixed, you'll need to manually resolve it based on the error.
  Error: /workspace/docker/docker-compose.oracle.yaml:19:1: [error] too many blank lines (1 > 0) (empty-lines)
  Error: /workspace/docker/docker-compose.yaml:589:1: [error] too many blank lines (1 > 0) (empty-lines)

Got, do it right now

takatost

thanks!

tmuife · 2024-06-22T03:09:12Z

thanks!

This feature, I mean support for Oracle 23 ai as a vector data source, which version will it be released with?

Co-authored-by: walter from vm <walter.jin@oracle.com>

* refs/heads/deploy/dev: (273 commits) chore: host required fix: monitor copywrite feat: make show citations enable default chore: use singular style in middleware config class name (#5502) refactor: extract hosted service configs into dify config (#5504) fix: summary of duckduckgo_search (#5488) fix: zhipu tool calling, this PR fixes the bug described in issue #5496 (#5469) fix: tongyi json output (#5396) fix: firecrawl apikey not start with fc- (#5498) fix lock refactor: extract db configs and celery configs into dify config (#5491) refactor: extract storage provider configs into dify configs (#5443) chore: use singular style in config class name (#5489) chore: remove pip support for api service (#5453) Add Oracle23ai as a vector datasource (#5342) build: support Poetry for depencencies tool in api's Dockerfile (#5105) feat: add ops trace encrypt_config decrypt_config obfuscate_config feat: add remove tracing app fix: max_tokens of qwen-plus & qwen-plus-chat (#5480) feat: add support for Vertex AI claude-3-5-sonnet@20240620 (#5475) ...

tmuife added 6 commits June 17, 2024 15:10

add Oracle23ai as a vector db datasource

d6e5288

add Oracle23ai as a vector db datasource

b3de804

add oracle23ai as a vector db datasouce

85cb335

add oracle docker service

2972532

move volumes block

5dc89b4

del print

bd791f3

crazywoola requested a review from takatost June 18, 2024 04:19

tmuife added 2 commits June 18, 2024 22:33

Merge branch 'main' into main

9bd5d8a

fix format check

0f889b5

takatost requested review from JohnJyong and removed request for takatost June 19, 2024 04:46

tmuife added 3 commits June 19, 2024 05:09

change file mode

da05fd1

fix pytest error

ea8efab

Merge branch 'main' into main

b635468

takatost requested changes Jun 19, 2024

View reviewed changes

add oracledb in pyproject.toml

6eed3b8

tmuife requested a review from takatost June 20, 2024 02:47

takatost previously approved these changes Jun 20, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jun 20, 2024

update poetry lock

111066a

tmuife dismissed takatost’s stale review via 111066a June 20, 2024 03:36

tmuife added 3 commits June 20, 2024 11:38

Merge branch 'main' into main

27b5ecb

add more processes

e2f24f0

XMerge branch 'main' of github.com:jinshcc/dify

230a06c

update poetry.lock

91bf7c9

tmuife added 2 commits June 20, 2024 05:30

update poetry.lock

13b7da9

add new line

267e2e5

add close connection

0d6a432

tmuife requested a review from takatost June 20, 2024 08:21

tmuife and others added 2 commits June 20, 2024 21:18

Merge branch 'main' into main

9d744ca

run test local & add docker-compose in api-tests.yaml

b534cf2

tmuife commented Jun 20, 2024

View reviewed changes

Walter-jin and others added 2 commits June 21, 2024 00:59

manually reformat yaml

ff4a924

del comment

908af86

takatost approved these changes Jun 21, 2024

View reviewed changes

takatost merged commit 6a09409 into langgenius:main Jun 21, 2024
7 checks passed

XiaoLey pushed a commit to XiaoLey/dify that referenced this pull request Jun 22, 2024

Add Oracle23ai as a vector datasource (langgenius#5342)

4aa4963

Co-authored-by: walter from vm <walter.jin@oracle.com>

takatost mentioned this pull request Jun 28, 2024

bump to 0.6.12 #5712

Merged

crazywoola mentioned this pull request Jul 9, 2024

Add Oracle23ai as a vector datasource #5344

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Oracle23ai as a vector datasource #5342

Add Oracle23ai as a vector datasource #5342

tmuife commented Jun 18, 2024

takatost commented Jun 19, 2024

tmuife commented Jun 19, 2024

takatost left a comment

tmuife commented Jun 19, 2024

tmuife commented Jun 19, 2024 •

edited

Loading

takatost commented Jun 19, 2024

tmuife commented Jun 20, 2024

takatost left a comment

takatost commented Jun 20, 2024

tmuife commented Jun 20, 2024

takatost commented Jun 20, 2024

tmuife commented Jun 20, 2024

tmuife commented Jun 20, 2024

takatost commented Jun 20, 2024

tmuife commented Jun 20, 2024

tmuife left a comment

takatost commented Jun 20, 2024

tmuife commented Jun 21, 2024

takatost left a comment

tmuife commented Jun 22, 2024

Add Oracle23ai as a vector datasource #5342

Add Oracle23ai as a vector datasource #5342

Conversation

tmuife commented Jun 18, 2024

Description

Type of Change

How Has This Been Tested?

Suggested Checklist:

takatost commented Jun 19, 2024

tmuife commented Jun 19, 2024

takatost left a comment

Choose a reason for hiding this comment

tmuife commented Jun 19, 2024

tmuife commented Jun 19, 2024 • edited Loading

takatost commented Jun 19, 2024

tmuife commented Jun 20, 2024

takatost left a comment

Choose a reason for hiding this comment

takatost commented Jun 20, 2024

tmuife commented Jun 20, 2024

takatost commented Jun 20, 2024

tmuife commented Jun 20, 2024

tmuife commented Jun 20, 2024

takatost commented Jun 20, 2024

tmuife commented Jun 20, 2024

tmuife left a comment

Choose a reason for hiding this comment

takatost commented Jun 20, 2024

tmuife commented Jun 21, 2024

takatost left a comment

Choose a reason for hiding this comment

tmuife commented Jun 22, 2024

tmuife commented Jun 19, 2024 •

edited

Loading