[MERGE AFTER SDK #159] Adding GCS dependency for backend and prompt service #1106

harini-venkataraman · 2025-01-30T09:48:16Z

What

Adding dependency version needed for Backend and prompt service.
...

Why

This will be needed while the integrator uses fsspec. Corresponding file storage dependency is to be added.
...

How

Pinned the dependency in toml.
...

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

No, these are additions of dependency
...

Database Migrations

Not applicable.
...

Env Config

Not applicable.
...

Relevant Docs

Related Issues or PRs

Not applicable.
...

Dependencies Versions

Not applicable.
...

Notes on Testing

In backend, this is already present as a transitive dependency through connectors.
Pinned the version.
...

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

ritwik-g · 2025-01-30T10:21:40Z

backend/pyproject.toml

@@ -33,6 +33,7 @@ dependencies = [
    "social-auth-app-django==5.3.0", # For OAuth
    "social-auth-core==4.4.2", # For OAuth
    "unstract-sdk~=0.56.0rc4",
+    "gcsfs==2024.6.0",


@harini-venkataraman won't we need the dependencies for Azure and S3 if this is needed? @gaya3-zipstack @hari-kuriakose

@gaya3-zipstack I got a doubt. Shouldn't we be adding this in the SDK?

@ritwik-g Adding it in SDK is a baggage for SDK as we are only using fsspec APIs. The integrator can decide what filesystem is required and cann the dependencies accordingly.
Eg, tools need minio and not GCS. Hence adding GCS in SDK will be unnecessary for tools.
However we could use a selective way to install SDK like explicitly mention what we need to be added. We could take it up later....

Okay but let's make sure to add the Azure and S3 dependencies as well.

s3 and azure is not used in backend yet, and we can add it when we make the integration.

@harini-venkataraman @gaya3-zipstack we will be requiring it as soon as we release it to production for on-prem customers. This is not a separate requirement. S3, GCS and Azure storage support for all 3 needs to be present. Testing wise we can test it later. So my suggestion would be to take care of this so that the effort to make it on-prem ready will be minimal. Where ever we need google storage we will need the S3 and Azure as well there.

Any way this is what I feel. If you think it is better to proceed without the same please go ahead.

#1106 (comment)
Adding the dependencies as optional group from SDK.

I would suggest, we address azure, once integration is stable
CC @gaya3-zipstack

prompt-service/pyproject.toml

for more information, see https://pre-commit.ci

unstract/connectors/pyproject.toml

backend/prompt_studio/prompt_studio_core_v2/prompt_ide_base_tool.py

backend/pyproject.toml

platform-service/pyproject.toml

prompt-service/pyproject.toml

unstract/connectors/pyproject.toml

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

Signed-off-by: Gayathri <142381512+gaya3-zipstack@users.noreply.github.com>

hari-kuriakose

@harini-venkataraman If gcsfs and s3fs etc are required only for code in SDK, then it will not be good for the long term to include them as direct dependencies downstream.

However, keeping packages for all cloud providers as default dependencies in SDK will also lead to bloating.

Thus I suggest that we leverage optional dependencies in SDK pyproject.toml to have a section like the following:

[project.optional-dependencies]
aws = [
  "s3fs~=2024.10.0"
]
azure = [
]
gcp = [
  "gcsfs~=2024.6"
]

This will allow downstream services to install SDK selectively by having dependencies like unstract-sdk, unstract-sdk[gcp], etc.

NOTE: It is relatively OK to release with only GCP support first.

cc @ritwik-g @gaya3-zipstack @muhammad-ali-e

harini-venkataraman · 2025-02-06T01:42:22Z

@harini-venkataraman If gcsfs and s3fs etc are required only for code in SDK, then it will not be good for the long term to include them as direct dependencies downstream.

However, keeping packages for all cloud providers as default dependencies in SDK will also lead to bloating.

Thus I suggest that we leverage optional dependencies in SDK pyproject.toml to have a section like the following:
[project.optional-dependencies]
aws = [
  "s3fs~=2024.10.0"
]
azure = [
]
gcp = [
  "gcsfs~=2024.6"
]
This will allow downstream services to install SDK selectively by having dependencies like unstract-sdk, unstract-sdk[gcp], etc.

NOTE: It is relatively OK to release with only GCP support first.

cc @ritwik-g @gaya3-zipstack @muhammad-ali-e

@hari-kuriakose Thanks for suggesting this.
Have made the requested changes in Zipstack/unstract-sdk#159

github-actions · 2025-02-06T02:54:26Z

filepath	function	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_logs}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_client\_init}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{runner/src/unstract/runner/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_run\_container}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$		$$\textcolor{#23d18b}{\tt{9}}$$	$$\textcolor{#23d18b}{\tt{9}}$$

sonarqubecloud · 2025-02-06T02:54:32Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

ritwik-g · 2025-02-09T11:50:35Z

unstract/connectors/pyproject.toml

    "PyDrive2[fsspec]==1.15.4", # For GDrive
    "oauth2client==4.1.3", # For GDrive
    "dropboxdrivefs==1.4.1", # For Dropbox
    "boxfs==0.2.1", # For Box
-    "gcsfs==2024.6.0", # For GoogleCloudStorage
+    "gcsfs==2024.6", # For GoogleCloudStorage


Suggested change

"gcsfs==2024.6", # For GoogleCloudStorage

"gcsfs~=2024.6", # For GoogleCloudStorage

ritwik-g · 2025-02-09T11:50:44Z

unstract/connectors/pyproject.toml

@@ -14,12 +14,12 @@ dependencies = [
    "google-cloud-secret-manager==2.16.1",
    "google-cloud-storage==2.9.0",
    # Filesystem connectors
-    "s3fs[boto3]==2024.6.0", # For Minio
+    "s3fs[boto3]==2024.10.0", # For Minio


Suggested change

"s3fs[boto3]==2024.10.0", # For Minio

"s3fs[boto3]~=2024.10", # For Minio

ritwik-g · 2025-02-09T11:51:28Z

tools/text_extractor/requirements.txt

@@ -3,4 +3,4 @@
 # Required for all unstract tools
 unstract-sdk~=0.56.0rc4
 # Required for remote storage support
-s3fs[boto3]==2024.6.0
+s3fs[boto3]~=2024.10.0


Suggested change

s3fs[boto3]~=2024.10.0

s3fs[boto3]~=2024.10

ritwik-g · 2025-02-09T11:51:37Z

tools/structure/requirements.txt

@@ -3,4 +3,4 @@
 # Required for all unstract tools
 unstract-sdk~=0.56.0rc4
 # Required for remote storage support
-s3fs[boto3]==2024.6.0
+s3fs[boto3]~=2024.10.0


Suggested change

s3fs[boto3]~=2024.10.0

s3fs[boto3]~=2024.10

ritwik-g · 2025-02-09T11:51:44Z

tools/classifier/requirements.txt

@@ -3,4 +3,4 @@
 # Required for all unstract tools
 unstract-sdk~=0.56.0rc4
 # Required for remote storage support
-s3fs[boto3]==2024.6.0
+s3fs[boto3]~=2024.10.0


Suggested change

s3fs[boto3]~=2024.10.0

s3fs[boto3]~=2024.10

ritwik-g

I have added few suggestions. But include it or not I will leave it up to you folks

harini-venkataraman · 2025-02-10T05:11:17Z

The dependencies have been moved to SDK as an optional dep - Zipstack/unstract-sdk#159, and other changes are cherrypicked to #1124
Addressed the review comments in the corresponding PRs, and support for Azure will be handled once the current versio is stable. Closing the PR with this comment.

Adding GCS dependency for backend and prompt service

af04650

harini-venkataraman requested review from gaya3-zipstack and muhammad-ali-e January 30, 2025 09:48

ritwik-g reviewed Jan 30, 2025

View reviewed changes

gaya3-zipstack reviewed Jan 30, 2025

View reviewed changes

prompt-service/pyproject.toml Outdated Show resolved Hide resolved

muhammad-ali-e approved these changes Jan 31, 2025

View reviewed changes

harini-venkataraman and others added 3 commits January 31, 2025 13:57

fix/unpin-deps

99e51fc

[pre-commit.ci] auto fixes from pre-commit.com hooks

7088a9e

for more information, see https://pre-commit.ci

Commit pdm.lock changes

9d5c60b

ritwik-g reviewed Jan 31, 2025

View reviewed changes

unstract/connectors/pyproject.toml Outdated Show resolved Hide resolved

ritwik-g reviewed Jan 31, 2025

View reviewed changes

unstract/connectors/pyproject.toml Outdated Show resolved Hide resolved

fix/unpin-deps

c842bdc

gaya3-zipstack reviewed Jan 31, 2025

View reviewed changes

backend/prompt_studio/prompt_studio_core_v2/prompt_ide_base_tool.py Show resolved Hide resolved

gaya3-zipstack reviewed Jan 31, 2025

View reviewed changes

backend/pyproject.toml Outdated Show resolved Hide resolved

ritwik-g reviewed Feb 3, 2025

View reviewed changes

platform-service/pyproject.toml Outdated Show resolved Hide resolved

ritwik-g reviewed Feb 3, 2025

View reviewed changes

prompt-service/pyproject.toml Outdated Show resolved Hide resolved

ritwik-g reviewed Feb 3, 2025

View reviewed changes

unstract/connectors/pyproject.toml Outdated Show resolved Hide resolved

ritwik-g and others added 2 commits February 3, 2025 09:20

Apply suggestions from code review

38457f6

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

Merge branch 'main' into fix/integration

9535e2b

Signed-off-by: Gayathri <142381512+gaya3-zipstack@users.noreply.github.com>

hari-kuriakose requested changes Feb 3, 2025

View reviewed changes

Fix highlight not working

eb83572

Reverting version pins

7a86b7d

harini-venkataraman changed the title ~~Adding GCS dependency for backend and prompt service~~ [MERGE WITH SDK #159] Adding GCS dependency for backend and prompt service Feb 6, 2025

harini-venkataraman changed the title ~~[MERGE WITH SDK #159] Adding GCS dependency for backend and prompt service~~ [MERGE AFTER SDK #159] Adding GCS dependency for backend and prompt service Feb 6, 2025

ritwik-g reviewed Feb 9, 2025

View reviewed changes

harini-venkataraman mentioned this pull request Feb 10, 2025

[FIX]Passing down execution source #1124

Merged

harini-venkataraman closed this Feb 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MERGE AFTER SDK #159] Adding GCS dependency for backend and prompt service #1106

[MERGE AFTER SDK #159] Adding GCS dependency for backend and prompt service #1106

harini-venkataraman commented Jan 30, 2025

ritwik-g Jan 30, 2025

ritwik-g Jan 30, 2025

gaya3-zipstack Jan 30, 2025

ritwik-g Jan 30, 2025

harini-venkataraman Jan 31, 2025

ritwik-g Jan 31, 2025 •

edited

Loading

ritwik-g Jan 31, 2025

harini-venkataraman Feb 6, 2025 •

edited

Loading

harini-venkataraman Feb 6, 2025

hari-kuriakose left a comment •

edited

Loading

harini-venkataraman commented Feb 6, 2025

github-actions bot commented Feb 6, 2025

sonarqubecloud bot commented Feb 6, 2025

ritwik-g Feb 9, 2025

ritwik-g Feb 9, 2025 •

edited

Loading

ritwik-g Feb 9, 2025

ritwik-g Feb 9, 2025

ritwik-g Feb 9, 2025

ritwik-g left a comment

harini-venkataraman commented Feb 10, 2025

	"gcsfs==2024.6", # For GoogleCloudStorage
	"gcsfs~=2024.6", # For GoogleCloudStorage

	"s3fs[boto3]==2024.10.0", # For Minio
	"s3fs[boto3]~=2024.10", # For Minio

[MERGE AFTER SDK #159] Adding GCS dependency for backend and prompt service #1106

[MERGE AFTER SDK #159] Adding GCS dependency for backend and prompt service #1106

Conversation

harini-venkataraman commented Jan 30, 2025

What

Why

How

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

Database Migrations

Env Config

Relevant Docs

Related Issues or PRs

Dependencies Versions

Notes on Testing

Screenshots

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ritwik-g Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harini-venkataraman Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hari-kuriakose left a comment • edited Loading

Choose a reason for hiding this comment

harini-venkataraman commented Feb 6, 2025

github-actions bot commented Feb 6, 2025

sonarqubecloud bot commented Feb 6, 2025

Quality Gate passed

Choose a reason for hiding this comment

ritwik-g Feb 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ritwik-g left a comment

Choose a reason for hiding this comment

harini-venkataraman commented Feb 10, 2025

ritwik-g Jan 31, 2025 •

edited

Loading

harini-venkataraman Feb 6, 2025 •

edited

Loading

hari-kuriakose left a comment •

edited

Loading

ritwik-g Feb 9, 2025 •

edited

Loading