Refactor clients and add support for FirecREST v2 (#148)

* Async client for FirecREST v2 (#126) * add unit tests (#127) * add v2 docs * Fix docstrings formatting warnings * Add status endpoints for v2 (#130) * Async client for FirecREST v2 * Fix log level * Add status endpoints * Apiv2 merge (#133) * Run unittests for all PRs, not only the ones to main (#131) * Update AsyncExternalStorage.py --------- Co-authored-by: rsarm <rafael.sarmiento@cscs.ch> * Add file system endpoints (#134) * Run unittests for all PRs, not only the ones to main (#131) * add fs endpoints * fix bugs * Fix small typos * Add responses for ls * Add responses for ls options * Add tests for /head * Small fixes and test files in tail * Small fixes in tail and test file * Small fixes in async client and checksum test file * Add file test output * Add test files for stat * add more endpoints * add responses * Fix rm * Add rm test * Fix typo * Fix simple upload and download * Rename param * Update _json_response * Fix simple upload --------- Co-authored-by: Eirini Koutsaniti <eirini.koutsaniti@cscs.ch> Co-authored-by: Eirini Koutsaniti <ekoutsaniti@gmail.com> * Add compute endpoints (#135) * Add submit endpoint * Fix typo * Add job info and metadata * Add job submissions output * Add cancel and attach to v2 client * Fix type * Add better support for transfer jobs (#136) * Add better support for transfer jobs in mv/cp/rm * Fix type * Add upload method * Add download method * Add aiofiles dep * Pin aiofiles to 23.2.1 to maintain support for python 3.7 * Fix mypy errors * Fix aiofiles version * Fix types-aiofiles version * Add sync client for v2 (#137) * Add better support for transfer jobs in mv/cp/rm * Fix type * Add upload method * Add download method * Add aiofiles dep * Pin aiofiles to 23.2.1 to maintain support for python 3.7 * Fix mypy errors * Fix aiofiles version * Fix types-aiofiles version * Add sync client * Fix mypy errors * Add docs for sync client * Add userinfo endpoint for api v2 (#138) * Run unittests for all PRs, not only the ones to main (#131) * add user info endpoint async * add user info endpoint sync * add unit tests * fix json format * Fix types --------- Co-authored-by: Eirini Koutsaniti <eirini.koutsaniti@cscs.ch> Co-authored-by: Eirini Koutsaniti <ekoutsaniti@gmail.com> * Add some fixes and updates in the api (#140) * Run unittests for all PRs, not only the ones to main (#131) * small fixes * bug fix * update ls rec resp --------- Co-authored-by: Eirini Koutsaniti <eirini.koutsaniti@cscs.ch> * Add extract and compress functions to v2 (#141) * Add extract and compress fucntions to v2 * Fix return types * Add missing endpoints mkdir and symlink (#143) * Run unittests for all PRs, not only the ones to main (#131) * add mkdir and symlink * Small fixes --------- Co-authored-by: Eirini Koutsaniti <eirini.koutsaniti@cscs.ch> Co-authored-by: Eirini Koutsaniti <ekoutsaniti@gmail.com> * Allow passing path to local script in client.submit (#142) * Run unittests for all PRs, not only the ones to main (#131) * submit with path to local file * fixes * fixes --------- Co-authored-by: Eirini Koutsaniti <eirini.koutsaniti@cscs.ch> Co-authored-by: Eirini Koutsaniti <ekoutsaniti@gmail.com> * Refactor versions of the api (#144) * Refactor package * Fix docs references for v1 * Fix type errors in docs * Update docs for v2 * Small fix * fix imports * Rename submit args * Retry requests in case of 429 (#147) * Retry in case of 429 in v2 * Fix workflow image * Small fix in systems method * Add missing line --------- Co-authored-by: rsarm <rafael.sarmiento@cscs.ch>
eth-cscs · Jan 22, 2025 · ab6ee81 · ab6ee81
1 parent d7b7a31
commit ab6ee81
Show file tree

Hide file tree

Showing 85 changed files with 3,331 additions and 96 deletions.
diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml
@@ -11,7 +11,7 @@ permissions:
 jobs:
   deploy:
 
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
 
     steps:
     - uses: actions/checkout@v3

diff --git a/.github/workflows/testing.yml b/.github/workflows/testing.yml
@@ -8,7 +8,7 @@ on:
 jobs:
   unittest:
 
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     strategy:
       matrix:
         python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
@@ -30,7 +30,7 @@ jobs:
 
   lint:
 
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
 
     steps:
     - uses: actions/checkout@v2
@@ -51,7 +51,7 @@ jobs:
 
   type-check:
 
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
 
     steps:
     - uses: actions/checkout@v2
@@ -65,4 +65,4 @@ jobs:
         pip install .[test]
     - name: type check with mypy
       run: |
-        mypy .
+        mypy firecrest
diff --git a/.mypy.ini b/.mypy.ini
@@ -0,0 +1,2 @@
+[mypy-unasync]
+ignore_missing_imports = True
diff --git a/README.md b/README.md
@@ -28,7 +28,7 @@ keycloak = f7t.ClientCredentialsAuth(
 )
 
 # Setup the client for the specific account
-client = f7t.Firecrest(
+client = f7t.v1.Firecrest(
     firecrest_url="http://localhost:8000", authorization=keycloak
 )
 

diff --git a/async-example.md b/async-example.md
@@ -47,7 +47,7 @@ async def workflow(client, i):
 
 async def main():
     auth = firecrest.ClientCredentialsAuth(client_id, client_secret, token_uri)
-    client = firecrest.AsyncFirecrest(firecrest_url, authorization=auth)
+    client = firecrest.v1.AsyncFirecrest(firecrest_url, authorization=auth)
 
     # Set up the desired polling rate for each microservice. The float number
     # represents the number of seconds between consecutive requests in each

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -14,11 +14,20 @@ You can also clone it from `Github <https://github.com/eth-cscs/pyfirecrest>`__
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents:
+   :caption: Getting started:
 
    authorization
    tutorial_index
-   reference_index
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Reference:
+
+   reference_auth
+   reference_v1_index
+   reference_v2_index
+
 
 Contact
 =======

diff --git a/docs/source/reference_async.rst b/docs/source/reference_async.rst
@@ -5,15 +5,15 @@ The library also provides an asynchronous API for the client:
 
 The ``AsyncFirecrest`` class
 ****************************
-.. autoclass:: firecrest.AsyncFirecrest
+.. autoclass:: firecrest.v1.AsyncFirecrest
     :members:
     :undoc-members:
     :show-inheritance:
 
 
 The ``AsyncExternalDownload`` class
 ***********************************
-.. autoclass:: firecrest.AsyncExternalDownload
+.. autoclass:: firecrest.v1.AsyncExternalDownload
     :inherited-members:
     :members:
     :undoc-members:
@@ -22,7 +22,7 @@ The ``AsyncExternalDownload`` class
 
 The ``AsyncExternalUpload`` class
 *********************************
-.. autoclass:: firecrest.AsyncExternalUpload
+.. autoclass:: firecrest.v1.AsyncExternalUpload
     :inherited-members:
     :members:
     :undoc-members:

diff --git a/docs/source/reference_async_v2.rst b/docs/source/reference_async_v2.rst
@@ -0,0 +1,11 @@
+Asynchronous FirecREST objects
+==============================
+
+The library also provides an asynchronous API for the client:
+
+The ``AsyncFirecrest`` class
+****************************
+.. autoclass:: firecrest.v2.AsyncFirecrest
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/source/reference_auth.rst b/docs/source/reference_auth.rst
@@ -0,0 +1,9 @@
+Authorization
+=============
+
+The ``ClientCredentialsAuth`` class
+***********************************
+.. autoclass:: firecrest.ClientCredentialsAuth
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/source/reference_basic.rst b/docs/source/reference_basic.rst
@@ -6,15 +6,15 @@ Together with the authorisation class it takes care of the token and makes the a
 
 The ``Firecrest`` class
 ***********************
-.. autoclass:: firecrest.Firecrest
+.. autoclass:: firecrest.v1.Firecrest
     :members:
     :undoc-members:
     :show-inheritance:
 
 
 The ``ExternalUpload`` class
 ****************************
-.. autoclass:: firecrest.ExternalUpload
+.. autoclass:: firecrest.v1.ExternalUpload
     :inherited-members:
     :members:
     :undoc-members:
@@ -23,21 +23,13 @@ The ``ExternalUpload`` class
 
 The ``ExternalDownload`` class
 ******************************
-.. autoclass:: firecrest.ExternalDownload
+.. autoclass:: firecrest.v1.ExternalDownload
     :inherited-members:
     :members:
     :undoc-members:
     :show-inheritance:
 
 
-The ``ClientCredentialsAuth`` class
-***********************************
-.. autoclass:: firecrest.ClientCredentialsAuth
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-
 Custom types of the library
 ***************************
 .. automodule:: firecrest.types

diff --git a/docs/source/reference_sync_v2.rst b/docs/source/reference_sync_v2.rst
@@ -0,0 +1,11 @@
+FirecREST objects
+==============================
+
+Here is the API for the client:
+
+The ``Firecrest`` class
+****************************
+.. autoclass:: firecrest.v2.Firecrest
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/source/reference_index.rst → docs/source/reference_v1_index.rst b/docs/source/reference_index.rst → docs/source/reference_v1_index.rst
@@ -1,5 +1,5 @@
-Reference
-=========
+API v1
+======
 
 .. toctree::
    :maxdepth: 2

diff --git a/docs/source/reference_v2_index.rst b/docs/source/reference_v2_index.rst
@@ -0,0 +1,9 @@
+API v2
+======
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   reference_sync_v2
+   reference_async_v2
diff --git a/docs/source/tutorial_async.rst b/docs/source/tutorial_async.rst
@@ -1,5 +1,5 @@
-How to use the asynchronous API [experimental]
-==============================================
+How to use the asynchronous API
+===============================
 
 In this tutorial, we will explore the asynchronous API of the pyFirecREST library.
 Asynchronous programming is a powerful technique that allows you to write more efficient and responsive code by handling concurrent tasks without blocking the main execution flow.
@@ -15,7 +15,7 @@ First you will need to create an ``AsyncFirecrest`` object, instead of the simpl
 
 .. code-block:: Python
 
-    client = fc.AsyncFirecrest(
+    client = fc.v1.AsyncFirecrest(
         firecrest_url=<firecrest_url>,
         authorization=MyAuthorizationClass()
     )

diff --git a/docs/source/tutorial_basic.rst → docs/source/tutorial_basic_v1.rst b/docs/source/tutorial_basic.rst → docs/source/tutorial_basic_v1.rst
@@ -1,5 +1,5 @@
-Simple tutorial
-===============
+Tutorial for FirecREST v1
+=========================
 
 Your starting point to use pyFirecREST will be the creation of a FirecREST object.
 This is simply a mini client that, in cooperation with the authorization object, will take care of the necessary requests that need to be made and handle the responses.
@@ -19,7 +19,7 @@ For this tutorial we will assume the simplest kind of authorization class, where
             return <TOKEN>
 
     # Setup the client with the appropriate URL and the authorization class
-    client = fc.Firecrest(firecrest_url=<firecrest_url>, authorization=MyAuthorizationClass())
+    client = fc.v1.Firecrest(firecrest_url=<firecrest_url>, authorization=MyAuthorizationClass())
 
 
 Simple blocking requests
-Original file line number
+Diff line change
@@ Expand Up / @@ -11,7 +11,7 @@ permissions: @@
     jobs:
       deploy:
-        runs-on: ubuntu-latest
+        runs-on: ubuntu-22.04
         steps:
         - uses: actions/checkout@v3
@@ Expand Down @@