Use Deephaven as a Python Library (#63)

* Progress towards using DHaaL * Trying to build whl files * Trying to build whl files * Working to fix sphinx * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Working to fix sphinx and build * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * Added docker builds * added deephaven_server to docs. * Refactored local docker. * Updated documentation. * Updated documentation. * Updated documentation. * Renamed the wheel version number for dev branches to make twine happy. * Set pypi publishing credential. * Updated readme with run instructions * Updated readme with better run instructions * Updated readme * Fixed readme bugs * Fixed docs on port exposure * Handle blocking contract error conditions properly. * Better error reporting on contract details problems. * Fixed example code.
deephaven-examples · Jul 26, 2022 · 89d15da · 89d15da
1 parent 8860c46
commit 89d15da
Show file tree

Hide file tree

Showing 23 changed files with 405 additions and 321 deletions.
diff --git a/.github/workflows/Dockerfile b/.github/workflows/Dockerfile
@@ -0,0 +1,14 @@
+
+FROM ubuntu:22.04
+
+RUN apt update && \
+    apt install -y openjdk-11-jdk python3-pip
+
+ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/
+
+COPY ./wheels /wheels
+
+RUN pip3 install /wheels/*.whl && \
+    rm -rf /wheels/
+
+CMD python3
diff --git a/.github/workflows/build-and-publish.yml b/.github/workflows/build-and-publish.yml
@@ -1,10 +1,16 @@
-name: Build and publish to PyPI
+name: Build & Publish
 
-on: push
+on:
+  pull_request:
+    branches:
+      - main
+  push:
+    branches:
+      - main
 
 jobs:
-  build-and-publish:
-    name: Build and publish to PyPI
+  build-whl:
+    name: Build WHL
     runs-on: ubuntu-20.04
     steps:
     - uses: actions/checkout@master
@@ -22,7 +28,8 @@ jobs:
         TAG_NAME=${{ github.event.release.tag_name }}
         if [ -z "${TAG_NAME}" ]
         then
-            echo "::set-output name=version::development"
+            PR_NUMBER=$(echo $GITHUB_REF | awk 'BEGIN { FS = "/" } ; { print $3 }')
+            echo "::set-output name=version::0.0.0.dev${PR_NUMBER}"
         else
             echo "::set-output name=version::${{ github.event.release.tag_name }}"
         fi
@@ -31,15 +38,122 @@ jobs:
         DH_IB_VERSION: ${{ steps.version.outputs.version }}
       run: |
         python -m build
-    - name: Archive production artifacts
+    - name: Archive build artifacts
       uses: actions/upload-artifact@v2
       with:
-        name: Artifacts
+        name: wheels
         path: |
           dist/*
-#TODO: publish
-#    - name: Publish
-#      if: xxxx
-#      python -m twine upload --repository testpypi dist/*
 
-
+  publish-whl:
+    name: Publish WHL
+    runs-on: ubuntu-20.04
+    needs: [build-whl]
+    if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
+    steps:
+    - name: Download Build Artifacts
+      uses: actions/download-artifact@v3
+      with:
+        name: build-artifacts
+        path: wheel/
+    - name: Publish WHL to PyPi
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        user: __token__
+        password: ${{ secrets.DEEPHAVENIB_PYPI_TOKEN }}
+        packages_dir: wheel/
+
+
+  build-sphinx:
+    name: Build Sphinx
+    runs-on: ubuntu-20.04
+    needs: [build-whl]
+    steps:
+    - uses: actions/checkout@v1
+    - name: Apt installs
+      run: |
+        sudo apt update
+        sudo apt install -y openjdk-11-jdk
+    - name: Pip installs
+      run:  pip3 install --upgrade sphinx==4.2.0 sphinx-autodoc-typehints furo==2021.10.9
+    - name: Download wheels
+      uses: actions/download-artifact@v3
+      with:
+        name: wheels
+    - name: Install Whl
+      run: pip3 install *.whl
+    - name: Run Sphinx
+      working-directory: ./sphinx
+      env:
+        JAVA_HOME: /usr/lib/jvm/java-11-openjdk-amd64
+      run: |
+        make html
+        touch build/html/.nojekyll
+    - name: Archive Sphinx artifacts
+      uses: actions/upload-artifact@v1
+      with:
+        name: documentation-html
+        path: sphinx/build/html/
+
+  publish-sphinx:
+    name: Publish Sphinx
+    runs-on: ubuntu-20.04
+    needs: [build-sphinx]
+    if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
+    steps:
+    - name: Download Sphinx Artifacts
+      uses: actions/download-artifact@v3
+      with:
+        name: documentation-html
+        path: html/
+    - name: Deploy Sphinx docs to gh-pages
+      uses: JamesIves/github-pages-deploy-action@v4.2.3
+      with:
+        branch: gh-pages
+        folder: html/
+
+  docker:
+    name: Build and Publish Docker
+    runs-on: ubuntu-20.04
+    needs: [build-whl]
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Get branch name
+        id: branch-name
+        uses: tj-actions/branch-names@v5.2
+      - name: Download wheels
+        uses: actions/download-artifact@v3
+        with:
+          name: wheels
+          path: wheels/
+      - name: Log in to the Container registry
+        uses: docker/login-action@v1.10.0
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta-base
+        uses: docker/metadata-action@v4.0.1
+        with:
+          images: ghcr.io/${{ github.repository }}
+          tags: |
+            type=schedule
+            type=ref,event=branch
+            type=ref,event=tag
+            type=ref,event=pr
+            type=raw,${{ steps.branch-name.outputs.current_branch }}
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v2.7.0
+        with:
+          context: .
+          file: ./.github/workflows/Dockerfile
+          push: true
+          tags: ${{ steps.meta-base.outputs.tags }}
+          labels: ${{ steps.meta-base.outputs.labels }}
+        env:
+          IMAGE_NAME: ${{ github.repository }}
diff --git a/.github/workflows/docker-cleanup.yml b/.github/workflows/docker-cleanup.yml
@@ -0,0 +1,26 @@
+# Delete Docker images after PR merge
+#
+
+name: 'Clean up Docker images from PR'
+
+on:
+  pull_request:
+    types: [closed]
+
+jobs:
+  purge-image:
+    name: Delete images from ghcr.io
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Delete image (Base)
+        uses: chipkent/action-cleanup-package@v1.0.2
+        with:
+          package-name: ${{ github.event.repository.name }}-base
+          tag: pr-${{ github.event.pull_request.number }}
+          github-token: ${{ secrets.CI_ACTION_TOKEN }}
+      - name: Delete image (Downloader)
+        uses: chipkent/action-cleanup-package@v1.0.2
+        with:
+          package-name: ${{ github.event.repository.name }}-downloader
+          tag: pr-${{ github.event.pull_request.number }}
+          github-token: ${{ secrets.CI_ACTION_TOKEN }}
diff --git a/.github/workflows/sphinx.yml b/.github/workflows/sphinx.yml
diff --git a/.gitignore b/.gitignore
@@ -1,4 +1,7 @@
 .idea
 venv
 dist
-src/deephaven_ib.egg-info
+src/deephaven_ib.egg-info
+docker/data
+docker/*/data
+docker/*/build
diff --git a/README.md b/README.md
@@ -1,4 +1,3 @@
-
 # deephaven-ib
 
 ![Deephaven Data Labs Logo](docs/assets/Deephaven-Logo-Wordmark-Community-OnLight.png)
@@ -174,33 +173,80 @@ upper right corner.  ![](docs/assets/config-gear.png)
 1) [For Paper Trading] Log into the [Interactive Brokers Web Interface](https://interactivebrokers.com/).
 1) [For Paper Trading] In the [Interactive Brokers Web Interface](https://interactivebrokers.com/), navigate to `Account->Settings->Paper Trading Account` and make sure that "Share real-time market data subscriptions with paper trading account?" is set to true.
 
+
 ## Launch
 To launch the system:
 
+### Launch with Docker
+
+This is the most tested way to launch.
+
 1) Launch [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php).
-1) Accept incoming connections to [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php).  (May not be required for all sessions.)
+2) Accept incoming connections to [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php).  (May not be required for all sessions.)
 ![](docs/assets/allow-connections.png)
-1) Build the Docker images:
+3) Create a directory for your data and scripts
+    ```bash
+    mkdir data
+    ```
+4) Launch the system (Option 1):
+    * On Mac:
     ```bash
-    ./docker/deephaven_ib_docker.sh build --dh-version <deephaven_version>
+    git clone git@github.com:deephaven-examples/deephaven-ib.git
+    cd deephaven-ib/docker/dev/build.sh
+    # Set jvm_args to the desired JVM memory for Deephaven
+    docker run -it -v data:/data -p 10000:10000 deephaven-examples/deephaven-ib:dev python3 -i -c "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
     ```
-1) Launch the system:
+    * On other platforms:
     ```bash
-    ./docker/deephaven_ib_docker.sh up --dh-version <deephaven_version>
+    # Set jvm_args to the desired JVM memory for Deephaven
+    docker run -it -v data:/data -p 10000:10000 ghcr.io/deephaven-examples/deephaven-ib python3 -i -c "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
     ```
-1) Launch the [Deephaven IDE](https://github.com/deephaven/deephaven-core/blob/main/README.md#run-deephaven-ide) by navigating to [http://localhost:10000/ide/](http://localhost:10000/ide/) in a browser.
+5) Launch the system and execute a custom script (Option 2):
+    * On Mac:
+    ```bash
+    git clone git@github.com:deephaven-examples/deephaven-ib.git
+    cd deephaven-ib/docker/dev/build.sh
+    # your_script.py must begin with: "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
+    # Set jvm_args to the desired JVM memory for Deephaven
+    cp path/to/your_script.py data/your_script.py
+    docker run -it -v data:/data -p 10000:10000 deephaven-examples/deephaven-ib:dev python3 -i /data/your_script.py
+    ```
+    * On other platforms:
+    ```bash
+    # your_script.py must begin with: "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
+    # Set jvm_args to the desired JVM memory for Deephaven
+    cp path/to/your_script.py data/your_script.py
+    docker run -it -v data:/data -p 10000:10000 ghcr.io/deephaven-examples/deephaven-ib python3 -i /data/your_script.py
+    ```
+7) Launch the [Deephaven IDE](https://github.com/deephaven/deephaven-core/blob/main/README.md#run-deephaven-ide) by navigating to [http://localhost:10000/ide/](http://localhost:10000/ide/) in a browser.
 
-## Shutdown
-To shut down the system:
-```bash
-./docker/deephaven_ib_docker.sh down --dh-version <deephaven_version>
-```
+### Launch with a local installation (No Docker)
 
-## Help
-To get help on running the system:
-```bash
-./docker/deephaven_ib_docker.sh help
-```
+> **_NOTE:_**  Deephaven pip install does not yet supported on all architectures.  This launch should work on Linux (AMD64 and ARM64) and Windows WSL.  It is not yet supported on Windows without WSL or Mac.  For these architectures, you should use the Docker installation.  As soon as Deephaven supports these architectures for pip, [deephaven-ib](https://github.com/deephaven-examples/deephaven-ib) will work.
+
+It is possible to use [deephaven-ib](https://github.com/deephaven-examples/deephaven-ib) without docker, but this is a 
+new feature and has not been well tested.  To do this:
+1) Launch [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php).
+2) Accept incoming connections to [IB Trader Workstation (TWS)](https://www.interactivebrokers.com/en/trading/tws.php).  (May not be required for all sessions.)
+![](docs/assets/allow-connections.png)
+3) Install [deephaven-ib](https://github.com/deephaven-examples/deephaven-ib):
+    ```bash
+    pip3 install deephaven-ib
+    ```
+4) Install Java 11 and set the appropriate `JAVA_HOME` environment variable.    
+5) Launch the system (Option 1):
+    ```bash
+    # Set jvm_args to the desired JVM memory for Deephaven
+    python3 -i -c "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
+    ```
+6) Launch the system and execute a custom script (Option 2):
+    ```bash
+    # your_script.py must begin with: "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"
+    # Set jvm_args to the desired JVM memory for Deephaven
+    python3 -i /data/your_script.py
+    ```
+7) Launch the [Deephaven IDE](https://github.com/deephaven/deephaven-core/blob/main/README.md#run-deephaven-ide) by navigating to [http://localhost:10000/ide/](http://localhost:10000/ide/) in a browser.
+8) Use `host=localhost` for the hostname in the examples
 
 # Use deephaven-ib
 

diff --git a/docker/Dockerfile b/docker/Dockerfile