Add developer's guide to upgrade dependencies and a workflow to test …

…it (#28)

.github/workflows/check_requirements.yml

@@ -0,0 +1,45 @@
+name: Check the dependencies in requirements.in and requirements.txt
+
+on:
+  workflow_call:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+  pull_request_target:
+    branches:
+      - main
+
+concurrency:
+  group: check-requirements-${{ github.head_ref || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  tests:
+    name: Run requirements tests on multiple architectures
+    runs-on: [ubuntu-latest, macos-latest]
+
+    steps:
+      - name: Checkout the repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install pip-tools pytest -y
+
+      - name: Generate requirements-generated.txt
+        run: |
+          pip-compile --output-file=./requirements/generated-requirements.txt ./requirements/requirements.in --strip-extras
+
+      - name: Check if requirements.txt is the same as requirements-generated.txt
+        run: |
+          pytest tests

Dockerfile

@@ -25,8 +25,15 @@ RUN alternatives --set python3 /usr/bin/python3.9 && \
     curl https://bootstrap.pypa.io/get-pip.py | python3
 
 # Install pip dependencies
-COPY requirements.txt requirements.txt
-RUN pip3 install --no-cache-dir -r requirements.txt
+COPY requirements requirements
+# Install requirements based on the architecture
+RUN if [ "${TARGETPLATFORM}" = "linux/arm64" ]; then \
+      pip3 install --no-cache-dir -r requirements/arm64-requirements.txt; \
+    elif [ "${TARGETPLATFORM}" = "linux/amd64" ]; then \
+      pip3 install --no-cache-dir -r requirements/amd64-gpu-requirements.txt; \
+    else \
+      echo "Unsupported platform: ${TARGETARCH}" && exit 1; \
+    fi
 
 # Setup scripts and execute them
 COPY scripts scripts

README.md

@@ -3,6 +3,45 @@ The dependencies and Dockerfile for creating the Marqo base image
 
 The marqo-base image is an image that has the necessary dependencies for the Marqo to be installed on and run. This speeds up the build process. 
 
+## Manage Dependencies
+
+The marqo-base image contains all the dependencies needed to run Marqo. We use `pip-compile` to 
+to manage the dependencies. It is a part of the `pip-tools` package and can be installed as
+```bash
+pip install pip-tools
+```
+In case you want to add a new dependency, you can add it to the `requirements.in` file and run
+```bash
+pip-compile requirements.in --output-file=requirements.txt --strip-extras
+```
+to generate the `requirements.txt` file. This file is used to install the dependencies in the Dockerfile.
+If any of the dependencies/sub-dependencies are not pinned and are updated when you run `pip-compile`, you should add
+the new version to the `requirements.in` file and run the above command again until they converge.
+We have an automated pipeline for this check. You **SHOULD NOT** manually update the `requirements.txt` file for dependencies.
+
+### Cross-platform Dependencies:
+We use [environment markers](https://peps.python.org/pep-0508/#environment-markers) to manage cross-platform
+dependencies. For example, we have
+```text
+torch==1.12.1+cu113; platform_machine == "x86_64"
+torch==1.12.1; platform_machine == "arm64" or platform_machine == "aarch64"
+```
+The `pip-comple` tool does not support cross-platform dependencies. If you run `pip-compile` on the `requirements.in` file 
+on an `x86_64` machine, it will generate the `torch==1.12.1+cu113; platform_machine == "x86_64"` line, but remove the
+`torch==1.12.1; platform_machine == "arm64" or platform_machine == "aarch64"` line, and vice versa. 
+
+In such cases, **you are supposed to run `pip-compile` on the `requirements.in` file on both `x86_64` and `arm64` machines** 
+and generate two `requirements.txt` files, e.g., `amd64-gpu-requirements.txt` and `arm64-requirements.txt`. We do not have
+seperated files to distinguish between `amd64` and `amd64-gpu` at the moment, but we might have in the future.
+Here is an example of how to generate the `amd64-gpu-requirements.txt` file:
+```bash
+pip-compile --output-file=./requirements/amd64-gpu-requirements.txt ./requirements/requirements.in --strip-extras
+```
+
+It is not recommended to merge the cross-platform dependencies into a single `requirements.txt` file 
+[here](https://github.com/jazzband/pip-tools?tab=readme-ov-file#cross-environment-usage-of-requirementsinrequirementstxt-and-pip-compile).
+Having separate files for different platforms is the best practice.
+
 ## Build and push a new Marqo-base version to Dockerhub
 To release a new version of Marqo-base to Dockerhub: