Skip to content

Commit

Permalink
Merge pull request #29 from nspcc-dev/build_sharp_from_sources
Browse files Browse the repository at this point in the history
build: add Dockerfile to build C# node from sources
  • Loading branch information
roman-khimov authored Sep 7, 2020
2 parents 2170b83 + 1716edf commit 1568ebc
Show file tree
Hide file tree
Showing 4 changed files with 181 additions and 1 deletion.
1 change: 0 additions & 1 deletion .docker/build/Dockerfile.sharp
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ RUN for mod in ${MODULES}; do \
# Copy RocksDBStore plugin with dependant dlls from Build
COPY --from=Build /tmp/neo-modules-master/src/RocksDBStore/bin/Release/netstandard2.1/publish/RocksDBStore.dll \
/tmp/neo-modules-master/src/RocksDBStore/bin/Release/netstandard2.1/publish/RocksDbSharp.dll \
/tmp/neo-modules-master/src/RocksDBStore/bin/Release/netstandard2.1/publish/RocksDbNative.dll \
/tmp/neo-modules-master/src/RocksDBStore/bin/Release/netstandard2.1/publish/RocksDbNative.dll /neo-cli/Plugins/
COPY --from=Build /tmp/neo-modules-master/src/RocksDBStore/bin/Release/netstandard2.1/publish/RocksDBStore/ /neo-cli/Plugins/RocksDBStore/

Expand Down
121 changes: 121 additions & 0 deletions .docker/build/Dockerfile.sharp.sources
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS Build

# Install system dependencies.
RUN set -x \
&& apt-get update \
&& apt-get install -y \
unzip \
wget

# Publish neo-cli from source as a self-contained deployment for linux-64 into /neo-cli folder (all dependant .dlls are included).
# See https://docs.microsoft.com/ru-ru/dotnet/core/deploying/#publish-self-contained for details.
ENV CLIBRANCH="v3.0.0-preview3"
RUN wget -O /tmp/neo-cli.zip https://github.com/neo-project/neo-node/archive/${CLIBRANCH}.zip && \
unzip -q -d /tmp/neo-node/ /tmp/neo-cli.zip && \
mkdir /build && \
cp -r /tmp/neo-node/*/neo-cli /build && \
cp -r tmp/neo-node/*/Neo.ConsoleService /build && \
cp tmp/neo-node/*/NuGet.Config /build/neo-cli && \
dotnet restore /build/neo-cli && \
dotnet publish -c Release -r linux-x64 /build/neo-cli -o /neo-cli && \
rm -rf /build /tmp/neo-cli.zip /tmp/neo-node

# Build neo-modules from source into /Plugins folder (only plugin .dll and plugin config are included, if you need other dependant .dlls, see the next step)
ENV MODULESBRANCH="v3.0.0-preview3-00"
ENV MODULES="LevelDBStore RocksDBStore RpcServer"
RUN wget -O /tmp/neo-modules.zip https://github.com/neo-project/neo-modules/archive/${MODULESBRANCH}.zip && \
unzip -q -d /tmp/neo-modules/ /tmp/neo-modules.zip && \
mkdir /Plugins && \
for mod in ${MODULES}; do \
dotnet restore /tmp/neo-modules/*/src/${mod}/; \
dotnet build -c Release /tmp/neo-modules/*/src/${mod}/; \
cp -r /tmp/neo-modules/*/src/${mod}/bin/Release/netstandard2.1/* /Plugins/; \
done

# Sometimes Plugins need extra .dlls which can not be retrieved from build folder.
# Use `dotnet publish` instead of `dotnet build` and copy all necessary .dlls from `publish` folder.
# For example, RpsServer Plugin needs Microsoft.AspNetCore.ResponseCompression.dll:
RUN dotnet publish -c Release /tmp/neo-modules/*/src/RpcServer/ && \
cp /tmp/neo-modules/*/src/RpcServer/bin/Release/netstandard2.1/publish/Microsoft.AspNetCore.ResponseCompression.dll /Plugins/ && \
dotnet publish -c Release /tmp/neo-modules/*/src/RocksDBStore/ && \
cp /tmp/neo-modules/*/src/RocksDBStore/bin/Release/netstandard2.1/publish/RocksDbSharp.dll /Plugins/ && \
cp /tmp/neo-modules/*/src/RocksDBStore/bin/Release/netstandard2.1/publish/RocksDbNative.dll /Plugins/ && \
rm -rf /tmp/neo-modules.zip /tmp/neo-modules

# Publish neo-vm from source as a self-contained deployment for linux-64 into /neo-vm folder (although neo-vm does not have dependant .dlls)
ENV NEOVMBRANCH="v3.0.0-preview3"
RUN wget -O /tmp/neo-vm.zip https://github.com/neo-project/neo-vm/archive/${NEOVMBRANCH}.zip && \
unzip -q -d /tmp/neo-vm/ /tmp/neo-vm.zip && \
dotnet restore /tmp/neo-vm/*/src/neo-vm/ && \
dotnet publish -c Release -r linux-x64 /tmp/neo-vm/*/src/neo-vm/ -o /neo-vm && \
rm -rf /tmp/neo-vm.zip tmp/neo-vm

# Publish neo from source as a self-contained deployment for linux-64 into /neo folder (all dependant .dlls are included)
ENV NEOBRANCH="v3.0.0-preview3"
RUN wget -O /tmp/neo.zip https://github.com/neo-project/neo/archive/${NEOBRANCH}.zip && \
unzip -q -d /tmp/neo/ /tmp/neo.zip && \
mkdir /build && \
cp -r /tmp/neo/*/src/neo/* /build && \
cp tmp/neo/*/NuGet.Config /build && \
dotnet restore /build && \
dotnet publish -c Release -r linux-x64 /build -o /neo && \
rm -rf /build /tmp/neo.zip tmp/neo

# All things are published, so build the final image by copying binaries from Build
FROM microsoft/dotnet:3.0-runtime-stretch-slim as Final

# Frontend non-interactive
ENV DEBIAN_FRONTEND noninteractive

# Disable dotnet usage information collection
# https://docs.microsoft.com/en-us/dotnet/core/tools/telemetry#behavior
ENV DOTNET_CLI_TELEMETRY_OPTOUT 1

# Install system dependencies. always should be done in one line
# https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run
RUN set -x \
&& apt-get update \
&& apt-get install -y \
libleveldb-dev \
screen \
jq \
libssl-dev \
libunwind8 \
librocksdb-dev \
libc6-dev \
# APT cleanup to reduce image size
&& rm -rf /var/lib/apt/lists/*

# Copy neo-cli and all dependant .dlls from Build
COPY --from=Build /neo-cli /neo-cli/

# Copy Plugins without dependant .dlls from Build
COPY --from=Build /Plugins /neo-cli/Plugins/

# Replace existing neo.dll (it's one of the neo-cli dependencies) with the built one from /neo/ folder of Build.
# Comment this step in case if you need neo.dll from neo-cli.
# IMPORTANT: if there's a mismatch between neo-cli's and neo's version, be careful with their dependencies.
# All neo dependent .dlls have version of neo-cli (as far as we copied the whole pack of neo-cli dependencies).
# If you need any particular .dll to have version of neo (not neo-cli), just copy this neo dependent .dll from /neo/ into /neo-ci/ folder, e.g.:
# COPY --from-Build /neo/NeoDependentLibrary.dll /neo-cli/
COPY --from=Build /neo/Neo.dll /neo-cli/

# Replace existing Neo.VM.dll (it's one of the neo-cli dependencies) with the built one from /neo-vm/ folder of Build.
# Comment this step in case if you need Neo.VM.dll from neo-cli.
COPY --from=Build /neo-vm/Neo.VM.dll /neo-cli/

# A welcome message for bash users
RUN echo "printf \"\n* Consensus nodes are running in screen sessions, check 'screen -ls'\"" >> /root/.bashrc
RUN echo "printf \"\n* Please report issues to https://github.com/CityOfZion/neo-local\n\n\"" >> /root/.bashrc

WORKDIR /neo-cli

RUN chmod +x /neo-cli/neo-cli

COPY ./dump.acc /
COPY ./single.acc /
COPY ./sharp.entrypoint.sh /entrypoint.sh
COPY ./sharp.healthcheck.sh /healthcheck.sh
COPY ./sharp.rpc.config.json /neo-cli/Plugins/RpcServer/config.json

ENTRYPOINT [ "/entrypoint.sh" ]
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ DC_SHARP_RPC=.docker/rpc/docker-compose.sharp.yml
DF_GO=.docker/build/Dockerfile.golang
DF_BENCH=.docker/build/Dockerfile.bench
DF_SHARP=.docker/build/Dockerfile.sharp
#DF_SHARP=.docker/build/Dockerfile.sharp.sources

TAG=bench
HUB=nspccdev/neo-node
Expand Down
59 changes: 59 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,3 +147,62 @@ TPS
Used for RPC requests.
Example: -t 30s
```

## Build options

By default, neo-bench uses released versions of Neo nodes to build Docker images.
However, you can easily test non-released branches or even separate commits for both Go and C# Neo nodes.

### Build Go node image from sources

To test non-released version of Go Neo node:

1. Set `ARG REV` variable of
[Go node Dockerfile](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.golang#L9)
to the desired branch, tag or commit from the [neo-go github repository](https://github.com/nspcc-dev/neo-go).
Example:
```
ARG REV="v0.91.0"
```

2. Build Go Neo node image with the following command:
```
$ make build
```

### Build C# node image from sources

To test non-released version of C# node:

1. Set `DF_SHARP` variable of [neo-bench Makefile](https://github.com/nspcc-dev/neo-bench/blob/master/Makefile#L18)
to `.docker/build/Dockerfile.sharp.sources`. It is necessary because neo-bench have two separate Dockerfiles to build C#
Neo node image from release and from sources.

2. Set `CLIBRANCH`, `MODULESBRANCH`, `NEOVMBRANCH` and `NEOBRANCH` variables of
[C# node-sources Dockerfile](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L12)
to the desired branch, tag or commit from the corresponding repositories. Refer the following table for the variables meaning:

| Variable | Purpose | Example |
| --- | --- | --- |
| [`CLIBRANCH`](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L12) | Branch, tag or commit from [C# neo-node github repository](https://github.com/neo-project/neo-node) to build neo-cli from the source code | `ENV CLIBRANCH="v3.0.0-preview3"` |
| [`MODULESBRANCH`](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L24) | Branch, tag or commit from [C# neo-modules github repository](https://github.com/neo-project/neo-modules) to build node Plugins from the source code | `ENV MODULESBRANCH="v3.0.0-preview3-00"` |
| [`NEOVMBRANCH`](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L46) | Branch, tag or commit from [C# neo-vm github repository](https://github.com/neo-project/neo-vm) to build neo VM (Neo.VM.dll) from the source code | `ENV NEOVMBRANCH="v3.0.0-preview3"` |
| [`NEOBRANCH`](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L54) | Branch, tag or commit from [C# neo github repository](https://github.com/neo-project/neo/) to build neo itself (Neo.dll) from the source code | `ENV NEOBRANCH="v3.0.0-preview3"` |

3. C# node image includes `LevellDBStore`, `BadgerDBStore` and `RpcServer` Plugins by default. If you need to install other
Plugins, add desired Plugin name to [`MODULES`](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L25)
variable of C# node-sources Dockerfile. This will use `dotnet build` command to build the specified plugin without dependencies.
If you need to build plugin with dependant .dll-s, refer to [this section](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L35-L43)
of C# node-sources Dockerfile.

4. ***Please, be careful while choosing branch, tag or commit on step 2.*** It is possible that one of the `neo-cli`, `neo-modules`, `neo` or `neo-vm` versions
is incompatible with the others. For example, some method required for `neo.dll` from master-branch is missing in `neo-vm.dll` from v3.0.0-preview3-branch. In this
case you either have to provide the compatible `neo-vm` branch via [`NEOVMBRANCH`](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L46)
variable *or* provide `neo-vm.dll` from the dependencies of built `neo` (not from built `neo-vm`) by editing
[these sections](https://github.com/nspcc-dev/neo-bench/blob/master/.docker/build/Dockerfile.sharp.sources#L89-L105) of C# node-sources Dockerfile.

5. Build C# Neo node image with the following command:
```
$ make build
```

0 comments on commit 1568ebc

Please sign in to comment.