Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

An error occurred between opentelemetry modules #8437

Closed
oct28-yjkim opened this issue Apr 6, 2022 · 48 comments · Fixed by #8585, #8606 or #8719
Closed

An error occurred between opentelemetry modules #8437

oct28-yjkim opened this issue Apr 6, 2022 · 48 comments · Fixed by #8585, #8606 or #8719
Assignees
@longwuyuan
Labels
kind/bug Categorizes issue or PR as related to a bug. kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@oct28-yjkim
Copy link

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

bash-5.1$ ./nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.1.3
  Build:         9d3a285f19a704524439c75b947e2189406565ab
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

[root@trace1 ingress-nginx]# kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:10:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:04:34Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration:
  • baremetal, kubernetes on centos7
[root@trace1 ingress-nginx]# cat /etc/*release
CentOS Linux release 7.9.2009 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.9.2009 (Core)
CentOS Linux release 7.9.2009 (Core)

  • Cloud provider or hardware configuration:

  • Kernel (e.g. uname -a):

  [root@trace1 ingress-nginx]# uname -a
Linux trace1 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:

    • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
    • kubespray 2.18
    • helm chart

  • Basic cluster related info:

    • kubectl get nodes -o wide
  [root@trace1 ingress-nginx]# kubectl get nodes
NAME     STATUS   ROLES                  AGE   VERSION
trace1   Ready    control-plane,master   3d    v1.22.8
trace2   Ready    <none>                 3d    v1.22.8
  • How was the ingress-nginx-controller installed:
    • If helm was used then please show output of helm ls -A | grep -i ingress
git clone https://github.com/kubernetes/ingress-nginx.git
cd ingress-nginx/charts/ingress-nginx
cat << EOF >> extra.yml 
  extraModules:
  - name: opentelemetry
    image: gcr.io/k8s-staging-ingress-nginx/opentelemetry:v20220331-controller-v1.1.2-36-g7517b7ecf
EOF

helm install ingress-nginx . -f extra.yml`


[root@trace1 ingress-nginx]# helm ls -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                  APP VERSION
ingress-nginx   default         1               2022-04-06 05:03:26.400791535 +0000 UTC deployed        ingress-nginx-4.0.19   1.1.3
  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
[root@trace1 ingress-nginx]# helm get values ingress-nginx
USER-SUPPLIED VALUES:
null

  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

  • Current State of the controller:

    • kubectl describe ingressclasses
    • kubectl -n <ingresscontrollernamespace> get all -A -o wide
    • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  [root@trace1 ingress-nginx]# kubectl get po -n default
NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-5c5846d6f7-2b7gr   1/1     Running   0          20m

[root@trace1 ingress-nginx]# kubectl describe po ingress-nginx-controller-5c5846d6f7-2b7gr 
Name:         ingress-nginx-controller-5c5846d6f7-2b7gr
Namespace:    default
Priority:     0
Node:         trace2/192.168.16.69
Start Time:   Wed, 06 Apr 2022 05:03:28 +0000
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=5c5846d6f7
Annotations:  cni.projectcalico.org/containerID: 6446671f8ac137a1024148588551902e25e715c3a22f35cbed1544a9a096682a
              cni.projectcalico.org/podIP: 10.233.94.74/32
              cni.projectcalico.org/podIPs: 10.233.94.74/32
Status:       Running
IP:           10.233.94.74
IPs:
  IP:           10.233.94.74
Controlled By:  ReplicaSet/ingress-nginx-controller-5c5846d6f7
Init Containers:
  opentelemetry:
    Container ID:  containerd://24b7aa7186e49c44cfa5ba59f7a580ac4e61f9f527e2052176d927196559b752
    Image:         yjkim1ntels/ingress-nginx:opentelemetry
    Image ID:      docker.io/yjkim1ntels/ingress-nginx@sha256:d86b3679ad13c510d5991f5e2c6e34dc5d2e957e60b493487cff875a138ed806
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      /usr/local/bin/init_module.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 06 Apr 2022 05:03:34 +0000
      Finished:     Wed, 06 Apr 2022 05:03:34 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /modules_mount from modules (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xn5cv (ro)
Containers:
  controller:
    Container ID:  containerd://96113bb6bdbd35d91534ef40c5794d2ac1c551b06b64bd0d91dbb57c4042453f
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
    Image ID:      k8s.gcr.io/ingress-nginx/controller@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Wed, 06 Apr 2022 05:03:35 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-5c5846d6f7-2b7gr (v1:metadata.name)
      POD_NAMESPACE:  default (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /modules_mount from modules (rw)
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xn5cv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True
Volumes:
  modules:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-xn5cv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                      Message
  ----    ------     ----  ----                      -------
  Normal  Scheduled  20m   default-scheduler         Successfully assigned default/ingress-nginx-controller-5c5846d6f7-2b7gr to trace2
  Normal  Pulling    20m   kubelet                   Pulling image "yjkim1ntels/ingress-nginx:opentelemetry"
  Normal  Pulled     20m   kubelet                   Successfully pulled image "yjkim1ntels/ingress-nginx:opentelemetry" in 5.495976532s
  Normal  Created    20m   kubelet                   Created container opentelemetry
  Normal  Started    20m   kubelet                   Started container opentelemetry
  Normal  Pulled     20m   kubelet                   Container image "k8s.gcr.io/ingress-nginx/controller:v1.1.3@sha256:31f47c1e202b39fadecf822a9b76370bd4baed199a005b3e7d4d1455f4fd3fe2" already present on machine
  Normal  Created    20m   kubelet                   Created container controller
  Normal  Started    20m   kubelet                   Started container controller
  Normal  RELOAD     20m   nginx-ingress-controller  NGINX reload triggered due to a change in configuration

What happened:

  • An error occurred while adding the openteletry module as a sidecar to ingress-nginx.
ls -al /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so
-rwxr-xr-x    1 root     root       8077416 Apr  6 04:20 /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so

# add nginx.conf load_module 

bash-5.1$ cat nginx.conf | head -n 3
load_module /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so;
# Configuration checksum: 6642706386070326205

nginx -s reload
 
# error 
2022/04/06 04:45:48 [emerg] 619#619: dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found) in /etc/nginx/nginx.conf:1

What you expected to happen:

  • I want the nginx-ingress-controller and opentelemetry module to work together.
@oct28-yjkim oct28-yjkim added the kind/bug Categorizes issue or PR as related to a bug. label Apr 6, 2022
@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Apr 6, 2022
@longwuyuan longwuyuan mentioned this issue Apr 6, 2022
Closed
8 tasks
@longwuyuan
Copy link
Contributor

/remove-kind bug
/kind feature

@dmathieu, based on the PR you submitted earlier, wondering if you have any comments on this

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed kind/bug Categorizes issue or PR as related to a bug. labels Apr 6, 2022
@oct28-yjkim
Copy link
Author

@dmathieu
currently i try manually build

docker run -it --rm alpine:3.14.4 bash 
/ # # copy /build.sh to /build.sh 

sh -x build.sh 

/ # sh -x build.sh 
+ set -o errexit
+ set -o nounset
+ set -o pipefail
+ export 'NGINX_VERSION=1.19.10'
+ export 'OPENTELEMETRY_CPP_VERSION=1.2.0'
+ export 'OPENTELEMETRY_CONTRIB_COMMIT=2656a4072e257b6794da86ddd1b773b49f5517b3'
+ export 'BUILD_PATH=/tmp/build'
+ rm -rf '/var/cache/debconf/*' '/var/lib/apt/lists/*' '/var/log/*' /tmp/build '/var/tmp/*'
+ mkdir -p /etc/nginx
+ mkdir --verbose -p /tmp/build
created directory: '/tmp/build'
+ cd /tmp/build
+ apk add curl git build-base
OK: 313 MiB in 74 packages
+ get_src e8d0290ff561986ad7cd6c33307e12e11b137186c4403a6a5ccdb4914c082d88 https://nginx.org/download/nginx-1.19.10.tar.gz
+ hash=e8d0290ff561986ad7cd6c33307e12e11b137186c4403a6a5ccdb4914c082d88
+ url=https://nginx.org/download/nginx-1.19.10.tar.gz
+ basename https://nginx.org/download/nginx-1.19.10.tar.gz
+ f=nginx-1.19.10.tar.gz
+ echo 'Downloading https://nginx.org/download/nginx-1.19.10.tar.gz'
Downloading https://nginx.org/download/nginx-1.19.10.tar.gz
+ curl -sSL https://nginx.org/download/nginx-1.19.10.tar.gz -o nginx-1.19.10.tar.gz
+ echo 'e8d0290ff561986ad7cd6c33307e12e11b137186c4403a6a5ccdb4914c082d88  nginx-1.19.10.tar.gz'
+ sha256sum -c -
nginx-1.19.10.tar.gz: OK
+ tar xzf nginx-1.19.10.tar.gz
+ rm -rf nginx-1.19.10.tar.gz
+ get_src 360cdcbd1a235ec62119cc53956b2d31b6ff5f41d44415be53acc544709d58b8 https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ hash=360cdcbd1a235ec62119cc53956b2d31b6ff5f41d44415be53acc544709d58b8
+ url=https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ basename https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ f=2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ echo 'Downloading https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz'
Downloading https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ curl -sSL https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz -o 2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ echo '360cdcbd1a235ec62119cc53956b2d31b6ff5f41d44415be53acc544709d58b8  2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz'
+ sha256sum -c -
2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz: OK
+ tar xzf 2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ rm -rf 2656a4072e257b6794da86ddd1b773b49f5517b3.tar.gz
+ grep -c ^processor /proc/cpuinfo
+ CORES=15
+ export 'MAKEFLAGS=-j15'
+ apk add protobuf-dev grpc grpc-dev gtest-dev c-ares-dev pcre-dev
OK: 313 MiB in 74 packages
+ cd /tmp/build
+ git clone --recursive https://github.com/open-telemetry/opentelemetry-cpp opentelemetry-cpp-1.2.0
Cloning into 'opentelemetry-cpp-1.2.0'...
remote: Enumerating objects: 12353, done.
remote: Counting objects: 100% (2384/2384), done.
remote: Compressing objects: 100% (1129/1129), done.
remote: Total 12353 (delta 1339), reused 1872 (delta 1052), pack-reused 9969
Receiving objects: 100% (12353/12353), 3.36 MiB | 10.26 MiB/s, done.
Resolving deltas: 100% (7307/7307), done.
Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/benchmark'
Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/googletest'
Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/ms-gsl'
Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/nlohmann-json'
Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-proto'
Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/prometheus-cpp'
Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'tools/vcpkg'
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/benchmark'...
remote: Enumerating objects: 7074, done.        
remote: Counting objects: 100% (1534/1534), done.        
remote: Compressing objects: 100% (266/266), done.        
remote: Total 7074 (delta 1392), reused 1273 (delta 1267), pack-reused 5540        
Receiving objects: 100% (7074/7074), 2.15 MiB | 9.40 MiB/s, done.
Resolving deltas: 100% (4751/4751), done.
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/googletest'...
remote: Enumerating objects: 24156, done.        
remote: Counting objects: 100% (614/614), done.        
remote: Compressing objects: 100% (293/293), done.        
remote: Total 24156 (delta 361), reused 474 (delta 303), pack-reused 23542        
Receiving objects: 100% (24156/24156), 10.29 MiB | 7.40 MiB/s, done.
Resolving deltas: 100% (17753/17753), done.
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/ms-gsl'...
remote: Enumerating objects: 4626, done.        
remote: Counting objects: 100% (228/228), done.        
remote: Compressing objects: 100% (155/155), done.        
remote: Total 4626 (delta 127), reused 139 (delta 72), pack-reused 4398        
Receiving objects: 100% (4626/4626), 1.57 MiB | 9.43 MiB/s, done.
Resolving deltas: 100% (3034/3034), done.
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/nlohmann-json'...
remote: Enumerating objects: 89408, done.        
remote: Counting objects: 100% (14798/14798), done.        
remote: Compressing objects: 100% (1806/1806), done.        
remote: Total 89408 (delta 11171), reused 14320 (delta 10846), pack-reused 74610
Receiving objects: 100% (89408/89408), 285.48 MiB | 9.99 MiB/s, done.
Resolving deltas: 100% (70756/70756), done.
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/opentelemetry-proto'...
remote: Enumerating objects: 1652, done.        
remote: Counting objects: 100% (391/391), done.        
remote: Compressing objects: 100% (204/204), done.        
remote: Total 1652 (delta 156), reused 256 (delta 75), pack-reused 1261        
Receiving objects: 100% (1652/1652), 499.47 KiB | 7.45 MiB/s, done.
Resolving deltas: 100% (614/614), done.
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/prometheus-cpp'...
remote: Enumerating objects: 4534, done.        
remote: Counting objects: 100% (779/779), done.        
remote: Compressing objects: 100% (402/402), done.        
remote: Total 4534 (delta 506), reused 486 (delta 326), pack-reused 3755        
Receiving objects: 100% (4534/4534), 1.03 MiB | 1.75 MiB/s, done.
Resolving deltas: 100% (2787/2787), done.
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/tools/vcpkg'...
remote: Enumerating objects: 146836, done.        
remote: Counting objects: 100% (94/94), done.        
remote: Compressing objects: 100% (21/21), done.        
remote: Total 146836 (delta 81), reused 73 (delta 73), pack-reused 146742        
Receiving objects: 100% (146836/146836), 55.37 MiB | 9.66 MiB/s, done.
Resolving deltas: 100% (91993/91993), done.
Submodule path 'third_party/benchmark': checked out 'c05843a9f622db08ad59804c190f98879b76beba'
Submodule path 'third_party/googletest': checked out 'a6dfd3aca7f2f91f95fc7ab650c95a48420d513d'
Submodule path 'third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
Submodule path 'third_party/nlohmann-json': checked out 'db78ac1d7716f56fc9f1b030b715f872f93964e4'
Submodule path 'third_party/opentelemetry-proto': checked out 'b43e9b18b76abf3ee040164b55b9c355217151f3'
Submodule path 'third_party/prometheus-cpp': checked out '4ea303fa66e4c26dc4df67045fa0edf09c2f3077'
Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/prometheus-cpp/3rdparty/civetweb'
Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/prometheus-cpp/3rdparty/googletest'
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/prometheus-cpp/3rdparty/civetweb'...
remote: Enumerating objects: 24520, done.        
remote: Counting objects: 100% (332/332), done.        
remote: Compressing objects: 100% (135/135), done.        
remote: Total 24520 (delta 200), reused 293 (delta 195), pack-reused 24188        
Receiving objects: 100% (24520/24520), 27.54 MiB | 7.64 MiB/s, done.
Resolving deltas: 100% (15909/15909), done.
Cloning into '/tmp/build/opentelemetry-cpp-1.2.0/third_party/prometheus-cpp/3rdparty/googletest'...
remote: Enumerating objects: 24156, done.        
remote: Counting objects: 100% (614/614), done.        
remote: Compressing objects: 100% (303/303), done.        
remote: Total 24156 (delta 356), reused 466 (delta 293), pack-reused 23542        
Receiving objects: 100% (24156/24156), 10.32 MiB | 9.98 MiB/s, done.
Resolving deltas: 100% (17744/17744), done.
Submodule path 'third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
Submodule path 'third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
Submodule path 'tools/vcpkg': checked out '5568f110b509a9fd90711978a7cb76bae75bb092'
+ cd opentelemetry-cpp-1.2.0
+ git checkout v1.2.0
M       third_party/prometheus-cpp
Note: switching to 'v1.2.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 033b16f Prepare for 1.2.0 release (#1188)
+ mkdir .build
+ cd .build
+ cmake '-DCMAKE_BUILD_TYPE=Release' '-DBUILD_TESTING=OFF' '-DWITH_EXAMPLES=OFF' '-DCMAKE_POSITION_INDEPENDENT_CODE=ON' '-DWITH_OTLP=ON' '-DWITH_OTLP_HTTP=OFF' ..
-- The C compiler identification is GNU 10.3.1
-- The CXX compiler identification is GNU 10.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building for architecture ARCH=x64
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
Using external nlohmann::json
-- Found Protobuf: /usr/lib/libprotobuf.so (found version "3.15.7") 
-- Found ZLIB: /lib/libz.so (found version "1.2.12") 
-- Found OpenSSL: /usr/lib/libcrypto.so (found version "1.1.1n")  
-- Could NOT find c-ares (missing: c-ares_DIR)
-- Found c-ares: /usr/include (found version "1.17.2") 
-- PROTOBUF_PROTOC_EXECUTABLE=/usr/bin/protoc
-- Performing Test check_cxx_compiler_flag_-Wno-type-limits
-- Performing Test check_cxx_compiler_flag_-Wno-type-limits - Success
-- Performing Test check_cxx_compiler_flag_-Wno-deprecated-declarations
-- Performing Test check_cxx_compiler_flag_-Wno-deprecated-declarations - Success
-- Performing Test check_cxx_compiler_flag_-Wno-unused-parameter
-- Performing Test check_cxx_compiler_flag_-Wno-unused-parameter - Success
-- gRPC_CPP_PLUGIN_EXECUTABLE=/usr/bin/grpc_cpp_plugin
Building with nostd types...
-- Could NOT find CURL (missing: CURL_LIBRARY CURL_INCLUDE_DIR) 
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/build/opentelemetry-cpp-1.2.0/.build
+ make
[  2%] Generating generated/third_party/opentelemetry-proto/opentelemetry/proto/common/v1/common.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/common/v1/common.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/resource/v1/resource.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/resource/v1/resource.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/trace/v1/trace.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/trace/v1/trace.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/logs/v1/logs.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/logs/v1/logs.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/metrics/v1/metrics.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/metrics/v1/metrics.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/trace/v1/trace_service.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/trace/v1/trace_service.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/trace/v1/trace_service.grpc.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/trace/v1/trace_service.grpc.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/logs/v1/logs_service.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/logs/v1/logs_service.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/logs/v1/logs_service.grpc.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/logs/v1/logs_service.grpc.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/metrics/v1/metrics_service.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/metrics/v1/metrics_service.pb.cc, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/metrics/v1/metrics_service.grpc.pb.h, generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/metrics/v1/metrics_service.grpc.pb.cc
[ 11%] Building CXX object sdk/src/common/CMakeFiles/opentelemetry_common.dir/core.cc.o[ 11%] Building CXX object sdk/src/common/CMakeFiles/opentelemetry_common.dir/random.cc.o

[ 11%] Building CXX object sdk/src/version/CMakeFiles/opentelemetry_version.dir/version.cc.o
[ 11%] Building CXX object sdk/src/common/CMakeFiles/opentelemetry_common.dir/platform/fork_unix.cc.o
[ 13%] Linking CXX static library libopentelemetry_version.a
[ 13%] Built target opentelemetry_version
[ 15%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/common/v1/common.pb.cc.o
[ 17%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/trace/v1/trace.pb.cc.o
[ 22%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/metrics/v1/metrics.pb.cc.o
[ 22%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/trace/v1/trace_service.pb.cc.o
[ 26%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/resource/v1/resource.pb.cc.o
[ 26%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/trace/v1/trace_service.grpc.pb.cc.o
[ 28%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/logs/v1/logs_service.pb.cc.o
[ 31%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/logs/v1/logs_service.grpc.pb.cc.o
[ 37%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/metrics/v1/metrics_service.grpc.pb.cc.o[ 37%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/logs/v1/logs.pb.cc.o

[ 37%] Building CXX object CMakeFiles/opentelemetry_proto.dir/generated/third_party/opentelemetry-proto/opentelemetry/proto/collector/metrics/v1/metrics_service.pb.cc.o
[ 40%] Linking CXX static library libopentelemetry_common.a
[ 40%] Built target opentelemetry_common
[ 46%] Building CXX object sdk/src/resource/CMakeFiles/opentelemetry_resources.dir/resource.cc.o
[ 46%] Building CXX object sdk/src/resource/CMakeFiles/opentelemetry_resources.dir/resource_detector.cc.o
[ 46%] Building CXX object sdk/src/metrics/CMakeFiles/opentelemetry_metrics.dir/meter_provider.cc.o
[ 48%] Building CXX object sdk/src/metrics/CMakeFiles/opentelemetry_metrics.dir/meter.cc.o
[ 51%] Building CXX object sdk/src/metrics/CMakeFiles/opentelemetry_metrics.dir/meter_context.cc.o
[ 53%] Linking CXX static library libopentelemetry_resources.a
[ 53%] Built target opentelemetry_resources
[ 60%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/tracer_context.cc.o[ 60%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/tracer.cc.o

[ 60%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/tracer_provider.cc.o
[ 62%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/batch_span_processor.cc.o
[ 68%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/samplers/parent.cc.o[ 68%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/span.cc.o

[ 68%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/samplers/trace_id_ratio.cc.o
[ 71%] Building CXX object sdk/src/trace/CMakeFiles/opentelemetry_trace.dir/random_id_generator.cc.o
[ 73%] Linking CXX static library libopentelemetry_metrics.a
[ 73%] Built target opentelemetry_metrics
[ 75%] Linking CXX static library libopentelemetry_proto.a
[ 75%] Built target opentelemetry_proto
[ 77%] Linking CXX static library libopentelemetry_trace.a
[ 77%] Built target opentelemetry_trace
[ 82%] Building CXX object exporters/otlp/CMakeFiles/opentelemetry_otlp_recordable.dir/src/otlp_log_recordable.cc.o
[ 82%] Building CXX object exporters/ostream/CMakeFiles/opentelemetry_exporter_ostream_span.dir/src/span_exporter.cc.o
[ 84%] Building CXX object exporters/otlp/CMakeFiles/opentelemetry_otlp_recordable.dir/src/otlp_recordable.cc.o
[ 86%] Building CXX object exporters/otlp/CMakeFiles/opentelemetry_otlp_recordable.dir/src/otlp_recordable_utils.cc.o
[ 88%] Linking CXX static library libopentelemetry_otlp_recordable.a
[ 88%] Built target opentelemetry_otlp_recordable
[ 91%] Building CXX object exporters/otlp/CMakeFiles/opentelemetry_exporter_otlp_grpc_log.dir/src/otlp_grpc_log_exporter.cc.o
[ 93%] Building CXX object exporters/otlp/CMakeFiles/opentelemetry_exporter_otlp_grpc.dir/src/otlp_grpc_exporter.cc.o
[ 95%] Linking CXX static library libopentelemetry_exporter_otlp_grpc_log.a
[ 95%] Built target opentelemetry_exporter_otlp_grpc_log
[ 97%] Linking CXX static library libopentelemetry_exporter_ostream_span.a
[ 97%] Built target opentelemetry_exporter_ostream_span
[100%] Linking CXX static library libopentelemetry_exporter_otlp_grpc.a
[100%] Built target opentelemetry_exporter_otlp_grpc
+ make install
Consolidate compiler generated dependencies of target opentelemetry_version
Consolidate compiler generated dependencies of target opentelemetry_common
Consolidate compiler generated dependencies of target opentelemetry_proto
[  4%] Built target opentelemetry_version
[ 13%] Built target opentelemetry_common
Consolidate compiler generated dependencies of target opentelemetry_resources
Consolidate compiler generated dependencies of target opentelemetry_metrics
[ 42%] Built target opentelemetry_proto
[ 48%] Built target opentelemetry_resources
[ 57%] Built target opentelemetry_metrics
Consolidate compiler generated dependencies of target opentelemetry_trace
[ 77%] Built target opentelemetry_trace
Consolidate compiler generated dependencies of target opentelemetry_exporter_ostream_span
Consolidate compiler generated dependencies of target opentelemetry_otlp_recordable
[ 82%] Built target opentelemetry_exporter_ostream_span
[ 91%] Built target opentelemetry_otlp_recordable
Consolidate compiler generated dependencies of target opentelemetry_exporter_otlp_grpc_log
Consolidate compiler generated dependencies of target opentelemetry_exporter_otlp_grpc
[ 95%] Built target opentelemetry_exporter_otlp_grpc_log
[100%] Built target opentelemetry_exporter_otlp_grpc
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libopentelemetry_proto.a
-- Up-to-date: /usr/local/include/opentelemetry
-- Up-to-date: /usr/local/include/opentelemetry/proto
-- Up-to-date: /usr/local/include/opentelemetry/proto/collector
-- Up-to-date: /usr/local/include/opentelemetry/proto/collector/logs
-- Up-to-date: /usr/local/include/opentelemetry/proto/collector/logs/v1
-- Installing: /usr/local/include/opentelemetry/proto/collector/logs/v1/logs_service.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/collector/logs/v1/logs_service.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/collector/logs/v1/logs_service_mock.grpc.pb.h
-- Up-to-date: /usr/local/include/opentelemetry/proto/collector/metrics
-- Up-to-date: /usr/local/include/opentelemetry/proto/collector/metrics/v1
-- Installing: /usr/local/include/opentelemetry/proto/collector/metrics/v1/metrics_service.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/collector/metrics/v1/metrics_service.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/collector/metrics/v1/metrics_service_mock.grpc.pb.h
-- Up-to-date: /usr/local/include/opentelemetry/proto/collector/trace
-- Up-to-date: /usr/local/include/opentelemetry/proto/collector/trace/v1
-- Installing: /usr/local/include/opentelemetry/proto/collector/trace/v1/trace_service.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/collector/trace/v1/trace_service.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/collector/trace/v1/trace_service_mock.grpc.pb.h
-- Up-to-date: /usr/local/include/opentelemetry/proto/common
-- Up-to-date: /usr/local/include/opentelemetry/proto/common/v1
-- Installing: /usr/local/include/opentelemetry/proto/common/v1/common.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/common/v1/common.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/common/v1/common_mock.grpc.pb.h
-- Up-to-date: /usr/local/include/opentelemetry/proto/logs
-- Up-to-date: /usr/local/include/opentelemetry/proto/logs/v1
-- Installing: /usr/local/include/opentelemetry/proto/logs/v1/logs.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/logs/v1/logs.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/logs/v1/logs_mock.grpc.pb.h
-- Up-to-date: /usr/local/include/opentelemetry/proto/metrics
-- Up-to-date: /usr/local/include/opentelemetry/proto/metrics/v1
-- Installing: /usr/local/include/opentelemetry/proto/metrics/v1/metrics.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/metrics/v1/metrics.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/metrics/v1/metrics_mock.grpc.pb.h
-- Up-to-date: /usr/local/include/opentelemetry/proto/resource
-- Up-to-date: /usr/local/include/opentelemetry/proto/resource/v1
-- Installing: /usr/local/include/opentelemetry/proto/resource/v1/resource.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/resource/v1/resource.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/resource/v1/resource_mock.grpc.pb.h
-- Up-to-date: /usr/local/include/opentelemetry/proto/trace
-- Up-to-date: /usr/local/include/opentelemetry/proto/trace/v1
-- Installing: /usr/local/include/opentelemetry/proto/trace/v1/trace.grpc.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/trace/v1/trace.pb.h
-- Installing: /usr/local/include/opentelemetry/proto/trace/v1/trace_mock.grpc.pb.h
-- Installing: /usr/local/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config.cmake
-- Installing: /usr/local/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-config-version.cmake
-- Installing: /usr/local/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-target.cmake
-- Installing: /usr/local/lib/cmake/opentelemetry-cpp/opentelemetry-cpp-target-release.cmake
-- Up-to-date: /usr/local/include/opentelemetry
-- Up-to-date: /usr/local/include/opentelemetry/_metrics
-- Installing: /usr/local/include/opentelemetry/_metrics/async_instruments.h
-- Installing: /usr/local/include/opentelemetry/_metrics/instrument.h
-- Installing: /usr/local/include/opentelemetry/_metrics/meter.h
-- Installing: /usr/local/include/opentelemetry/_metrics/meter_provider.h
-- Installing: /usr/local/include/opentelemetry/_metrics/noop.h
-- Installing: /usr/local/include/opentelemetry/_metrics/observer_result.h
-- Installing: /usr/local/include/opentelemetry/_metrics/provider.h
-- Installing: /usr/local/include/opentelemetry/_metrics/sync_instruments.h
-- Up-to-date: /usr/local/include/opentelemetry/baggage
-- Installing: /usr/local/include/opentelemetry/baggage/baggage.h
-- Installing: /usr/local/include/opentelemetry/baggage/baggage_context.h
-- Up-to-date: /usr/local/include/opentelemetry/baggage/propagation
-- Installing: /usr/local/include/opentelemetry/baggage/propagation/baggage_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/common
-- Installing: /usr/local/include/opentelemetry/common/attribute_value.h
-- Installing: /usr/local/include/opentelemetry/common/key_value_iterable.h
-- Installing: /usr/local/include/opentelemetry/common/key_value_iterable_view.h
-- Installing: /usr/local/include/opentelemetry/common/kv_properties.h
-- Installing: /usr/local/include/opentelemetry/common/macros.h
-- Installing: /usr/local/include/opentelemetry/common/spin_lock_mutex.h
-- Installing: /usr/local/include/opentelemetry/common/string_util.h
-- Installing: /usr/local/include/opentelemetry/common/timestamp.h
-- Installing: /usr/local/include/opentelemetry/config.h
-- Up-to-date: /usr/local/include/opentelemetry/context
-- Installing: /usr/local/include/opentelemetry/context/context.h
-- Installing: /usr/local/include/opentelemetry/context/context_value.h
-- Up-to-date: /usr/local/include/opentelemetry/context/propagation
-- Installing: /usr/local/include/opentelemetry/context/propagation/composite_propagator.h
-- Installing: /usr/local/include/opentelemetry/context/propagation/global_propagator.h
-- Installing: /usr/local/include/opentelemetry/context/propagation/noop_propagator.h
-- Installing: /usr/local/include/opentelemetry/context/propagation/text_map_propagator.h
-- Installing: /usr/local/include/opentelemetry/context/runtime_context.h
-- Up-to-date: /usr/local/include/opentelemetry/detail
-- Installing: /usr/local/include/opentelemetry/detail/preprocessor.h
-- Up-to-date: /usr/local/include/opentelemetry/logs
-- Installing: /usr/local/include/opentelemetry/logs/logger.h
-- Installing: /usr/local/include/opentelemetry/logs/logger_provider.h
-- Installing: /usr/local/include/opentelemetry/logs/noop.h
-- Installing: /usr/local/include/opentelemetry/logs/provider.h
-- Installing: /usr/local/include/opentelemetry/logs/severity.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail
-- Installing: /usr/local/include/opentelemetry/nostd/detail/all.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/decay.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/dependent_type.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/functional.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/invoke.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/trait.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/type_pack_element.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/valueless.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/variant_alternative.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/variant_fwd.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/variant_size.h
-- Installing: /usr/local/include/opentelemetry/nostd/detail/void.h
-- Installing: /usr/local/include/opentelemetry/nostd/function_ref.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/attributes.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/config.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal/identity.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal/inline_variable.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal/invoke.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/macros.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/optimization.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/options.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/policy_checks.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/base/port.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/meta
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/meta/type_traits.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/types
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/types/bad_variant_access.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/types/internal
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/types/internal/variant.h
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/types/variant.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/utility
-- Installing: /usr/local/include/opentelemetry/nostd/internal/absl/utility/utility.h
-- Installing: /usr/local/include/opentelemetry/nostd/shared_ptr.h
-- Installing: /usr/local/include/opentelemetry/nostd/span.h
-- Installing: /usr/local/include/opentelemetry/nostd/string_view.h
-- Installing: /usr/local/include/opentelemetry/nostd/type_traits.h
-- Installing: /usr/local/include/opentelemetry/nostd/unique_ptr.h
-- Installing: /usr/local/include/opentelemetry/nostd/utility.h
-- Installing: /usr/local/include/opentelemetry/nostd/variant.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail
-- Installing: /usr/local/include/opentelemetry/plugin/detail/dynamic_library_handle.h
-- Installing: /usr/local/include/opentelemetry/plugin/detail/dynamic_load_unix.h
-- Installing: /usr/local/include/opentelemetry/plugin/detail/dynamic_load_windows.h
-- Installing: /usr/local/include/opentelemetry/plugin/detail/loader_info.h
-- Installing: /usr/local/include/opentelemetry/plugin/detail/tracer_handle.h
-- Installing: /usr/local/include/opentelemetry/plugin/detail/utility.h
-- Installing: /usr/local/include/opentelemetry/plugin/dynamic_load.h
-- Installing: /usr/local/include/opentelemetry/plugin/factory.h
-- Installing: /usr/local/include/opentelemetry/plugin/hook.h
-- Installing: /usr/local/include/opentelemetry/plugin/tracer.h
-- Up-to-date: /usr/local/include/opentelemetry/std
-- Installing: /usr/local/include/opentelemetry/std/shared_ptr.h
-- Installing: /usr/local/include/opentelemetry/std/span.h
-- Installing: /usr/local/include/opentelemetry/std/string_view.h
-- Installing: /usr/local/include/opentelemetry/std/type_traits.h
-- Installing: /usr/local/include/opentelemetry/std/unique_ptr.h
-- Installing: /usr/local/include/opentelemetry/std/utility.h
-- Installing: /usr/local/include/opentelemetry/std/variant.h
-- Up-to-date: /usr/local/include/opentelemetry/trace
-- Installing: /usr/local/include/opentelemetry/trace/canonical_code.h
-- Installing: /usr/local/include/opentelemetry/trace/context.h
-- Installing: /usr/local/include/opentelemetry/trace/default_span.h
-- Installing: /usr/local/include/opentelemetry/trace/experimental_semantic_conventions.h
-- Installing: /usr/local/include/opentelemetry/trace/noop.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation
-- Installing: /usr/local/include/opentelemetry/trace/propagation/b3_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation/detail
-- Installing: /usr/local/include/opentelemetry/trace/propagation/detail/hex.h
-- Installing: /usr/local/include/opentelemetry/trace/propagation/detail/string.h
-- Installing: /usr/local/include/opentelemetry/trace/propagation/http_trace_context.h
-- Installing: /usr/local/include/opentelemetry/trace/propagation/jaeger.h
-- Installing: /usr/local/include/opentelemetry/trace/provider.h
-- Installing: /usr/local/include/opentelemetry/trace/scope.h
-- Installing: /usr/local/include/opentelemetry/trace/span.h
-- Installing: /usr/local/include/opentelemetry/trace/span_context.h
-- Installing: /usr/local/include/opentelemetry/trace/span_context_kv_iterable.h
-- Installing: /usr/local/include/opentelemetry/trace/span_context_kv_iterable_view.h
-- Installing: /usr/local/include/opentelemetry/trace/span_id.h
-- Installing: /usr/local/include/opentelemetry/trace/span_metadata.h
-- Installing: /usr/local/include/opentelemetry/trace/span_startoptions.h
-- Installing: /usr/local/include/opentelemetry/trace/trace_flags.h
-- Installing: /usr/local/include/opentelemetry/trace/trace_id.h
-- Installing: /usr/local/include/opentelemetry/trace/trace_state.h
-- Installing: /usr/local/include/opentelemetry/trace/tracer.h
-- Installing: /usr/local/include/opentelemetry/trace/tracer_provider.h
-- Installing: /usr/local/include/opentelemetry/version.h
-- Up-to-date: /usr/local/include/opentelemetry
-- Up-to-date: /usr/local/include/opentelemetry/_metrics
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/async_instruments.h
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/instrument.h
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/meter.h
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/meter_provider.h
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/noop.h
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/observer_result.h
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/provider.h
-- Up-to-date: /usr/local/include/opentelemetry/_metrics/sync_instruments.h
-- Up-to-date: /usr/local/include/opentelemetry/baggage
-- Up-to-date: /usr/local/include/opentelemetry/baggage/baggage.h
-- Up-to-date: /usr/local/include/opentelemetry/baggage/baggage_context.h
-- Up-to-date: /usr/local/include/opentelemetry/baggage/propagation
-- Up-to-date: /usr/local/include/opentelemetry/baggage/propagation/baggage_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/common
-- Up-to-date: /usr/local/include/opentelemetry/common/attribute_value.h
-- Up-to-date: /usr/local/include/opentelemetry/common/key_value_iterable.h
-- Up-to-date: /usr/local/include/opentelemetry/common/key_value_iterable_view.h
-- Up-to-date: /usr/local/include/opentelemetry/common/kv_properties.h
-- Up-to-date: /usr/local/include/opentelemetry/common/macros.h
-- Up-to-date: /usr/local/include/opentelemetry/common/spin_lock_mutex.h
-- Up-to-date: /usr/local/include/opentelemetry/common/string_util.h
-- Up-to-date: /usr/local/include/opentelemetry/common/timestamp.h
-- Up-to-date: /usr/local/include/opentelemetry/config.h
-- Up-to-date: /usr/local/include/opentelemetry/context
-- Up-to-date: /usr/local/include/opentelemetry/context/context.h
-- Up-to-date: /usr/local/include/opentelemetry/context/context_value.h
-- Up-to-date: /usr/local/include/opentelemetry/context/propagation
-- Up-to-date: /usr/local/include/opentelemetry/context/propagation/composite_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/context/propagation/global_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/context/propagation/noop_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/context/propagation/text_map_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/context/runtime_context.h
-- Up-to-date: /usr/local/include/opentelemetry/detail
-- Up-to-date: /usr/local/include/opentelemetry/detail/preprocessor.h
-- Up-to-date: /usr/local/include/opentelemetry/metrics
-- Installing: /usr/local/include/opentelemetry/metrics/meter.h
-- Installing: /usr/local/include/opentelemetry/metrics/meter_provider.h
-- Installing: /usr/local/include/opentelemetry/metrics/provider.h
-- Installing: /usr/local/include/opentelemetry/metrics/async_instruments.h
-- Installing: /usr/local/include/opentelemetry/metrics/noop.h
-- Installing: /usr/local/include/opentelemetry/metrics/observer_result.h
-- Installing: /usr/local/include/opentelemetry/metrics/sync_instruments.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/all.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/decay.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/dependent_type.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/functional.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/invoke.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/trait.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/type_pack_element.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/valueless.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/variant_alternative.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/variant_fwd.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/variant_size.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/detail/void.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/function_ref.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/attributes.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/config.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal/identity.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal/inline_variable.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/internal/invoke.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/macros.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/optimization.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/options.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/policy_checks.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/base/port.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/meta
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/meta/type_traits.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/types
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/types/bad_variant_access.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/types/internal
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/types/internal/variant.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/types/variant.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/utility
-- Up-to-date: /usr/local/include/opentelemetry/nostd/internal/absl/utility/utility.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/shared_ptr.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/span.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/string_view.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/type_traits.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/unique_ptr.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/utility.h
-- Up-to-date: /usr/local/include/opentelemetry/nostd/variant.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail/dynamic_library_handle.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail/dynamic_load_unix.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail/dynamic_load_windows.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail/loader_info.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail/tracer_handle.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/detail/utility.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/dynamic_load.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/factory.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/hook.h
-- Up-to-date: /usr/local/include/opentelemetry/plugin/tracer.h
-- Up-to-date: /usr/local/include/opentelemetry/std
-- Up-to-date: /usr/local/include/opentelemetry/std/shared_ptr.h
-- Up-to-date: /usr/local/include/opentelemetry/std/span.h
-- Up-to-date: /usr/local/include/opentelemetry/std/string_view.h
-- Up-to-date: /usr/local/include/opentelemetry/std/type_traits.h
-- Up-to-date: /usr/local/include/opentelemetry/std/unique_ptr.h
-- Up-to-date: /usr/local/include/opentelemetry/std/utility.h
-- Up-to-date: /usr/local/include/opentelemetry/std/variant.h
-- Up-to-date: /usr/local/include/opentelemetry/trace
-- Up-to-date: /usr/local/include/opentelemetry/trace/canonical_code.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/context.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/default_span.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/experimental_semantic_conventions.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/noop.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation/b3_propagator.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation/detail
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation/detail/hex.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation/detail/string.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation/http_trace_context.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/propagation/jaeger.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/provider.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/scope.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/span.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/span_context.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/span_context_kv_iterable.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/span_context_kv_iterable_view.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/span_id.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/span_metadata.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/span_startoptions.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/trace_flags.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/trace_id.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/trace_state.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/tracer.h
-- Up-to-date: /usr/local/include/opentelemetry/trace/tracer_provider.h
-- Up-to-date: /usr/local/include/opentelemetry/version.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common
-- Installing: /usr/local/include/opentelemetry//sdk/common/atomic_shared_ptr.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/atomic_unique_ptr.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/circular_buffer.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/circular_buffer_range.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/empty_attributes.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/env_variables.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/exporter_utils.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/global_log_handler.h
-- Installing: /usr/local/include/opentelemetry//sdk/common/attribute_utils.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/instrumentationlibrary
-- Installing: /usr/local/include/opentelemetry//sdk/instrumentationlibrary/instrumentation_library.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/logs
-- Installing: /usr/local/include/opentelemetry//sdk/logs/exporter.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/log_record.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/logger_context.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/multi_log_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/multi_recordable.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/recordable.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/simple_log_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/batch_log_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/logger.h
-- Installing: /usr/local/include/opentelemetry//sdk/logs/logger_provider.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/aggregator
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/aggregator/aggregator.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/instrument_selector.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/meter_selector.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/predicate.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/predicate_factory.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/view_registry.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/aggregation.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/attributes_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/view/view.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/instruments.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/meter.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/meter_context.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/meter_provider.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/metric_exporter.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/metric_reader.h
-- Installing: /usr/local/include/opentelemetry//sdk/metrics/recordable.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/resource
-- Installing: /usr/local/include/opentelemetry//sdk/resource/experimental_semantic_conventions.h
-- Installing: /usr/local/include/opentelemetry//sdk/resource/resource.h
-- Installing: /usr/local/include/opentelemetry//sdk/resource/resource_detector.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace
-- Installing: /usr/local/include/opentelemetry//sdk/trace/exporter.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/id_generator.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/multi_recordable.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/multi_span_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/random_id_generator.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/recordable.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/sampler.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/samplers
-- Installing: /usr/local/include/opentelemetry//sdk/trace/samplers/always_off.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/samplers/always_on.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/samplers/parent.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/samplers/trace_id_ratio.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/simple_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/span_data.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/tracer.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/tracer_context.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/batch_span_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/trace/tracer_provider.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/version
-- Installing: /usr/local/include/opentelemetry//sdk/version/version.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk
-- Up-to-date: /usr/local/include/opentelemetry//sdk/_metrics
-- Up-to-date: /usr/local/include/opentelemetry//sdk/_metrics/aggregator
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/aggregator/aggregator.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/aggregator/counter_aggregator.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/aggregator/exact_aggregator.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/aggregator/gauge_aggregator.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/aggregator/histogram_aggregator.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/aggregator/min_max_sum_count_aggregator.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/aggregator/sketch_aggregator.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/async_instruments.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/exporter.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/instrument.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/meter.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/meter_provider.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/record.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/sync_instruments.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/ungrouped_processor.h
-- Installing: /usr/local/include/opentelemetry//sdk/_metrics/controller.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/atomic_shared_ptr.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/atomic_unique_ptr.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/circular_buffer.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/circular_buffer_range.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/empty_attributes.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/env_variables.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/exporter_utils.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/global_log_handler.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/common/attribute_utils.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/instrumentationlibrary
-- Up-to-date: /usr/local/include/opentelemetry//sdk/instrumentationlibrary/instrumentation_library.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/aggregator
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/aggregator/aggregator.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/instrument_selector.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/meter_selector.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/predicate.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/predicate_factory.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/view_registry.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/aggregation.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/attributes_processor.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/view/view.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/instruments.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/meter.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/meter_context.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/meter_provider.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/metric_exporter.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/metric_reader.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/metrics/recordable.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/resource
-- Up-to-date: /usr/local/include/opentelemetry//sdk/resource/experimental_semantic_conventions.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/resource/resource.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/resource/resource_detector.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/exporter.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/id_generator.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/multi_recordable.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/multi_span_processor.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/processor.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/random_id_generator.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/recordable.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/sampler.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/samplers
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/samplers/always_off.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/samplers/always_on.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/samplers/parent.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/samplers/trace_id_ratio.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/simple_processor.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/span_data.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/tracer.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/tracer_context.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/batch_span_processor.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/trace/tracer_provider.h
-- Up-to-date: /usr/local/include/opentelemetry//sdk/version
-- Up-to-date: /usr/local/include/opentelemetry//sdk/version/version.h
-- Installing: /usr/local/lib/libopentelemetry_common.a
-- Installing: /usr/local/lib/libopentelemetry_trace.a
-- Installing: /usr/local/lib/libopentelemetry_metrics.a
-- Installing: /usr/local/lib/libopentelemetry_version.a
-- Installing: /usr/local/lib/libopentelemetry_resources.a
-- Up-to-date: /usr/local/include/opentelemetry//ext
-- Up-to-date: /usr/local/include/opentelemetry//ext/http
-- Up-to-date: /usr/local/include/opentelemetry//ext/http/client
-- Up-to-date: /usr/local/include/opentelemetry//ext/http/client/curl
-- Installing: /usr/local/include/opentelemetry//ext/http/client/curl/http_client_curl.h
-- Installing: /usr/local/include/opentelemetry//ext/http/client/curl/http_operation_curl.h
-- Installing: /usr/local/include/opentelemetry//ext/http/client/http_client.h
-- Installing: /usr/local/include/opentelemetry//ext/http/client/http_client_factory.h
-- Up-to-date: /usr/local/include/opentelemetry//ext/http/common
-- Installing: /usr/local/include/opentelemetry//ext/http/common/url_parser.h
-- Up-to-date: /usr/local/include/opentelemetry//ext/http/server
-- Installing: /usr/local/include/opentelemetry//ext/http/server/file_http_server.h
-- Installing: /usr/local/include/opentelemetry//ext/http/server/http_server.h
-- Installing: /usr/local/include/opentelemetry//ext/http/server/socket_tools.h
-- Up-to-date: /usr/local/include/opentelemetry//ext/zpages
-- Installing: /usr/local/include/opentelemetry//ext/zpages/latency_boundaries.h
-- Up-to-date: /usr/local/include/opentelemetry//ext/zpages/static
-- Installing: /usr/local/include/opentelemetry//ext/zpages/static/tracez_index.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/static/tracez_script.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/static/tracez_style.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/threadsafe_span_data.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/tracez_data.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/tracez_data_aggregator.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/tracez_http_server.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/tracez_processor.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/tracez_shared_data.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/zpages.h
-- Installing: /usr/local/include/opentelemetry//ext/zpages/zpages_http_server.h
-- Installing: /usr/local/lib/libopentelemetry_otlp_recordable.a
-- Installing: /usr/local/lib/libopentelemetry_exporter_otlp_grpc.a
-- Installing: /usr/local/lib/libopentelemetry_exporter_otlp_grpc_log.a
-- Up-to-date: /usr/local/include/opentelemetry/exporters//otlp
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_grpc_exporter.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_grpc_exporter_options.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_grpc_log_exporter.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_log_recordable.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_recordable_utils.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/protobuf_include_prefix.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/protobuf_include_suffix.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_environment.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_http_client.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_http_exporter.h
-- Installing: /usr/local/include/opentelemetry/exporters//otlp/otlp_http_log_exporter.h
-- Installing: /usr/local/lib/libopentelemetry_exporter_ostream_span.a
-- Up-to-date: /usr/local/include/opentelemetry/exporters//ostream
-- Installing: /usr/local/include/opentelemetry/exporters//ostream/log_exporter.h
-- Installing: /usr/local/include/opentelemetry/exporters//ostream/span_exporter.h
-- Up-to-date: /usr/local/include/opentelemetry/exporters//memory
-- Installing: /usr/local/include/opentelemetry/exporters//memory/in_memory_span_data.h
-- Installing: /usr/local/include/opentelemetry/exporters//memory/in_memory_span_exporter.h
+ cd /tmp/build/nginx-1.19.10
+ ./configure '--prefix=/usr/local/nginx' --with-compat '--add-dynamic-module=/tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx'
checking for OS
 + Linux 3.10.0-1160.59.1.el7.x86_64 x86_64
checking for C compiler ... found
 + using GNU C compiler
 + gcc version: 10.3.1 20210424 (Alpine 10.3.1_git20210424) 
checking for gcc -pipe switch ... found
checking for -Wl,-E switch ... found
checking for gcc builtin atomic operations ... found
checking for C99 variadic macros ... found
checking for gcc variadic macros ... found
checking for gcc builtin 64 bit byteswap ... found
checking for unistd.h ... found
checking for inttypes.h ... found
checking for limits.h ... found
checking for sys/filio.h ... not found
checking for sys/param.h ... found
checking for sys/mount.h ... found
checking for sys/statvfs.h ... found
checking for crypt.h ... found
checking for Linux specific features
checking for epoll ... found
checking for EPOLLRDHUP ... found
checking for EPOLLEXCLUSIVE ... found
checking for eventfd() ... found
checking for O_PATH ... found
checking for sendfile() ... found
checking for sendfile64() ... found
checking for sys/prctl.h ... found
checking for prctl(PR_SET_DUMPABLE) ... found
checking for prctl(PR_SET_KEEPCAPS) ... found
checking for capabilities ... not found
checking for crypt_r() ... found
checking for sys/vfs.h ... found
checking for nobody group ... found
checking for poll() ... found
checking for /dev/poll ... not found
checking for kqueue ... not found
checking for crypt() ... found
checking for F_READAHEAD ... not found
checking for posix_fadvise() ... found
checking for O_DIRECT ... found
checking for F_NOCACHE ... not found
checking for directio() ... not found
checking for statfs() ... found
checking for statvfs() ... found
checking for dlopen() ... found
checking for sched_yield() ... found
checking for sched_setaffinity() ... found
checking for SO_SETFIB ... not found
checking for SO_REUSEPORT ... found
checking for SO_ACCEPTFILTER ... not found
checking for SO_BINDANY ... not found
checking for IP_TRANSPARENT ... found
checking for IP_BINDANY ... not found
checking for IP_BIND_ADDRESS_NO_PORT ... found
checking for IP_RECVDSTADDR ... not found
checking for IP_SENDSRCADDR ... not found
checking for IP_PKTINFO ... found
checking for IPV6_RECVPKTINFO ... found
checking for TCP_DEFER_ACCEPT ... found
checking for TCP_KEEPIDLE ... found
checking for TCP_FASTOPEN ... found
checking for TCP_INFO ... found
checking for accept4() ... found
checking for int size ... 4 bytes
checking for long size ... 8 bytes
checking for long long size ... 8 bytes
checking for void * size ... 8 bytes
checking for uint32_t ... found
checking for uint64_t ... found
checking for sig_atomic_t ... found
checking for sig_atomic_t size ... 4 bytes
checking for socklen_t ... found
checking for in_addr_t ... found
checking for in_port_t ... found
checking for rlim_t ... found
checking for uintptr_t ... uintptr_t found
checking for system byte ordering ... little endian
checking for size_t size ... 8 bytes
checking for off_t size ... 8 bytes
checking for time_t size ... 8 bytes
checking for AF_INET6 ... found
checking for setproctitle() ... not found
checking for pread() ... found
checking for pwrite() ... found
checking for pwritev() ... found
checking for strerrordesc_np() ... not found
checking for sys_nerr ... not found
checking for _sys_nerr ... not found
checking for localtime_r() ... found
checking for clock_gettime(CLOCK_MONOTONIC) ... found
checking for posix_memalign() ... found
checking for memalign() ... found
checking for mmap(MAP_ANON|MAP_SHARED) ... found
checking for mmap("/dev/zero", MAP_SHARED) ... found
checking for System V shared memory ... found
checking for POSIX semaphores ... found
checking for struct msghdr.msg_control ... found
checking for ioctl(FIONBIO) ... found
checking for ioctl(FIONREAD) ... found
checking for struct tm.tm_gmtoff ... found
checking for struct dirent.d_namlen ... not found
checking for struct dirent.d_type ... found
checking for sysconf(_SC_NPROCESSORS_ONLN) ... found
checking for sysconf(_SC_LEVEL1_DCACHE_LINESIZE) ... not found
checking for openat(), fstatat() ... found
checking for getaddrinfo() ... found
configuring additional dynamic modules
adding module in /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx
 + otel_ngx_module was configured
checking for PCRE library ... found
checking for PCRE JIT support ... found
checking for zlib library ... found
creating objs/Makefile

Configuration summary
  + using system PCRE library
  + OpenSSL library is not used
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

+ make modules
make -f objs/Makefile modules
make[1]: Entering directory '/tmp/build/nginx-1.19.10'
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/agent_config.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/agent_config.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/nginx_config.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/nginx_config.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/otel_ngx_module.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/otel_ngx_module.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/propagate.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/propagate.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/script.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/script.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/toml.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/toml.c
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/trace_context.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/trace_context.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/otel_ngx_module_modules.o \
        objs/otel_ngx_module_modules.c
cc -o objs/otel_ngx_module.so \
objs/addon/src/agent_config.o \
objs/addon/src/nginx_config.o \
objs/addon/src/otel_ngx_module.o \
objs/addon/src/propagate.o \
objs/addon/src/script.o \
objs/addon/src/toml.o \
objs/addon/src/trace_context.o \
objs/otel_ngx_module_modules.o \
-lstdc++ -lopentelemetry_common -lopentelemetry_resources -lopentelemetry_trace -lopentelemetry_exporter_otlp_grpc -lopentelemetry_otlp_recordable \
-shared
make[1]: Leaving directory '/tmp/build/nginx-1.19.10'
+ mkdir -p /etc/nginx/modules
+ cp objs/otel_ngx_module.so /etc/nginx/modules/otel_ngx_module.so
+ find /usr/local -name '*.a' -print
+ xargs /bin/rm
/ # ldd /etc/nginx/modules/otel_ngx_module.so 
        /lib/ld-musl-x86_64.so.1 (0x7fa5077f2000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7fa5075b7000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fa50759d000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fa5077f2000)
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_array_create: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18Resource8CopyFromERKS3_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18ResourceC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v113ResourceSpansEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_get_indexed_variable: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18Resource12InternalSwapEPS3_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14SpanD1Ev: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN4grpc13CreateChannelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_18ChannelCredentialsEE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: pcre_exec: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto6common2v18AnyValueEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_script_run: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span12InternalSwapEPS3_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_add_variable: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v18AnyValue11clear_valueEv: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14SpanC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v16StatusEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v126ExportTraceServiceResponseC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf8internal20RepeatedPtrFieldBase7ReserveEi: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN4grpc13ClientContext11AddMetadataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18ResourceD1Ev: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v19Span_LinkEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v122InstrumentationLibraryD1Ev: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN4grpc26InsecureChannelCredentialsEv: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_get_variable_index: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v125ExportTraceServiceRequestC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v122InstrumentationLibrary12InternalSwapEPS3_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_script_compile: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v126ExportTraceServiceResponseD1Ev: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_palloc: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v127InstrumentationLibrarySpansEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v110Span_EventEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto6common2v110ArrayValueEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto6common2v18KeyValueEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto6common2v122InstrumentationLibraryEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry2v13sdk6common6Random20GenerateRandomBufferENS0_5nostd4spanIhLm18446744073709551615EEE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_pool_cleanup_add: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_array_push: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_pcalloc: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_atoi: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v122InstrumentationLibraryC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_log_error_core: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN4grpc13ClientContextC1Ev: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN4grpc18Timepoint2TimespecERKNSt6chrono10time_pointINS0_3_V212system_clockENS0_8durationIlSt5ratioILl1ELl1000000000EEEEEEP12gpr_timespec: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_script_variables_count: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf8internal14ArenaStringPtr3SetENS2_12EmptyDefaultERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_5ArenaE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf8internal14ArenaStringPtr3SetENS2_12EmptyDefaultEONSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_5ArenaE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_strncasecmp: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v112TraceService7NewStubERKSt10shared_ptrIN4grpc16ChannelInterfaceEERKNS6_11StubOptionsE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v14SpanEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_regex_compile: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v122InstrumentationLibrary8CopyFromERKS3_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN4grpc14SslCredentialsERKNS_21SslCredentialsOptionsE: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_conf_log_error: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto8resource2v18ResourceEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v125ExportTraceServiceRequestD1Ev: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN4grpc13ClientContextD1Ev: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_module: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_http_core_module: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: _ZN6google8protobuf8internal26fixed_address_empty_stringB5cxx11E: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_conf_set_flag_slot: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_conf_set_flag_slot: symbol not found
Error relocating /etc/nginx/modules/otel_ngx_module.so: ngx_conf_set_flag_slot: symbol not found
/ #

@dmathieu
Copy link
Contributor

dmathieu commented Apr 6, 2022

It looks like some modules are built in the sidecar, but aren't moved into the main image.
We need to figure out which those ones are and make them available.

@oct28-yjkim
Copy link
Author

@dmathieu
I got a err message


/tmp/build # cd nginx-1.19.10/
/tmp/build/nginx-1.19.10 # ls
CHANGES     LICENSE     README      conf        contrib     man         src
CHANGES.ru  Makefile    auto        configure   html        objs
/tmp/build/nginx-1.19.10 # ./configure '--prefix=/usr/local/nginx' --with-compat '--add-dynamic-module=/tmp/build/opent
elemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx'
checking for OS
 + Linux 3.10.0-1160.59.1.el7.x86_64 x86_64
checking for C compiler ... found
 + using GNU C compiler
 + gcc version: 10.3.1 20210424 (Alpine 10.3.1_git20210424) 
checking for gcc -pipe switch ... found
checking for -Wl,-E switch ... found
checking for gcc builtin atomic operations ... found
checking for C99 variadic macros ... found
checking for gcc variadic macros ... found
checking for gcc builtin 64 bit byteswap ... found
checking for unistd.h ... found
checking for inttypes.h ... found
checking for limits.h ... found
checking for sys/filio.h ... not found
checking for sys/param.h ... found
checking for sys/mount.h ... found
checking for sys/statvfs.h ... found
checking for crypt.h ... found
checking for Linux specific features
checking for epoll ... found
checking for EPOLLRDHUP ... found
checking for EPOLLEXCLUSIVE ... found
checking for eventfd() ... found
checking for O_PATH ... found
checking for sendfile() ... found
checking for sendfile64() ... found
checking for sys/prctl.h ... found
checking for prctl(PR_SET_DUMPABLE) ... found
checking for prctl(PR_SET_KEEPCAPS) ... found
checking for capabilities ... not found
checking for crypt_r() ... found
checking for sys/vfs.h ... found
checking for nobody group ... found
checking for poll() ... found
checking for /dev/poll ... not found
checking for kqueue ... not found
checking for crypt() ... found
checking for F_READAHEAD ... not found
checking for posix_fadvise() ... found
checking for O_DIRECT ... found
checking for F_NOCACHE ... not found
checking for directio() ... not found
checking for statfs() ... found
checking for statvfs() ... found
checking for dlopen() ... found
checking for sched_yield() ... found
checking for sched_setaffinity() ... found
checking for SO_SETFIB ... not found
checking for SO_REUSEPORT ... found
checking for SO_ACCEPTFILTER ... not found
checking for SO_BINDANY ... not found
checking for IP_TRANSPARENT ... found
checking for IP_BINDANY ... not found
checking for IP_BIND_ADDRESS_NO_PORT ... found
checking for IP_RECVDSTADDR ... not found
checking for IP_SENDSRCADDR ... not found
checking for IP_PKTINFO ... found
checking for IPV6_RECVPKTINFO ... found
checking for TCP_DEFER_ACCEPT ... found
checking for TCP_KEEPIDLE ... found
checking for TCP_FASTOPEN ... found
checking for TCP_INFO ... found
checking for accept4() ... found
checking for int size ... 4 bytes
checking for long size ... 8 bytes
checking for long long size ... 8 bytes
checking for void * size ... 8 bytes
checking for uint32_t ... found
checking for uint64_t ... found
checking for sig_atomic_t ... found
checking for sig_atomic_t size ... 4 bytes
checking for socklen_t ... found
checking for in_addr_t ... found
checking for in_port_t ... found
checking for rlim_t ... found
checking for uintptr_t ... uintptr_t found
checking for system byte ordering ... little endian
checking for size_t size ... 8 bytes
checking for off_t size ... 8 bytes
checking for time_t size ... 8 bytes
checking for AF_INET6 ... found
checking for setproctitle() ... not found
checking for pread() ... found
checking for pwrite() ... found
checking for pwritev() ... found
checking for strerrordesc_np() ... not found
checking for sys_nerr ... not found
checking for _sys_nerr ... not found
checking for localtime_r() ... found
checking for clock_gettime(CLOCK_MONOTONIC) ... found
checking for posix_memalign() ... found
checking for memalign() ... found
checking for mmap(MAP_ANON|MAP_SHARED) ... found
checking for mmap("/dev/zero", MAP_SHARED) ... found
checking for System V shared memory ... found
checking for POSIX semaphores ... found
checking for struct msghdr.msg_control ... found
checking for ioctl(FIONBIO) ... found
checking for ioctl(FIONREAD) ... found
checking for struct tm.tm_gmtoff ... found
checking for struct dirent.d_namlen ... not found
checking for struct dirent.d_type ... found
checking for sysconf(_SC_NPROCESSORS_ONLN) ... found
checking for sysconf(_SC_LEVEL1_DCACHE_LINESIZE) ... not found
checking for openat(), fstatat() ... found
checking for getaddrinfo() ... found
configuring additional dynamic modules
adding module in /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx
 + otel_ngx_module was configured
checking for PCRE library ... found
checking for PCRE JIT support ... found
checking for zlib library ... found
creating objs/Makefile

Configuration summary
  + using system PCRE library
  + OpenSSL library is not used
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

/tmp/build/nginx-1.19.10 # ls -al /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrum
entation/nginx/
CMakeLists.txt  README.md       ci/             config          nginx.cmake     src/            test/

/tmp/build/nginx-1.19.10 # ls -al /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrum
entation/nginx/
total 24
drwxrwxr-x    5 root     root           111 Apr  6 07:07 .
drwxrwxr-x    5 root     root            61 Apr  6 07:07 ..
-rw-rw-r--    1 root     root          1019 Mar 29 04:20 CMakeLists.txt
-rw-rw-r--    1 root     root          8090 Mar 29 04:20 README.md
drwxrwxr-x    2 root     root            34 Apr  6 07:07 ci
-rw-rw-r--    1 root     root           908 Mar 29 04:20 config
-rw-rw-r--    1 root     root          1170 Mar 29 04:20 nginx.cmake
drwxrwxr-x    2 root     root          4096 Apr  6 07:07 src
drwxrwxr-x    6 root     root           112 Apr  6 07:07 test

/tmp/build/nginx-1.19.10 # make modules
make -f objs/Makefile modules
make[1]: Entering directory '/tmp/build/nginx-1.19.10'
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/agent_config.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/agent_config.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/nginx_config.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/nginx_config.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/otel_ngx_module.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/otel_ngx_module.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/propagate.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/propagate.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/script.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/script.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/toml.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/toml.c
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/addon/src/trace_context.o \
        /tmp/build/opentelemetry-cpp-contrib-2656a4072e257b6794da86ddd1b773b49f5517b3/instrumentation/nginx/src/trace_context.cpp
cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules \
        -o objs/otel_ngx_module_modules.o \
        objs/otel_ngx_module_modules.c
cc -o objs/otel_ngx_module.so \
objs/addon/src/agent_config.o \
objs/addon/src/nginx_config.o \
objs/addon/src/otel_ngx_module.o \
objs/addon/src/propagate.o \
objs/addon/src/script.o \
objs/addon/src/toml.o \
objs/addon/src/trace_context.o \
objs/otel_ngx_module_modules.o \
-lstdc++ -lopentelemetry_common -lopentelemetry_resources -lopentelemetry_trace -lopentelemetry_exporter_otlp_grpc -lopentelemetry_otlp_recordable \
-shared
/usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lopentelemetry_common
/usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lopentelemetry_resources
/usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lopentelemetry_trace
/usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lopentelemetry_exporter_otlp_grpc
/usr/lib/gcc/x86_64-alpine-linux-musl/10.3.1/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find -lopentelemetry_otlp_recordable
collect2: error: ld returned 1 exit status
make[1]: *** [objs/Makefile:1210: objs/otel_ngx_module.so] Error 1
make[1]: Leaving directory '/tmp/build/nginx-1.19.10'
make: *** [Makefile:16: modules] Error 2

Is this a related error?

@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 6, 2022 via email

@longwuyuan
Copy link
Contributor

/kind bug
/triage accepted

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 7, 2022
@dmathieu
Copy link
Contributor

dmathieu commented Apr 7, 2022

The exact missing module isn't clear to me here (I'm also not a C developer).
The likely thing is that those modules are installed when the image is built. But lost when we move things into the core container.

@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 7, 2022 •
edited
Loading

ok will check. Wonder if the build is against glibc and run is against musl.
/assign

@longwuyuan longwuyuan mentioned this issue Apr 8, 2022
Closed
@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 8, 2022 •
edited
Loading

open-telemetry/opentelemetry-cpp-contrib#141

@longwuyuan
Copy link
Contributor

longwuyuan commented Apr 8, 2022 •
edited
Loading

@dmathieu are you on slack. Do you know, if building the Otel_ngx_module on alpine but copying the binary to a busbox image, for sidecar loading, is causing the missing symbols. I don't think busy box has must. Best-case would be busy box would have glib. I am checking. Do you think the missing symbols problem would be fixed, if the sidecar was also based on alpine.

@dmathieu
Copy link
Contributor

The only thing that I see may cause issues with this difference in base images would be if one uses musl and the other gcc.

@longwuyuan
Copy link
Contributor

Hi @iquirino,

I saw https://github.com/iquirino/nginx-opentelemetry/commits?author=iquirino .
I was wondering if you are interested to help us solve this problem here.

@longwuyuan longwuyuan mentioned this issue May 7, 2022
Closed
9 tasks
@longwuyuan
Copy link
Contributor

open-telemetry/opentelemetry-cpp-contrib#141 (comment)

@longwuyuan
Copy link
Contributor

open-telemetry/opentelemetry-cpp#1382

@NissesSenap NissesSenap mentioned this issue May 9, 2022
Open
@dmathieu dmathieu mentioned this issue May 12, 2022
Closed
@longwuyuan longwuyuan mentioned this issue May 13, 2022
Closed
@esigo
Copy link
Member

esigo commented May 14, 2022

Hi, I'm working on a fix. I was wondering if the image needs nginx binaries shipped too, or just otel_ngx_module.so with its dependencies?

@k8s-ci-robot k8s-ci-robot reopened this May 20, 2022
@k8s-ci-robot
Copy link
Contributor

@longwuyuan: Reopened this issue.

In response to this:

/reopen

Will close after testing the opentelemetry module to load

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@longwuyuan
Copy link
Contributor

longwuyuan commented May 20, 2022 •
edited
Loading

@esigo we have some progress but looks like no option but to ask you more questions. In summary, the otel module loads from the new image built yesterday but its looking for the grpc module and the feature fails, with error as ;

(Error loading shared library libopentelemetry_exporter_otlp_grpc.so: No such file or directory (needed by /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so)) in /etc/nginx/nginx.conf:2

Is the grpc module supposed to be in the image. Are we forgetting to copy the grpc module over ;

% k exec -ti ingress-nginx-controller-5f846c75b5-lc7lx -- sh 
Defaulted container "controller" out of: controller, opentelemetry (init)
/etc/nginx $ ls -l /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so 
-rwxr-xr-x    1 root     root        732000 May 20 01:48 /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so
/etc/nginx $ vi nginx.conf
/etc/nginx $ nginx -s reload
2022/05/20 01:51:00 [emerg] 112#112: dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error loading shared library libopentelemetry_exporter_otlp_grpc.so: No such file or directory (needed by /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so)) in /etc/nginx/nginx.conf:2
nginx: [emerg] dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error loading shared library libopentelemetry_exporter_otlp_grpc.so: No such file or directory (needed by /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so)) in /etc/nginx/nginx.conf:2
/etc/nginx $ ls -l /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so ^C
/etc/nginx $ find / -name  libopentelemetry_exporter_otlp_grpc.so
find: /root: Permission denied
find: /proc/tty/driver: Permission denied
/etc/nginx $ sudo find / -name  libopentelemetry_exporter_otlp_grpc.so
sh: sudo: not found
/etc/nginx $ find / -name  libopentelemetry_exporter_otlp_grpc.so
find: /root: Permission denied
find: /proc/tty/driver: Permission denied
/etc/nginx $ head -3 nginx.conf

load_module /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so;
# Configuration checksum: 3143774805836829111
/etc/nginx $

@longwuyuan
Copy link
Contributor

longwuyuan commented May 20, 2022 •
edited
Loading

@esigo, I see that we do have the grpc module in the opentelemetry image, just to state the obvious, as we built grpc;

m@mypc [~] 
% docker images
REPOSITORY                                       TAG         IMAGE ID       CREATED         SIZE
gcr.io/k8s-staging-ingress-nginx/opentelemetry   <none>      5a9078a96d0f   3 hours ago     48.9MB
kindest/node                                     <none>      f5aa68ba122a   10 hours ago    911MB
gcr.io/k8s-staging-ingress-nginx/controller      1.0.0-dev   e4ecc7b2241c   32 hours ago    294MB
kindest/node                                     v1.24.0     f607f70d3a1a   10 days ago     934MB
k8s.gcr.io/ingress-nginx/e2e-test-runner         <none>      9826129be03e   7 weeks ago     1.14GB
kindest/node                                     <none>      32b8b755dee8   12 months ago   1.12GB
m@mypc [~] 
% docker exec -ti 5a9078a96d0f sh
Error: No such container: 5a9078a96d0f
m@mypc [~] 
% docker run -ti 5a9078a96d0f sh 
/ # find / -name libopentelemetry*
/usr/lib/libopentelemetry_trace.so
/usr/lib/libopentelemetry_resources.so
/usr/lib/libopentelemetry_exporter_otlp_grpc.so
/usr/lib/libopentelemetry_common.so
/usr/lib/libopentelemetry_otlp_recordable.so

But the grpc module is not under /etc/nginx/modules of the ingress-nginx-controller container of the pod. Neither is it copied over to /usr/lib of the ingress-nginx-controller container of the pod. Instead the grpc module is under /usr/lib of the opentelemetry container of the pod. And the init_module.sh is not copying it over to the /modules_mount location that the ingress-nginx-controller container, of the pod, is using, to get the shared library over.

I don't know the history of the init_module.sh script yet. Will check asap. But do you want to alter build to also place the grpc module under /etc/nginx or do you suggest adding one more cp command to the init_module.sh.

I suspect, even after copying to /mount_modules, the grpc module will have to be moved/copied over to /usr/lib on the ingress-nginx-controller container of the pod.

Personally, I want to first get it running.

Thanks tons again. Please advise and will do accordingly.

@longwuyuan
Copy link
Contributor

@esigo , on looking again, maybe we can just add another line 46 in Dockerfile, that looks like ;

COPY --from=nginx /opt/third_party/install/lib/libopentelemetry* /etc/nginx/modules

@Tobrek
Copy link
Contributor

Tobrek commented May 20, 2022 •
edited
Loading

@longwuyuan just copying the libopentelemetry* files to /etc/nginx/modules didn't worked for me. Local path seems to be not in the lib search path. Also it would give strange structures if you copy the libs to /etc/nginx/modules in the sidecar container - init_module.sh would copy the files to /modules_mount/etc/nginx/modules/< filename/< filename>.
Also when i understood it correctly, just these 5 files (libopentelementry*.so) will be not enough. You will need more then these.

I solved it that way:

  • copy all libs from opentelemetry-cpp container /usr/lib to /modules_mount/usr/lib in the init_module.sh
  • add /modules_mount/usr/lib to lib search path using ld-musl

@longwuyuan
Copy link
Contributor

You are right. My problem is we can't hack it so although I hinted at next steps, I did not submit a PR for any change there.

I am waiting for @esigo & @rikatz to advise.

This does not end at copying the libs over. The module has to actually start shipping signals like metrics or logs on demand and I have no clue on how to test that.

@esigo
Copy link
Member

esigo commented May 20, 2022

@longwuyuan I thought the container will be used as it is, so I copied everything to:
COPY --from=nginx /opt/third_party/install/lib /usr/lib
this command should put every .so file needed to /usr/lib.
If you are using the .so files somewhere else we can simply use:
COPY --from=nginx /opt/third_party/install/lib /etc/nginx/modules
so that everything is copied to the same place. But then you need to update LD_LIBRARY_PATH.

@esigo esigo mentioned this issue May 20, 2022
Merged
9 tasks
@longwuyuan
Copy link
Contributor

longwuyuan commented May 20, 2022 •
edited
Loading

Thank you so very much @esigo. After your PR #8612 merges and a new opentelemetry image is built, I have to check how to copy/move all those *.so to /usr/lib because LD_LIBRARY_PATH is default and it will be a hack to add /etc/nginx/modules to the LD_LIBRARY_PATH . @rikatz please comment if you have any thoughts.

As a recap, this is what is happening ;

  • We install ingress-nginx controller helm chart with a values file that enables opentelemetry as a extraModule
% cat /tmp/values.ot.yaml 
controller:
  extraModules:
    - name: opentelemetry
      image: gcr.io/k8s-staging-ingress-nginx/opentelemetry@sha256:0bf055c82d4060dac3dc8ae72d61a7fb4ea1289b907f2fe255b193cc1635aff8

Installing example from within my clone of ingress-nginx controller git workspace

m@mypc [~/Documents/github/longwuyuan/ingress-nginx/charts/ingress-nginx] opentelemetry
% helm install ingress-nginx . -f /tmp/values.ot.yaml
NAME: ingress-nginx
LAST DEPLOYED: Sat May 21 00:06:34 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls
  • This causes the ingress-nginx-controller to be deployed, as a 2 container pod, instead of the normal 1 container pod, with the opentelemetry container being the init container. This is non default
% k get po ingress-nginx-controller-5f846c75b5-v2ws9 -o yaml | grep -a10 initContainerStatuses
    imageID: k8s.gcr.io/ingress-nginx/controller@sha256:d8196e3bc1e72547c5dec66d6556c0ff92a23f6d0919b206be170bc90d5f9185
    lastState: {}
    name: controller
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-05-20T18:36:40Z"
  hostIP: 192.168.105.6
  initContainerStatuses:
  - containerID: containerd://950b903695b3c02f88e0223e8ce867673a6f42ea2e32303e2bfac80810e33804
    image: sha256:5a9078a96d0fba84462620706741e069bf6feedc1e3c7f29ddf39ba41a1862fa
    imageID: gcr.io/k8s-staging-ingress-nginx/opentelemetry@sha256:0bf055c82d4060dac3dc8ae72d61a7fb4ea1289b907f2fe255b193cc1635aff8
    lastState: {}
    name: opentelemetry
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://950b903695b3c02f88e0223e8ce867673a6f42ea2e32303e2bfac80810e33804
  • During the pod init stage, the init_modules.sh copies the shared library otel_nginx_mod.so, from the opentelemetry container, to /modules_mount/ and the opentelemetry container terminates, being just a init container
  • Now with your PR move OpenTelemetry so files under /etc/nginx/modules #8612 , what will happen is that all the other required *.so files, which are only present in the opetelemetry init container, will also be copied to /modules_mount
  • But those grpc and other shared libraries *.so files, will become available to the main container, running ingress-nginx-controller, in the location /etc/nginx/modules, but not under /usr/lib
  • Hence your suggestion to add /etc/nginx/modules also, to the LD_LIBRARY_PATH
  • And I have floated the thought of copying all those files to /usr/lib in the main container, that runs the ingress-nginx-controller, so that the default LD_LIBRARY_PATH works

I will wait for the new image to get built and then check

@longwuyuan
Copy link
Contributor

@tao12345666333 please check and comment for LD_LIBRARY_PATH situation as per above comment

@longwuyuan
Copy link
Contributor

@rikatz the LD_LIBRARY_PATH situation above is special so please comment/advise

@Tobrek Tobrek mentioned this issue May 23, 2022
Closed
9 tasks
This was referenced May 24, 2022
Closed
Merged
@longwuyuan
Copy link
Contributor

longwuyuan commented May 26, 2022 •
edited
Loading

Hi @esigo , after your PR #8628 to add LD_LIBRARY_PATH in the nginx docker file, merged, the status is below ;

  • the required LD_LIBRARY_PATH is "/modules_mount/etc/nginx/modules/modules/" ------> # note 2 times "modules"
  • the LD_LIBRARY_PATH value set by your PR is "/modules_mount/etc/nginx/modules" ------> #note 1 time "modules"
  • I think we should put the LD_LIBRARY_PATH in /rootfs/Dockerfile and /rootfs/Dockerfile.chroot
  • I think we should remove the LD_LIBRARY_PATH from /images/nginx/rootfs/Dockerfile

A screenshot of my test is below
image

@esigo
Copy link
Member

esigo commented May 27, 2022

Hi @longwuyuan Sorry for the late reply. I'm on vacation right now. I'll ask my sister if she could kindly take over from here.

@ssaei would it be possible to apply changes per @longwuyuan's suggestions, also please see my suggestions below.
thanks

diff --git a/images/nginx/rootfs/Dockerfile b/images/nginx/rootfs/Dockerfile
index 2d1d12178..75c63cff1 100644
--- a/images/nginx/rootfs/Dockerfile
+++ b/images/nginx/rootfs/Dockerfile
@@ -24,7 +24,6 @@ RUN apk update \
 FROM alpine:3.14.6
 
 ENV PATH=$PATH:/usr/local/luajit/bin:/usr/local/nginx/sbin:/usr/local/nginx/bin
-ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/modules_mount/etc/nginx/modules
 
 ENV LUA_PATH="/usr/local/share/luajit-2.1.0-beta3/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/lib/lua/?.lua;;"
 ENV LUA_CPATH="/usr/local/lib/lua/?/?.so;/usr/local/lib/lua/?.so;;"
diff --git a/rootfs/Dockerfile b/rootfs/Dockerfile
index 1eab94c58..0bf9a0304 100644
--- a/rootfs/Dockerfile
+++ b/rootfs/Dockerfile
@@ -31,6 +31,8 @@ LABEL org.opencontainers.image.revision="${COMMIT_SHA}"
 
 LABEL build_id="${BUILD_ID}"
 
+ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/modules_mount/etc/nginx/modules/modules
+
 WORKDIR  /etc/nginx
 
 RUN apk update \
diff --git a/rootfs/Dockerfile.chroot b/rootfs/Dockerfile.chroot
index 5ea3ce835..04adccf8a 100644
--- a/rootfs/Dockerfile.chroot
+++ b/rootfs/Dockerfile.chroot
@@ -44,6 +44,7 @@ LABEL build_id="${BUILD_ID}"
 ENV LUA_PATH="/usr/local/share/luajit-2.1.0-beta3/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/lib/lua/?.lua;;"
 ENV LUA_CPATH="/usr/local/lib/lua/?/?.so;/usr/local/lib/lua/?.so;;"
 ENV PATH=$PATH:/usr/local/luajit/bin:/usr/local/nginx/sbin:/usr/local/nginx/bin
+ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/modules_mount/etc/nginx/modules/modules
 
 RUN apk update \
   && apk upgrade \

@ssaei ssaei mentioned this issue May 27, 2022
Merged
9 tasks
@ssaei
Copy link
Contributor

ssaei commented May 27, 2022 •
edited
Loading

FYI @longwuyuan @esigo
#8641

@longwuyuan
Copy link
Contributor

longwuyuan commented May 27, 2022 via email

@longwuyuan
Copy link
Contributor

longwuyuan commented May 30, 2022 •
edited
Loading

Hi @esigo @ssaei, @tao12345666333

Even after LD_LIBRARY_PATH is set to the location, nginx is still unable to load the grpc module. I am checking if nginx.conf itself needs some config. Below is my test

/modules_mount/etc/nginx/modules/modules $ echo $LD_LIBRARY_PATH
:/modules_mount/etc/nginx/modules/modules
/modules_mount/etc/nginx/modules/modules $ uname -a
Linux ingress-nginx-controller-6946b96576-tcn9z 4.19.202 #1 SMP Tue Feb 8 19:13:02 UTC 2022 x86_64 Linux
/modules_mount/etc/nginx/modules/modules $ pwd
/modules_mount/etc/nginx/modules/modules
/modules_mount/etc/nginx/modules/modules $ ls libopentelemetry_*
libopentelemetry_common.so              libopentelemetry_otlp_recordable.so     libopentelemetry_trace.so
libopentelemetry_exporter_otlp_grpc.so  libopentelemetry_resources.so
/modules_mount/etc/nginx/modules/modules $ nginx -t
2022/05/30 01:19:52 [emerg] 117#117: dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error loading shared library libopentelemetry_exporter_otlp_grpc.so: No such file or directory (needed by /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so)) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error loading shared library libopentelemetry_exporter_otlp_grpc.so: No such file or directory (needed by /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so)) in /etc/nginx/nginx.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

Note, the error message says Error loading shared library libopentelemetry_exporter_otlp_grpc.so, even though that file exists in the LD_LIBRARY_PATH

@Tobrek Tobrek mentioned this issue Jun 21, 2022
Merged
9 tasks
@Tobrek Tobrek mentioned this issue Jun 23, 2022
Closed
9 tasks
@kuzaxak
Copy link

kuzaxak commented Sep 8, 2022

@longwuyuan I can confirm that on the latest image gcr.io/k8s-staging-ingress-nginx/opentelemetry:v20220906-controller-v1.3.1-3-g981ce38a7 problem with libopentelemetry_exporter_otlp_grpc.so is reproducible.

@longwuyuan
Copy link
Contributor

The new openteemetry image with the fixes has not been published yet. Hope to get it published in a day or 2.

@longwuyuan
Copy link
Contributor 

docker pull gcr.io/k8s-staging-ingress-nginx/opentelemetry@sha256:aa079daa7efd93aa830e26483a49a6343354518360929494bad1d0ad3303142e
OR
docker pull gcr.io/k8s-staging-ingress-nginx/opentelemetry:v20220906-controller-v1.3.1-3-g981ce38a7

But not published yet and not even tested if its ok so if you want you can test

@longwuyuan
Copy link
Contributor

@kuzaxak This issue is closed. Please track all opentelemetry things in #9016

@kuzaxak
Copy link

kuzaxak commented Sep 8, 2022

But not published yet and not even tested if its ok so if you want you can test

I tested exactly the same image, and it isn't working gcr.io/k8s-staging-ingress-nginx/opentelemetry:v20220906-controller-v1.3.1-3-g981ce38a7

@esigo
Copy link
Member

esigo commented Sep 8, 2022

But not published yet and not even tested if its ok so if you want you can test

I tested exactly the same image, and it isn't working gcr.io/k8s-staging-ingress-nginx/opentelemetry:v20220906-controller-v1.3.1-3-g981ce38a7

Hi @kuzaxak you need both the new nginx image (which is not published yet) and new opentelemetry image for testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@longwuyuan longwuyuan

Labels
kind/bug Categorizes issue or PR as related to a bug. kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
8 participants
@dmathieu @longwuyuan @kuzaxak @Tobrek @k8s-ci-robot @esigo @oct28-yjkim @ssaei