symbol not found when loading opentelemetry module #141

longwuyuan · 2022-04-08T05:38:28Z

Hi,

This is related to the Kubernetes community ingress-nginx project An error occurred between opentelemetry modules kubernetes/ingress-nginx#8437
We use opentelemetry as a plugin to the ingress-nginx controller
We build the opentelemetry image with this https://github.com/kubernetes/ingress-nginx/tree/main/images/opentelemetry
We then optionally include it using this https://github.com/kubernetes/ingress-nginx/blob/02ccd22616ca47d447fcaffefa9efbdc545f2d20/charts/ingress-nginx/values.yaml#L578
We can see the module being available in the sidecar of the ingress-nginx-controller pod

Steps to reproduce

Install the ingress-nginx-controller on Minikube/kind/other Kubernetes cluster like so

% helm install ingress-nginx-2 ingress-nginx/ingress-nginx  \       
--namespace ingress-nginx-2 \
--set controller.ingressClassResource.name=nginx-two \
--set controller.ingressClassResource.controllerValue="example.com/ingress-nginx-2" \
--set controller.ingressClassResource.enabled=true \
--set controller.ingressClassByName=true \
--values /tmp/ot.yaml

Where the ot.yaml looks like this

% cat /tmp/ot.yaml 
controller:
  extraModules: 
    - name: opentelemetry
      image: "k8s.gcr.io/ingress-nginx/opentelemetry:v20220331-controller-v1.1.2-36-g7517b7ecf@sha256:e3f635474b5da24ccd0ea6b078fb190dae68b8b4a44b52bea19ec2561f0102ec"

Then follow the below steps ;

% k -n ingress-nginx-2 get po ingress-nginx-2-controller-f68f874c8-z6mc7
NAME                                         READY   STATUS    RESTARTS   AGE
ingress-nginx-2-controller-f68f874c8-z6mc7   1/1     Running   0          21m
[~/Documents/github/longwuyuan/ingress-nginx] issue-8437
% k -n ingress-nginx-2 exec -ti ingress-nginx-2-controller-f68f874c8-z6mc7 -- sh
Defaulted container "controller" out of: controller, opentelemetry (init)
/etc/nginx $ ls -l /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so
-rwxr-xr-x    1 root     root       8077416 Apr  8 05:05 /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so
/etc/nginx $ head -3  nginx.conf

==> nginx.conf <==
load_module /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so;
 \# Configuration checksum: 11658230481990042565

/etc/nginx $ ldd /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so

What is the expected behavior?
I expected the module to load since we built on alpine and there was no build error .

What is the actual behavior?
The module fails to load and attempt to load module reports missing symbols

/etc/nginx $ ldd /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so                                                                                                                                                                            
        /lib/ld-musl-x86_64.so.1 (0x7ffaadd1a000)                                                                                                                                                                                                       
        /usr/local/lib/libmimalloc.so => /usr/local/lib/libmimalloc.so (0x7ffaadc48000)                                                                                                                                                                 
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7ffaadaa7000)                                                                                                                                                                                      
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7ffaada8d000)                                                                                                                                                                                        
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7ffaadd1a000)                                                                                                                                                                              
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: ngx_array_create: symbol not found                                                                                                                                        
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found                                                                                                   
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18Resource8CopyFromERKS3_: symbol not found                                                                                            
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18ResourceC1EPN6google8protobuf5ArenaE: symbol not found                                                                               
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v113ResourceSpansEJEEEPT_PS1_DpOT0_: symbol not found                                          
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: ngx_http_get_indexed_variable: symbol not found                                                                                                                           
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18Resource12InternalSwapEPS3_: symbol not found                                                                                        
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14SpanD1Ev: symbol not found                                                                                                              
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN4grpc13CreateChannelERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt10shared_ptrINS_18ChannelCredentialsEE: symbol not found                                
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: pcre_exec: symbol not found                                                                                                                                               
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto6common2v18AnyValueEJEEEPT_PS1_DpOT0_: symbol not found                                               
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: ngx_http_script_run: symbol not found                                                                                                                                     
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span12InternalSwapEPS3_: symbol not found                                                                                               
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: ngx_http_add_variable: symbol not found                                                                                                                                   
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v18AnyValue11clear_valueEv: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14SpanC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v16StatusEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v126ExportTraceServiceResponseC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf8internal20RepeatedPtrFieldBase7ReserveEi: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN4grpc13ClientContext11AddMetadataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto8resource2v18ResourceD1Ev: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v19Span_LinkEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v122InstrumentationLibraryD1Ev: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN4grpc26InsecureChannelCredentialsEv: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: ngx_http_get_variable_index: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v125ExportTraceServiceRequestC1EPN6google8protobuf5ArenaE: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto6common2v122InstrumentationLibrary12InternalSwapEPS3_: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: ngx_http_script_compile: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto9collector5trace2v126ExportTraceServiceResponseD1Ev: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: ngx_palloc: symbol not found                  
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v127InstrumentationLibrarySpansEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto5trace2v110Span_EventEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto6common2v110ArrayValueEJEEEPT_PS1_DpOT0_: symbol not found
Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN6google8protobuf5Arena18CreateMaybeMessageIN13opentelemetry5proto6common2v18KeyValueEJEEEPT_PS1_DpOT0_: symbol not found

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

longwuyuan · 2022-04-08T17:07:57Z

@esigo @lalitb any advise you can provide will help.
Completely clueless on why we are observing "symbol not found" for everything. Is it because the libc/musl headers are not present, on the busy box image, to which we are copying the binary otl_ngx_module.so , to load as sidecar ?

longwuyuan · 2022-04-21T05:09:43Z

maybe related #146

longwuyuan · 2022-05-06T10:04:59Z

@esigo @lalitb hoping you can comment on this. thanks

mahaisong · 2022-05-06T14:25:34Z

@longwuyuan I need to deploy nginx on the upper part of k8s to integrate opentelemetry. I plan to use two methods. The first is to directly configure it in the pod of ingress nginx controller according to the requirements of opentelemetry CPP contrib, but nginx reports an error: the file ld-linux-x86-64 cannot be found so. 2。 So I failed. The second is the way you mentioned above. Have you succeeded? Can you help me? thank you.

longwuyuan · 2022-05-06T15:11:53Z

my status is same as reported originally

esigo · 2022-05-06T20:58:04Z

Hi @longwuyuan, can you give us some context here please. Your Dockerfile doesn't seem to have the build result. I suppose you use it for building and storing artifacts somewhere else but I couldn't find it.

longwuyuan · 2022-05-06T22:21:55Z

Hi @esigo,

The link to the image built by the ingress-nginx project CI is here https://console.cloud.google.com/gcr/images/k8s-staging-ingress-nginx/global/opentelemetry

The build is a success. We then make the use of the image available as an option, in a sidecar container. A user can enable this option to load the module in the nginx-ingress-controller like this https://github.com/kubernetes/ingress-nginx/blob/ec1b01092ef2c2ff36fe296c91c45d9b2d394bbd/charts/ingress-nginx/values.yaml#L583 ;

bash-5.1$ ls -al /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so
-rwxr-xr-x    1 root     root       8077416 Apr  6 04:20 /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so

# add nginx.conf load_module 

bash-5.1$ cat nginx.conf | head -n 3
load_module /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so;
# Configuration checksum: 6642706386070326205

bash-5.1$ nginx -s reload

But the module fails to load. There are several errors stating "symbol not found" _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found);

# error 
2022/04/06 04:45:48 [emerg] 619#619: dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found) in /etc/nginx/nginx.conf:1

esigo · 2022-05-07T17:05:43Z

@longwuyuan, I can't pull your image. Also I was not able to build opentelemetry using the build script you have provided. You may add -DBUILD_SHARED_LIBS=ON to your current script or keep your script as it is but build gRPC using our script instead of installing using apk. With both thses changes I was able to build and run otlp_grpc example in an alpine container.

longwuyuan · 2022-05-07T17:49:36Z

Hi @esigo,

Thank you very much for updating.

I was able to pull the image like thus ;

% docker pull k8s.gcr.io/ingress-nginx/opentelemetry:v20220415-controller-v1.2.0-beta.0-2-g81c2afd97@sha256:ce61e2cf0b347dffebb2dcbf57c33891d2217c1bad9c0959c878e5be671ef941
k8s.gcr.io/ingress-nginx/opentelemetry@sha256:ce61e2cf0b347dffebb2dcbf57c33891d2217c1bad9c0959c878e5be671ef941: Pulling from ingress-nginx/opentelemetry
8663204ce13b: Already exists 
ad2d99f72212: Pull complete 
b0c99e6c383e: Pull complete 
Digest: sha256:ce61e2cf0b347dffebb2dcbf57c33891d2217c1bad9c0959c878e5be671ef941
Status: Downloaded newer image for k8s.gcr.io/ingress-nginx/opentelemetry@sha256:ce61e2cf0b347dffebb2dcbf57c33891d2217c1bad9c0959c878e5be671ef941
k8s.gcr.io/ingress-nginx/opentelemetry:v20220415-controller-v1.2.0-beta.0-2-g81c2afd97@sha256:ce61e2cf0b347dffebb2dcbf57c33891d2217c1bad9c0959c878e5be671ef941

If you are going to pull, kindly try like above and update and I will work on the problem.

About the build failing for you, Kindly let me know if you meant that the Dockerfile and the dependendant build.sh script is what you are referring to, when you said that it does not work https://github.com/kubernetes/ingress-nginx/tree/ec1b01092ef2c2ff36fe296c91c45d9b2d394bbd/images/opentelemetry/rootfs . Or if you meant something else. I suspect that this being a Kubernetes projects specific CI instrumentation, that uses the CI software called "Prow" & Cloudbuild, there could be reasons that conventional builds fail. I will wait for your update/advice on specifics of how build failed and then work on it accordingly.
I think that adding -DBUILD_SHARED_LIBS=on may solve the problem so I will check that change in to the project CI build script asap and update you on the results

Thanks tons for this help.

esigo · 2022-05-07T18:27:46Z

You may check the PR I've raised in our repo: open-telemetry/opentelemetry-cpp#1382

longwuyuan · 2022-05-07T18:29:02Z

Hi @esigo,

I added the flag and I used the latest commit, at the time of writing this post, but I am getting a checksum error on the src tgz. Any suggestions/comments ;

m@mypc [~/Documents/github/longwuyuan/ingress-nginx/images/opentelemetry/rootfs] issue-8437                                                                                                                                                             
% ls -l /tmp/build                                                                                                                                                                                                                                      
ls: /tmp/build: No such file or directory                                                                                                                                                                                                               
m@mypc [~/Documents/github/longwuyuan/ingress-nginx/images/opentelemetry/rootfs] issue-                                                                                                                                                                 
m@mypc [~/Documents/github/longwuyuan/ingress-nginx/images/opentelemetry/rootfs] issue-8437                                                                                                                                                             
% docker build -t m/opentelemetry:dev-0 .                                                                                                                                                                                                               
[+] Building 25.2s (7/9)                                                                                                                                                                                                                                
 => [internal] load build definition from Dockerfile                              0.0s                                                                                                                                                                  
 => => transferring dockerfile: 37B                                               0.0s                                      
 => [internal] load .dockerignore                                                 0.0s                                      
 => => transferring context: 2B                                                   0.0s                                      
 => [internal] load metadata for docker.io/library/alpine:3.14.6                  3.2s                                      
 => [internal] load build context                                                 0.0s                                      
 => => transferring context: 2.98kB                                               0.0s                                      
 => CACHED [builder 1/3] FROM docker.io/library/alpine:3.14.6@sha256:06b5d462c92  0.0s                                      
 => [builder 2/3] COPY . /                                                        0.0s                                      
 => ERROR [builder 3/3] RUN apk update  && apk upgrade  && apk add -U bash  &&   21.8s                                      
------                                                        
 > [builder 3/3] RUN apk update         && apk upgrade  && apk add -U bash      && /build.sh:                                                                                  
#7 0.301 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz                                      
#7 1.866 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz                                                                                    
#7 3.183 v3.14.6-29-gf7b7ffe6e8 [https://dl-cdn.alpinelinux.org/alpine/v3.14/main]                                          
#7 3.183 v3.14.6-22-gd06d5b9286 [https://dl-cdn.alpinelinux.org/alpine/v3.14/community]                                     
#7 3.184 OK: 14958 distinct packages available                                                                              
#7 3.408 OK: 6 MiB in 14 packages                             
#7 3.629 (1/4) Installing ncurses-terminfo-base (6.2_p20210612-r0)                                                          
#7 3.904 (2/4) Installing ncurses-libs (6.2_p20210612-r0)                                                                   
#7 4.096 (3/4) Installing readline (8.1.0-r0)                                                                               
#7 4.171 (4/4) Installing bash (5.1.16-r0)                    
#7 4.306 Executing bash-5.1.16-r0.post-install                                                                              
#7 4.309 Executing busybox-1.33.1-r7.trigger                                                                                
#7 4.312 OK: 8 MiB in 18 packages                             
#7 4.355 created directory: '/tmp/build'                      
#7 4.553 (1/28) Installing libgcc (10.3.1_git20210424-r2)                                                                   
#7 4.863 (2/28) Installing libstdc++ (10.3.1_git20210424-r2)                                                                
#7 5.070 (3/28) Installing binutils (2.35.2-r2)                                                                             
#7 5.280 (4/28) Installing libmagic (5.40-r1)                                                                               
#7 5.364 (5/28) Installing file (5.40-r1)                     
#7 5.424 (6/28) Installing libgomp (10.3.1_git20210424-r2)                                                                  
#7 5.488 (7/28) Installing libatomic (10.3.1_git20210424-r2)                                                                
#7 5.547 (8/28) Installing libgphobos (10.3.1_git20210424-r2)                                                               
#7 5.709 (9/28) Installing gmp (6.2.1-r1)                     
#7 5.778 (10/28) Installing isl22 (0.22-r0)                                                                                 
#7 5.856 (11/28) Installing mpfr4 (4.1.0-r0)                                                                                
#7 5.999 (12/28) Installing mpc1 (1.2.1-r0)                                                                                 
#7 6.061 (13/28) Installing gcc (10.3.1_git20210424-r2)                                                                     
#7 9.174 (14/28) Installing musl-dev (1.2.2-r3)                                                                             
#7 9.553 (15/28) Installing libc-dev (0.7.2-r3)                                                                             
#7 9.611 (16/28) Installing g++ (10.3.1_git20210424-r2)                                                                     
#7 11.60 (17/28) Installing make (4.3-r0)                     
#7 11.67 (18/28) Installing fortify-headers (1.1-r1)                                                                        
#7 11.73 (19/28) Installing patch (2.7.6-r7)                                                                                
#7 11.80 (20/28) Installing build-base (0.5-r2)                                                                             
#7 11.86 (21/28) Installing ca-certificates (20211220-r0)                                                                   
#7 11.94 (22/28) Installing brotli-libs (1.0.9-r5)                                                                          
#7 12.03 (23/28) Installing nghttp2-libs (1.43.0-r0)                                                                        
#7 12.10 (24/28) Installing libcurl (7.79.1-r1)                                                                             
#7 12.17 (25/28) Installing curl (7.79.1-r1)                                                                                
#7 12.24 (26/28) Installing expat (2.4.7-r0)
#7 12.31 (27/28) Installing pcre2 (10.36-r0)
#7 12.39 (28/28) Installing git (2.32.1-r0)
#7 12.98 Executing busybox-1.33.1-r7.trigger
#7 12.98 Executing ca-certificates-20211220-r0.trigger
#7 13.02 OK: 212 MiB in 46 packages
#7 13.05 Downloading https://nginx.org/download/nginx-1.19.10.tar.gz
#7 18.80 nginx-1.19.10.tar.gz: OK
#7 18.88 Downloading https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/2619b5e0edf56872d5883325a3079c13c25dceac.tar.gz
#7 21.54 2619b5e0edf56872d5883325a3079c13c25dceac.tar.gz: FAILED
#7 21.54 sha256sum: WARNING: 1 of 1 computed checksums did NOT match
------

longwuyuan · 2022-05-07T18:33:53Z

You may check the PR I've raised in our repo: open-telemetry/opentelemetry-cpp#1382

Ah, just saw the PR. That should help a lot in simplifying the build. Thank you very much.

esigo · 2022-05-07T18:46:02Z

No comment on hash.
I updated my PR with commands you need for using the result of build.
Let me know if you'd like me to raise a similar PR in your repo.

longwuyuan · 2022-05-07T18:54:58Z

Ah, thank you again. Only now I realise there are 2 components, needing to be built.
While I can attempt to edit our Dockerfile as per what I am learning here, I would rather love to have you raise a PR in the ingress-nginx project. I suspect you will suggest changes needed under images/opentelemetry/rootfs .

longwuyuan · 2022-05-10T18:06:04Z

@esigo thanks tons. Wanted to repeat clear opinion that if you can raise a similar PR in our repo, will be much obliged and grateful.

esigo · 2022-05-10T18:54:34Z

Hi @longwuyuan, sure I'm working on it.

longwuyuan · 2022-05-10T18:56:18Z

@esigo Thank you very much. Its a huge help

mahaisong · 2022-05-11T15:24:46Z

Hi @esigo,

The link to the image built by the ingress-nginx project CI is here https://console.cloud.google.com/gcr/images/k8s-staging-ingress-nginx/global/opentelemetry

The build is a success. We then make the use of the image available as an option, in a sidecar container. A user can enable this option to load the module in the nginx-ingress-controller like this https://github.com/kubernetes/ingress-nginx/blob/ec1b01092ef2c2ff36fe296c91c45d9b2d394bbd/charts/ingress-nginx/values.yaml#L583 ;
bash-5.1$ ls -al /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so
-rwxr-xr-x    1 root     root       8077416 Apr  6 04:20 /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so

# add nginx.conf load_module 

bash-5.1$ cat nginx.conf | head -n 3
load_module /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so;
# Configuration checksum: 6642706386070326205

bash-5.1$ nginx -s reload
But the module fails to load. There are several errors stating "symbol not found" _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found);
# error 
2022/04/06 04:45:48 [emerg] 619#619: dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/modules_mount/etc/nginx/modules/modules/otel_ngx_module.so" failed (Error relocating /modules_mount/etc/nginx/modules/modules/otel_ngx_module.so: _ZN13opentelemetry5proto5trace2v14Span8CopyFromERKS3_: symbol not found) in /etc/nginx/nginx.conf:1

Have you solved this problem? Can your image make ot work normally? I hope you succeed.

longwuyuan · 2022-05-11T15:37:49Z

hi @mahaisong, This is work in progress as you can see from the comments. I think the problem will be solved in a few weeks.

esigo · 2022-05-14T10:44:53Z

kubernetes/ingress-nginx#8585

esigo · 2022-05-14T14:12:12Z

@mahaisong FYI kubernetes/ingress-nginx#8585

esigo · 2022-05-14T14:12:46Z

@longwuyuan please close the issue if it's fixed.

longwuyuan · 2022-05-19T06:49:12Z

Hi @esigo ,

Apologies for troubling again but we are having problems making progress. After merging your PR in ingress-nginx, the CI there is failing to build the opentelemetry image stating checksum error on downloading src tgz

#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//otlp/otlp_recordable_utils.h
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//otlp/otlp_environment.h
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//otlp/otlp_grpc_exporter_options.h
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//otlp/otlp_http_client.h
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//otlp/protobuf_include_prefix.h
#26 42.41 -- Installing: /opt/third_party/install/lib/libopentelemetry_exporter_ostream_span.so
#26 42.41 -- Set runtime path of "/opt/third_party/install/lib/libopentelemetry_exporter_ostream_span.so" to ""
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//ostream
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//ostream/metric_exporter.h
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//ostream/span_exporter.h
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//ostream/log_exporter.h
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//memory
#26 42.41 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//memory/in_memory_span_exporter.h
#26 42.42 -- Installing: /opt/third_party/install/include/opentelemetry/exporters//memory/in_memory_span_data.h
#26 DONE 43.0s

#27 [linux/amd64 nginx 2/3] COPY --from=otel-cpp /opt/third_party/install/ /usr
#27 DONE 0.1s

#28 [linux/amd64 nginx 3/3] RUN bash /opt/third_party/build.sh -n
#28 0.116 created directory: '/tmp/build'
#28 0.119 Downloading https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/6467ec2e4d67b08b44580b7eb7a298786f4eef91.tar.gz
#28 0.653 6467ec2e4d67b08b44580b7eb7a298786f4eef91.tar.gz: FAILED
#28 0.653 sha256sum: WARNING: 1 of 1 computed checksums did NOT match
#28 ERROR: process "/bin/sh -c bash /opt/third_party/build.sh -n" did not complete successfully: exit code: 10

#12 [linux/arm64 grpc 1/1] RUN bash /opt/third_party/build.sh -g v1.43.2
#12 CANCELED
------
 > [linux/amd64 nginx 3/3] RUN bash /opt/third_party/build.sh -n:
#28 0.116 created directory: '/tmp/build'
#28 0.119 Downloading https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/6467ec2e4d67b08b44580b7eb7a298786f4eef91.tar.gz
#28 0.653 6467ec2e4d67b08b44580b7eb7a298786f4eef91.tar.gz: FAILED
#28 0.653 sha256sum: WARNING: 1 of 1 computed checksums did NOT match
------
Dockerfile:40
--------------------
  38 |     COPY --from=grpc /opt/third_party/install/ /usr
  39 |     COPY --from=otel-cpp /opt/third_party/install/ /usr
  40 | >>> RUN bash /opt/third_party/build.sh -n
  41 |     
  42 |     FROM alpine:3.14.6
--------------------
error: failed to solve: process "/bin/sh -c bash /opt/third_party/build.sh -n" did not complete successfully: exit code: 10
make: *** [Makefile:37: build] Error 1
ERROR
ERROR: build step 0 "gcr.io/k8s-staging-test-infra/gcb-docker-gcloud:v20211118-2f2d816b90" failed: step exited with non-zero status: 2

kubernetes/ingress-nginx#8437 (comment)

https://00f74ba44b89321eca01421360815588dc1fba736b-apidata.googleusercontent.com/download/storage/v1/b/k8s-staging-ingress-nginx-gcb/o/logs%2Flog-8a89d93d-b08f-49af-bad6-abecc3082fbe.txt?jk=AFshE3WO198D4w4E00oU7h-TDSWJEten8JUi_sbd7o7xXESzzWK5WukQZ_9r5gWuZiTQfKPVd2Av3itSmNVbVvPRoYOfgSuTtK1LvlvkSzs9f8ndZTytswIQ2OllZybh5PcUR-pooIx1SJgPBzsWxEgbnBra1Ye6FAUwAvPhYZbtEeewshRnJn_vJdLAXbOoNs6tMAUzQ2Rx10YmBLSSl1d9_7fRI0z0UzPzi9G4RR0LJyGH3otAz-MjEFRgQtNN61RHgRwAm0EION2cb2dWLVN6Yu5RFzaIR6XeWwKTNInk2m2b3pTQ7zyOIEBY5R2QfVXitr4mKd9ly6vNBiBw4w8xbv7X1B1OLlRKQVQ2Px3446QxWEr5D2rSHcoTpDo4FW_LLcuw3S0BSjCmA0iV1idOz1HB6VoyH1ftcV_KLM86NkLhE4vis0G1x2SewE_YYJv-oQh6UVtFZrM7XPrtyn8BYFLp4_naQcCmdB_8FcZTWwW5KhYlr0UpGN3rL_2XhE0nMErGeObdG6dDLjl1yL4JUY3ssqidMBh_JVWepH5UbagpJcQQnFp0CMSE1mz5V4U8qOXh1BR3QpntB1SxzjHSwIQKPOW7vA5-Wz04JImIM4irLmLIRBn_xUOP1iam_fph3iDQcqkr117F_S0aaROfTEKubiMRxtnUFQx0vbFmf1aJE-oKqzRTuQgU52m2f8Lr0BZ4sB7m-k1BdVCRcCiimJiCWZh_DbC_Ha41rl7Wr71XM-jiezxTe3tGHmL0pIPa2jMCE6sFQeLXmKNWDeHnuYFdg-7RKx0Sx4tctRSu7yG0SpZuTPjESm16fEt_S_5yqRAPcoT7TGfN-81qaKsMfzAP981Um8FlFhkK2gynLPjp40Hc2mq_RdC3K682aX9KrcmuyQhjNCZrLZZjM_158LcZ5izx6Et2X_uaiNhlyLFudf5Z-QX3LYHRMxlXx9hplzz9h1ZypwT_ifeh95lpvFZcPTKVmwEbJPjnqiJ1RdOhwR0YAxbhsmCjJOstNsotrOigR4UI49WxCi9cV7gIE3wgN9DxoIH0ptUj0dKtzv9eyA&isca=1

longwuyuan · 2022-05-20T00:29:44Z

kubernetes/k8s.io#3746. we have a successful build . Much much gratitude @esigo

/close

longwuyuan added the bug Something isn't working label Apr 8, 2022

longwuyuan mentioned this issue Apr 8, 2022

An error occurred between opentelemetry modules kubernetes/ingress-nginx#8437

Closed

longwuyuan mentioned this issue May 7, 2022

added recommended flag to opentelemetry build kubernetes/ingress-nginx#8560

Closed

9 tasks

NissesSenap mentioned this issue May 9, 2022

Add tracing to Nginx Ingresses XenitAB/terraform-modules#668

Open

longwuyuan mentioned this issue May 14, 2022

Opentelemetry module build kubernetes/ingress-nginx#8585

Merged

9 tasks

esigo mentioned this issue May 19, 2022

opentelemetry use git instead of curl kubernetes/ingress-nginx#8606

Merged

9 tasks

longwuyuan closed this as completed May 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

symbol not found when loading opentelemetry module #141

symbol not found when loading opentelemetry module #141

longwuyuan commented Apr 8, 2022

longwuyuan commented Apr 8, 2022

longwuyuan commented Apr 21, 2022

longwuyuan commented May 6, 2022

mahaisong commented May 6, 2022

longwuyuan commented May 6, 2022

esigo commented May 6, 2022

longwuyuan commented May 6, 2022 •

edited

Loading

esigo commented May 7, 2022 •

edited

Loading

longwuyuan commented May 7, 2022

esigo commented May 7, 2022

longwuyuan commented May 7, 2022

longwuyuan commented May 7, 2022

esigo commented May 7, 2022

longwuyuan commented May 7, 2022

longwuyuan commented May 10, 2022

esigo commented May 10, 2022

longwuyuan commented May 10, 2022

mahaisong commented May 11, 2022

longwuyuan commented May 11, 2022

esigo commented May 14, 2022

esigo commented May 14, 2022

esigo commented May 14, 2022

longwuyuan commented May 19, 2022

longwuyuan commented May 20, 2022

symbol not found when loading opentelemetry module #141

symbol not found when loading opentelemetry module #141

Comments

longwuyuan commented Apr 8, 2022

longwuyuan commented Apr 8, 2022

longwuyuan commented Apr 21, 2022

longwuyuan commented May 6, 2022

mahaisong commented May 6, 2022

longwuyuan commented May 6, 2022

esigo commented May 6, 2022

longwuyuan commented May 6, 2022 • edited Loading

esigo commented May 7, 2022 • edited Loading

longwuyuan commented May 7, 2022

esigo commented May 7, 2022

longwuyuan commented May 7, 2022

longwuyuan commented May 7, 2022

esigo commented May 7, 2022

longwuyuan commented May 7, 2022

longwuyuan commented May 10, 2022

esigo commented May 10, 2022

longwuyuan commented May 10, 2022

mahaisong commented May 11, 2022

longwuyuan commented May 11, 2022

esigo commented May 14, 2022

esigo commented May 14, 2022

esigo commented May 14, 2022

longwuyuan commented May 19, 2022

longwuyuan commented May 20, 2022

longwuyuan commented May 6, 2022 •

edited

Loading

esigo commented May 7, 2022 •

edited

Loading