CI build should fail if 'make test' fails with segmentation fault

[SergeyKleyman]

It seems that 'make test' step of CI build marked as successful even when it actually fails with Segmentation fault. For example: PR elastic/apm-agent-php#26 build 5 is green even though Segmentation fault was fixed only in build 6.

[v1v]

I've just seen this particular ticket. Not sure if this is still the case.

As far as I see there are two reasons why the segmentation fault could happen:

• An environmental issue, the docker container could be causing the issue. The CI Jenkinsfile should catch any errors if the command to run the container failed.
• The source code itself could cause the issue.

The CI uses the make command, so therefore if the make command does not report the errorlevel then that's a problem.

What do I mean about the make command?

• https://github.com/elastic/apm-agent-php/blob/master/Dockerfile#L23

I've just tested the docker container scenario and the errorlevel is reported

$ docker build --build-arg PHP_VERSION=7.2 --tag test-php:7.2 .
$ docker run --rm -u $(id -u):$(id -g) -t -v $(pwd):/app test-php:7.2 make foo
make: *** No rule to make target 'foo'.  Stop.
$ echo $?                                                                     
2

Regarding the make command seems to be have as expected when there is a compilation error, for instance, I just changed a couple of files

diff --git a/src/ext/basic_macros.h b/src/ext/basic_macros.h
index 8d33dda..d69fa1d 100644
--- a/src/ext/basic_macros.h
+++ b/src/ext/basic_macros.h
@@ -9,7 +9,7 @@
    +----------------------------------------------------------------------+
  */
 
-#define ELASTICAPM_UNUSED( var ) (void)(var)
+define ELASTICAPM_UNUSED( var ) (void)(var)
 
 #define ELASTICAPM_PP_STRINGIZE_IMPL( token ) #token
 #define ELASTICAPM_PP_STRINGIZE( token ) ELASTICAPM_PP_STRINGIZE_IMPL( token )
diff --git a/src/ext/constants.h b/src/ext/constants.h
index a8ba8d4..daba0ad 100644
--- a/src/ext/constants.h
+++ b/src/ext/constants.h
@@ -9,7 +9,7 @@
    +----------------------------------------------------------------------+
  */
 
-#pragma once
+pragma once

and when I run the docker container to build the project it fails

$ docker run --rm -u $(id -u):$(id -g) -t -v $(pwd):/app test-php:7.2
...
                 ^~~~~
/app/src/ext/basic_types.h:18:22: error: declaration for parameter 'UInt' but no such parameter
 typedef unsigned int UInt;
                      ^~~~
/app/src/ext/ConfigManager.c:1332: error: expected '{' at end of input
 }
 
make: *** [Makefile:194: ConfigManager.lo] Error 1
$ echo $?                                                            
2

Please let me know if there is something else to be analysed or whether we can close this issue

[v1v]

I'm trying to reproduce the segmentation fault based for that particular PR #26 build 5 that was based on:

• bf6bc33

For such I did run the below commands:

$ hub pr checkout 26
$ git checkout bf6bc335396d7192ae5358b769c52cf9698a029a
$ docker build --tag apm-agent-php .
$ docker run --rm -ti -v $(pwd):/app apm-agent-php
$ docker run --rm -ti -v $(pwd):/app apm-agent-php make test

For the build the logs are the below ones:

Expand to view

/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/elasticapm.c -o elasticapm.lo 
mkdir .libs
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/elasticapm.c  -fPIC -DPIC -o .libs/elasticapm.o
/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/elasticapm_assert.c -o elasticapm_assert.lo 
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/elasticapm_assert.c  -fPIC -DPIC -o .libs/elasticapm_assert.o
/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/lifecycle.c -o lifecycle.lo 
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/lifecycle.c  -fPIC -DPIC -o .libs/lifecycle.o
/app/src/ext/lifecycle.c: In function 'elasticApmThrowExceptionHook':
/app/src/ext/lifecycle.c:148:5: warning: implicit declaration of function 'UNUSED_LOCAL_VAR' [-Wimplicit-function-declaration]
  148 |     UNUSED_LOCAL_VAR( resultCode );
      |     ^~~~~~~~~~~~~~~~
/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/log.c -o log.lo 
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/log.c  -fPIC -DPIC -o .libs/log.o
/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/php_error.c -o php_error.lo 
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/php_error.c  -fPIC -DPIC -o .libs/php_error.o
/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/supportability.c -o supportability.lo 
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/supportability.c  -fPIC -DPIC -o .libs/supportability.o
/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/SystemMetrics.c -o SystemMetrics.lo 
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/SystemMetrics.c  -fPIC -DPIC -o .libs/SystemMetrics.o
/bin/sh /app/src/ext/libtool --mode=compile cc  -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2   -c /app/src/ext/utils.c -o utils.lo 
 cc -I. -I/app/src/ext -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -DHAVE_CONFIG_H -g -O2 -c /app/src/ext/utils.c  -fPIC -DPIC -o .libs/utils.o
/bin/sh /app/src/ext/libtool --mode=link cc -DPHP_ATOM_INC -I/app/src/ext/include -I/app/src/ext/main -I/app/src/ext -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib  -DHAVE_CONFIG_H  -g -O2    -o elasticapm.la -export-dynamic -avoid-version -prefer-pic -module -rpath /app/src/ext/modules -lcurl elasticapm.lo elasticapm_assert.lo lifecycle.lo log.lo php_error.lo supportability.lo SystemMetrics.lo utils.lo 
cc -shared  .libs/elasticapm.o .libs/elasticapm_assert.o .libs/lifecycle.o .libs/log.o .libs/php_error.o .libs/supportability.o .libs/SystemMetrics.o .libs/utils.o  -lcurl  -Wl,-soname -Wl,elasticapm.so -o .libs/elasticapm.so
creating elasticapm.la
(cd .libs && rm -f elasticapm.la && ln -s ../elasticapm.la elasticapm.la)
/bin/sh /app/src/ext/libtool --mode=install cp ./elasticapm.la /app/src/ext/modules
cp ./.libs/elasticapm.so /app/src/ext/modules/elasticapm.so
cp ./.libs/elasticapm.lai /app/src/ext/modules/elasticapm.la
PATH="$PATH:/sbin" ldconfig -n /app/src/ext/modules
----------------------------------------------------------------------
Libraries have been installed in:
   /app/src/ext/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,--rpath -Wl,LIBDIR' linker flag

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------

Build complete.
Don't forget to run 'make test'.

For the test the logs are:

Expand to view

Build complete.
Don't forget to run 'make test'.

[24-Jun-2020 10:10:51 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'elasticapm.so' (tried: /app/src/ext/modules/elasticapm.so (Error relocating /app/src/ext/modules/elasticapm.so: UNUSED_LOCAL_VAR: symbol not found), /app/src/ext/modules/elasticapm.so.so (Error loading shared library /app/src/ext/modules/elasticapm.so.so: No such file or directory)) in Unknown on line 0
[24-Jun-2020 10:10:51 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'elasticapm.so' (tried: /app/src/ext/modules/elasticapm.so (Error relocating /app/src/ext/modules/elasticapm.so: UNUSED_LOCAL_VAR: symbol not found), /app/src/ext/modules/elasticapm.so.so (Error loading shared library /app/src/ext/modules/elasticapm.so.so: No such file or directory)) in Unknown on line 0

=====================================================================
PHP         : /usr/local/bin/php 
Warning: PHP Startup: Unable to load dynamic library 'elasticapm.so' (tried: /app/src/ext/modules/elasticapm.so (Error relocating /app/src/ext/modules/elasticapm.so: UNUSED_LOCAL_VAR: symbol not found), /app/src/ext/modules/elasticapm.so.so (Error loading shared library /app/src/ext/modules/elasticapm.so.so: No such file or directory)) in Unknown on line 0

PHP_SAPI    : cli
PHP_VERSION : 7.3.19
ZEND_VERSION: 3.3.19
PHP_OS      : Linux - Linux 0a5b21d9a795 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64
INI actual  : /app/src/ext/tmp-php.ini
More .INIs  :   
---------------------------------------------------------------------
PHP         : /usr/local/bin/phpdbg PHP Warning:  PHP Startup: Unable to load dynamic library 'elasticapm.so' (tried: /app/src/ext/modules/elasticapm.so (Error relocating /app/src/ext/modules/elasticapm.so: UNUSED_LOCAL_VAR: symbol not found), /app/src/ext/modules/elasticapm.so.so (Error loading shared library /app/src/ext/modules/elasticapm.so.so: No such file or directory)) in Unknown on line 0

PHP_SAPI    : 
Warning: PHP Startup: Unable to load dynamic library 'elasticapm.so' (tried: /app/src/ext/modules/elasticapm.so (Error relocating /app/src/ext/modules/elasticapm.so: UNUSED_LOCAL_VAR: symbol not found), /app/src/ext/modules/elasticapm.so.so (Error loading shared library /app/src/ext/modules/elasticapm.so.so: No such file or directory)) in Unknown on line 0
phpdbg
PHP_VERSION : 7.3.19
ZEND_VERSION: 3.3.19
PHP_OS      : Linux - Linux 0a5b21d9a795 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64
INI actual  : /app/src/ext/tmp-php.ini
More .INIs  : 
---------------------------------------------------------------------
CWD         : /app/src/ext
Extra dirs  : 
VALGRIND    : Not used
=====================================================================
TIME START 2020-06-24 10:10:51
=====================================================================

=================
TEST /app/src/ext/tests/001.phpt
SKIP Check if elasticapm is loaded [tests/001.phpt] 

=================
TEST /app/src/ext/tests/002.phpt
SKIP elasticapm should work even if service_name configuration option is not set [tests/002.phpt] 

=================
TEST /app/src/ext/tests/003.phpt
SKIP Check if elasticapm provides transaction_id and trace_id [tests/003.phpt] 
=====================================================================
TIME END 2020-06-24 10:10:51

=====================================================================
TEST RESULT SUMMARY
---------------------------------------------------------------------
Exts skipped    :    0
Exts tested     :   33
---------------------------------------------------------------------

Number of tests :    3                 0
Tests skipped   :    3 (100.0%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :    0 (  0.0%) (  0.0%)
---------------------------------------------------------------------
Time taken      :    0 seconds
=====================================================================

But I cannot see any segmentation faults, any hints how I can reproduce it locally?

[SergeyKleyman]

It's already implemented.