Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use TSAN-enabled glib-2.0 libs for Matter CI TSAN checks #23529

Closed
wants to merge 7 commits into from
Closed

Use TSAN-enabled glib-2.0 libs for Matter CI TSAN checks #23529

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f5aa7b4
Partially revert commit 0e41b1911
arkq Nov 4, 2022
e30d22d
Fix race condition when setting GIO channel
arkq Nov 4, 2022
c4c5841
Run Linux TSAN tests with TSAN-enabled glib-2.0 libs
arkq Nov 7, 2022
0867c15
Move "-test" suffix to variant matrix declaration
arkq Dec 2, 2022
f0ff9c0
Allow to specify LD_PRELOAD env variable when launching test script
arkq Dec 6, 2022
8717be9
Install python bindings built with build_examples
arkq Dec 6, 2022
3bb32f1
Use GCC for TSAN Python bindings CI check
arkq Jan 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 89 additions & 28 deletions .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
env:
BUILD_VARIANT: ${{matrix.build_variant}}
CHIP_TOOL_VARIANT: ${{matrix.chip_tool}}
TSAN_OPTIONS: "halt_on_error=1 suppressions=scripts/tests/chiptest/tsan-linux-suppressions.txt"
TSAN_OPTIONS: halt_on_error=1
LSAN_OPTIONS: detect_leaks=1

if: github.actor != 'restyled-io[bot]'
Expand Down Expand Up @@ -199,6 +199,9 @@ jobs:
- name: Run Tests
timeout-minutes: 65
run: |
if echo ${BUILD_VARIANT} | grep -q tsan; then
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH_TSAN
fi
./scripts/run_in_build_env.sh \
"./scripts/tests/run_test_suite.py \
--chip-tool ./out/linux-x64-chip-tool${CHIP_TOOL_VARIANT}-${BUILD_VARIANT}/chip-tool \
Expand Down Expand Up @@ -240,7 +243,7 @@ jobs:
env:
BUILD_VARIANT: ${{matrix.build_variant}}
CHIP_TOOL_VARIANT: ${{matrix.chip_tool}}
TSAN_OPTIONS: "halt_on_error=1"
TSAN_OPTIONS: halt_on_error=1
LSAN_OPTIONS: detect_leaks=1 suppressions=scripts/tests/chiptest/lsan-mac-suppressions.txt

if: github.actor != 'restyled-io[bot]'
Expand Down Expand Up @@ -345,8 +348,17 @@ jobs:
name: REPL Tests - Linux
timeout-minutes: 130

strategy:
matrix:
# NOTE: This CI checks Python bindings with TSAN. Since TSAN check requires
# init code to be executed before anything else, we cannot use Python
# module with statically linked TSAN with not-instrumented CPython.
# As an easy workaround, we are going to use dynamically linked TSAN
# supported by GCC (we need non -clang variant here).
build_variant: [ipv6only-no-ble-no-wifi-tsan-test]
env:
TSAN_OPTIONS: "halt_on_error=1 suppressions=scripts/tests/chiptest/tsan-linux-suppressions.txt"
BUILD_VARIANT: ${{ matrix.build_variant }}
TSAN_OPTIONS: halt_on_error=1

if: github.actor != 'restyled-io[bot]'
runs-on: ubuntu-latest
Expand Down Expand Up @@ -374,29 +386,69 @@ jobs:
uses: actions/upload-artifact@v3
if: ${{ always() && !env.ACT }}
with:
name: bootstrap-logs-linux-${{ matrix.build_variant }}${{ matrix.chip_tool }}
name: bootstrap-logs-linux-${{ matrix.build_variant }}
path: |
.environment/gn_out/.ninja_log
.environment/pigweed-venv/*.log
- name: Build Python REPL and example apps
timeout-minutes: 50
run: |
scripts/run_in_build_env.sh './scripts/build_python.sh --install_wheel build-env'
./scripts/run_in_build_env.sh \
"./scripts/build/build_examples.py \
--target linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test \
--target linux-x64-python-bindings \
build \
--copy-artifacts-to objdir-clone \
"
"./scripts/build/build_examples.py \
--target linux-x64-all-clusters-${BUILD_VARIANT} \
--target linux-x64-python-bindings-${BUILD_VARIANT} \
build \
--copy-artifacts-to objdir-clone \
"
./scripts/run_in_build_env.sh \
"pip install --force-reinstall \
out/linux-x64-python-bindings-${BUILD_VARIANT}/controller/python/chip*.whl \
"
- name: Run Tests
timeout-minutes: 10
run: |
scripts/run_in_build_env.sh './scripts/tests/run_python_test.py --app out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factoryreset --script-args "--log-level INFO -t 3600 --disable-test ClusterObjectTests.TestTimedRequestTimeout"'
scripts/run_in_build_env.sh './scripts/tests/run_python_test.py --app out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factoryreset --app-args "--discriminator 1234 --KVS kvs1 --trace_decode 1" --script "src/python_testing/TC_RR_1_1.py" --script-args "--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021"'
scripts/run_in_build_env.sh './scripts/tests/run_python_test.py --app out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factoryreset --app-args "--discriminator 1234 --KVS kvs1 --trace_decode 1" --script "src/python_testing/TC_SC_3_6.py" --script-args "--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021"'
scripts/run_in_build_env.sh './scripts/tests/run_python_test.py --app out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factoryreset --app-args "--discriminator 1234 --KVS kvs1 --trace_decode 1" --script "src/python_testing/TC_DA_1_7.py" --script-args "--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --bool-arg allow_sdk_dac:true"'
scripts/run_in_build_env.sh './scripts/tests/run_python_test.py --app out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factoryreset --app-args "--discriminator 1234 --KVS kvs1 --trace_decode 1 --enable-key 000102030405060708090a0b0c0d0e0f" --script "src/python_testing/TC_TestEventTrigger.py" --script-args "--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --bool-arg allow_sdk_dac:true"'
if echo ${BUILD_VARIANT} | grep -q tsan; then
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH_TSAN
export CHIP_SCRIPT_LD_PRELOAD=libtsan.so.0
fi
./scripts/run_in_build_env.sh \
"./scripts/tests/run_python_test.py \
--app out/linux-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app \
--factoryreset \
--script-args '--log-level INFO -t 3600 --disable-test ClusterObjectTests.TestTimedRequestTimeout' \
"
./scripts/run_in_build_env.sh \
"./scripts/tests/run_python_test.py \
--app out/linux-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app \
--factoryreset \
--app-args '--discriminator 1234 --KVS kvs1 --trace_decode 1' \
--script 'src/python_testing/TC_RR_1_1.py' \
--script-args '--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021' \
"
./scripts/run_in_build_env.sh \
"./scripts/tests/run_python_test.py \
--app out/linux-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app\
--factoryreset \
--app-args '--discriminator 1234 --KVS kvs1 --trace_decode 1' \
--script 'src/python_testing/TC_SC_3_6.py' \
--script-args '--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021' \
"
./scripts/run_in_build_env.sh \
"./scripts/tests/run_python_test.py \
--app out/linux-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app \
--factoryreset \
--app-args '--discriminator 1234 --KVS kvs1 --trace_decode 1' \
--script 'src/python_testing/TC_DA_1_7.py' \
--script-args '--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --bool-arg allow_sdk_dac:true' \
"
./scripts/run_in_build_env.sh \
"./scripts/tests/run_python_test.py \
--app out/linux-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app \
--factoryreset \
--app-args '--discriminator 1234 --KVS kvs1 --trace_decode 1 --enable-key 000102030405060708090a0b0c0d0e0f' \
--script 'src/python_testing/TC_TestEventTrigger.py' \
--script-args '--storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --bool-arg allow_sdk_dac:true' \
"
- name: Uploading core files
uses: actions/upload-artifact@v3
if: ${{ failure() && !env.ACT }}
Expand All @@ -418,9 +470,13 @@ jobs:
name: Java Tests - Linux
timeout-minutes: 130

strategy:
matrix:
build_variant: [ipv6only-no-ble-no-wifi-tsan-clang-test]
env:
TSAN_OPTIONS: "halt_on_error=1 suppressions=scripts/tests/chiptest/tsan-linux-suppressions.txt"
BUILD_VARIANT: ${{ matrix.build_variant }}
JAVA_PATH: /usr/lib/jvm/java-8-openjdk-amd64
TSAN_OPTIONS: halt_on_error=1

if: github.actor != 'restyled-io[bot]'
runs-on: ubuntu-latest
Expand Down Expand Up @@ -448,17 +504,16 @@ jobs:
uses: actions/upload-artifact@v3
if: ${{ always() && !env.ACT }}
with:
name: bootstrap-logs-linux-${{ matrix.build_variant }}${{ matrix.chip_tool }}
name: bootstrap-logs-linux-${{ matrix.build_variant }}
path: |
.environment/gn_out/.ninja_log
.environment/pigweed-venv/*.log
- name: Build Java Mattter Controller and all clusters app
timeout-minutes: 50
run: |
scripts/run_in_build_env.sh './scripts/build_python.sh --install_wheel build-env'
./scripts/run_in_build_env.sh \
"./scripts/build/build_examples.py \
--target linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test \
--target linux-x64-all-clusters-${BUILD_VARIANT} \
--target linux-x64-java-matter-controller \
build \
"
Expand All @@ -467,7 +522,7 @@ jobs:
run: |
scripts/run_in_build_env.sh \
'./scripts/tests/run_java_test.py \
--app out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app \
--app out/linux-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app \
--app-args "--discriminator 3840 --interface-id -1" \
--tool-path out/linux-x64-java-matter-controller \
--tool-cluster "discover" \
Expand All @@ -479,7 +534,7 @@ jobs:
run: |
scripts/run_in_build_env.sh \
'./scripts/tests/run_java_test.py \
--app out/linux-x64-all-clusters-ipv6only-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app \
--app out/linux-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app \
--app-args "--discriminator 3840 --interface-id -1" \
--tool-path out/linux-x64-java-matter-controller \
--tool-cluster "pairing" \
Expand Down Expand Up @@ -509,10 +564,10 @@ jobs:

strategy:
matrix:
build_variant: [no-ble-no-wifi-tsan-clang]
build_variant: [no-ble-no-wifi-tsan-clang-test]
env:
BUILD_VARIANT: ${{matrix.build_variant}}
TSAN_OPTIONS: "halt_on_error=1"
TSAN_OPTIONS: halt_on_error=1

if: github.actor != 'restyled-io[bot]' && false
runs-on: macos-latest
Expand Down Expand Up @@ -547,24 +602,30 @@ jobs:
uses: actions/upload-artifact@v3
if: ${{ always() && !env.ACT }}
with:
name: bootstrap-logs-darwin-${{ matrix.build_variant }}${{ matrix.chip_tool }}
name: bootstrap-logs-darwin-${{ matrix.build_variant }}
path: |
.environment/gn_out/.ninja_log
.environment/pigweed-venv/*.log
- name: Build Python REPL and example apps
timeout-minutes: 50
run: |
scripts/run_in_build_env.sh './scripts/build_python.sh --install_wheel build-env'
./scripts/run_in_build_env.sh './scripts/build_python.sh --install_wheel build-env'
./scripts/run_in_build_env.sh \
"./scripts/build/build_examples.py \
--target darwin-x64-all-clusters-${BUILD_VARIANT}-test \
--target darwin-x64-all-clusters-${BUILD_VARIANT} \
build \
--copy-artifacts-to objdir-clone \
"
- name: Run Tests
timeout-minutes: 30
run: |
scripts/run_in_build_env.sh './scripts/tests/run_python_test.py --app out/darwin-x64-all-clusters-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factoryreset --app-args "--discriminator 3840 --interface-id -1" --script-args "-t 3600 --disable-test ClusterObjectTests.TestTimedRequestTimeout"'
./scripts/run_in_build_env.sh \
"./scripts/tests/run_python_test.py \
--app out/darwin-x64-all-clusters-${BUILD_VARIANT}/chip-all-clusters-app \
--factoryreset \
--app-args '--discriminator 3840 --interface-id -1' \
--script-args '-t 3600 --disable-test ClusterObjectTests.TestTimedRequestTimeout' \
"
- name: Uploading core files
uses: actions/upload-artifact@v3
if: ${{ failure() && !env.ACT }}
Expand Down
10 changes: 0 additions & 10 deletions scripts/tests/chiptest/tsan-linux-suppressions.txt
View file
Open in desktop

This file was deleted.

65 changes: 50 additions & 15 deletions scripts/tests/run_python_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ def EnqueueLogOutput(fp, tag, q):
try:
timestamp = float(line[1:18].decode())
line = line[19:]
except Exception as ex:
except Exception:
pass
sys.stdout.buffer.write(
(f"[{datetime.datetime.fromtimestamp(timestamp).isoformat(sep=' ')}]").encode() + tag + line)
Expand All @@ -67,13 +67,42 @@ def DumpProgramOutputToQueue(thread_list: typing.List[threading.Thread], tag: st


@click.command()
@click.option("--app", type=click.Path(exists=True), default=None, help='Path to local application to use, omit to use external apps.')
@click.option("--factoryreset", is_flag=True, help='Remove app config and repl configs (/tmp/chip* and /tmp/repl*) before running the tests.')
@click.option("--app-args", type=str, default='', help='The extra arguments passed to the device.')
@click.option("--script", type=click.Path(exists=True), default=os.path.join(DEFAULT_CHIP_ROOT, 'src', 'controller', 'python', 'test', 'test_scripts', 'mobile-device-test.py'), help='Test script to use.')
@click.option("--script-args", type=str, default='', help='Path to the test script to use, omit to use the default test script (mobile-device-test.py).')
@click.option("--script-gdb", is_flag=True, help='Run script through gdb')
def main(app: str, factoryreset: bool, app_args: str, script: str, script_args: str, script_gdb: bool):
@click.option(
"--app",
type=click.Path(exists=True),
help='Path to local application to use, omit to use external apps.')
@click.option(
"--factoryreset",
is_flag=True,
help='Remove app config and repl configs (/tmp/chip* and /tmp/repl*) before running the tests.')
@click.option(
"--app-args",
type=str,
default='',
help='The extra arguments passed to the device.')
@click.option(
"--script",
type=click.Path(exists=True),
default=os.path.join(DEFAULT_CHIP_ROOT, 'src', 'controller', 'python', 'test', 'test_scripts', 'mobile-device-test.py'),
help='Test script to use.')
@click.option(
"--script-args",
type=str,
default='',
help='Path to the test script to use, omit to use the default test script (mobile-device-test.py).')
@click.option(
"--script-gdb",
is_flag=True,
help='Run script through gdb')
@click.option(
"--script-ld-preload",
type=str,
help=(
'The LD_PRELOAD env variable value used when running the test script. This '
'option can be used to e.g. preload TSAN library without affecting python '
'test runner.'))
def main(app: str, factoryreset: bool, app_args: str, script: str, script_args: str,
script_gdb: bool, script_ld_preload: str):
if factoryreset:
# Remove native app config
retcode = subprocess.call("rm -rf /tmp/chip* /tmp/repl*", shell=True)
Expand Down Expand Up @@ -122,19 +151,25 @@ def main(app: str, factoryreset: bool, app_args: str, script: str, script_args:
'--log-format', '%(message)s'] + shlex.split(script_args)

if script_gdb:
#
# When running through Popen, we need to preserve some space-delimited args to GDB as a single logical argument. To do that, let's use '|' as a placeholder
# for the space character so that the initial split will not tokenize them, and then replace that with the space char there-after.
#
script_command = "gdb -batch -return-child-result -q -ex run -ex thread|apply|all|bt --args python3".split() + script_command
# When running through Popen, we need to preserve some space-delimited
# args to GDB as a single logical argument. To do that, let's use '|'
# as a placeholder for the space character so that the initial split
# will not tokenize them, and then replace that with the space char
# there-after.
launcher = "gdb -batch -return-child-result -q -ex run -ex thread|apply|all|bt --args python3".split()
else:
script_command = "/usr/bin/env python3".split() + script_command
launcher = "python3".split()

script_command = launcher + script_command
final_script_command = [i.replace('|', ' ') for i in script_command]

env = os.environ.copy()
if script_ld_preload:
env['LD_PRELOAD'] = script_ld_preload

logging.info(f"Execute: {final_script_command}")
test_script_process = subprocess.Popen(
final_script_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
final_script_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
DumpProgramOutputToQueue(log_cooking_threads, Fore.GREEN + "TEST" + Style.RESET_ALL,
test_script_process, log_queue)

Expand Down
Loading