Disabling/Skipping failing Windows tests

[Blast545]

Desired behavior

Running the ign-gazebo buildfarm should be green as long as no new regressions are added to the code.

Alternatives considered

1. Using SKIP option with Cmake to specifically skip running some tests for Windows.
2. Using Labels to skip running some tests in our CI jobs, something like mark flaky / broken tests in a way to allow skipping them ros2/ros2#900.
3. Remove the failing tests from the Windows CMakeLists.txt, so they don't compile at all for windows. (it would be good to avoid this one).

Implementation suggestion

I'm OK with either option 1 or 2, I don't know in advance which one would be preferred by the Ignition team or if there are other disabled tests in the stack so I could rely on infrastructure already in place. So I would like some feedback on it before starting.

Additional context

As of https://build.osrfoundation.org/job/ign_gazebo-ign-6-win/20/, the failing tests for ign-gazebo6 Windows are:

ign-gazebo6 tests failing, 60 tests failed out of 174

         7 - UNIT_Component_TEST (Failed)
         11 - UNIT_EntityComponentManager_TEST (Failed)
         21 - UNIT_SdfGenerator_TEST (Failed)
         25 - UNIT_Server_TEST (Failed)
         27 - UNIT_SimulationRunner_TEST (Failed)
         39 - UNIT_ign_TEST (Failed)
         51 - UNIT_EntityFeatureMap_TEST (Failed)
         53 - INTEGRATION_ackermann_steering_system (Exit code 0xc0000374)
         55 - INTEGRATION_air_pressure_system (Failed)
         57 - INTEGRATION_altimeter_system (Failed)
         59 - INTEGRATION_apply_joint_force_system (Failed)
         61 - INTEGRATION_battery_plugin (Failed)
         63 - INTEGRATION_breadcrumbs (Failed)
         65 - INTEGRATION_buoyancy (Failed)
         67 - INTEGRATION_buoyancy_engine (Failed)
         69 - INTEGRATION_collada_world_exporter (Failed)
         73 - INTEGRATION_contact_system (Failed)
         75 - INTEGRATION_detachable_joint (Failed)
         77 - INTEGRATION_diff_drive_system (Failed)
         79 - INTEGRATION_each_new_removed (Exit code 0xc0000374)
         81 - INTEGRATION_entity_erase (Failed)
         83 - INTEGRATION_events (Failed)
         85 - INTEGRATION_examples_build (Failed)
         87 - INTEGRATION_follow_actor_system (Failed)
         89 - INTEGRATION_force_torque_system (Failed)
         93 - INTEGRATION_halt_motion (Failed)
         95 - INTEGRATION_imu_system (Failed)
         97 - INTEGRATION_joint_controller_system (Failed)
         99 - INTEGRATION_joint_position_controller_system (Failed)
        101 - INTEGRATION_joint_state_publisher_system (Failed)
        103 - INTEGRATION_joint_trajectory_controller_system (Failed)
        105 - INTEGRATION_kinetic_energy_monitor_system (Failed)
        107 - INTEGRATION_lift_drag_system (Failed)
        109 - INTEGRATION_level_manager (Failed)
        111 - INTEGRATION_level_manager_runtime_performers (Failed)
        115 - INTEGRATION_logical_camera_system (Failed)
        117 - INTEGRATION_logical_audio_sensor_plugin (Failed)
        119 - INTEGRATION_magnetometer_system (Failed)
        123 - INTEGRATION_multicopter (Failed)
        125 - INTEGRATION_multiple_servers (SEGFAULT)
        127 - INTEGRATION_nested_model_physics (Failed)
        129 - INTEGRATION_network_handshake (Failed)
        131 - INTEGRATION_odometry_publisher (Failed)
        133 - INTEGRATION_particle_emitter (Failed)
        135 - INTEGRATION_particle_emitter2 (Failed)
        137 - INTEGRATION_performer_detector (Failed)
        139 - INTEGRATION_physics_system (Failed)
        143 - INTEGRATION_pose_publisher_system (Failed)
        145 - INTEGRATION_save_world (Failed)
        147 - INTEGRATION_scene_broadcaster_system (Failed)
        149 - INTEGRATION_sdf_frame_semantics (Failed)
        151 - INTEGRATION_sdf_include (Failed)
        153 - INTEGRATION_spherical_coordinates (Failed)
        155 - INTEGRATION_touch_plugin (Failed)
        157 - INTEGRATION_triggered_publisher (Failed)
        159 - INTEGRATION_user_commands (Failed)
        161 - INTEGRATION_velocity_control_system (Failed)
        163 - INTEGRATION_log_system (Failed)
        165 - INTEGRATION_wheel_slip (Failed)
        167 - INTEGRATION_wind_effects (Failed)

As of https://build.osrfoundation.org/job/ign_gazebo-ign-5-win/45/, the failing tests for ign-gazebo5 Windows are:

ign-gazebo5 tests failing, 59 tests failed out of 168

          3 - UNIT_Component_TEST (Failed)
          9 - UNIT_EntityComponentManager_TEST (Timeout)
         13 - UNIT_ign_TEST (Failed)
         19 - UNIT_ModelCommandAPI_TEST (Failed)
         23 - UNIT_SdfGenerator_TEST (Failed)
         25 - UNIT_Server_TEST (Failed)
         29 - UNIT_SimulationRunner_TEST (Failed)
         47 - UNIT_Gui_TEST (Timeout)
         51 - UNIT_EntityFeatureMap_TEST (Failed)
         53 - INTEGRATION_ackermann_steering_system (Exit code 0xc0000409)
         55 - INTEGRATION_air_pressure_system (Failed)
         57 - INTEGRATION_altimeter_system (Failed)
         59 - INTEGRATION_apply_joint_force_system (Failed)
         61 - INTEGRATION_battery_plugin (Failed)
         63 - INTEGRATION_breadcrumbs (Failed)
         65 - INTEGRATION_buoyancy (Failed)
         67 - INTEGRATION_collada_world_exporter (Failed)
         71 - INTEGRATION_contact_system (Failed)
         73 - INTEGRATION_detachable_joint (Failed)
         75 - INTEGRATION_diff_drive_system (Failed)
         77 - INTEGRATION_each_new_removed (Exit code 0xc0000374)
         79 - INTEGRATION_entity_erase (Failed)
         81 - INTEGRATION_events (Failed)
         83 - INTEGRATION_examples_build (Failed)
         85 - INTEGRATION_follow_actor_system (Failed)
         89 - INTEGRATION_halt_motion (Failed)
         91 - INTEGRATION_imu_system (Failed)
         93 - INTEGRATION_joint_controller_system (Failed)
         95 - INTEGRATION_joint_position_controller_system (Failed)
         97 - INTEGRATION_joint_state_publisher_system (Failed)
         99 - INTEGRATION_joint_trajectory_controller_system (Failed)
        101 - INTEGRATION_kinetic_energy_monitor_system (Failed)
        103 - INTEGRATION_lift_drag_system (Exit code 0xc0000374)
        105 - INTEGRATION_level_manager (Failed)
        107 - INTEGRATION_level_manager_runtime_performers (Failed)
        111 - INTEGRATION_logical_camera_system (Failed)
        113 - INTEGRATION_logical_audio_sensor_plugin (Failed)
        115 - INTEGRATION_magnetometer_system (Failed)
        119 - INTEGRATION_multicopter (Failed)
        123 - INTEGRATION_nested_model_physics (Failed)
        125 - INTEGRATION_network_handshake (Exit code 0xc0000374)
        127 - INTEGRATION_odometry_publisher (Failed)
        129 - INTEGRATION_particle_emitter (Failed)
        131 - INTEGRATION_particle_emitter2 (Failed)
        133 - INTEGRATION_performer_detector (Failed)
        135 - INTEGRATION_physics_system (Failed)
        139 - INTEGRATION_pose_publisher_system (Failed)
        141 - INTEGRATION_save_world (Failed)
        143 - INTEGRATION_scene_broadcaster_system (Failed)
        145 - INTEGRATION_sdf_frame_semantics (Failed)
        147 - INTEGRATION_sdf_include (Failed)
        149 - INTEGRATION_touch_plugin (Failed)
        151 - INTEGRATION_triggered_publisher (Failed)
        153 - INTEGRATION_user_commands (Failed)
        155 - INTEGRATION_velocity_control_system (Failed)
        157 - INTEGRATION_log_system (Failed)
        159 - INTEGRATION_wheel_slip (Failed)
        161 - INTEGRATION_wind_effects (Failed)
        167 - PERFORMANCE_level_manager (Failed)

Probably some of them overlap, so I'll start with the ign-gazebo5 tests and then forward port.

[chapulina]

Thanks for ticketing the issue and looking into this, @Blast545 !

I recommend using the test macros provided by ign-cmake to disable tests:

https://github.com/ignitionrobotics/ign-cmake/blob/ign-cmake2/include/ignition/utilities/ExtraTestMacros.hh

Here's an example:

https://github.com/ignitionrobotics/ign-gazebo/blob/c36d4aae1075aa7dce0c950a107ff05bf9ec7f54/test/integration/thermal_sensor_system.cc#L76-L77

This makes sure that the tests are still compiled on all platforms, they just don't run.

[Blast545]

@chapulina I didn't know those existed, I'll use them. Thanks!

[Blast545]

Mmm, is it important for this ign-gazebo5 test to create 100000 components? I also noted this test it not longer available in ign-gazebo6 🕵️

https://github.com/ignitionrobotics/ign-gazebo/blob/ba53127c5b1418c17ca0fdfdcdd04667c09b06ff/src/EntityComponentManager_TEST.cc#L184

[chapulina]

Mmm, is it important for this ign-gazebo5 test to create 100000 components?

I guess we could reduce the number, is this causing issues on Windows?

I also noted this test it not longer available in ign-gazebo6 detective

That's because from Fortress the components aren't stored adjacently anymore

[Blast545]

@chapulina Yeah, it's timing out on Windows. In particular, this check is failing:

https://github.com/ignitionrobotics/ign-gazebo/blob/ba53127c5b1418c17ca0fdfdcdd04667c09b06ff/src/EntityComponentManager_TEST.cc#L195-L200

There's a problem registering one of the components in the test in Windows so the test fails all of the times in windows, also printing thousands of lines to the output log.

I'm disabling the test for Windows so it won't be a problem there, just wanted to make sure that high number makes sense for that particular test.

[chapulina]

Got it. It sounds like Windows has other problems besides the number of components. So I think for now it's enough to disable the test there, and we can revisit it later.