Tricky new test-case requirements for Hydra?

[lukego]

I have a couple of test cases that I would like to define and I wonder how to best support them with Hydra.

Side effects on lab servers

Tests that have potential side-effects on the machine state that are important to clean up before the next test.

For example, I would like to run the same test at many different CPU frequencies (low, medium, high, "Turbo Boost") to be able to see how performance scales with clock speed. However, I would adjust the frequency via /sys/devices/system/cpu/... and this setting would persist until explicitly undone. If this setting would "leak" into other tests then it would cause mass confusion.

Is there a way that we can ensure that tests always start in a "known good" state? (Perhaps a pre-test hook that ensures default settings?)

Reusing expensive setup

Tests that depend on expensive setup that should be reused for several tests.

The case I have in mind is testing Mellanox NICs with many different firmwares. On the one hand I want to test Snabb with every available supported firmware. On the other hand it takes time (~ 1 minute) to burn a firmware and maybe frequently burning firmware reduces the mean time between failures of the NICs.

Is there a good way that we can group the tests together such that we achieve a high rate of testing but only rarely burn new firmwares onto the cards?

Related issue would be testing configuration options that require a reboot to change e.g. tests that compare with and without kernel boot-time options like IOMMU on/off/passthrough. On the one hand we want to test all of the different settings and on the other hand we don't want to reboot for every test.

[domenkozar]

Pushing Hydra/Nix to the limits :) Give me a few days to research the options.

[domenkozar]

For side effects I see two solutions at the moment:

1. Set pre-build-hook in /etc/nix/nix.conf, this hook is exectued before each build so it can setup a clean environment. The drawback is the fact that our integration into non-nixos stack is getting harder with this, but if users won't run such tests then it won't be needed.

2. Specify a bash snippet that's run as part of each benchmark in begining, also setting the environment. Drawback here is that solution is not general enough and in future refactorings may break the promise that each derivation ran sets up environment correctly.

For reusing expensive setup only one things comes to mind. On eiger there would be a timer/cronjob that would:

• disconnect build slave from hydra master
• ssh into the build slave
• using lock command wait for builds to finish
• alter system state to desired state (reboot, burn firmware)
• re-add the build slave under a different feature name

This way builds would require features like firmwareXXX or kernel4.X and they'd wait in the queue until a build slave would appear online to match the required feature(s).