Testing

Detect has three main types of tests:

• Unit
• Functional
• Battery

Ideally every Detectable in detect has at least one Battery test, several functional and many unit tests.

Unit

Unit tests operate on a very small scope and generally should have no external dependencies such as resource files and should not be testing cooperation between units.

A good unit test generally tests a single method on a single class. For example, a parser unit test might call parseLine("name:version") and verify the created Dependency had it's name and version correctly set.

Pros:

• Easy to implement.
• Easy to fix and debug.
• Easy to understand.

Cons:

• Breaks when refactoring.
• Don't provide much confidence in system as a whole.
• Does not test unit interactions.

Functional

Functional tests operate on a slightly larger scale and may test the coordination of multiple units. They may rely on external dependencies such as files in resources.

A good functional test generally replicates a single edge case with the smallest amount of data needed for the edge case to occur. For example, a functional tests wants to check that if a lock file is missing a dependency the transformer handles it. In this case, our functional test creates a small handcrafted lock file and runs that through a Parser and Transformer to verify the edge case is handled correctly.

Inline strings are preferred as it keeps the text "close" to the unit test - but if escaping or size becomes unwieldy it should be moved to a resource file.

Pros:

• Increases confidence in system as a whole.
• Can tests interaction of units.
• Complex edge cases can be tested.

Cons:

• Still breaks when refactoring.
• More difficult to fix when broken.

Battery

Battery tests operate on the largest scale and perform a full run of detect. These tests use real output generated from external systems. Generally these tests are for consistency to verify that things are still working as they were. Changes to the way we communicate with external systems breaks these tests.

A good battery test uses real output from external systems to test that the same bdio file is generated each time. For example, a real lock file and package json file are added to resources and given to detect. The generated bdio file is then added as a resource and the test checks the bdio file has not changed.

The battery test suite includes additional tooling to make it easier to maintain the battery tests. When a battery test breaks it is usually a manual and tedious process to determine why a file changed. Large real world output files contain thousands of dependencies and a battery test simply operates against a known bdio file.

Pros:

• Large confidence that overall system is working.
• Easy to take real output and create a test. (Especially from a diagnostic zip)
• Should never break when refactoring.
• Confidence that refactoring did not break existing customers.

Cons:

• Breaks are notoriously difficult to debug, are usually fixed by large scale file replacement.
• Breaks when external system communication changes (a new command is called).
• Breaks when BDIO changes, requiring all files to be updated.
• Adding a battery test requires getting real output from external systems.
• Has large overhead of needing to provide detect everything it needs for a given run.