Skip to content

Testing Strategy

Jump to bottom
Kateřina Pilátová edited this page Dec 13, 2023 · 66 revisions

In this document, the way we test Code PushUp is covered and guidelines are included.

Test suites

On this project, we distinguish three types of functional tests by scope - unit tests, integration tests and E2E tests. All tests are written using Vitest.

Unit tests

  • Their scope refers to an isolated unit of code.
  • Aim to only test the internal logic and use mock where needed in order to narrow down the testing scope (file system and other).
  • These tests are located next to their production code counterpart and have the .unit.test.ts suffix.
  • One can run unit tests for a given package via npx nx run <package>:unit-test
  • One can run all unit tests via npx nx run-many -t unit-test

Integration tests

  • Their scope refers to a bigger unit of code in isolation or an integration with a 3rd-party tool.
  • Aim to test the output of the unit as a whole. When communicating with a 3rd-party tool, only tests its usage in CodePushup.
  • These tests are also located next to their production code counterpart and have the .integration.test.ts suffix.
  • Run unit tests for a given package via npx nx run <package>:integration-test
  • Run all unit tests via npx nx run-many -t integration-test

End-to-end tests

  • Test the whole CLI application and use actual deployed environment.
  • These tests are located in a separate project cli-e2e and their test files have the .e2e.test.ts suffix.
  • Run all E2E tests via npx nx run cli-e2e:e2e.

Automation

The test suites are automatically run within the CI workflow in the following scenarios:

  • All affected tests shall be run inside a pull request for all major OS (Linux, Windows, MacOS) (see here).
  • Each affected test suite shall have its linter run inside a pull request.
  • All tests are run on one platform after a pull request is merged to update the code coverage (see here).

Testing utilities

In order to keep the maintenance cost lower, there is a package dedicated to testing utilities called testing-utils. Any fixtures, helper functions, constants or mocks usable by multiple tests should be added here.

The folder structure should be fixtures for fixed data or examples and utils for utility functions.

The exception to this may be a utility which is applicable only to the current package. In this case, each package shall use the mocks folder to mirror the hierarchy of the testing-utils - fixtures go into fixtures subfolder and functions into utils subfolder. TypeScript is set up to include this folder only for its test configuration.

Mocking

Each test suite has a predefined set of setup files which mock appropriate parts of the logic. These mock setups may be found in the setup folder of the testing-utils library (see here).

One may apply them inside vitest.(unit|integration|e2e).config.ts in the following way:

export default defineConfig({
  // ...
  test: {
    // ...
    setupFiles: [
      // any setup files go here, e.g. for mocking the console
      '../../testing-utils/src/lib/setup/console.mock.ts',
    ],
  },
});

There is an additional file reset.mocks.ts which may be included in the setupFiles to automatically reset your mocks. It is located here.

Test quality

Good practices

The one message that overlaps all the good practices is the optimisation for DAMP (Descriptive and Meaningful Phrases). In test code, most time is spent by reading it and maintaining it. Your test code is most likely to be looked at by other people than yourself. Therefore, the most important quality of your test code is readability. The following principles should be followed when writing tests:

  • The test description should clearly define the use case for which the test was written (no should return correct data). It should not merely describe the implementation (no when X is set to true, Y is returned).
  • The test implementation should cover the defined use case (and only that use case) in an expected way. This means no need for investigation of what the test is doing - it should be obvious.
  • A test should only focus on and verify relevant parts of a given use case.
  • The test flow should be linear (no loops or conditionals) and one should be able to read it from top to bottom without having to scroll around or look into other files too much.
  • Use explicit values. A test knows which exact values it uses. Using too many helper variables and functions can obscure the logic and coverage and increases the time necessary to understand the test.
  • Realistic values should be used (no mock-data or similar). The test should tell a story which involves an actual user interacting with the application.
  • If it is necessary to use variables or helper functions, their name should be self-explanatory (no mock()). The nature of the entity should also be clear (refrain from using names similar to production code).

Now, that doesn't mean that DRY (Don't Repeat Yourself) doesn't have its place in the test code. It is a balance (see relevant article here). Here are some of our good practices based on the DRY principle:

  • There should only be one test that verifies a specific behaviour. That test should be implemented for the smallest unit possible where the use case is testable.
  • If the same function or flow is tested with different sets of inputs (most of the implementation is the same), dynamic testing (it.each) should be used.
  • Reusable testing utilities should live in testing-utils, see Testing utilities section.
  • Shared setup should be configured in a vitest configuration file, see Mocking section.

ESLint rules

CI

Vitest is a powerful tool. Its methods should be fully utilised to serve the tests. In order to ensure the quality of our test code, we have a set of ESLint rules in place for our test files. We use our own set of ESLint rules which can be found and reused from @code-pushup/eslint-config. The test rules can be found in @code-pushup/eslint-config/vitest.

Run ESLint on a specific package via npx nx run <package>:lint.

Code coverage

codecov

Code coverage is a metric that measures the percentage of lines of code run in automated tests. This is not to be confused with test coverage which aims to measure how much of the application logic based on business criteria is covered by tests.

We use codecov to measure the ratio of Code PushUp logic executed within our test suites. The code coverage is measured for unit and integration tests only (E2E tests only test critical happy paths).

The code coverage trends may be found here.

Errors

Aside from happy paths, the tests should also verify the application behaviour when something goes wrong. Vitest provides a wide variety of methods for this and should be utilised (toThrow, rejects and other).

Clone this wiki locally