Skip to content
This repository has been archived by the owner on Aug 24, 2021. It is now read-only.

Commit

Permalink
docs: Add a roadmap (#180)
Browse files Browse the repository at this point in the history
  • Loading branch information
jcornaz authored Aug 2, 2020
1 parent 7949c9f commit d0276dd
Show file tree
Hide file tree
Showing 2 changed files with 88 additions and 0 deletions.
3 changes: 3 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,11 @@ The project is incubating and its API is subject to changes.

Please give it a try and write a feedback in the issues_ or discuss on gitter_.

For mode details about the future of kwik, here's a roadmap_.

.. _issues: https://github.com/jcornaz/kwik/issues
.. _gitter: https://gitter.im/kwik-test/community
.. _roadmap: https://github.com/jcornaz/kwik/blob/main/ROADMAP.md

How it looks like
-----------------
Expand Down
85 changes: 85 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# The road map for version 1.0.0

This document details the direction `kwik` is taking and the `API` it would provide when reaching version `1.0.0`.

## How it will look like

### Basic test

```kotlin
@Test
fun isAssociative() = forAny(Arb.pair(Arb.int(), Arb.int())) { (x, y) ->
(x + y) alwaysEquals (y + x)
}
```

Few points to note:

* Instead of providing multiple overloads of `forAny` with different number of arguments, it would take exactly one fuzzer argument, I'd let the user create explicit combination.
* The lambda `forAny` would need to return an instance of `TestResult` (which in this example, is created by the call of `alwaysEqual`)

This has for consequences, that correct usage can be fully checked at compile time. The compiler will make sure for us that:

* Data can be generated (so the test can pass)
* An expectation/assertion has been formulated (so the test can fail)

All of that without loosing flexibility, fluency or information in case of failure.

### Skipping an evaluation

To skip an evaluation the user could either `filter` the arbitrary or return `TestResult.Skip` in evaluation:

```kotlin
forAny(Arb.pair(Arb.int(), Arb.int())) { (x, y) ->
if (x == y) {
TestResult.Skip
} else {
(x - y) neverEquals 0
}
}
```

### Test result

As demonstrated in the 2 previous examples, a `TestResult` api would be provided to let the user formulate expectation and control the evaluation.

This is heavily inspired from `TestResult` in rust's `quickcheck` and `Expectation` in `elm-test`.

An api for fluent formulation of expectation would be provided. Mostly inspired from `Kluent`, with slight different naming using `always` and `never` instead of `should` and `shouldNot`.

The different naming has two advantages:

* Don't cause confusion in codebases where `kwik` would cohabit with a fluent assertion library such as Kluent, Kotest or similar.
* Emphasis the *invariant* flavor of the test. Since it is not an example, but a rule that should always (or never) be true.

### Configuration

I'd give priority on configuration made via system property and environment variable.
It proved to be very powerful and can remove the need to edit the source code (with hard-coded config) when playing to reproducing errors.

It would still be possible to configure a *min* and *max* number of iterations from code.

```kotlin
forAny(Arb.int(), iterations = 10..100) { x ->
-(-x) alwaysEqual x
}
```

The actual number of iterations depends then on the global configuration. The code only defines the sensible limits in regard of the system under test.

The random generation would still be seeded. But the seed could only be set via system property or environment variable. Not from code.

### Composing and configuring fuzzers

`map`, `filter`, `andThen` (aka flatmap), `combine` and `merge` would be the main building block that should allow easy and safe creation of a wide variety of fuzzers.

`withSamples` and common aliases such as `withNull`, `withNaN`, etc. proved to be very useful and would be ported in the new api.

## Major tasks

* [ ] Mark the old API as Obsolete and remove experimental flag for the new API (#185)
* [ ] Introduce TestResult with a minimal api (#85)
* [ ] Provide a fuzzer API that can be used the same way as generator are currently used
* [ ] Provide decent support for input simplification (aka shrinking) (#62, #173, #174, #64)
* [ ] Move the concept of 'sample' from generators to fuzzers (#186)
* [ ] Deprecate old API and document how to migrate the new API (the more can be automated via Kotlin's deprecation system, the better)

0 comments on commit d0276dd

Please sign in to comment.