Open Payments: E2E conformance testing

[matdehaast]

Summary

The goal of this issue is to have full E2E conformance tests for the open payment apis, where we can ensure that an implementation is conformant to the API spec.

This is still being fleshed out and awaiting a full proposal from @omertoast

Intended Outcomes

• Pre-defined config files to standardize testing cases
• Modular test creation
• Automate tests using CI/CD
• Have scenarios (test modules) consisting of multiple requests and responses to simulate a real-life scenario.

How will it work?

• We are planning to use conformance suite as the primary testing tool. It's created and used by OpenID to test OpenID Connect and higher-level specs like FAPI. We have to create test modules for each case we want to test. After having a Rafiki instance up and seeded with demo values, we can run the conformance suite against it.

Links

• https://gitlab.com/openid/conformance-suite
• https://gitlab.com/openid/conformance-suite/-/wikis/Developers/Build-&-Run
• https://gitlab.com/openid/conformance-suite/-/wikis/Authlete-Example-Configuration

[omertoast]

I created a fork of the conformance suite and a few scenarios you can try to run.