Skip to content

Latest commit

 

History

History
60 lines (47 loc) · 3.79 KB

README.md

File metadata and controls

60 lines (47 loc) · 3.79 KB

WASM Spec Tests

This repo provides a set of unit tests that can be used to check a WASM Backend's conformance to the WebAssembly spec.

Tests

How tests are generated

  1. The JSON file for a spec test suite is read.
  2. For each spec test wasm defined in the JSON:
    • All the spec tests are created in a C++ file to match the function declaration as interpreted from the JSON.
    • Test are categorized into assert_trap/assert_exhaustion and assert_return.
    • Tests are split up into sub_apply functions based on the rules defined below.
    • An apply function is written that switches on the third parameter to decide which sub_apply function to run.
    • A map of the function name and the index in order is created to be used for merging.
  3. Unit Tests are generated based on the rules below.
  4. The generated C++ files are compiled and linked, without optimizations to prevent the empty functions from being optimized out.
  5. The generated WASM is combined with the original test wasm.
    • The imports and apply functions (and any helper functions) from the generated wasm are combined with the test function definitions from the spec test wasm.
    • Any necessary shifting of type/import/function/call/exports numbers is done.
      • This is where the generated map from above is used.
  6. The newly created merged wasms and unit test C++ files are copied into the appropriate directory in the Leap repo.

How tests are split up

  • Within a spec test suite, each assert_trap and assert_exhaustion test case is given a unique sub_apply function.

    • All tests in a suite are in the same WASM file, so the test that is run is based on the test.name passed in to apply (which calls the correct sub_apply).

  • Within a test suite, assert_return tests are grouped into sets of 100.

    • This is due to the limit on 1024 locals and 1024 func defs built into nodeos. Some spec tests had too many functions to have a sub_apply per test, and some had too many variables to be put all into one sub_apply.
    • 100 was found to be the number that did not exceed this maximum for all the tests.
    • The tests also have some reliance on ordering (a store may need to be called before a load for example).
    • 100 also works out to make sure the right ordering is achieved.

  • The unit tests are split into 2 groups. All of the assert_trap tests are grouped into one BOOST_DATA_TEST_CASE and all the assert_return tests are grouped into a second BOOST_DATA_TEST_CASE

  • The unit test files are grouped by test suite (all address tests are together, all call tests together, etc.)

How to generate tests

  • Run the setup_eosio_tests.py script with no options to see the help text.

Known Issues

  • memory_grow.3 -- Will fail if not deleted from generated tests.

    • Unclear how to hand alter this to have memory properly zeroed where expected.

  • start.7 -- Will fail if not deleted from generated tests.

    • Imports "print" from "spectest". Changing to any of the eosio::print functions results in "start function must be nullary" due to their requiring a parameter.

  • globals.2 -- Delete from generated tests or it segfaults due to missing wasm.

    • eosio-wasm2wast error "mutable globals cannot be exported" when converting to wast.
      • wasm2wat provided by WABT handles this correctly, implying an error in CDT.

  • globals.3 -- Delete from generated tests or it segfaults due to missing wasm.

    • eosio-wasm2wast error "mutable globals cannot be exported" when converting to wast.
      • wasm2wat provided by WABT handles this correctly, implying an error in CDT.

  • globals.14 -- Delete from generated tests or it segfaults due to missing wasm.

    • Imports "global_i32" from "spectest".
      • Unclear what an appropriate substition would be.

License

MIT