Tests failing on Windows

[ahmedkrmn]

When running MSBuild.exe check-hermes.vcxproj on Windows, all tests fail.

Below is an example for one of the tests:

********************
Testing:
FAIL: Hermes :: AST/es6/simplify-rest-param.js (16 of 1771)
******************** TEST 'Hermes :: AST/es6/simplify-rest-param.js' FAILED ********************
Script:
--
echo 'RUN: at line 8' > nul && C:/Users/AhmedKaraman/Desktop/version-control/hermes_workingdir/hermes/build/bin/Debug/hermesc -dump-transformed-ast -pretty-json C:/Users/AhmedKaraman/Desktop/version-control/hermes_workingdir/hermes/test/AST/es6/simplify-rest-param.js | C:/Users/AhmedKaraman/Desktop/version-control/hermes_workingdir/hermes/build/bin/Debug//FileCheck --match-full-lines C:/Users/AhmedKaraman/Desktop/version-control/hermes_workingdir/hermes/test/AST/es6/simplify-rest-param.js
--
Exit Code: 255

CommandOutput(stderr):
--
'C:'is not recognized as an internal or external command,
operable program or batch file.

--

• I tried rebuilding Hermes with --distribute and they still fail.

• I tried replacing the / with \ for the above test and rerunning it separately and it worked, so it might be a problem with using forward slashes for paths, but I'm not sure.

[haozhun]

We recognize that we didn't provide any instructions on running LIT tests on Windows. This is an improvement area.

On Windows, unit tests of Hermes can run in Visual Studio and other environments. However, LIT tests of Hermes can only run from Cygwin or MinGW (Git bash).

In PowerShell, I ran (you can likely skip these steps):

git clone https://github.com/facebook/hermes.git
mv hermes hermes-oss
cd hermes-oss
utils/build/configure.py --build-system='Visual Studio 15 2017 Win64'
cd build
cmake --build .
echo "'use strict'; function hello() { print('Hello World'); } hello();" | .\bin\Debug\hermes

After that, you can use either Cygwin or MinGW (Git bash). Note that it is necessary to add non-Cygwin Python and non-Cygwin CMake to the front of the path, as shown in the instructions below.

For Cygwin:

cd /cygdrive/c/open/hermes-oss/
cd build
export PATH="/cygdrive/c/Python27:/cygdrive/c/Program Files/cmake/bin:$PATH"
cmake --build . --target check-hermes

For MinGW (Git bash):

cd /c/open/hermes-oss
cd build
export PATH="/c/Python27:/c/Program Files/cmake/bin:$PATH"
cmake --build . --target check-hermes

[ahmedkrmn]

Thanks Haozhun for the detailed instructions.
It might useful if this is added to the Building and Running Hermes page to prevent anyone from facing this issue in the future.

I ran the tests on both Linux and Windows and I thought it would be helpful if I share the output with you because they both produce 1 failing testcase.

Linux (WSL):

[0/1] Running the Hermes regression tests
FAIL: Hermes-Unit :: VMRuntime/./HermesVMRuntimeTests/AlignedStorageTest.AdviseUnused (976 of 1274)
******************** TEST 'Hermes-Unit :: VMRuntime/./HermesVMRuntimeTests/AlignedStorageTest.AdviseUnused' FAILED ********************
Note: Google Test filter = AlignedStorageTest.AdviseUnused
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from AlignedStorageTest
[ RUN      ] AlignedStorageTest.AdviseUnused
../unittests/VMRuntime/AlignedStorageTest.cpp:124: Failure
Expected: (initial) != (FAILED), actual: 18446744073709551615 vs 18446744073709551615
[  FAILED  ] AlignedStorageTest.AdviseUnused (4 ms)
[----------] 1 test from AlignedStorageTest (4 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (5 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] AlignedStorageTest.AdviseUnused

 1 FAILED TEST

********************
Testing Time: 48.89s
********************
Failing Tests (1):  
    Hermes-Unit :: VMRuntime/./HermesVMRuntimeTests/AlignedStorageTest.AdviseUnused

  Expected Passes    : 1217
  Unsupported Tests  : 56
  Unexpected Failures: 1

Windows:

Running the Hermes regression tests
  -- Testing: 1265 tests, 12 threads --
  Testing: 0 .. 10.. 20.. 30.. 40
  FAIL: Hermes :: hermes/hermesc.js (532 of 1265)
  ******************** TEST 'Hermes :: hermes/hermesc.js' FAILED ********************
  Script:
  --
  : 'RUN: at line 8';   diff <(C:/Users/AhmedKaraman/Desktop/version-control/hermes_workingdir/hermes/build_release/bin/Debug/hermes -target=HBC -dump-bytecode C:/Users/AhmedKaraman/Desktop/version-control
  /hermes_workingdir/hermes/test/hermes/hermesc.js) <(C:/Users/AhmedKaraman/Desktop/version-control/hermes_workingdir/hermes/build_release/bin/Debug/hermesc -target=HBC -dump-bytecode C:/Users/AhmedKaraman
  /Desktop/version-control/hermes_workingdir/hermes/test/hermes/hermesc.js)
  --
  Exit Code: 2

  Command Output (stderr):
  --
  diff: /dev/fd/63: No such file or directory
  diff: /dev/fd/62: No such file or directory
CUSTOMBUILD : LLVM error : IO failure on output stream: invalid argument [C:\Users\AhmedKaraman\Desktop\version-control\hermes_workingdir\hermes\build_release\check-hermes.vcxproj]
  

  --
  
  ********************
  Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
  Testing Time: 663.31s
  ********************
  Failing Tests (1):
      Hermes :: hermes/hermesc.js

    Expected Passes    : 1204
    Expected Failures  : 4
    Unsupported Tests  : 56
    Unexpected Failures: 1

[haozhun]

I don't understand what's going on here. Both are very mysterious.

The first test failure is complaining that the two numbers (2^64 - 1) are not equal. I checked and both numbers have size_t type.

The second test failure is emitted by diff CLI tool, complaining that the arguments it receives (synthesized by the shell for process substitution) does not exist. What environment (Cygwin, MinGW) did you use for the second test failure? Can you see if process substitution works at all in that environment?

I do not have WSL set up. And I cannot reproduce either bug on Windows, Linux, or Mac.

[ahmedkrmn]

Hi @haozhun,
I'm using Git bash for running Windows test cases and it indeed supports process substitution.