Unicode sys tests and fixes

[Aurel300]

Closes #8094.

General

• define target.unicode - true by default, false on Neko or with (cpp && !cppia && !hxcpp_smart_strings) (extracted into a separate PR target.unicode define #8244)

Test setup

• create a script to generate a test suite with Unicode-named files and directories
  • no NUL bytes
  • no /:*"?<>|
  • UTF-8 boundary conditions
  • non-BMP characters
  • NFC/NFD ambiguous characters
  • compatibility equivalent characters? (out of scope)

Filesystem

• Sys
  • setCwd, getCwd (cd into Uni directory, read back path)
  • programPath (symlink in a Uni directory, read back path) this does not work over symlinks – we could copy binaries into Uni directories and run them there, but this is problematic to cover for all targets; getCwd should mostly cover this anyway
• sys.FileSystem
  • absolutePath (after setCwd, nested)
  • exists
  • fullPath (with symlinks in Uni directories)
  • isDirectory
  • readDirectory (test dir + nested Uni directories)
  • stat
  • rename
  • deleteFile
  • deleteDirectory
  • createDirectory
• sys.io.File
  • copy (this should probably be in sys.FileSystem)
• haxe.io.Path
  • construct + check directory components, name components, extension components

I/O API

Read from a known source. Write to a temporary file and compare on Bytes against a known correct output.

• trace
• sys.io.File
  • getBytes
  • getContent
  • read (+ haxe.io.Input test)
  • saveContent
  • update, write, append (+ haxe.io.Output test)
  • Unicode paths for the above
• haxe.io.Input
  • readLine
  • readString
  • readUntil
• haxe.io.Output
  • writeString
• Sys
  • stdin, stderr, stdout
  • print, println
  • getChar (not Unicode related)

Environment-related API

• Sys
  • args
  • environment
  • getEnv, putEnv

Process API

• Sys
  • command - Uni executable, Uni arguments
• sys.io.Process
  • new
  • stdin, stderr, stdout

Miscellaneous

• move sys Unicode tests into TestUnicode
• genTestRes.sh Windows port commit test suite to git just use Python
• some common scaffolding for running process of the same platform (currently in TestUnicode as well as TestCommandBase)
• replace ExitCode with UtilityProcess

---

Check targets

• Travis
  • cpp
  • cs
  • eval
  • hl
  • lua
  • java
  • jvm
  • php
  • python
• AppVeyor
  • cpp
  • cs
  • eval
  • hl
  • java
  • jvm
  • php
  • python

Issues / PRs spawned

• (C++) trace, Sys.print, Sys.println, String.split, UTF-8 decoding OOB Unicode fixes hxcpp#781
• (Neko / HL) Utf8.iter (in native code) throws for \x7F Utf.iter8 throws for 0x7F on Neko #8139
• Input.readString should read read N characters Input.readString should be in characters #8199 (bytes it is!)
• (PHP) StringTools.endsWith fails on Unicode strings [php] StringTools.endsWith fails with Unicode #8211
• (PHP) Bytes.getString with non-BMP sequences [php] haxe.io.Bytes.getString issues #8212
• (HL) haxe.io.Path uses StringTools.fastCodeAt without handling surrogates (fix is part of this PR)
• (C++) Sys.args does not decode UTF-8 arguments correctly Sys.args with unicode hxcpp#783
• (C++) UTF-16 surrogates are incorrectly detected Fix faulty detection of UTF-16 surrogates hxcpp#782
• (HL) Surrogate issues Unicode fixes hashlink#246
• (HL) U+10000 swallows next character [hl] Unicode char swallows next character #8245
• (Python) Output.writeString writes bytes individually as codepoints Output.writeString issue with Unicode strings on Python #8197
• (Python) Sys.stdin().readLine() fails on Unicode input [python] Fix #8373 #8374
• (Python, PHP, eval) Sys.println() should append \r\n on Windows (fix is part of this PR)
• (Python) trace defaults to cp1252 on Windows when piped (fix is part of this PR)
• (C#, Java, JVM) Sys.stdout may not default to UTF-8 on Windows (fix is part of this PR)
• (C++) UTF-16 decoding OOB
• (C#, Lua) FileSystem.fullPath does not resolve symlinks [cs,lua] FileSystem.fullPath does not resolve symlinks #8215
• (C#) FileSystem.readDirectory reports bad encoding [cs] readDirectory bad encoding #8247
• (Lua) Sys.environment segfault [lua] Sys.environment segfault #8216
• (C++, HL) Sys.print duplicates CR on Windows Sys.print duplicates CR on Windows #8379
• (C++) Sys.setCwd, .deleteFile, .deleteDirectory, FileSystem.exists, .rename fail on Unicode path [cpp] Sys.setCwd, .deleteFile, .deleteDirectory, FileSystem.exists, .rename fail on Unicode path #8400

[RealyUniqueName]

sys.io.File is missing on your list.

[RealyUniqueName]

Also, a windows script for unicode filenames generation is needed.

[Aurel300]

sys.io.File is missing on your list.

Yup, I only listed the methods for FileSystem, didn't go through File in detail yet.

Also, a windows script for unicode filenames generation is needed.

I might port it yet, although the idea was partly to generate the test suite and commit the resulting files and directories to git.

[ncannasse]

Given there's not much fixes to do I'm putting that in Release 4.0 to monitor the changes. In case we have some very target specific issue yet unresolved we will still not block final release because of this.

[Aurel300]

So the non-Windows part of this is basically done. The various target failures should be covered by the "Issues / PRs spawned" list. I am not sure how to proceed with Windows support here (cc @andyli):

The last commit attempted to simply commit the test suite directory with its Unicode-named files and sub-directories. But AppVeyor fails to even git clone the repository with this. Apparently chcp 65001 should be run beforehand, but git clone is not even in the CI file, it is run automatically.

I also tried to put the test-res directory into a ZIP archive. Unfortunately this seems to fail on OS X already (it tries to override existing files, because \x01 is completely ignored, I think?) … Apparently this is best "solved" by using a different program, e.g. 7z.

Porting genTestRes.sh would be best, but outputting the correct Unicode filenames seems to be rather painful in Windows batch. I found workarounds to simply put binary data into a file (e.g. https://stackoverflow.com/questions/47750732/write-hex-values-to-file-in-windows-batch), but nothing yet to actually create files and directories with the correct names.

[andyli]

7z is pre-installed in AppVeyor: https://www.appveyor.com/docs/windows-images-software/#tools

You can also use whatever program that can correctly handle Unicode file names correctly (e.g. Python?).

One tip about working with AppVeyor: You can remote desktop into the build and do whatever testing/debugging there. Just make sure you're not doing it in the HF repo, but use a fork of your own, to prevent other people connect into it and print out our secret variables (those wouldn't be available in fork builds).

[ncannasse]

I have merged the HL PR and restarted the build to update results.

[ncannasse]

Uhm, something is weird there then, how and why were the tests were broken?

…

On Tue, Apr 30, 2019 at 3:57 PM Aurel ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In std/haxe/io/Path.hx <#8135 (comment)>: > @@ -230,8 +230,13 @@ class Path { var acc = new StringBuf(); var colon = false; var slashes = false; + #if hl It was (in bba3863 <bba3863>), but that broke the Java tests. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#8135 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHZXQE6JBZCL3QM322L3VLPTBF5BANCNFSM4HEWZOGQ> .

[Aurel300]

Uhm, something is weird there then, how and why were the tests were broken?

Here are the failing tests: https://travis-ci.org/HaxeFoundation/haxe/jobs/524618714#L3553

I didn't investigate why they failed, but you are right, it really should work on Java. Interestingly enough, the failures are e.g. expected 0x1F619, but got 0xF619 (forgetting to add the 0x10000), and this is through the Unicode-safe string iterator, I think.

I'll investigate if there is a problem with the iterator on Java tomorrow. I'm also wondering why it even works as-is on Java.

[Aurel300]

Ok, fixed that with #8239.

[Aurel300]

It would be best if somebody with a working Haxe setup on Windows looked into this. It is difficult for me to debug the targets from AppVeyor results.

---

Update: I got a Vagrant set up with Windows 10 that mostly works, so hopefully I can progress on this again.

[Aurel300]

Travis failure is unrelated (brew problem when installing Flash player).