Utf8 input

[jerch]

This PR adds UTF8 input to the terminal.

API is as simple as this:

Terminal.writeUtf8(data: Uint8Array);

The typed array can be picked directly from the websocket (set binaryType to 'arraybuffer') or from node-pty for local apps. "Binary strings" are not supported as kinda all tooling in JS world expects buffers for raw UTF8 data.

To actually see this in action with the demo several changes are needed:

• server.js
  • set pty encoding in server.js to null
  • change input buffer to work on Buffer instead of strings (see here: https://github.com/jerch/xterm.js/blob/c08c298061bd9ef3cdedf1e893535cf4b4d0555a/demo/server.js#L62-L76)
• attach addon:
  • set binaryType of the websocket to 'arraybuffer'
  • invoke .writeUtf8 instead of .write

PR relies on #1878, changes start at commit 6bdd380.

Edit:
Note that the UTF8 input currently does no flow control, thus can easily been flooded and "stall" the terminal. Fixed.

[jerch]

The last commit (e6e5ecc) applies the changes to the demo. This will most likely not be part of this PR.

Here are some numbers:

• without UTF8

   Context "lib/xterm_perfcases/input.js"
      Context "Terminal: ls -lR /usr/lib"
         Case "#1" : 1 - runtime: 2063.41 ms
         Case "#1" : 1 - throughput: 23.36 MB/s
         Case "#1" : 2 - runtime: 1884.72 ms
         Case "#1" : 2 - throughput: 25.57 MB/s
         Case "#1" : 3 - runtime: 1853.08 ms
         Case "#1" : 3 - throughput: 26.01 MB/s
         Case "#1" : 3 runs - average runtime: 1933.73 ms
         Case "#1" : 3 runs - average throughput: 24.98 MB/s

• with UTF8

   Context "lib/xterm_perfcases/input_utf8.js"
      Context "Terminal: ls -lR /usr/lib"
         Context "input throughtput"
            Case "#1" : 1 - runtime: 1443.94 ms
            Case "#1" : 1 - throughput: 33.41 MB/s
            Case "#1" : 2 - runtime: 1301.88 ms
            Case "#1" : 2 - throughput: 37.05 MB/s
            Case "#1" : 3 - runtime: 1279.51 ms
            Case "#1" : 3 - throughput: 37.70 MB/s
            Case "#1" : 3 runs - average runtime: 1341.78 ms
            Case "#1" : 3 runs - average throughput: 36.06 MB/s

To make sense of those numbers: The test measures the time taken for 50MB of ls -lR /usr/lib data to be written to an offscreen terminal object, means this includes all input parsing and storing in the terminal buffer, but no rendering. An input speed of >30 MB/s is quite impressive and actually faster than many native emulators.
This speedup relies heavily on the buffer reworks, as this table shows:

	Input throughput
JS Array buffer (v3.8)	~7 MB/s
TypedArray buffer (v3.9)	~18 MB/s
+ UTF32 buffer for parser (#1878, pending)	~25 MB/s
+ UTF8 input (this PR)	~35 MB/s

😸😸

One might wonder why raw UTF8 input gives that much of a further speedup compared to string input. The reason is basically the string to charCode conversion, which happens to be way more expensive than a UTF8 to UTF32 conversion if the data is already a bunch of charCodes.

For integrators this might be relevant as it will save several forth and back conversions going on from pty to xterm.js, example for our demo with websocket with string input:

• pty spits out UTF8 bytes
• node-pty reads them and does a UTF8 to JS string conversion (happens in Socket)
• server.js reads the string chunks from node-pty and forwards them to the websocket server (express-ws)
• websocket creates a string frame and actually decodes the string back to UTF8 (lol)
• browser gets the websocket frame as string message type and again converts the UTF8 data to JS strings
• terminal gets the JS string and converts it to UTF32 for further processing (only this is covered by the test above!)

With raw UTF8 this is more like (without going into details):

• pty spits out UTF8 bytes
• node-pty reads them and emits a Buffer object created on the memory
• server.js reads the buffer chunks from node-pty and forwards them to the websocket server (express-ws)
• websocket copies memory into the outgoing socket with some binary framing (no conversion)
• browser gets the websocket frame as binary message type and creates an ArrayBuffer
• terminal gets the buffer, creates a Uint8Array instance and converts it to UTF32 for further processing

Lesser conversions, lesser data shuffling around in the memory. Even for remote transport UTF8 is clearly the winner, as terminal data typically contains a high percentage of ASCII chars which will save bandwidth/time. UTF8 for president. 🚀

[Tyriar]

Remaining for this:

• Merge conflict (probably just messed up imports)
• Adding time-based limiting to writeUtf8

[jerch]

@Tyriar
Now with input limit logic (taken from Terminal.write). Not yet happy with this, it doubles the buffering logic and makes it impossible to mix string and utf8 byte inputs during heavy input flow (while the buffers are not empty). Any idea how to solve this? A mixed type buffer would solve it but looks quite wrong to me.

[Tyriar]

Looks good, just 2 comments then we can merge. I also did a little clean up and added an API test 🙂

[jerch]

@Tyriar Any clue why this test times out? Happened yday too, but worked in the second run.

[jerch]

Reminder to myself: Remove e6e5ecc before merging.

[Tyriar]

Any clue why this test times out? Happened yday too, but worked in the second run.

I saw that too, but rebuilding fixed it 🤔