Problem with haxe.io.BytesInput

[robertshady]

Somewhere in Main.hx

    var bytes:openfl.utils.ByteArray = Assets.getBytes("assets/skins/default.skn");
    var newBytes:haxe.io.Bytes = haxe.io.Bytes.ofString(bytes.toString());

    for (x in 0...5)
    {
        trace("Byte #" + x + ": " + newBytes.toString().charCodeAt(x));     
    }

    var blah:Reader = new Reader(new haxe.io.BytesInput(newBytes));
    blah.read();

Reader.hx

package;

class Reader
{
var i : haxe.io.Input;

public function new(i)
{
    this.i = i;
}

public function read():Void
{
    var myByte;

    for (x in 0...5)
    {
        myByte = i.readByte();
        trace("Byte #" + x + ": " + myByte);
    }
}

}

MY OUTPUT

Main.hx:48: Byte #0 31
Main.hx:48: Byte #1 139
Main.hx:48: Byte #2 8
Main.hx:48: Byte #3 null
Main.hx:48: Byte #4 null

Reader.hx:19: Byte #0: 31
Reader.hx:19: Byte #1: 194
Reader.hx:19: Byte #2: 139
Reader.hx:19: Byte #3: 8
Reader.hx:19: Byte #4: 0

Where is the 194 byte in Reader.hx Byte #1 coming from?!?!?!? Shouldn't both outputs be the same?

NOTE: All of this stems from trying to read in a tar.gz file from the assets directory and parse it with the Haxe format library so I can let the user skin the application on the fly... Format expects the input to be of type haxe.io.Input, and in order to read assets cross-platform I need to use Assets.getBytes (as far as I understand). Either way, the data appears to be getting corrupted at some point.

[andyli]

Which one, haxe.io.Bytes or openfl.utils.ByteArray, do you think is correct?
Can you post the asset here such that we can reproduce the issue?

[robertshady]

The asset is just a standard tar.gz file whose first 3 bytes are in fact 31, 139, 8.
I think you misunderstood my explanation, if you look at the output - both of them are working on the newBytes array, but when I loop thru the bytes with readByte() it reads them incorrectly adding a 194 after the 31. There is no 194 in the file...

[andyli]

Ok. But still we need a small example that can reproduce the problem. For example:

import haxe.io.*;
import flash.utils.ByteArray;

class Test
{
    static function main()
    {
        var ba = new ByteArray(5);
        ba.writeByte(31);
        ba.writeByte(139);
        ba.writeByte(8);
        ba.writeByte(0);
        ba.writeByte(0);

        var b = Bytes.ofString(ba.toString());


        for (i in 0...b.length) {
            trace(b.toString().charCodeAt(i));
        }

        var input = new BytesInput(b);
        for (i in 0...b.length) {
            trace(input.readByte());
        }
    }
}

compiles with:

-cpp bin
-main Test
-lib openfl-native
--macro allowPackage('flash')
-cmd ./bin/Test

It correctly outputs:

Test.hx:19: 31
Test.hx:19: 139
Test.hx:19: 8
Test.hx:19: 0
Test.hx:19: 0
Test.hx:24: 31
Test.hx:24: 139
Test.hx:24: 8
Test.hx:24: 0
Test.hx:24: 0

Does the above outputs the same for you?
I'm using latest dev branch of haxe and hxcpp, compile and run on a Mac.

[robertshady]

I'm not sure what you are referring to.. I included the source code above, and the "asset" file in question is ANY standard tar.gz file (they all begin with 31, 139, 8 as the first 3 bytes)...

[andyli]

Now I tried to create a project that is exactly what you've described, but still it outputted correctly for me without the extra byte you mentioned. Here is my project: https://dl.dropboxusercontent.com/u/2661116/TestHaxe3148.zip
Please let me know if it behave correctly or not.

[robertshady]

Ok, I downloaded your zip, compiled it with "lime build flash" and ran it in Flash Player Debugger 11.2.202.228 (12.0) and got the following output (which matches my problem I described above)...

Main.hx:17: Byte #0: 31
Main.hx:17: Byte #1: 139
Main.hx:17: Byte #2: 8
Main.hx:17: Byte #3: null
Main.hx:17: Byte #4: null
Reader.hx:17: Byte #0: 31
Reader.hx:17: Byte #1: 194
Reader.hx:17: Byte #2: 139
Reader.hx:17: Byte #3: 8
Reader.hx:17: Byte #4: 0

I have the following libraries installed:

actuate: 1.7.3 [1.7.5]
ash: [1.5.3]
away3d: git [dev:/usr/lib/haxe/lib/away3d/git]
awe6: [2.3.642]
box2d: [1.2.0]
format: [3.0.5]
gameconsole: 1.1.0 2.0.0 3.0.0 3.0.1 3.0.2 [3.1.3]
haxelib_client: [3.1.0-rc.4]
hscript: [2.0.1]
hxcpp: 3.1.37 [3.1.39]
layout: [1.0.4]
lime-tools: 1.4.0 1.5.4 1.5.6 [1.5.7]
lime: 0.9.7 1.0.0 [1.0.1]
openfl-html5: [1.4.2-beta]
openfl-native: [1.4.0]
openfl-samples: [1.3.0]
openfl-stage3d: [0.1.6]
openfl-tiled: [0.4.2]
openfl: 1.2.2 1.4.0 2.0.0 [2.0.1]
spritesheet: [1.2.0]
stage3d-glsl: [0.1.0]
swf: [1.5.2]

Also, I am on a Mac if that matters...

[andyli]

I see. I thought you were targeting cpp. I can now reproduce it with other targets.
Following is a minimised example without openfl or asset dependency.

import haxe.io.Bytes;

class Test {
    static function main():Void {
        var b = Bytes.alloc(2);
        b.set(0, 31);
        b.set(1, 139);

        var bstr = b.toString();
        trace(bstr.length); //2, correct

        var b2 = Bytes.ofString(bstr);
        trace(b2.length); //3, but should be 2

        for (i in 0...b2.length) {
            trace(b2.get(i));
        }
    }
}

Targeting macro, cpp, php (correct):

2
2
31
139

Targeting flash:

2
3
31
194
139

Targeting JS, runs with node:

2
3
31
203
128

Targeting java, c#:

2
4
31
239
191
189

Looks like some crossplatform string issue.
Assigning to Nicolas.

[robertshady]

Ah, yes. Sorry for the confusion. Thanks for looking into this.

[andyli]

Btw, since you're reading a binary file, converting to and from string should be avoided anyway... You could get a haxe.io.Bytes by:

#if flash
//for flash, BytesData = flash.utils.ByteArray
var bytes = haxe.io.Bytes.ofData(openfl.Assets.getBytes("assets/Test.tgz")); 
#else
//for openfl-native, ByteArray extends Bytes
var bytes = openfl.Assets.getBytes("assets/Test.tgz");
#end

[robertshady]

Yes, I knew that using a string was a bad idea, but I was just testing trying to get it to read the file at all and format was complaining that I wasn't feeding it properly.

Before I even submitted this I switched to converting the ByteArray to haxe.io.bytes which seems to work cross-platform, albeit probably not as fast as your method which I'll give a try in a little bit.

Thanks for the info!

[ncannasse]

This is normal behavior, because the ofString assume UTF8 encoding but here you're inserting an invalid utf8 char into the bytes. In flash/js that would fail when doing bytes.toString but CPP does not check that. We are planning to review this as part of our work on Unicode (see #3072)