Stack errors with very large CSV files

[gtracy]

I'm getting a RangeError when I read very large - 570k - CSV files.

Reading from myfile.csv

/..../node_modules/fast-csv/lib/parser/parser_stream.js:277
emit: function (event) {
^
RangeError: Maximum call stack size exceeded

I do not have the same problem when I write the file of this size - only when I turn around and read it back.

Here's the pattern I'm using :

var file_name = process.argv[2];
var instream = fs.createReadStream(file_name);
var readStream = csv.parse({headers:true});

readStream.on('data', function(data) {
// transform the data and write it to a new file
});

readStream.on('end', function() {
console.log('\ndone.');
});

instream.pipe(readStream);

[gtracy]

Note that event if the on-data function does absolutely nothing, I'll still run into the RangeError

[doug-martin]

What version of node are you using?

[gtracy]

0.10.28

[phof]

+1, same issue here while running the following against a 6M lines csv file (1.51GB) on Node v0.10.33:

var output = fs.createWriteStream(resultFile, { encoding: 'utf-8' });
var input = fs.createReadStream(file);

csv
.fromStream(input, {headers: true})
.transform(function(obj, next){
    process.nextTick(function() {
        if(obj.TIMESTAMP) { obj.TIMESTAMP = moment(obj.TIMESTAMP, 'YYYYMMDDHHmmss.SSSZ').toISOString(); }
        return next(null, obj);
    });
});
.pipe(csv.createWriteStream({headers: true}))
.pipe(output);

/node_modules/fast-csv/lib/parser/parser_stream.js:277
    emit: function (event) {
                   ^
RangeError: Maximum call stack size exceeded

[doug-martin]

So I found the issue which had to do with emit calling itself if an error was thrown within an event listener. I changed this and removed the try catch from emit so the error should bubble up and added new tests to prevent any regressions.

Please let me know if the problem persists, but this was the only way I was able to reproduce.

-Doug

[doug-martin]

I should note you will have to upgrade to v0.6.0 to get the fix.

[gtracy]

awesome. i've verified the fix with my example.

thanks!

[doug-martin]

Awesome...thanks for the bug report

-Doug

[phof]

v0.6.0 solved the issue, thank you!

[doug-martin]

👍

[afanasy]

Solved for me too, thanks!