Update to DataFrames v0.11, CSV 0.2.1, RData 0.3 #48

alyst · 2017-09-19T16:43:51Z

A part of JuliaData/DataFrames.jl#1232.
Still have to update .yml so that CI loads master versions of DataFrames, CSV and RData.

randyzwitch · 2017-09-19T16:48:54Z

Thanks @alyst. When this is ready for merging, let me know

alyst · 2017-09-19T16:50:48Z

@quinnj I've switched from readtable() to CSV.read(), but now I've got errors like:

DataStreams.Data.NullException("encountered a null value for a non-null column type on row = 6, col = 1")
 Stacktrace:
   [1] (::CSV.##4#5)(::Int64, ::Int64) at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:124
   [2] parsefield at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:233 [inlined]
   [3] parsefield at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:315 [inlined]
   [4] parsefield at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:130 [inlined] (repeats 2 times)
   [5] streamfrom at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:205 [inlined]
   [6] macro expansion at /fs/home/<>/.julia/v0.6/DataStreams/src/DataStreams.jl:272 [inlined]
   [7] stream!(::CSV.Source{Base.AbstractIOBuffer{Array{UInt8,1}},Nulls.Null}, ::Type{DataStreams.Data.Field}, ::DataFrames.DataFrameStream{Tuple{CategoricalArrays.CategoricalArray{String,1,UInt32,String,Union{}},WeakRefStrings.WeakRefStringArray{Union{Nulls.Null, WeakRefString{UInt8}},1}}}, ::DataStreams.Data.Schema{true,Tuple{CategoricalArrays.CategoricalValue{String,UInt32},Union{Nulls.Null, WeakRefString{UInt8}}}}, ::Int64, ::Tuple{Base.#identity,Base.#identity}, ::DataStreams.Data.##15#16, ::Array{Any,1}) at /fs/home/<>/.julia/v0.6/DataStreams/src/DataStreams.jl:344
   [8] #stream!#17(::Bool, ::Dict{Int64,Function}, ::Function, ::Array{Any,1}, ::Array{Any,1}, ::Function, ::CSV.Source{Base.AbstractIOBuffer{Array{UInt8,1}},Nulls.Null}, ::Type{DataFrames.DataFrame}) at /fs/home/<>/.julia/v0.6/DataStreams/src/DataStreams.jl:222
   [9] (::DataStreams.Data.#kw##stream!)(::Array{Any,1}, ::DataStreams.Data.#stream!, ::CSV.Source{Base.AbstractIOBuffer{Array{UInt8,1}},Nulls.Null}, ::Type{DataFrames.DataFrame}) at ./<missing>:0
   [10] #read#39(::Bool, ::Dict{Int64,Function}, ::Bool, ::Array{Any,1}, ::Function, ::String, ::Type{T} where T) at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:311
   [11] dataset(::String, ::String) at /fs/home/<>/.julia/v0.6/RDatasets/src/dataset.jl:13
   [12] macro expansion at /fs/gpfs03/lv06/pool/pool-innate-bioinfo2/<>/.julia/v0.6/RDatasets/test/dataset.jl:15 [inlined]
...

or

CSV.ParsingException("Unexpected start of quote (34), use \"9234\" to type \"34\"")
  Stacktrace:
   [1] readsplitline!(::Array{CSV.RawField,1}, ::Base.AbstractIOBuffer{Array{UInt8,1}}, ::UInt8, ::UInt8, ::UInt8, ::Base.AbstractIOBuffer{Array{UInt8,1}}) at /fs/home/<>/.julia/v0.6/CSV/src/io.jl:98
   [2] #Source#22(::String, ::CSV.Options{Nulls.Null}, ::Int64, ::Int64, ::Array{Type,1}, ::Nulls.Null, ::Bool, ::Int64, ::Int64, ::Int64, ::Bool, ::Type{T} where T) at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:99
   [3] (::Core.#kw#Type)(::Array{Any,1}, ::Type{CSV.Source}) at ./<missing>:0
   [4] #Source#21(::UInt8, ::UInt8, ::UInt8, ::String, ::Int64, ::Int64, ::Array{Type,1}, ::Nulls.Null, ::Nulls.Null, ::UInt8, ::String, ::String, ::Bool, ::Int64, ::Int64, ::Int64, ::Bool, ::Type{T} where T, ::String) at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:27
   [5] CSV.Source(::String) at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:24
   [6] #read#39(::Bool, ::Dict{Int64,Function}, ::Bool, ::Array{Any,1}, ::Function, ::String, ::Type{T} where T) at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:310
   [7] dataset(::String, ::String) at /fs/home/<>/.julia/v0.6/RDatasets/src/dataset.jl:13
   [8] macro expansion at /fs/gpfs03/lv06/pool/pool-innate-bioinfo2/<>/.julia/v0.6/RDatasets/test/dataset.jl:15 [inlined]
...

or

ArgumentError: Symbol name may not contain \0
 Stacktrace:
   [1] macro expansion at ./broadcast.jl:153 [inlined]
   [2] macro expansion at ./simdloop.jl:73 [inlined]
   [3] macro expansion at ./broadcast.jl:147 [inlined]
   [4] _broadcast! at ./broadcast.jl:139 [inlined]
   [5] broadcast_t at ./broadcast.jl:268 [inlined]
   [6] broadcast_c at ./broadcast.jl:314 [inlined]
   [7] broadcast at ./broadcast.jl:434 [inlined]
   [8] close!(::DataFrames.DataFrameStream{Tuple{CategoricalArrays.CategoricalArray{String,1,UInt32,String,Union{}}}}) at /fs/home/<>/.julia/v0.6/DataFrames/src/abstractdataframe/io.jl:302
   [9] #read#39(::Bool, ::Dict{Int64,Function}, ::Bool, ::Array{Any,1}, ::Function, ::String, ::Type{T} where T) at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:312
   [10] dataset(::String, ::String) at /fs/home/<>/.julia/v0.6/RDatasets/src/dataset.jl:13
   [11] macro expansion at /fs/gpfs03/lv06/pool/pool-innate-bioinfo2/<>/.julia/v0.6/RDatasets/test/dataset.jl:15 [inlined]
...

Maybe you have an easy fix/explanations to those.

alyst · 2017-09-21T09:56:20Z

I've failed to resist the temptation of caching the datasets() and packages() tables in global variables. I'm not sure, though, the way I implemented it follows the recommended code pattern.

quinnj · 2017-09-21T09:57:28Z

@alyst, can you list the individual files you saw the errors on? Happy to take a look.

alyst · 2017-09-21T10:04:25Z

The csv.gz issues listed above are resolved. By default I'm using 200 rows for type detection now, and there are some datasets that need more rows to properly infer types.
12 datasets are failing, but mostly because null are encountered beyond 200-th row.
So far the only dataset that requires fixing JuliaData/CSV.jl#64 is datasets/attenu.csv.gz, the inferred type of Station column is CategoricalArrays.CategoricalValue{String,UInt32}:

DataStreams.Data.NullException("encountered a null value for a non-null column type on row = 79, col = 3")
  Stacktrace:
   [1] (::CSV.##4#5)(::Int64, ::Int64) at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:124
   [2] parsefield at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:235 [inlined]
   [3] parsefield at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:317 [inlined]
   [4] parsefield at /fs/home/<>/.julia/v0.6/CSV/src/parsefields.jl:131 [inlined] (repeats 2 times)
   [5] streamfrom at /fs/home/<>/.julia/v0.6/CSV/src/Source.jl:216 [inlined]

(the CSV.jl line numbers could be off, because I've made some changes to support all-nulls columns):

alyst · 2017-11-27T00:15:26Z

Some datasets need JuliaData/CSV.jl#115 to load properly

tlnagy · 2017-12-06T20:46:35Z

Thanks for working on this. I believe RDatasets is only package that needs to be updated before we can patch up Gadfly to work on top of Dataframes v0.11

alyst · 2017-12-06T20:54:01Z

I'm waiting for JuliaData/CSV.jl#115 to remove [WIP] and consider merging. I hope that PR would be reviewed in a few days.

randyzwitch · 2017-12-07T14:30:21Z

Just ping me when this is ready for merging

Nosferican · 2017-12-24T12:06:36Z

@alyst JuliaData/CSV.jl#115 has been merged.

randyzwitch · 2017-12-24T12:41:37Z

Even with the PR being merged, it still needs to be tagged. And that tag should be the minimum version in REQUIRES here.

Nosferican · 2017-12-24T12:44:58Z

@quinnj could you tag a minor release? It would help with updating RDatasets and Gadfly.

alyst · 2017-12-24T12:45:15Z

@Nosferican Thanks! Actually, I haven't forgotten this PR at all :) Besides JuliaData/CSV.jl#115 there are a few other outstanding issues (notably JuliaData/CSV.jl#132), which we are trying to resolve. Once it's done and CI is green for DataFrames and CSV, I'll update this PR and it's ok to merge.

quinnj · 2018-01-10T14:39:12Z

A new tag of CSV was just merged.

randyzwitch · 2018-01-10T16:10:56Z

@alyst Travis fails overall because it still mentions 0.4 and 0.5, but 0.6 passes. If you could please modify .travis.yml to just test 0.6 and master, then we should be able to tag a new version of RDatasets

alyst · 2018-01-10T16:14:21Z

@randyzwitch I hope to have a look tomorrow.

for JuliaStats#48 (comment)

for #48 (comment)

randyzwitch · 2018-01-10T16:50:55Z

Thanks to @StoneCypher, this is now passing on 0.6 as well. Up to you @alyst if there are more updates you want to do, or if I should merge this in and tag a new release.

tlnagy · 2018-01-11T00:44:26Z

It would be great to have this merged and a new release tagged as soon as possible. Thanks everyone for getting this up and running. (esp @alyst!)

- decompress csv.gz and feed the stream to CSV using CodecZlib - raise row detection limit for csv files to 15000 as vcd::Bundesliga needs to read upto row #10804 to detect that 7th column can contain missing values

so that test results for non-problematic packages are collapsed into a single row in the report

allow digits in the first position to make psych::cubits pass, but disallow whitespace at any other position

use 20 rows by default and define Dataset_typedetect_rows dictionary that stores the exceptions

alyst · 2018-01-11T16:54:26Z

I've updated REQUIRE, removed enclosing modules from tests (now all test.jl files are executed in their own testset) and finally removed [WIP].

randyzwitch · 2018-01-11T17:39:06Z

JuliaLang/METADATA.jl#12836

alyst mentioned this pull request Sep 21, 2017

IOStream API completeness JuliaIO/CodecZlib.jl#7

Closed

alyst force-pushed the update_to_0.6 branch from 0c34701 to fd57db9 Compare November 26, 2017 20:59

alyst mentioned this pull request Nov 29, 2017

Type detection fixes JuliaData/CSV.jl#115

Merged

nalimilan mentioned this pull request Nov 30, 2017

Port dependencies to Missing and CategoricalArray JuliaData/DataFrames.jl#1232

Closed

tlnagy mentioned this pull request Dec 6, 2017

Dataframes v0.11 is out GiovineItalia/Gadfly.jl#1065

Closed

andreasnoack mentioned this pull request Jan 8, 2018

Add deprecation for vcat(Vector{<:AbstractDataFrame}) JuliaData/DataFrames.jl#1342

Merged

randyzwitch mentioned this pull request Jan 9, 2018

Update to latest changes in DataFrames JuliaLang/METADATA.jl#12808

Merged

randyzwitch closed this Jan 10, 2018

randyzwitch reopened this Jan 10, 2018

StoneCypher added a commit to StoneCypher/RDatasets.jl that referenced this pull request Jan 10, 2018

change to only run 0.6 and nightly

5c822f4

for JuliaStats#48 (comment)

StoneCypher mentioned this pull request Jan 10, 2018

change to only run 0.6 and nightly #49

Merged

randyzwitch pushed a commit that referenced this pull request Jan 10, 2018

change to only run 0.6 and nightly (#49)

b99af9d

for #48 (comment)

randyzwitch closed this Jan 10, 2018

randyzwitch reopened this Jan 10, 2018

DilumAluthge mentioned this pull request Jan 10, 2018

Update to newest version of RDatasets.jl bcbi/PredictMD.jl#20

Closed

alyst added 8 commits January 11, 2018 16:04

switch to DataFrames v0.11, CSV, Julia v0.6

37b9873

- decompress csv.gz and feed the stream to CSV using CodecZlib - raise row detection limit for csv files to 15000 as vcd::Bundesliga needs to read upto row #10804 to detect that 7th column can contain missing values

update tests to use @testset

d0a4f4f

add testset layer for all datasets in a package

655a8eb

so that test results for non-problematic packages are collapsed into a single row in the report

cache datasets and packages dataframes

3b171ef

enable precompilation

acdf7f4

remove available_datasets deprecation

9fe9f48

tests: change correct column check

ff239fb

allow digits in the first position to make psych::cubits pass, but disallow whitespace at any other position

minimize rows used to detect column types

18c5a33

use 20 rows by default and define Dataset_typedetect_rows dictionary that stores the exceptions

alyst force-pushed the update_to_0.6 branch from 045a0cf to 18c5a33 Compare January 11, 2018 16:39

alyst changed the title ~~[WIP] Update to Julia v0.6, DataFrames v0.11~~ Update to DataFrames v0.11, CSV 0.2.1, RData 0.3 Jan 11, 2018

randyzwitch merged commit a3e584c into JuliaStats:master Jan 11, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to DataFrames v0.11, CSV 0.2.1, RData 0.3 #48

Update to DataFrames v0.11, CSV 0.2.1, RData 0.3 #48

alyst commented Sep 19, 2017

randyzwitch commented Sep 19, 2017

alyst commented Sep 19, 2017

alyst commented Sep 21, 2017

quinnj commented Sep 21, 2017

alyst commented Sep 21, 2017 •

edited

Loading

alyst commented Nov 27, 2017

tlnagy commented Dec 6, 2017

alyst commented Dec 6, 2017

randyzwitch commented Dec 7, 2017

Nosferican commented Dec 24, 2017

randyzwitch commented Dec 24, 2017

Nosferican commented Dec 24, 2017

alyst commented Dec 24, 2017 •

edited

Loading

quinnj commented Jan 10, 2018

randyzwitch commented Jan 10, 2018

alyst commented Jan 10, 2018

randyzwitch commented Jan 10, 2018

tlnagy commented Jan 11, 2018 •

edited

Loading

alyst commented Jan 11, 2018

randyzwitch commented Jan 11, 2018

Update to DataFrames v0.11, CSV 0.2.1, RData 0.3 #48

Update to DataFrames v0.11, CSV 0.2.1, RData 0.3 #48

Conversation

alyst commented Sep 19, 2017

randyzwitch commented Sep 19, 2017

alyst commented Sep 19, 2017

alyst commented Sep 21, 2017

quinnj commented Sep 21, 2017

alyst commented Sep 21, 2017 • edited Loading

alyst commented Nov 27, 2017

tlnagy commented Dec 6, 2017

alyst commented Dec 6, 2017

randyzwitch commented Dec 7, 2017

Nosferican commented Dec 24, 2017

randyzwitch commented Dec 24, 2017

Nosferican commented Dec 24, 2017

alyst commented Dec 24, 2017 • edited Loading

quinnj commented Jan 10, 2018

randyzwitch commented Jan 10, 2018

alyst commented Jan 10, 2018

randyzwitch commented Jan 10, 2018

tlnagy commented Jan 11, 2018 • edited Loading

alyst commented Jan 11, 2018

randyzwitch commented Jan 11, 2018

alyst commented Sep 21, 2017 •

edited

Loading

alyst commented Dec 24, 2017 •

edited

Loading

tlnagy commented Jan 11, 2018 •

edited

Loading