Sub-partitioning of Parquet file for ADAM

[jpdna]

The Spark-SQL programming guide describe an optimization of parquet usage that involves splitting parquet file into directories corresponding to different column values. here

This issue is meant as a place for discussion of this topic and to determine if we should prototype such a parquet directory layout, for example dividing the parquet file into individual files per chromosome.

Look forward to any comments and/or links to earlier discussions of this topic

[fnothaft]

Closing as dupe of #651.

[heuermh]

@jpdna Take a look at https://github.com/tomwhite/genomics-analytics/blob/master/adam.md

[tomwhite]

@jpdna @heuermh that's pretty old now - I think using Spark to do the partitioning is the way forward, and Impala supports nested types so flattening is not necessary. See #651 (comment)