SPARK-1293 [SQL] Parquet support for nested types

[AndreSchumacher]

It should be possible to import and export data stored in Parquet's columnar format that contains nested types. For example:

message AddressBook {
   required binary owner;
   optional group ownerPhoneNumbers {
      repeated binary array;
   }
   optional group contacts {
      repeated group array {
         required binary name;
         optional binary phoneNumber;
      }
   }
   optional group nameToApartmentNumber {
      repeated group map {
         required binary key;
         required int32 value;
      }
   }
}

The example could model a type (AddressBook) that contains records made of strings (owner), lists (ownerPhoneNumbers) and a table of contacts (e.g., a list of pairs or a map that can contain null values but keys must not be null). The list of tasks are as follows:

Implement support for converting nested Parquet types to Spark/Catalyst types:

- [x] Structs - [x] Lists - [x] Maps (note: currently keys need to be Strings)

Implement import (via ``parquetFile``) of nested Parquet types (first version in this PR)

- [x] Initial version

Implement export (via ``saveAsParquetFile``)

- [x] Initial version

Test support for AvroParquet, etc.

- [x] Initial testing of import of avro-generated Parquet data (simple + nested)

Example:

val data = TestSQLContext
  .parquetFile("input.dir")
  .toSchemaRDD
data.registerAsTable("data")
sql("SELECT owner, contacts[1].name, nameToApartmentNumber['John'] FROM data").collect()

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AndreSchumacher]

@marmbrus I had to do some changes to both the attribute resolution and the SQLParser. Would be great if you could have a look. I think it would be actually much better to parse nested fields together with their datatypes, as it's now done for everything else that is not a simple nested struct.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13893/

[marmbrus]

Hey @AndreSchumacher, Thanks for working on this. I don't think we are going to be able to include it for 1.0, but it will be an awesome feature in 1.1. I will take a detailed look at this as soon as we get all the critical bug fixes in for 1.0.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14056/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14094/

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14095/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14098/

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14099/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[rxin]

@AndreSchumacher do u mind removing the [WIP] tag from the pull request?

Unfortunately due to the avro version bump, we can't include this in 1.0.1.

[pwendell]

@AndreSchumacher do you mind updating the maven build as well?

[AndreSchumacher]

@rxin the avro dependency is for the tests only (to make sure we can read parquet files with avro objects in them). I can remove the one test if that blocks it from being included. When the rest of the build has caught up with the version we can add it again. What do you think?

[rxin]

That sounds good. If you can just comment that test out for now, that'd be great.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AndreSchumacher]

@pwendell good point, forgot about the pom. But now that the avro dependency is removed no changes are neccesary.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15939/

[rxin]

Ok I'm going to merge this in master & branch-1.0 now. Kinda scary but the change is very isolated.

[marmbrus]

Yay for finally merging this large change in! Thanks @AndreSchumacher!

BTW, I've been testing this PR on some pretty large / complex schemas for a while so hopefully not too scary merging it at the last minute.

[AndreSchumacher]

Thanks @rxin for merging! Great to see it finally in master. I understand that it's quite a bit larger than the average PR. Hopefully there won't be too many issues though. Thanks to @marmbrus and @aarondav for reviewing!