Code

Switch parquet library#19

Code

Merged

jshiv merged 10 commits intorivian:mainrivian/delta-go:mainfrom chelseajonesr:switch-parquet-librarychelseajonesr/delta-go:switch-parquet-libraryCopy head branch name to clipboardJul 11, 2023

+1,080-440

Commits

Commits on Jun 26, 2023

• Start switching parquet library

  chelseajonesr

  committed
  5589bb4

  Copy full SHA for 5589bb4

• Remove delta data type aliases

  chelseajonesr

  committed
  6dc0e16

  Copy full SHA for 6dc0e16

• Remove _parsed fields, DeltaDataTypeDate, drop writer version to 1

  chelseajonesr

  committed
  78928d4

  Copy full SHA for 78928d4

Commits on Jun 30, 2023

• Manually parse actions since the schema isn't fixed

  chelseajonesr

  committed
  fce73de

  Copy full SHA for fce73de

Commits on Jul 5, 2023

• Restore DeltaDataType type definitions. Convert all action struct members to pointers so that parquet schema is consistent with Spark/Rust

  chelseajonesr

  committed
  49a15f0

  Copy full SHA for 49a15f0

• Fix up map and list reading to work with either pointer or non-pointer entries

  chelseajonesr

  committed
  2c88b3f

  Copy full SHA for 2c88b3f

• go mod tidy

  chelseajonesr

  committed
  f9118dc

  Copy full SHA for f9118dc

• Use consistent version type

  chelseajonesr

  committed
  ec88cac

  Copy full SHA for ec88cac

• Return error if add size is zero or missing when creating a checkpoint

  chelseajonesr

  committed
  6522c19

  Copy full SHA for 6522c19

Commits on Jul 7, 2023

• Make fields that are required in the protocol also required in Parquet. Remove timestamp logical type to match Spark schema. Remove Delta data type definitions.

  chelseajonesr

  committed
  0bf30fc

  Copy full SHA for 0bf30fc