[ADAM-651] Implement Hive-style partitioning by genomic range of Parquet backed datasets #1948

fnothaft · 2018-03-07T23:43:36Z

Fixes #651.

Continuation of #1922.

AmplabJenkins · 2018-03-08T00:28:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2714/
Test PASSed.

heuermh

Minor typos/nits. Note the Github UI doesn't seem to be showing all the comments here correctly due to the new Load more... function. They all look to be there in the Files changed tab.

heuermh · 2018-03-08T04:16:08Z

adam-cli/src/main/scala/org/bdgenomics/adam/cli/TransformAlignments.scala

@@ -131,6 +131,10 @@ class TransformAlignmentsArgs extends Args4jBase with ADAMSaveAnyArgs with Parqu
  var storageLevel: String = "MEMORY_ONLY"
  @Args4jOption(required = false, name = "-disable_pg", usage = "Disable writing a new @PG line.")
  var disableProcessingStep = false
+  @Args4jOption(required = false, name = "-partition_by_start_pos", usage = "Save the data partitioned by genomic range bins based on start pos using Hive-style partitioning.")
+  var partitionByStartPos: Boolean = false
+  @Args4jOption(required = false, name = "-partition_bin_size", usage = "Partition bin sized used in Hive-style partitioning.")


add Defaults to 1Mbp (1,000,000 base pairs). to usage

heuermh · 2018-03-08T04:16:24Z

adam-cli/src/main/scala/org/bdgenomics/adam/cli/TransformGenotypes.scala

+  @Args4jOption(required = false, name = "-partition_by_start_pos", usage = "Save the data partitioned by genomic range bins based on start pos using Hive-style partitioning.")
+  var partitionByStartPos: Boolean = false
+
+  @Args4jOption(required = false, name = "-partition_bin_size", usage = "Partition bin sized used in Hive-style partitioning.")


add Defaults to 1Mbp (1,000,000 base pairs). to usage

heuermh · 2018-03-08T04:18:53Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

@@ -3052,4 +3192,44 @@ class ADAMContext(@transient val sc: SparkContext) extends Serializable with Log
      loadParquetFragments(pathName, optPredicate = optPredicate, optProjection = optProjection)
    }
  }
+
+  /**
+   * Return integer size of partitions if the specified path of Parquet + Avro files is partitioned.


Didn't you have a better suggestion for this doc previously? Return length of partitions in base pairs... perhaps?

heuermh · 2018-03-08T04:20:10Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+   *
+   * @param pathName Path in which to look for partitioned flag.
+   * @return Return true if the specified path of Parquet + Avro files is partitioned.
+   * Behavior is undefined if some paths in glob are contain paritioned flag and some do not.


paritioned → partitioned

heuermh · 2018-03-08T04:22:13Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala

+  val optPartitionBinSize: Option[Int]
+  val optLookbackPartitions: Option[Int]
+
+  private def referenceRegionsToDatasetQueryString(regions: Iterable[ReferenceRegion], partitionSize: Int = 1000000, partitionedLookBackNum: Int = 1): String = {


shouldn't this method use the vals optPartitionBinSize and optLookbackPartitions above?

eeek, yes you are correct.

Well actually, it was working correctly before because the optPartitionedBinSize was what was passed as a parameter when called fro filterByoverlappingRegions but given that both functions are in the same trait, no no need for the parameter.

heuermh · 2018-03-08T04:37:03Z

docs/api/genomicRdd.rst

+The existence of the file ``_partitionedByStartPos`` can be tested with the public
+function ``ADAMContext.isPartitioned(path)`` and can be used to determine explicitly
+if an ADAM Parquet dataset is partitioned using this scheme. The parition size used
+when the dataset was written to disk is stored in ``_paritionedByStartPos`` and is


paritionedByStartPos → partitionedByStartPos

heuermh · 2018-03-08T04:37:34Z

docs/api/genomicRdd.rst

+
+The Spark dataset API recognizes that the field ``positionBin`` is defined implicitly
+by the Parquet files' paritioning scheme, and makes ``positionBin`` available as a field
+that can be queried through the Spark SQL API. ``positionBin`` is used interally by


interally → internally

heuermh · 2018-03-08T04:37:48Z

docs/api/genomicRdd.rst

+The Spark dataset API recognizes that the field ``positionBin`` is defined implicitly
+by the Parquet files' paritioning scheme, and makes ``positionBin`` available as a field
+that can be queried through the Spark SQL API. ``positionBin`` is used interally by
+the public function ``GenomicRDD.filterByOverlappinRegions``. User code in ADAM-shell or


filterByOverlappinRegions → filterByOverlappingRegions

heuermh · 2018-03-08T04:38:32Z

docs/api/genomicRdd.rst

+by the Parquet files' paritioning scheme, and makes ``positionBin`` available as a field
+that can be queried through the Spark SQL API. ``positionBin`` is used interally by
+the public function ``GenomicRDD.filterByOverlappinRegions``. User code in ADAM-shell or
+a user applcations could similarly utilize the ``positionBin`` field when creating SPARK


a user applcations → user applications
SPARK → Spark

heuermh · 2018-03-08T04:39:23Z

docs/api/genomicRdd.rst

+    val mydata = loadPartitionedParquetAlignments("alignmets.adam")
+    val filteredCount1 = mydata.filterByOverlappingRegions(regions1).dataset.count
+    val filteredCount2 = mydata.filterByOverlappingRegions(regions2).dataset.count
+


remove blank lines

coveralls · 2018-03-08T13:44:38Z

Coverage decreased (-0.07%) to 82.465% when pulling f745bca on jpdna-hive_partitioned_v11 into 9ea870c on master.

AmplabJenkins · 2018-03-08T14:13:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2716/

Build result: ABORTED

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f1d6be1 # timeout=10Checking out Revision f1d6be1 (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f1d6be122ce302eda8ea2c4d88fc1644e1377ef5First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result ABORTEDADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2018-03-08T14:46:59Z

Jenkins, retest this please.

AmplabJenkins · 2018-03-08T15:27:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2717/
Test PASSed.

heuermh · 2018-03-08T21:41:52Z

docs/api/genomicRdd.rst

+The existence of the file ``_partitionedByStartPos`` can be tested with the public
+function ``ADAMContext.isPartitioned(path)`` and can be used to determine explicitly
+if an ADAM Parquet dataset is partitioned using this scheme. The parition size used
+when the dataset was written to disk is stored in ``_parititionedByStartPos`` and is


parititionedByStartPos → partitionedByStartPos

fixed -thanks, I honestly need to increase my font size so I can see typos.

AmplabJenkins · 2018-03-09T17:26:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2718/
Test PASSed.

AmplabJenkins · 2018-03-09T18:04:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2719/
Test PASSed.

AmplabJenkins · 2018-03-09T19:16:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2720/
Test PASSed.

AmplabJenkins · 2018-03-09T19:54:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2721/
Test PASSed.

AmplabJenkins · 2018-03-09T20:33:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2722/
Test PASSed.

AmplabJenkins · 2018-03-09T21:06:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2723/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains b589bfc # timeout=10Checking out Revision b589bfc (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f b589bfc > /home/jenkins/git2/bin/git rev-list 0367f44 # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

jpdna · 2018-03-09T21:07:06Z

I'm getting an error when I'm testing this downstream in Mango, I'll be looking into it later this evening.

jpdna · 2018-03-10T03:47:07Z

Resolved problem I had with using in Mango.

This is ready for another review pass.

AmplabJenkins · 2018-03-10T04:06:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2724/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 14d75a9 # timeout=10Checking out Revision 14d75a9 (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 14d75a9 > /home/jenkins/git2/bin/git rev-list b589bfc # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

jpdna · 2018-03-14T15:52:11Z

Ping for review here.

akmorrow13 · 2018-03-14T16:02:40Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+    require(partitionSizes.nonEmpty, "Input Parquet files (%s) are not partitioned.".format(pathName))
+    require(partitionSizes.size == 1, "Found multiple partition sizes (%s).".format(partitionSizes.mkString(", ")))
+    partitionSizes.head


If partitionSizes is empty this will throw java.util.NoSuchElementException: next on empty iterator

But on like 3219 doesn't the require check for partitionSizes being empty and produce the intended message when the require condition is not met?

But on like 3219 doesn't the require check for partitionSizes being empty and produce the intended message when the require condition is not met?

Correct, you're fine here.

heuermh · 2018-03-14T19:51:37Z

Jenkins, retest this please.

AmplabJenkins · 2018-03-14T20:11:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2725/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 14d75a9 # timeout=10Checking out Revision 14d75a9 (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 14d75a9 > /home/jenkins/git2/bin/git rev-list 14d75a9 # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

heuermh · 2018-03-14T20:12:24Z

Looks like there is a code style issue in GenomicRDD.scala

++ git status --porcelain
+ test -n ' M adam-core/src/main/scala/org/bdgenomics/adam/rdd/GenomicRDD.scala'

jpdna · 2018-03-14T20:25:05Z

@heuermh - you online? I'm not sure how to get details about and how to resolve this code style issue.

heuermh · 2018-03-14T20:43:57Z

If you run the maven build from the command line, git will tell you if a file changed due to the formatter that is run as part of the build

fnothaft · 2018-03-14T20:52:11Z

If you run the maven build from the command line, git will tell you if a file changed due to the formatter that is run as part of the build

If this doesn't show any changes, run ./scripts/format-sources.

partitionSizes is checked for nonEmpty before head is called.

fnothaft · 2018-03-14T21:00:29Z

@jpdna don't worry about it; I'll clean the source formatting issue.

Resolves #651.

AmplabJenkins · 2018-03-14T21:21:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2726/

Build result: FAILURE

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f461abf # timeout=10Checking out Revision f461abf (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f461abf > /home/jenkins/git2/bin/git rev-list 14d75a9 # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2018-03-14T21:45:49Z

Jenkins, test this please.

AmplabJenkins · 2018-03-14T22:33:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2727/

Build result: ABORTED

[...truncated 7 lines...] > /home/jenkins/git2/bin/git init /home/jenkins/workspace/ADAM-prb # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/heads/:refs/remotes/origin/ # timeout=15 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10 > /home/jenkins/git2/bin/git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10 > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1948/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains f461abf # timeout=10Checking out Revision f461abf (origin/pr/1948/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f f461abf > /home/jenkins/git2/bin/git rev-list f461abf # timeout=10Triggering ADAM-prb ? 2.6.2,2.10,2.2.1,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.1,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.1,centosADAM-prb ? 2.6.2,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.6.2,2.11,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.10,2.2.1,centos completed with result SUCCESSADAM-prb ? 2.7.3,2.11,2.2.1,centos completed with result ABORTEDNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

fnothaft · 2018-03-14T22:40:55Z

Jenkins, test this please.

AmplabJenkins · 2018-03-14T23:24:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2728/
Test PASSed.

heuermh · 2018-03-14T23:30:35Z

Thank you, @jpdna @fnothaft!

fnothaft added this to the 0.24.0 milestone Mar 7, 2018

fnothaft requested a review from heuermh March 7, 2018 23:43

heuermh requested changes Mar 8, 2018

View reviewed changes

jpdna mentioned this pull request Mar 10, 2018

Use Hive-style partitioning bigdatagenomics/mango#370

Closed

akmorrow13 previously requested changes Mar 14, 2018

View reviewed changes

heuermh mentioned this pull request Mar 14, 2018

[ADAM-651] Implement Hive-style partitioning by genomic range of Parquet backed datasets #1922

Closed

heuermh changed the title ~~Implement Hive-style partitioning by genomic range of Parquet backed datasets~~ [ADAM-651] Implement Hive-style partitioning by genomic range of Parquet backed datasets Mar 14, 2018

heuermh approved these changes Mar 14, 2018

View reviewed changes

[ADAM-651] Add support for Hive-style partitioning.

452d760

Resolves #651.

fnothaft force-pushed the jpdna-hive_partitioned_v11 branch from f745bca to 452d760 Compare March 14, 2018 21:01

heuermh merged commit c54f1a1 into master Mar 14, 2018

fnothaft deleted the jpdna-hive_partitioned_v11 branch March 14, 2018 23:33

[ADAM-651] Implement Hive-style partitioning by genomic range of Parquet backed datasets #1948

[ADAM-651] Implement Hive-style partitioning by genomic range of Parquet backed datasets #1948

Conversation

fnothaft commented Mar 7, 2018 • edited by heuermh Loading

AmplabJenkins commented Mar 8, 2018

heuermh left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Mar 8, 2018 • edited Loading

AmplabJenkins commented Mar 8, 2018

Build result: ABORTED

fnothaft commented Mar 8, 2018

AmplabJenkins commented Mar 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Mar 9, 2018

AmplabJenkins commented Mar 9, 2018

AmplabJenkins commented Mar 9, 2018

AmplabJenkins commented Mar 9, 2018

AmplabJenkins commented Mar 9, 2018

AmplabJenkins commented Mar 9, 2018

Build result: FAILURE

jpdna commented Mar 9, 2018

jpdna commented Mar 10, 2018

AmplabJenkins commented Mar 10, 2018

Build result: FAILURE

jpdna commented Mar 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heuermh commented Mar 14, 2018

AmplabJenkins commented Mar 14, 2018

Build result: FAILURE

heuermh commented Mar 14, 2018

jpdna commented Mar 14, 2018

heuermh commented Mar 14, 2018

fnothaft commented Mar 14, 2018

fnothaft commented Mar 14, 2018

AmplabJenkins commented Mar 14, 2018

Build result: FAILURE

fnothaft commented Mar 14, 2018

AmplabJenkins commented Mar 14, 2018

Build result: ABORTED

fnothaft commented Mar 14, 2018

AmplabJenkins commented Mar 14, 2018

heuermh commented Mar 14, 2018

fnothaft commented Mar 7, 2018 •

edited by heuermh

Loading

heuermh left a comment •

edited

Loading

coveralls commented Mar 8, 2018 •

edited

Loading