refactored ReferenceFile to require SequenceDictionary #1086

akmorrow13 · 2016-07-19T15:34:58Z

NucleotideContigFragmentRDD now extends ReferenceFile
ReferenceFile requires Sequence Dictionary

AmplabJenkins · 2016-07-19T15:56:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1354/

Build result: FAILURE

GitHub pull request #1086 of commit 2f065ff automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > /home/jenkins/git2/bin/git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > /home/jenkins/git2/bin/git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > /home/jenkins/git2/bin/git --version # timeout=10 > /home/jenkins/git2/bin/git -c core.askpass=true fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1086/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 53fefde # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1086/merge^{commit} # timeout=10Checking out Revision 53fefde (origin/pr/1086/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 53fefde4c3a8171af3150fe52313af2f29a38f52First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

akmorrow13 · 2016-07-19T18:25:13Z

This is erroring out in 1.5.2 for files I did not modify.
For example: [ERROR] Error formatting /home/jenkins/workspace/ADAM-prb/HADOOP_VERSION/2.6.0/SCALAVER/2.10/SPARK_VERSION/1.5.2/label/centos/adam-cli/src/main/scala/org/bdgenomics/adam/cli/PrintADAM.scala: java.nio.charset.MalformedInputException: Input length = 1

fnothaft · 2016-07-19T18:28:06Z

That error is fine. Keep reading down the log ;):

Please run './scripts/format-source'

heuermh · 2016-07-19T20:13:29Z

Maven builds from the command line do that for you. Give up your IDE! 😄

heuermh · 2016-07-19T20:21:25Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/contig/NucleotideContigFragmentRDD.scala

@@ -169,7 +170,7 @@ case class NucleotideContigFragmentRDD(
   * @param region Reference region over which to get sequence.
   * @return String of bases corresponding to reference sequence.
   */
-  def getReferenceString(region: ReferenceRegion): String = {
+  def extract(region: ReferenceRegion): String = {


Wouldn't this read so much better if this class was a SequenceRDD (or SliceRDD, depending on how the data were loaded) and this method was

def slice(region: ReferenceRegion): Slice = { ... }

See bigdatagenomics/bdg-formats#83

So you would like ReferenceFile to change its interface to be slice instead of extract?

I think it's fine for now, with an eye to refactor if/when the Slice stuff is added, between versions 0.20 and 0.21 presumably

I can definitely change it. I just am wondering if slice is generic enough for the 2bitFile, because this extends ReferenceFile as well.

extract is fine with me

AmplabJenkins · 2016-07-22T00:13:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1360/
Test PASSed.

akmorrow13 · 2016-07-29T18:48:53Z

+1?

heuermh · 2016-07-29T21:18:12Z

adam-core/src/main/scala/org/bdgenomics/adam/util/ReferenceContigMap.scala

 import org.bdgenomics.formats.avro.NucleotideContigFragment

 case class ReferenceContigMap(contigMap: Map[String, Seq[NucleotideContigFragment]]) extends ReferenceFile {
+
+  /**
+   * Sequence dictionary extracted from contigMap


doc strings should be complete sentences

heuermh · 2016-07-29T21:22:37Z

Added some minor doc-related review comments.

AmplabJenkins · 2016-07-29T22:01:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1363/

Build result: FAILURE

[...truncated 24 lines...]Triggering ADAM-prb ? 2.3.0,2.10,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.10,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.3.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.3.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.4.1,centosTriggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.3.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.5.2,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.4.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2016-07-30T19:54:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1364/
Test PASSed.

AmplabJenkins · 2016-07-30T20:44:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1365/
Test PASSed.

akmorrow13 · 2016-07-30T21:05:02Z

I fixed all the comments @heuermh

akmorrow13 · 2016-08-02T17:48:54Z

bump @heuermh or @fnothaft

heuermh · 2016-08-02T19:37:35Z

LGTM

jpdna · 2016-08-04T18:38:12Z

+1

fnothaft · 2016-08-04T18:44:07Z

So NucleotideContigFragmentRDD is not going to be usable as a ReferenceFile from inside an RDD. Is this OK? Because of such, I'm -0.9 on this PR, unless I misunderstand the use case for this PR.

AmplabJenkins · 2016-08-04T23:30:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1369/
Test PASSed.

fnothaft · 2016-08-05T03:52:53Z

Thanks for updating this PR with the documentation fixes we discussed, @akmorrow13. I'm retracting my -0.9 and have just merged this as e7e1adf. Thank you!

heuermh reviewed Jul 19, 2016
View reviewed changes

akmorrow13 force-pushed the master branch from 2f065ff to f1f5fea Compare July 21, 2016 23:44

heuermh reviewed Jul 29, 2016
View reviewed changes

akmorrow13 force-pushed the master branch from f1f5fea to b577896 Compare July 29, 2016 21:34

akmorrow13 force-pushed the master branch from b577896 to 038595a Compare July 30, 2016 19:32

akmorrow13 force-pushed the master branch from 038595a to 0520525 Compare July 30, 2016 20:18

refactored ReferenceFile to require SequenceDictionary

d5a67dd

akmorrow13 force-pushed the master branch from 0520525 to d5a67dd Compare August 4, 2016 22:51

fnothaft closed this Aug 5, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactored ReferenceFile to require SequenceDictionary #1086

refactored ReferenceFile to require SequenceDictionary #1086

akmorrow13 commented Jul 19, 2016

AmplabJenkins commented Jul 19, 2016

akmorrow13 commented Jul 19, 2016

fnothaft commented Jul 19, 2016

heuermh commented Jul 19, 2016

heuermh Jul 19, 2016

akmorrow13 Jul 27, 2016

heuermh Jul 27, 2016

akmorrow13 Jul 27, 2016

heuermh Jul 27, 2016

AmplabJenkins commented Jul 22, 2016

akmorrow13 commented Jul 29, 2016

heuermh Jul 29, 2016

heuermh commented Jul 29, 2016

AmplabJenkins commented Jul 29, 2016

AmplabJenkins commented Jul 30, 2016

AmplabJenkins commented Jul 30, 2016

akmorrow13 commented Jul 30, 2016

akmorrow13 commented Aug 2, 2016

heuermh commented Aug 2, 2016

jpdna commented Aug 4, 2016

fnothaft commented Aug 4, 2016

AmplabJenkins commented Aug 4, 2016

fnothaft commented Aug 5, 2016

refactored ReferenceFile to require SequenceDictionary #1086

refactored ReferenceFile to require SequenceDictionary #1086

Conversation

akmorrow13 commented Jul 19, 2016

AmplabJenkins commented Jul 19, 2016

Build result: FAILURE

akmorrow13 commented Jul 19, 2016

fnothaft commented Jul 19, 2016

heuermh commented Jul 19, 2016

heuermh Jul 19, 2016

Choose a reason for hiding this comment

akmorrow13 Jul 27, 2016

Choose a reason for hiding this comment

heuermh Jul 27, 2016

Choose a reason for hiding this comment

akmorrow13 Jul 27, 2016

Choose a reason for hiding this comment

heuermh Jul 27, 2016

Choose a reason for hiding this comment

AmplabJenkins commented Jul 22, 2016

akmorrow13 commented Jul 29, 2016

heuermh Jul 29, 2016

Choose a reason for hiding this comment

heuermh commented Jul 29, 2016

AmplabJenkins commented Jul 29, 2016

Build result: FAILURE

AmplabJenkins commented Jul 30, 2016

AmplabJenkins commented Jul 30, 2016

akmorrow13 commented Jul 30, 2016

akmorrow13 commented Aug 2, 2016

heuermh commented Aug 2, 2016

jpdna commented Aug 4, 2016

fnothaft commented Aug 4, 2016

AmplabJenkins commented Aug 4, 2016

fnothaft commented Aug 5, 2016