Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mango using Partitioned parquet ADAM #358

Closed
wants to merge 12 commits into from
Closed

Mango using Partitioned parquet ADAM #358

wants to merge 12 commits into from

Conversation

jpdna
Copy link
Member

@jpdna jpdna commented Jan 30, 2018

This version of Mango works with the "partitioned" ADAM PR:
bigdatagenomics/adam#1878
at commit 23a3bcc as of testing just now.

Example mango run

mango-submit --master yarn --num-executors 10 --executor-cores 8 --executor-memory 10g --driver-memory 20g  -- /home/eecs/akmorrow/builds/hg19.2bit -genes http://www.biodalliance.org/datasets/ensGene.bb -reads hdfs://{headnodepath}/user/jpaschall/mango1/HG00096_Jan30_v2.adam -show_genotypes

The problem with "chr" fix was resolved by querying both with and without "chr" prefix

This was referenced Jan 30, 2018
Closed
Closed
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/563/
Test FAILed.

akmorrow13
akmorrow13 requested changes Feb 2, 2018
View reviewed changes
mango-cli/src/main/scala/org/bdgenomics/mango/cli/VizReads.scala
@@ -297,6 +297,9 @@ class VizReadsArgs extends Args4jBase with ParquetArgs {
@Args4jOption(required = false, name = "-preload", usage = "Chromosomes to prefetch, separated by commas (,).")
var preload: String = null

@Args4jOption(required = false, name = "-parquetIsBinned", usage = "This turns on binned parquet pre-fetch warmup step")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

mango-cli/src/main/scala/org/bdgenomics/mango/cli/VizReads.scala
@@ -667,6 +670,11 @@ class VizReads(protected val args: VizReadsArgs) extends BDGSparkCommand[VizRead
VizReads.genes = Some(args.genePath)
}

// initialize binned parquet by doing a small query to force warm-up
if (args.parquetIsBinned) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to foreach(readsFile -> if isPartitioned(readsFile) then VizReads.materializer.getReads()....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or if one of them is partitioned

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/568/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/569/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/570/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/571/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/572/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/573/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/574/
Test FAILed.

@jpdna jpdna mentioned this pull request Feb 13, 2018
Closed
@akmorrow13 akmorrow13 closed this Mar 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akmorrow13 akmorrow13 akmorrow13 requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jpdna @AmplabJenkins @akmorrow13