[ADAM-962] Fix corrupt single-file BAM output.

[fnothaft]

It seems like we were doing something incorrectly when writing the header. Additionally, we now write a correct end-of-file. Resolves #962.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1098/

Build result: FAILURE

GitHub pull request #964 of commit f27ffe6 automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/964/merge^{commit} # timeout=10 > git branch -a --contains 7031d8874959dffd46097e0aaa7c72d1db5aded6 # timeout=10 > git rev-parse remotes/origin/pr/964/merge^{commit} # timeout=10Checking out Revision 7031d8874959dffd46097e0aaa7c72d1db5aded6 (origin/pr/964/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f 7031d8874959dffd46097e0aaa7c72d1db5aded6First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1099/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1100/

Build result: FAILURE

GitHub pull request #964 of commit 3687d7e automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/964/merge^{commit} # timeout=10 > git branch -a --contains b40c1ab # timeout=10 > git rev-parse remotes/origin/pr/964/merge^{commit} # timeout=10Checking out Revision b40c1ab (origin/pr/964/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f b40c1ab97a031932bc12a9fe1ec10f356324bfeeFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[fnothaft]

Jenkins, retest this please.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1101/

Build result: FAILURE

GitHub pull request #964 of commit 3687d7e automatically merged.Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos spark-test) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/964/merge^{commit} # timeout=10 > git branch -a --contains b40c1ab # timeout=10 > git rev-parse remotes/origin/pr/964/merge^{commit} # timeout=10Checking out Revision b40c1ab (origin/pr/964/merge) > git config core.sparsecheckout # timeout=10 > git checkout -f b40c1ab > git rev-list b40c1ab # timeout=10Triggering ADAM-prb ? 2.6.0,2.11,1.5.2,centosTriggering ADAM-prb ? 2.6.0,2.10,1.5.2,centosTouchstone configurations resulted in FAILURE, so aborting...Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

[heuermh]

This worked for me in at least one instance.

Is this just waiting on fixing unit test failures? I would guess there is simply a problem with AlignmentRecordRDDFunctionsSuite.checkFiles not handling binary file encoding correctly. I can look into that.

[fnothaft]

Is this just waiting on fixing unit test failures? I would guess there is simply a problem with AlignmentRecordRDDFunctionsSuite.checkFiles not handling binary file encoding correctly. I can look into that.

IIRC, this was waiting on HadoopGenomics/Hadoop-BAM#80, but honestly, I'm a bit foggy.

[heuermh]

In a branch with this PR applied and Hadoop-BAM at version 7.4.1-SNAPSHOT (including the PR linked above), I still see similar test failures.

[fnothaft]

Ah, OK, that confirms that I am remembering wrong... If you wouldn't mind looking into checkFiles, I'd be appreciative.

[heuermh]

Could you rebase and push this? I'd like to confirm that there are still issues with Hadoop-BAM version 7.5.0 before changing checkFiles.

[fnothaft]

This is good to go!

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1338/
Test PASSed.

[heuermh]

Some thought to ordering between this and #1060 may be necessary, since the latter bumps Hadoop-BAM to 7.6.0. I propose merging this first and then dealing with any 7.6.0-related changes in behaviour separately.

[fnothaft]

I don't think Hadoop-BAM 7.6.0 will impact this patch, but I am OK with merging this before #1060.

[fnothaft]

Rebased!

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1350/
Test PASSed.

[heuermh]

Thank you, @fnothaft!