Planned Data Release: V16 #601

jharenza · 2020-03-04T20:08:03Z

What data file(s) does this issue pertain to?

pbta-histologies.tsv
pbta-tcga-manifest.tsv
pbta-tcga-snv-lancet.vep.maf.gz
pbta-tcga-snv-mutect2.vep.maf.gz
pbta-tcga-snv-strelka2.vep.maf.gz

What release are you using?

V15

Put your question or report your issue here.

Updated TCGA MAFs per TCGA BED file used for variant calling #568
Update TCGA manifest file to include BED file used per sample
Add new TCGA BED files, padded and unpadded
Remove old TCGA BED file gencode.v19.basic.exome.hg38liftover.100bp_padded.bed
Update histologies file to include additional information per tickets Add information about diagnosis changes to histologies file #607, Add cell line composition type to pbta-histologies.tsv #610, Updates in v16 for pbta-histologies.tsv #637 ~~potentially Updated analysis: molecular subtyping for cell lines #509 and Updated analysis: chordoma subtyping #608 Proposed Analysis: Molecularly subtype ependymoma tumors #245~~ (@jaclyn-taroni edit per Planned Data Release: V16 #601 (comment))

Placeholder for V16 release to include D3b team: @baileyckelly @chris-s-friedman @yuankunzhu @allisonheath

The text was updated successfully, but these errors were encountered:

jaclyn-taroni · 2020-03-06T16:28:15Z

Because we made changes to the pbta-fusion-putative-oncogenic.tsv with v15, we've had to update the fusion-summary files that are included in a release (fusion_summary_embryonal_foi.tsv and fusion_summary_ependymoma_foi.tsv): https://github.com/AlexsLemonade/OpenPBTA-analysis/tree/516201f91f174c495f6d7964f4a03d6c7c6fb227/analyses/fusion-summary/results

allisonheath · 2020-03-11T13:11:32Z

For the v16 release, @baileyckelly and @chris-s-friedman are moving the histology file generation over to a new database workflow to make the manual changes more trackable. We are working on replicating v15 first, expected to be done by the end of this week and then will start working on the new issues noted in the ticket early next. Will update if this changes.

allisonheath · 2020-03-11T13:19:06Z

We'll also do #624.

jaclyn-taroni · 2020-03-19T23:18:34Z

@tkoganti gave me a heads up that the updated TCGA files are available pre-v16 if folks would like to take a look.

Here's the link to the s3 bucket and folder https://s3.amazonaws.com/kf-openaccess-us-east-1-prd-pbta/data/TCGA_mar-12-2020/

And the file list:

README.md
intersected_whole_exome_agilent_designed_120_AND_tcga_6k_genes.Gh38.bed
intersected_whole_exome_agilent_plus_tcga_6k_AND_tcga_6k_genes.Gh38.bed
pbta-tcga-manifest_all319.txt
pbta-tcga-snv-lancet.maf.gz
pbta-tcga-snv-mutect2.maf.gz
pbta-tcga-snv-strelka2.maf.gz
tcga_6k_genes.targetIntervals.bed
whole_exome_agilent_1.1_refseq_plus_3_boosters.targetIntervals.Gh38.bed
whole_exome_agilent_1.1_refseq_plus_3_boosters.targetIntervals.bed
whole_exome_agilent_designed_120.targetIntervals.Gh38.bed
whole_exome_agilent_designed_120.targetIntervals.bed
whole_exome_agilent_plus_tcga_6k.targetIntervals.Gh38.bed

Where the README includes an explanation of what each of these files are.

yuankunzhu · 2020-03-20T16:42:30Z

[note-to-self]

1. sync data from previous release

last_release='release-v15-20200228'
new_release='release-v16-20200320'
bucket='s3://kf-openaccess-us-east-1-prd-pbta/data'
aws s3 sync $bucket/$last_release/ $bucket/$new_release/

2. new data

2.1 TCGA MAF

get overlapped file names between v15 and TCGA_mar-12-2020/

## get all files from v15 and TCGA_mar-12-2020/
all_files=`aws s3 ls --recursive $bucket | awk '{print $4}' | egrep "$new_release|TCGA_mar-12-2020/"`
echo $all_files | xargs -i basename {} | sort | uniq -dc

it turns 0 output, no overlaps? check all TCGA MAF files:

echo $all_files | grep -i tcga | grep -i maf

returns below

data/TCGA_mar-12-2020/pbta-tcga-snv-lancet.maf.gz
data/TCGA_mar-12-2020/pbta-tcga-snv-mutect2.maf.gz
data/TCGA_mar-12-2020/pbta-tcga-snv-strelka2.maf.gz
data/release-v16-20200320/pbta-tcga-snv-lancet.vep.maf.gz
data/release-v16-20200320/pbta-tcga-snv-mutect2.vep.maf.gz
data/release-v16-20200320/pbta-tcga-snv-strelka2.vep.maf.gz

MAF files named differently in TCGA_mar-12-2020/ folder and release folder, rename/overwrite TCGA_mar-12-2020/ MAF with vep extension to keep consistence

for caller in 'lancet' 'mutect2' 'strelka2'
do
    aws s3 cp $bucket/TCGA_mar-12-2020/pbta-tcga-snv-$caller.maf.gz $bucket/$new_release/pbta-tcga-snv-$caller.vep.maf.gz
done

2.2 TCGA BED

echo $all_files | grep -i bed | xargs -i basename {} | sort | uniq -dc
## no overlaps, copy all bed to release folder
aws s3 sync $bucket/TCGA_mar-12-2020/ $bucket/$new_release --exclude "*" --include "*.bed"

## remove old tcga BED
aws s3 rm $bucket/$new_release/gencode.v19.basic.exome.hg38liftover.100bp_padded.bed

2.3 TCGA Manifest check

$ curl -s https://s3.amazonaws.com/kf-openaccess-us-east-1-prd-pbta/data/TCGA_mar-12-2020/pbta-tcga-manifest_all319.txt | head
Normal_BAM	Tumor_BAM	tumorID	Primary_diagnosis
C494.TCGA-S9-A6U5-10A-01D-A33W-08.1_gdc_realn.bam	C494.TCGA-S9-A6U5-01A-12D-A33T-08.1_gdc_realn.bam	TCGA-S9-A6U5	Astrocytoma-NOS
C494.TCGA-TM-A7C4-10A-01D-A329-08.1_gdc_realn.bam	C494.TCGA-TM-A7C4-01A-11D-A32B-08.1_gdc_realn.bam	TCGA-TM-A7C4	Astrocytoma-NOS
C494.TCGA-HW-7490-10A-01D-2024-08.1_gdc_realn.bam	C494.TCGA-HW-7490-01A-11D-2024-08.1_gdc_realn.bam	TCGA-HW-7490	Astrocytoma-NOS
C494.TCGA-S9-A7R3-10A-01D-A34M-08.3_gdc_realn.bam	C494.TCGA-S9-A7R3-01A-11D-A34J-08.3_gdc_realn.bam	TCGA-S9-A7R3	Astrocytoma-NOS
C494.TCGA-HT-7680-10A-01D-2253-08.1_gdc_realn.bam	C494.TCGA-HT-7680-01A-11D-2253-08.1_gdc_realn.bam	TCGA-HT-7680	Astrocytoma-NOS
C494.TCGA-CS-4944-10A-01D-1468-08.3_gdc_realn.bam	C494.TCGA-CS-4944-01A-01D-1468-08.3_gdc_realn.bam	TCGA-CS-4944	Astrocytoma-NOS
C494.TCGA-P5-A5EW-10A-01D-A27N-08.4_gdc_realn.bam	C494.TCGA-P5-A5EW-01A-11D-A27K-08.4_gdc_realn.bam	TCGA-P5-A5EW	Astrocytoma-NOS
C494.TCGA-CS-6667-10A-01D-2024-08.1_gdc_realn.bam	C494.TCGA-CS-6667-01A-12D-2024-08.1_gdc_realn.bam	TCGA-CS-6667	Astrocytoma-NOS
C494.TCGA-S9-A7R7-10A-01D-A34M-08.3_gdc_realn.bam	C494.TCGA-S9-A7R7-01A-11D-A34J-08.3_gdc_realn.bam	TCGA-S9-A7R7	Astrocytoma-NOS

looks like it's missing BED information, add it up, modified get-tcga-capture_kit.py to get all 319 samples' capture kit at as new-tcga-capture-kit-info.tsv.

add that as Capture_Kit column for the manifest:
pbta-tcga-manifest.txt

upload this to release folder and overwrite old pbta-tcga-manifest.tsv

aws s3 cp pbta-tcga-manifest.txt $bucket/$new_release/pbta-tcga-manifest.tsv

~~### 2.4 histologies file~~
move histologies file update to V17 #656

new fusion results

$fusions_path='analyses/fusion_filtering/results'
aws s3 cp $fusions_path/pbta-fusion-putative-oncogenic.tsv $bucket/$new_release/
aws s3 cp $fusions_path/pbta-fusion-recurrently-fused-genes-byhistology.tsv $bucket/$new_release/
aws s3 cp $fusions_path/pbta-fusion-recurrently-fused-genes-bysample.tsv $bucket/$new_release/


## 3. update release doc
- [x] md5sum
- [x] release note
- [ ] download script

yuankunzhu · 2020-03-20T16:58:24Z

@tkoganti for the new TCGA BED files you put on TCGA_mar-12-2020/
I didn't see the Gh38 version for tcga_6k_genes.targetIntervals.bed and the original/un-liftover'd for whole_exome_agilent_plus_tcga_6k.targetIntervals.Gh38.bed?

yuankunzhu · 2020-03-20T17:15:52Z

~~@tkoganti I think we need to also add the BED information to the manifest.~~
Seems like the TCGA manifest is missing the capture kit information, I will add it for this release

tkoganti · 2020-03-20T17:25:03Z

@tkoganti for the new TCGA BED files you put on TCGA_mar-12-2020/
I didn't see the Gh38 version for tcga_6k_genes.targetIntervals.bed and the original/un-liftover'd for whole_exome_agilent_plus_tcga_6k.targetIntervals.Gh38.bed?

Just uploaded data/TCGA_mar-12-2020/tcga_6k_genes.targetIntervals.padded.Gh38.bed

This file from March 12 is in the. s3 bucket - data/TCGA_mar-12-2020/whole_exome_agilent_plus_tcga_6k.targetIntervals.bed - Is this what you are asking for?

yuankunzhu · 2020-03-20T17:27:12Z

This file from March 12 is in the. s3 bucket - data/TCGA_mar-12-2020/whole_exome_agilent_plus_tcga_6k.targetIntervals.bed - Is this what you are asking for?

$ aws s3 ls $bucket/TCGA_mar-12-2020/ | grep whole_exome_agilent_plus_tcga_6k.targetIntervals.bed          <aws:saml>
2020-03-20 13:22:34    7684576 whole_exome_agilent_plus_tcga_6k.targetIntervals.bed

it looks like it's just uploaded?

But anyway thanks for checking, i think we have all the BED files in place now

tkoganti · 2020-03-20T17:38:43Z

This file from March 12 is in the. s3 bucket - data/TCGA_mar-12-2020/whole_exome_agilent_plus_tcga_6k.targetIntervals.bed - Is this what you are asking for?
$ aws s3 ls $bucket/TCGA_mar-12-2020/ | grep whole_exome_agilent_plus_tcga_6k.targetIntervals.bed          <aws:saml>
2020-03-20 13:22:34    7684576 whole_exome_agilent_plus_tcga_6k.targetIntervals.bed
it looks like it's just uploaded?

Hmm It said March 12 when I first looked and I uploaded again. Not sure if it was overwritten.

This is the capture kit info used for the new samples. But might be easier to generate again with all 319 samples. Let me know if you want me to create that -
tcga-capture_kit-info_NEW.txt

kgaonkar6 · 2020-03-20T19:53:33Z

@yuankunzhu @jaclyn-taroni Adding files updated from fusion-filtering analysis https://github.com/AlexsLemonade/OpenPBTA-analysis/tree/master/analyses/fusion_filtering/results

Changes #621 :
A QC step to identify and remove fusions in multi (more than 4) histologies is added to the fusion-filtering analysis:
Update files:

pbta-fusion-putative-oncogenic.tsv
pbta-fusion-recurrently-fused-genes-byhistology.tsv
pbta-fusion-recurrently-fused-genes-bysample.tsv

baileyckelly · 2020-03-24T16:21:37Z

Hi there -
From reviewing tickets, can we confirm that the following tickets are likely not going to be ready for v16 for the pbta-histologies file?
#608 - blocked
#509 - Target for v17
#245 - Target for v17

We will be adding in the following ticket(s), though for the histologies file:
#637 - Data refresh for pbta-histologies file

jaclyn-taroni · 2020-03-24T17:11:20Z

Hi there -
From reviewing tickets, can we confirm that the following tickets are likely not going to be ready for v16 for the pbta-histologies file?
#608 - blocked
#509 - Target for v17
#245 - Target for v17

We will be adding in the following ticket(s), though for the histologies file:
#637 - Data refresh for pbta-histologies file

Yes, sounds good. I will edit the original post to match this.

yuankunzhu · 2020-03-27T19:04:06Z

D3b data team is still working on clinical data updating, we are suggesting move all changes related pbta-histologies.tsv to v17(#656), so for this release, it will have changes for:

Updated TCGA MAFs per TCGA BED file used for variant calling #568
Update TCGA manifest file to include BED file used per sample
Add new TCGA BED files, padded and unpadded
Remove old TCGA BED file gencode.v19.basic.exome.hg38liftover.100bp_padded.bed
new fusions files from Qc step to identify and remove fused genes in multi (more than 4) histologies #621:
- pbta-fusion-putative-oncogenic.tsv
- pbta-fusion-recurrently-fused-genes-byhistology.tsv
- pbta-fusion-recurrently-fused-genes-bysample.tsv

cc @jaclyn-taroni and @baileyckelly

jaclyn-taroni · 2020-04-12T19:26:23Z

Closed via #657

jharenza added the data label Mar 4, 2020

This was referenced Mar 5, 2020

Remove CI logic from fusion-summary; rerun with v15 data #605

Merged

Updated analysis: fusion-summary with v15 fusion files and downstream #578

Closed

This was referenced Mar 9, 2020

Updated analysis: Use TCGA SNV consensus calls to get mutational signatures #481

Closed

Proposed Analysis: Run brain TCGA SNV data through PBTA SNV caller pipeline #257

Closed

jharenza mentioned this issue Mar 10, 2020

CNS EFT-CIC removal from pbta-histologies file #624

Closed

jaclyn-taroni added the in progress Someone is working on this issue, but feel free to propose an alternative approach! label Mar 11, 2020

cansavvy mentioned this issue Mar 16, 2020

chromosomal-instability cdf function updates #625

Merged

5 tasks

kgaonkar6 mentioned this issue Mar 20, 2020

Qc step to identify and remove fused genes in multi (more than 4) histologies #621

Merged

5 tasks

yuankunzhu mentioned this issue Mar 27, 2020

Data release/v16 #657

Merged

4 tasks

jaclyn-taroni closed this as completed Apr 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Planned Data Release: V16 #601

Planned Data Release: V16 #601

jharenza commented Mar 4, 2020 •

edited by jaclyn-taroni

Loading

jaclyn-taroni commented Mar 6, 2020

allisonheath commented Mar 11, 2020

allisonheath commented Mar 11, 2020 •

edited

Loading

jaclyn-taroni commented Mar 19, 2020

yuankunzhu commented Mar 20, 2020 •

edited

Loading

yuankunzhu commented Mar 20, 2020

yuankunzhu commented Mar 20, 2020 •

edited

Loading

tkoganti commented Mar 20, 2020 •

edited

Loading

yuankunzhu commented Mar 20, 2020 •

edited

Loading

tkoganti commented Mar 20, 2020

kgaonkar6 commented Mar 20, 2020 •

edited

Loading

baileyckelly commented Mar 24, 2020

jaclyn-taroni commented Mar 24, 2020

yuankunzhu commented Mar 27, 2020 •

edited

Loading

jaclyn-taroni commented Apr 12, 2020

Planned Data Release: V16 #601

Planned Data Release: V16 #601

Comments

jharenza commented Mar 4, 2020 • edited by jaclyn-taroni Loading

What data file(s) does this issue pertain to?

What release are you using?

Put your question or report your issue here.

jaclyn-taroni commented Mar 6, 2020

allisonheath commented Mar 11, 2020

allisonheath commented Mar 11, 2020 • edited Loading

jaclyn-taroni commented Mar 19, 2020

yuankunzhu commented Mar 20, 2020 • edited Loading

1. sync data from previous release

2. new data

2.1 TCGA MAF

2.2 TCGA BED

2.3 TCGA Manifest check

new fusion results

yuankunzhu commented Mar 20, 2020

yuankunzhu commented Mar 20, 2020 • edited Loading

tkoganti commented Mar 20, 2020 • edited Loading

yuankunzhu commented Mar 20, 2020 • edited Loading

tkoganti commented Mar 20, 2020

kgaonkar6 commented Mar 20, 2020 • edited Loading

baileyckelly commented Mar 24, 2020

jaclyn-taroni commented Mar 24, 2020

yuankunzhu commented Mar 27, 2020 • edited Loading

jaclyn-taroni commented Apr 12, 2020

jharenza commented Mar 4, 2020 •

edited by jaclyn-taroni

Loading

allisonheath commented Mar 11, 2020 •

edited

Loading

yuankunzhu commented Mar 20, 2020 •

edited

Loading

yuankunzhu commented Mar 20, 2020 •

edited

Loading

tkoganti commented Mar 20, 2020 •

edited

Loading

yuankunzhu commented Mar 20, 2020 •

edited

Loading

kgaonkar6 commented Mar 20, 2020 •

edited

Loading

yuankunzhu commented Mar 27, 2020 •

edited

Loading