NCBI Index Compression

[morsecodist]

NCBI Index NT and NR compression code

Changes:

• rust code (ncbi-compress) package to:

  1. split NT and NR into taxids based on provided NCBI mapping files
  2. sort individual taxids with longest sequences on top
  3. compress individual taxids based on containment above a specified similarity threshold
  4. combine individual taxids back into a compressed version of NT and NR
  5. Shuffle resulting compressed NT and NR (to avoid large blocks of SC2 accessions on one diamond or minimap chunked index)

• Download NT/NR

  1. NCBI is moving blast DBs off of the ftp site, we needed to adjust to download NT and NR with NCBI supported cli tools.

• debugging jupyter notebooks:

  1. notebook to query NCBI (helpful for spot checking the 'all taxa with neither family nor genus' classification' in CZID to verify that all accessions in that bucket indeed to not have genus or species level classifications.
  2. notebook to easily query marisa trie files (helpful to find if a particular accessionID is included in NT or NR)
  3. notebook to compare diamond and minimap alignment times between two projects - helpful for determining the run time difference between samples run on two different index versions
  4. Notebook to compare NCBI compress runs with multiple different parameters / input data
  5. Notebook to create taxon-lineage changelog for comp bio QA purposes

[morsecodist]

Looks great overall! Just left two tiny comments. It was great working with you on all of this.