MRG: transition internal signature loading functions #3161

ctb · 2024-05-13T22:47:00Z

Note: PR into #3153

Tackles some signature loading and saving cleanup throughout the codebase. Most changes are in the tests, and this is a significant cleanup of the test code!

Fixes #1062.

Goals:

deprecate external use of sourmash.signature load/save functions, because they are JSON-specific and inflexible.
simplify and standardize signature load/save function usage during tests;
get rid of deprecation messages during tests;

In brief,

rename sourmash.signature.load_signatures to load_signatures_from_json;
rename sourmash.signature.load_one_signature to load_one_signature_from_json;
rename sourmash.signature.save_signatures to save_signatures_to_json;
deprecate sourmash.save_signatures and sourmash.load_one_signature for 5.0 (joining load_signatures, which was already deprecated);
reduce/eliminate deprecations by transitioning internal test code to use these three functions directly from sourmash.signature instead of from the top-level sourmash import.
bonus: eliminate zipfile UserWarning around overwriting files, which causes lots of warnings when running tests.

Done:

in sourmash.signature submodule, rename load_signatures to load_signatures_from_json, load_one_signature to load_one_signatures_from_json, and save_signatures to save_signatures_to_json; make tests pass.
deprecate sourmash.load_one_signature and sourmash.save_signatures.
catch zipfile UserWarning for duplicate filenames in ZipStorage.save

TODO:

transition internal sourmash code+tests away from deprecated functions
create issue around changing API documentation prior to 5.0;

for more information, see https://pre-commit.ci

codecov · 2024-05-13T22:58:30Z

Codecov Report

Attention: Patch coverage is 92.68293% with 3 lines in your changes missing coverage. Please review.

Project coverage is 86.66%. Comparing base (2542c69) to head (922bfc2).
Report is 80 commits behind head on latest.

Files with missing lines	Patch %	Lines
src/sourmash/sbt_storage.py	85.71%	1 Missing and 1 partial ⚠️
src/sourmash/__init__.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           latest    #3161   +/-   ##
=======================================
  Coverage   86.66%   86.66%           
=======================================
  Files         136      136           
  Lines       15821    15830    +9     
  Branches     2713     2716    +3     
=======================================
+ Hits        13711    13719    +8     
- Misses       1800     1801    +1     
  Partials      310      310

Flag	Coverage Δ
hypothesis-py	`25.35% <56.09%> (+0.05%)`	⬆️
python	`92.33% <92.68%> (+<0.01%)`	⬆️
rust	`62.05% <ø> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

for more information, see https://pre-commit.ci

…e_sig_cmds

ctb · 2024-05-14T13:23:44Z

This feels done! It changes a lot of code in dumb ways, and cleans up the warnings emitted by the test, too; probably shouldn't be combined with any more changes 😅

Ready for review, @sourmash-bio/devs

bluegenes

lgtm

**Note:** PR into #3161 This PR cleans up and regularizes naming output on `sig` subcommands. Specifically, it: - [x] adds `--set-name` to `sig intersect` and `sig subtract` to set names on output signatures. - [x] confirms and tests default names for output from `sig inflate`, `sig filter`, `sig downsample`, and `sig flatten` - [x] updates documentation appropriately - [x] aliases `--name` to `--set-name` on `sig merge`, and nominates `--name` for deprecation and removal on v5 - [x] nominates `--name` on `sig extract`, `sig filter`, and `sig flatten` for deprecation and removal on v5 - [x] highlights `--name` on `sketch dna`, `sketch protein`, and `sketch translate` - ref #3152 Fixes #1801 Fixes #3152 Related issues: * #3173 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Minor new features: * add `--set-name` to `sig intersect` and `sig subtract` (#3162) * upgrade `sig overlap` and `sig subtract` to load more than JSON signatures (#3153) * force continue past `tax genome` classification errors (#3100) Bug fixes: * fix `remaining_bp` output from sourmash gather (#3195) * fix RocksDB-based gather & other rust-based infelicities revealed by plugins (#3193, #3197) * use correct denominator in f_unique_to_query (#3138) Cleanup and documentation updates: * update JOSS for sourmash v4 (#3114, #3203, #3209) * fix links to taxonomy spreadsheets (#3119) * fix description of `f_unique_weighted` (#3164) Developer updates: * transition internal signature loading functions (#3161) * allow get/set record.filename (#3121) * round a number that is losing precision in 15th place in `test_distance_utpy` (#3126) * disable ppc64le wheel building (#3127) * prepare to remove `sourmash compute` for sourmash v5.0 (#3103) * add rustup target x86_64-apple-darwin (#3148) * mv `.cargo/config` to `config.toml` (#3147) * fix clippy warnings about max_value (#3146) * bump to v4.8.9-dev (#3135) * update src/core/CHANGELOG.md for sourmash-rs core release r0.14.0 (#3199) Dependabot updates: * Bump DeterminateSystems/nix-installer-action from 11 to 12 (#3184) * Bump DeterminateSystems/magic-nix-cache-action from 6 to 7 (#3185) * Bump statrs from 0.16.0 to 0.16.1 (#3186) * Bump serde from 1.0.202 to 1.0.203 (#3175) * Bump ouroboros from 0.18.3 to 0.18.4 (#3176) * Bump itertools from 0.12.1 to 0.13.0 (#3166) * Bump camino from 1.1.6 to 1.1.7 (#3169) * Bump serde from 1.0.201 to 1.0.202 (#3168) * Bump thiserror from 1.0.60 to 1.0.61 (#3167) * Bump pypa/cibuildwheel from 2.18.0 to 2.18.1 (#3165) * Bump DeterminateSystems/magic-nix-cache-action from 4 to 6 (#3157) * Bump DeterminateSystems/nix-installer-action from 10 to 11 (#3156) * Bump pypa/cibuildwheel from 2.17.0 to 2.18.0 (#3155) * Bump serde_json from 1.0.116 to 1.0.117 (#3159) * Bump thiserror from 1.0.59 to 1.0.60 (#3158) * Bump serde from 1.0.200 to 1.0.201 (#3160) * Bump roaring from 0.10.3 to 0.10.4 (#3142) * Bump histogram from 0.10.0 to 0.10.1 (#3141) * Bump getrandom from 0.2.14 to 0.2.15 (#3143) * Bump num-iter from 0.1.44 to 0.1.45 (#3140) * Bump jinja2 from 3.1.3 to 3.1.4 (#3145) * Bump serde from 1.0.199 to 1.0.200 (#3144) * Bump serde from 1.0.198 to 1.0.199 (#3130) * Bump conda-incubator/setup-miniconda from 3.0.3 to 3.0.4 (#3131) * Update pytest requirement from <8.2.0,>=6.2.4 to >=6.2.4,<8.3.0 (#3132) * Bump myst-parser from 2.0.0 to 3.0.1 (#3133) * Bump thiserror from 1.0.58 to 1.0.59 (#3123) * Bump serde_json from 1.0.115 to 1.0.116 (#3124) * Bump serde from 1.0.197 to 1.0.198 (#3122) * Update docutils requirement from <0.21,>=0.17.1 to >=0.17.1,<0.22 (#3116)

ctb and others added 9 commits May 12, 2024 09:14

support more general sig loading in sig overlap

333832b

fix up subtract as well

9a45829

add sourmash_args.load_one_signature

6da2d3a

[pre-commit.ci] auto fixes from pre-commit.com hooks

0082c45

for more information, see https://pre-commit.ci

fix one more call to load_one_signature

093ed65

add tests

6d7ac35

[pre-commit.ci] auto fixes from pre-commit.com hooks

9a6a61c

for more information, see https://pre-commit.ci

rename load/save fns

76aaafe

[pre-commit.ci] auto fixes from pre-commit.com hooks

64d2786

for more information, see https://pre-commit.ci

ctb changed the base branch from latest to upgrade_sig_cmds May 13, 2024 22:52

ctb and others added 5 commits May 13, 2024 16:17

rename to use _json names

fb28a1b

ignore zipfile UserWarnings about dup file names

dcf5920

[pre-commit.ci] auto fixes from pre-commit.com hooks

266d78a

for more information, see https://pre-commit.ci

Merge branch 'latest' of github.com:sourmash-bio/sourmash into upgrad…

35dc642

…e_sig_cmds

Merge branch 'upgrade_sig_cmds' into rename_sig_json_fns

e3e7d28

ctb mentioned this pull request May 14, 2024

make load_one_signature use load_file_as_signatures API #1062

Closed

stop unnecessary renaming ;)

1501704

ctb mentioned this pull request May 14, 2024

separate out load_signatures code to distinguish loading-from-file and parsing-from-string #1060

Open

ctb changed the title ~~WIP: transition internal signature loading functions~~ MRG: transition internal signature loading functions May 14, 2024

ctb mentioned this pull request May 14, 2024

MRG: add --set-name to sig intersect and sig subtract #3162

Merged

6 tasks

bluegenes approved these changes Jun 4, 2024

View reviewed changes

Base automatically changed from upgrade_sig_cmds to latest June 4, 2024 18:03

Merge branch 'latest' into rename_sig_json_fns

922bfc2

ctb merged commit 9283dc4 into latest Jun 4, 2024
39 of 40 checks passed

ctb deleted the rename_sig_json_fns branch June 4, 2024 18:26

ctb mentioned this pull request Jun 4, 2024

update API documentation around saving/loading sketches for v5 #3189

Open

ctb mentioned this pull request Jun 10, 2024

MRG: 4.8.9 release branch #3200

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MRG: transition internal signature loading functions #3161

MRG: transition internal signature loading functions #3161

ctb commented May 13, 2024 •

edited

Loading

codecov bot commented May 13, 2024 •

edited

Loading

ctb commented May 14, 2024

bluegenes left a comment

MRG: transition internal signature loading functions #3161

MRG: transition internal signature loading functions #3161

Conversation

ctb commented May 13, 2024 • edited Loading

codecov bot commented May 13, 2024 • edited Loading

Codecov Report

ctb commented May 14, 2024

bluegenes left a comment

Choose a reason for hiding this comment

ctb commented May 13, 2024 •

edited

Loading

codecov bot commented May 13, 2024 •

edited

Loading