ENH: Migrated partition and collating actions from `q2-moshpit` to `q2-types` #334

VinzentRisch · 2024-05-28T09:32:23Z

closes #332

Moved the actions partition_feature_data_mags, collate_feature_data_mags, collate_ortholog_annotations, collate_orthologs, partition_orthologs, partition_sample_data_mags, collate_sample_data_mags, _validate_mag_ids and _validate_num_partitions from q2-moshpit to q2-types.
Moved all tests that are associated with these actions from q2-moshpit to q2-types.
Created new tests for the actions collate_orthologs and partition_orthologs.
Renamed collate_annotations to collate_ortholog_annotations.
Changed shutil.move to duplicate in collate_orthologs action to be consistent with other collate actions.
Added tests directory where tests for helper functions in util.py live.

Run it locally

First, clone the repo and checkout the PR branch:

git clone https://github.com/qiime2/q2-types.git
cd q2-types
git fetch origin pull/334/head:pr-334
git checkout pr-334
pip install -e .

Download test files
PR-334-types.zip
Test it out!

qiime types partition-sample-data-mags --i-mags sample_data_mags.qza --output-dir sample_data_mags
qiime types partition-feature-data-mags --i-mags feature_data_mag.qza --output-dir feature_data_mag
qiime types partition-orthologs --i-orthologs genome_data_orthologs.qza --output-dir orthologs
qiime types collate-sample-data-mags --i-mags sample_data_mags/partitioned_mags --o-collated-mags collated-sample-data-mags.qza
qiime types collate-feature-data-mags --i-mags feature_data_mag/partitioned_mags --o-collated-mags collated-feature-data-mags.qza
qiime types collate-orthologs --i-orthologs orthologs/partitioned_orthologs --o-collated-orthologs collated-orthologs.qza

VinzentRisch · 2024-05-30T09:16:57Z

Hello @lizgehret,
I am trying to move these partition and collating actions from moshpit to types. And i am encountering a problem with circular imports. The errors occur when I add the imports in plugin setup. I don't have any experience with resolving this kind of problem and I wanted to ask if you could have a look at the code and help me resolve this issue? I normally would ask Michal but he is on holiday.

Thank you!

The error is this:

Traceback (most recent call last):
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 279, in get_plugin_manager
    return qiime2.sdk.PluginManager.reuse_existing()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 58, in reuse_existing
    raise UninitializedPluginManagerError
qiime2.sdk.plugin_manager.UninitializedPluginManagerError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/builtin/dev.py", line 33, in refresh_cache
    import q2cli.core.cache
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in <module>
    CACHE = DeploymentCache()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 61, in __init__
    self._state = self._get_cached_state(refresh=refresh)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 107, in _get_cached_state
    self._cache_current_state(current_requirements)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 205, in _cache_current_state
    state = self._get_current_state()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 253, in _get_current_state
    plugin_manager = q2cli.util.get_plugin_manager()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 291, in get_plugin_manager
    return qiime2.sdk.PluginManager()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 67, in __new__
    self._init(add_plugins=add_plugins)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 105, in _init
    plugin = entry_point.load()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2516, in load
    return self.resolve()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2522, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/__init__.py", line 18, in <module>
    importlib.import_module('q2_types.feature_data')
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data/__init__.py", line 11, in <module>
    from ._format import (
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data/_format.py", line 18, in <module>
    from ..plugin_setup import plugin
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/plugin_setup.py", line 17, in <module>
    from q2_types.feature_data_mag import MAG, NOG
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data_mag/__init__.py", line 11, in <module>
    from ._format import (
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data_mag/_format.py", line 11, in <module>
    from q2_types.feature_data import DNAFASTAFormat
ImportError: cannot import name 'DNAFASTAFormat' from partially initialized module 'q2_types.feature_data' (most likely due to a circular import) (/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data/__init__.py)
(qiime2-tiny-2024.2) rischv@fsbt007 q2-types % git commit -a -m "added everything but bug cicular imports"
[332_move_partition_collating 57234cc] added everything but bug cicular imports
 2 files changed, 112 insertions(+), 3 deletions(-)
(qiime2-tiny-2024.2) rischv@fsbt007 q2-types % qiime dev refresh-cache                                    
QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Traceback (most recent call last):
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 279, in get_plugin_manager
    return qiime2.sdk.PluginManager.reuse_existing()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 58, in reuse_existing
    raise UninitializedPluginManagerError
qiime2.sdk.plugin_manager.UninitializedPluginManagerError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/builtin/dev.py", line 33, in refresh_cache
    import q2cli.core.cache
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in <module>
    CACHE = DeploymentCache()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 61, in __init__
    self._state = self._get_cached_state(refresh=refresh)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 107, in _get_cached_state
    self._cache_current_state(current_requirements)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 205, in _cache_current_state
    state = self._get_current_state()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/core/cache.py", line 253, in _get_current_state
    plugin_manager = q2cli.util.get_plugin_manager()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2cli/util.py", line 291, in get_plugin_manager
    return qiime2.sdk.PluginManager()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 67, in __new__
    self._init(add_plugins=add_plugins)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/qiime2/sdk/plugin_manager.py", line 105, in _init
    plugin = entry_point.load()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2516, in load
    return self.resolve()
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2522, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/__init__.py", line 18, in <module>
    importlib.import_module('q2_types.feature_data')
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data/__init__.py", line 11, in <module>
    from ._format import (
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data/_format.py", line 18, in <module>
    from ..plugin_setup import plugin
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/plugin_setup.py", line 17, in <module>
    from q2_types.feature_data_mag import MAG, NOG
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data_mag/__init__.py", line 11, in <module>
    from ._format import (
  File "/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data_mag/_format.py", line 11, in <module>
    from q2_types.feature_data import DNAFASTAFormat
ImportError: cannot import name 'DNAFASTAFormat' from partially initialized module 'q2_types.feature_data' (most likely due to a circular import) (/Users/rischv/miniconda3/envs/qiime2-tiny-2024.2/lib/python3.8/site-packages/q2_types/feature_data/__init__.py)

lizgehret · 2024-05-30T19:07:57Z

Hey @VinzentRisch,

Happy to help! I'll take a look at this shortly.

lizgehret · 2024-06-18T20:57:18Z

Hey @VinzentRisch,

Apologies for the delay on this! I've been playing around with this locally, and this is definitely a tricky one. I'm going to try a few different things to work around these circular imports, but this may be something we want to discuss in our joint lab meeting next month. I suspect this will be something that may come up more as we continue to add types/formats from the metagenomic ecosystem.

EDIT: One additional thing I'm thinking about is that all of the additions in this PR are methods, rather than strictly being types/formats/transformers. I'm wondering if these are things we even want to be moving into q2-types - but I'll discuss with other folks on the team about this and circle back!

ebolyen · 2024-06-20T18:11:28Z

I'm going to give this a shot as well. q2-types has caused import issues for basically everyone at this point. I think a refactor may be in order.

ebolyen · 2024-07-24T22:05:06Z

Once #342 is merged, it should be relatively straight-forward to rebase this PR ontop of it. The registration will end up in _deferred_setup/__init__.py for the relevant submodules.

ebolyen · 2024-07-31T21:18:23Z

Hey @VinzentRisch,

We should be all set to continue working on this one. I've fixed the merge conflicts and we no longer end up with import cycles.

Of note, I did delete one method in 08e3fd1 which operated on a format that was removed in #338

We might consider moving the plugin method registrations into the smaller _deferred_setup/__init__.py files, but it honestly doesn't matter all that much, so your call there.

§ qiime types 
Usage: qiime types [OPTIONS] COMMAND [ARGS]...

  Description: This QIIME 2 plugin defines semantic types and transformers
  supporting microbiome analysis.

  Plugin website: https://github.com/qiime2/q2-types

  Getting user support: Please post to the QIIME 2 forum for help with this
  plugin: https://forum.qiime2.org

Options:
  --version            Show the version and exit.
  --example-data PATH  Write example data and exit.
  --citations          Show citations and exit.
  --help               Show this message and exit.

Commands:
  collate-feature-data-mags    Collate mags
  collate-orthologs            Collate Orthologs
  collate-sample-data-mags     Collate mags
  partition-feature-data-mags  Partition MAGs
  partition-orthologs          Partition orthologs
  partition-sample-data-mags   Partition MAGs

VinzentRisch · 2024-08-02T12:16:11Z

Hi @ebolyen
Thanks for the refactor! now everything seems to be working fine.
I finished everything and added some data and code in the PR description to test all actions.

gregcaporaso · 2024-08-22T17:41:40Z

Thanks for all of the work on this @VinzentRisch!

@misialq, any chance you could review this one? @ebolyen has already reviewed the structure and said that it all looks good, so it's just the functionality at this point that needs review and we think you might have the most context there. If you can't get to it, I can review this one but it'll be a few weeks as I'm about to head out to teach a workshop next week and then am tied up the week after that - so we're looking at the 2nd week of September before I can get to it.

misialq · 2024-08-26T07:30:37Z

Hey @gregcaporaso, sure thing - I'll give it a go this week!

misialq

Hey all, this looks good to me too - thanks! 🚀

VinzentRisch added 3 commits May 28, 2024 11:18

migrated partition and collating actions from moshpit to types

045ec95

added everything but bug cicular imports

57234cc

removed import

8103975

VinzentRisch mentioned this pull request May 30, 2024

MAINT: move partitioning and collating actions to q2-types bokulich-lab/q2-annotate#175

Merged

ebolyen self-assigned this Jun 20, 2024

ebolyen added 2 commits July 31, 2024 14:01

Merge branch 'dev' into 332_move_partition_collating

c5e4057

SQUASH: fix type drift from qiime2#338

08e3fd1

ebolyen assigned VinzentRisch and unassigned ebolyen Jul 31, 2024

VinzentRisch added 4 commits August 2, 2024 11:55

Merge branch 'dev' into 332_move_partition_collating

376265e

typos in plugin setup

de69df7

lint

2831f46

added package data

b9992c1

VinzentRisch marked this pull request as ready for review August 2, 2024 12:16

ebolyen unassigned VinzentRisch Aug 22, 2024

hagenjp requested a review from gregcaporaso August 22, 2024 17:37

VinzentRisch added 3 commits September 3, 2024 15:40

added collate_ortholog_annotations from moshpit

2139199

added registration for collate_ortholog_annotations

6af9325

added package data for ortholog-annotations-collating

bb17456

misialq approved these changes Sep 4, 2024

View reviewed changes

ebolyen assigned gregcaporaso Sep 5, 2024

ebolyen merged commit 0982c4a into qiime2:dev Sep 5, 2024
4 checks passed

misialq mentioned this pull request Nov 7, 2024

MAINT: move partitioning and collating actions to q2-types bokulich-lab/q2-annotate#172

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Migrated partition and collating actions from `q2-moshpit` to `q2-types` #334

ENH: Migrated partition and collating actions from `q2-moshpit` to `q2-types` #334

VinzentRisch commented May 28, 2024 •

edited

Loading

VinzentRisch commented May 30, 2024 •

edited

Loading

lizgehret commented May 30, 2024

lizgehret commented Jun 18, 2024 •

edited

Loading

ebolyen commented Jun 20, 2024

ebolyen commented Jul 24, 2024

ebolyen commented Jul 31, 2024

VinzentRisch commented Aug 2, 2024

gregcaporaso commented Aug 22, 2024

misialq commented Aug 26, 2024

misialq left a comment

ENH: Migrated partition and collating actions from q2-moshpit to q2-types #334

ENH: Migrated partition and collating actions from q2-moshpit to q2-types #334

Conversation

VinzentRisch commented May 28, 2024 • edited Loading

Run it locally

VinzentRisch commented May 30, 2024 • edited Loading

lizgehret commented May 30, 2024

lizgehret commented Jun 18, 2024 • edited Loading

ebolyen commented Jun 20, 2024

ebolyen commented Jul 24, 2024

ebolyen commented Jul 31, 2024

VinzentRisch commented Aug 2, 2024

gregcaporaso commented Aug 22, 2024

misialq commented Aug 26, 2024

misialq left a comment

Choose a reason for hiding this comment

ENH: Migrated partition and collating actions from `q2-moshpit` to `q2-types` #334

ENH: Migrated partition and collating actions from `q2-moshpit` to `q2-types` #334

VinzentRisch commented May 28, 2024 •

edited

Loading

VinzentRisch commented May 30, 2024 •

edited

Loading

lizgehret commented Jun 18, 2024 •

edited

Loading