Merge branch 'main' into inscopix

catalystneuro · Mar 4, 2024 · e21fe8f · e21fe8f
2 parents 1790769 + 7aa2a63
commit e21fe8f
Show file tree

Hide file tree

Showing 53 changed files with 534 additions and 337 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -60,10 +60,10 @@ body:
     attributes:
       label: Python Version
       options:
-        - 3.9
-        - 3.10
-        - 3.11
-        - 3.12
+        - "3.9"
+        - "3.10"
+        - "3.11"
+        - "3.12"
     validations:
       required: true
   - type: textarea
@@ -77,13 +77,22 @@ body:
 
         You can attach images or log files by clicking this area to highlight it and then dragging files in.
         If GitHub upload is not working, you can also copy and paste the output into this section.
-  - type: checkboxes
-    id: terms
+  - type: dropdown
+    id: code_of_conduct
     attributes:
       label: Code of Conduct
-      description: By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/catalystneuro/roiextractors/blob/master/.github/CODE_OF_CONDUCT.rst)
+      description: Do you agree to follow our [Code of Conduct](https://github.com/catalystneuro/roiextractors/blob/master/.github/CODE_OF_CONDUCT.rst)?
+      options:
+        - "Yes"
+    validations:
+      required: true
+  - type: dropdown
+    id: duplicated_issue_check
+    attributes:
+      label: Duplicated Issue Check
+      description: Have you ensured this bug was not already [reported](https://github.com/catalystneuro/roiextractors/issues)?
       options:
-        - label: I agree to follow this project's [Code of Conduct](https://github.com/catalystneuro/roiextractors/blob/master/.github/CODE_OF_CONDUCT.rst)
-          required: true
-        - label: Have you ensured this bug was not already [reported](https://github.com/catalystneuro/roiextractors/issues)?
-          required: true
+        - "No"
+        - "Yes"
+    validations:
+      required: true
diff --git a/.github/workflows/update-version.yml b/.github/workflows/update-version.yml
@@ -0,0 +1,41 @@
+name: Update Dev Version on Release
+
+on:
+    workflow_dispatch:
+    workflow_run:
+        workflows: [auto-publish]
+        types: [completed]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Checkout New Branch
+        run: |
+          git checkout -b update_dev_version
+      - name: Update dev version
+        run: |
+          git fetch --prune --unshallow --tags
+          tags="$(git tag --list)"
+          latest_tag=${tags: -6 : 6}
+          old_version=${latest_tag:1:5}
+          echo "Old Version: $old_version"
+          old_major_version=${old_version:0:1}
+          old_minor_version=${old_version:2:1}
+          old_patch_version=${old_version:4:1}
+          new_patch_version=`expr $old_patch_version + 1`
+          new_version="$old_major_version.$old_minor_version.$new_patch_version"
+          echo "New Version: $new_version"
+          sed -i -e "s/version=\"$old_version\"/version=\"$new_version\"/g" setup.py
+          echo "old_version=$old_version" >> "$GITHUB_ENV"
+          echo "new_version=$new_version" >> "$GITHUB_ENV"
+      - name: Commit Changes and Create Pull Request
+        run: |
+          git config --global user.email 41898282+github-actions[bot]@users.noreply.github.com
+          git config --global user.name github-actions[bot]
+          git commit . -m "Update dev version from $old_version to $new_version"
+          git push origin update_dev_version
+          gh pr create --title "[Github.CI] Update dev version from $old_version to $new_version" --body "version: $old_version --> $new_version"
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -6,7 +6,7 @@ repos:
     -   id: end-of-file-fixer
     -   id: trailing-whitespace
 -   repo: https://github.com/psf/black
-    rev: 23.12.1
+    rev: 24.2.0
     hooks:
     -   id: black
         exclude: ^docs/

diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -1,12 +1,14 @@
-build:
-    image: latest
-
-version: 2
+version: "2"
 
-sphinx:
-  configuration: docs/source/conf.py
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.10"
 
 python:
   version: 3.9
   install:
     - requirements: docs/requirements_doc.txt
+
+sphinx:
+  configuration: docs/source/conf.py
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -30,6 +30,11 @@
 
 * Added support for Miniscope AVI files with the `MiniscopeImagingExtractor`. [PR #225](https://github.com/catalystneuro/roiextractors/pull/225)
 
+* Added support for incomplete file ingestion for the `Suite2pSegmentationExtractor`. [PR #227](https://github.com/catalystneuro/roiextractors/pull/227)
+
+* Bug fix for the `CaimanSegmentationExtractor`: Change reshaping from 'C' to 'F' (Fortran). [PR #227](https://github.com/catalystneuro/roiextractors/pull/227)
+
+* Bug fix for the `CaimanSegmentationExtractor`: Added importing of `self._image_correlation` and changed how `self._image_mean` to import the background component image. [PR #227](https://github.com/catalystneuro/roiextractors/pull/227)
 
 
 # v0.5.2

diff --git a/README.md b/README.md
@@ -5,141 +5,64 @@
 [![documentation](https://readthedocs.org/projects/roiextractors/badge/?version=latest)](https://roiextractors.readthedocs.io/en/latest/)
 [![License](https://img.shields.io/pypi/l/pynwb.svg)](https://github.com/catalystneuro/roiextractors/license.txt)
 
-# ROI Extractors
-Python-based module for extracting from, converting between, and handling recorded and optical imaging data from several file formats. Inspired by [SpikeExtractors](https://github.com/SpikeInterface/spikeextractors).
-
-Developed by [CatalystNeuro](http://catalystneuro.com/).
-
-## Getting Started:
-#### Installation:
-`pip install roiextractors`
-
-## Usage:
-### Supported file types:
-#### Imaging
-1. HDF5
-2. TIFF
-3. STK
-4. FLI
-5. SBX
-
-#### Segmentation
-1. [calciumImagingAnalysis](https://github.com/bahanonu/calciumImagingAnalysis) (CNMF-E, EXTRACT)
-2. [SIMA](http://www.losonczylab.org/sima/1.3.2/)
-3. [NWB](https://pynwb.readthedocs.io/en/stable/)
-4. [suite2p](https://github.com/MouseLand/suite2p)
-45. Numpy (a data format for manual input of optical physiology data as various numpy datasets)
-
-#### Functionality:
-Interconversion amongst the various data formats as well as conversion to the NWB format and back.
-
-#### Features:
-1. SegmentationExtractor object:
-    * `seg_obj.get_channel_names()` :
-    List of optical channel names
-    * `seg_obj.get_num_channels()` :
-    Number of channels
-    * `seg_obj.get_movie_framesize()`:
-    (height, width) of raw movie
-    * `seg_obj.get_movie_location()`:
-    Location of storage of movie/tiff images
-    * `seg_obj.get_image_masks(self, roi_ids=None)`:
-    Image masks as (ht, wd, num_rois) with each value as the weight given during segmentation operation.
-    * `seg_obj.get_pixel_masks(roi_ids=None)`:
-    Get pixel masks as (total_pixels(ht*wid), no_rois)
-    * `seg_obj.get_traces(self, roi_ids=None, start_frame=None, end_frame=None)`:
-    df/F trace as (num_rois, num_frames)
-    * `seg_obj.get_sampling_frequency()`:
-    Sampling frequency of movie/df/F trace.
-    * `seg_obj.get_roi_locations()`:
-    Centroid pixel location of the ROI (Regions Of Interest) as (x,y).
-    * `seg_obj.get_num_rois()`:
-    Total number of ROIs after segmentation operation.
-    * `seg_obj.get_roi_ids()`:
-    Any integer tags associated with an ROI, defaults to `0:num_of_rois`
-
-#### SegmentationExtractor object creation:
-```python
-import roiextractors
-import numpy as np
-
-seg_obj_cnmfe = roiextractors.CnmfeSegmentationExtractor('cnmfe_filename.mat') # cnmfe
-seg_obj_extract = roiextractors.ExtractSegmentationExtractor('extract_filename.mat') # extract
-seg_obj_sima = roiextractors.SimaSegmentationExtractor('sima_filename.sima') # SIMA
-seg_obj_numpy = roiextractors.NumpySegmentationExtractor(
-        filepath = 'path-to-file',
-        masks=np.random.rand(movie_size[0],movie_size[1],no_rois),
-        signal=np.random.randn(num_rois,num_frames),
-        roi_idx=np.random.randint(no_rois,size=[1,no_rois]),
-        no_of_channels=None,
-        summary_image=None,
-        channel_names=['Blue'],
-) # Numpy object
-seg_obj_nwb = roiextractors.NwbSegmentationExtractor(
-                    filepath_of_nwb, optical_channel_name=None, # optical channel to extract and store info from
-                    imaging_plane_name=None, image_series_name=None, # imaging plane to extract and store data from
-                    processing_module_name=None,
-                    neuron_roi_response_series_name=None, # roi_response_series name to extract and store data from
-                    background_roi_response_series_name=None) # nwb object
-```
-#### Data format conversion: SegmentationExtractor to NWB:
-```python
-roiextractors.NwbSegmentationExtractor.write_segmentation(seg_obj, saveloc,
-                    propertydict=[{'name': 'ROI feature 1,
-                                   'description': 'additional attribute of each ROI',
-                                   'data': np.random.rand(1,no_rois),
-                                   'id': seg_obj.get_roi_ids()},
-                                  {'name': 'ROI feature 2,
-                                   'description': 'additional attribute of each ROI',
-                                   'data': np.random.rand(1,no_rois),
-                                   'id': seg_obj.get_roi_ids()}],
-                    nwbfile_kwargs={'session_description': 'nwbfiledesc',
-                                    'experimenter': 'experimenter name',
-                                    'lab': 'test lab',
-                                    'session_id': 'test sess id'},
-                    emission_lambda=400.0, excitation_lambda=500.0)
-```
-## Example Datasets:
-Example datasets are maintained at https://gin.g-node.org/CatalystNeuro/ophys_testing_data.
-
-To download test data on your machine,
+# ROIExtractors
+<p align="center">
+  <h3 align="center">Automatically read optical imaging/segmentation data into a common API</h3>
+</p>
+<p align="center">
+   <a href="roiextractors.readthedocs.io"><strong>Explore our documentation »</strong></a>
+</p>
 
-1. Install the gin client (instructions [here](https://gin.g-node.org/G-Node/Info/wiki/GIN+CLI+Setup#linux))
-2. Use gin to download data.
-```shell
-gin get CatalystNeuro/ophys_testing_data
-cd ophys_testing_data
-gin get-content
-```
+<!-- TABLE OF CONTENTS -->
 
-3. Change the file at `roiextractors/tests/gin_test_config.json` to point to the path of this test data
+## Table of Contents
 
-To update data later, `cd` into the test directory and run `gin get-content`
+- [About](#about)
+- [Installation](#installation)
+- [Documentation](#documentation)
+- [Funding](#funding)
+- [License](#license)
 
+## About
 
-## Class descriptions:
+ROIExtractors provides a common API for various optical imaging and segmentation formats to streamline conversion and data analysis. ROI stands for Region Of Interest, which is the region in a set of acquired fluorescence images which the segmentation software has determined as a neuron.
 
-*  **SegmentationExtractor:** An abstract class that contains all the meta-data and output data from the ROI segmentation operation when applied to the pre-processed data. It also contains methods to read from and write to various data formats ouput from  the processing pipelines like SIMA, CaImAn, Suite2p, CNNM-E.
+Features:
 
-*  **NumpySegmentationExtractor:** Contains all data coming from a file format for which there is currently no support. To construct this, all data must be entered manually as arguments.
+* Reads data from 10+ popular optical imaging and segmentation data formats into a common API.
+* Extracts relevant metadata from each format.
+* Provides a handy set of methods to analyze data such as `get_roi_locations()`
 
-*  **CnmfeSegmentationExtractor:** This class inherits from the SegmentationExtractor class, having all its funtionality specifically applied to the dataset output from the 'CNMF-E' ROI segmentation method.
+## Installation
 
-*  **ExtractSegmentationExtractor:** This class inherits from the SegmentationExtractor class, having all its funtionality specifically applied to the dataset output from the 'EXTRACT' ROI segmentation method.
+To install the latest stable release of **roiextractors** though PyPI, type:
+```shell
+pip install roiextractors
+```
 
-*  **SimaSegmentationExtractor:** This class inherits from the SegmentationExtractor class, having all its funtionality specifically applied to the dataset output from the 'SIMA' ROI segmentation method.
+For more flexibility we recommend installing the latest version directly from GitHub. The following commands create an environment with all the required dependencies and the latest updates:
 
-*  **NwbSegmentationExtractor:** Extracts data from the NWB data format. Also implements a static method to write any format specific object to NWB.
+```shell
+git clone https://github.com/catalystneuro/roiextractors
+cd roiextractors
+conda env create roiextractors_env
+conda activate roiextractors_env
+pip install -e .
+```
+Note that this will install the package in [editable mode](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs).
 
-* **Suite2PSegmentationExtractor:** Extracts data from suite2p format.
+Finally, if you prefer to avoid `conda` altogether, the following commands provide a clean installation within the current environment:
+```shell
+pip install git+https://github.com/catalystneuro/roiextractors.git@main
+```
 
-## Troubleshooting
-##### Installing SIMA with python>=3.7:
-Will need a manual installation for package dependency **SIMA** since it does not currently support python 3.7:
-1.   Download SIMA wheels distribution [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/#sima).
-2.  `pip install <download-path-to-wheels.whl>`
-3.  `pip install roiextractors`
+## Documentation
+See our [ReadTheDocs page](https://roiextractors.readthedocs.io/en/latest/) for full documentation, including a gallery of all supported formats.
 
-### Funded by
+## Funding
+ROIExtractors is funded by
 * Stanford University as part of the Ripple U19 project (U19NS104590).
 * LBNL as part of the NWB U24 (U24NS120057).
+
+## License
+ROIExtractors is distributed under the BSD3 License. See [LICENSE](https://github.com/catalystneuro/roiextractors/blob/main/LICENSE.txt) for more information.
diff --git a/docs/requirements_doc.txt b/docs/requirements_doc.txt
diff --git a/docs/source/compatible.rst b/docs/source/compatible.rst
@@ -1,25 +1,8 @@
 Supported formats
 =================
 
-Roiextractors supports various image formats obtained from different 2photon acquisition systems.
-It also supports well known post processing cell extraction/segmentation pipelines. If you use any other format and would like to see it being supported we would love to know! (:doc:`Contact <contact>`)
+The following is a live record of all the formats supported by ROIExtractors as well as other relevant ecosystem support.
 
-Imaging
--------
-1. HDF5
-2. TIFF
-3. STK
-4. FLI
+.. raw:: html
 
-Segmentation
-------------
-#. `calciumImagingAnalysis <https://github.com/bahanonu/calciumImagingAnalysis/>`_ (CNMF-E, EXTRACT)
-#. `Caiman <https://github.com/flatironinstitute/CaImAn/>`_
-#. `SIMA <http://www.losonczylab.org/sima/1.3.2//>`_
-#. `NWB <https://pynwb.readthedocs.io/en/stable//>`_
-#. `suite2p <https://github.com/MouseLand/suite2p/>`_
-#. Numpy (for any currently un-supported format)
-
-Example Datasets
-----------------
-Example datasets for each fo the file formats can be downloaded `here <https://gin.g-node.org/CatalystNeuro/ophys_testing_data/src/master/segmentation_datasets/>`_
+    <iframe style="width: 100%; height: 70vh; border: none;" src="https://catalystneuro.github.io/format-support-table/?tab=ophys-imaging"></iframe>