Add new examples and Update docs (#109)

* move graph ops to runtime * add/move graph ops tests * fix onnx helper imports * add `parse` to requirements & update gitignore * update graph ops init * update onnx exporter * add runtime wrapper * add vortex metric base * add metric registry * add lightning profiler * add metric base to init * update onnx-based runtime * add classification metric with plotting support * add mnasnet example * update example * add validation_epoch_end for mnasnet example * remove old examples * move mnasnet example * add export and inference example * move helper function to runtime * update onnxruntime requirements * add detr model dataset and utility * add detr module using vortex model base * add helper script to download voc dataset * add training config for detr example * update onnx exporter * add runtime wrapper * add vortex metric base * add metric registry * add lightning profiler * add metric base to init * update onnx-based runtime * add classification metric with plotting support * add mnasnet example * update example * add validation_epoch_end for mnasnet example * remove old examples * move mnasnet example * move metric compute at val epoch end only * rename export example * move dataset, model, and utils to separate dir for detection example * update docs for examples * remove old examples * update docstring for onnx exporter & runtime helper * remove old factory and pipelines * update docs for exporter, runtime, and graph_ops * update gitignore * remove docs for init method * remove unused test for old example * remove unused classification metrics * update api toc order * add more docstring for model base * update formatting to rst * add metadata documentation * add colab link for detection example * update docs front page * add alias function for common embed metadata * remove test metrics * remove modules doc that is no longer relevant * remove user guides that is no longer relevant * minor formatting fix * reorder toc * update formatting * update README * restore icon for readme * manually reorder exporter method * remove broken import for metrics * update model docstring * fix typo in predict docs
nodefluxio · Apr 14, 2021 · 9b959d8 · 9b959d8
1 parent 6847c39
commit 9b959d8
Show file tree

Hide file tree

Showing 67 changed files with 4,932 additions and 4,648 deletions.
diff --git a/.gitignore b/.gitignore
@@ -284,4 +284,8 @@ experiments/outputs/**/reports
 !tests/test_dataset/*
 lightning_logs
 checkpoints
-logs
+logs
+
+*.xml
+*.tar
+*.txt
diff --git a/README.md b/README.md
@@ -2,39 +2,23 @@
 
 # Vortex ( Visual-Cortex )
 
-A Deep Learning Model Development Framework for Computer Vision
+A Deep Learning Model Development Framework for Computer Vision.
 
-Key Features :
+Features:
+- Easy to use, scalable, and flexible utilizing [PyTorchLightning](https://pytorchlightning.ai/)
+- Unified execution for various computer vision task, such as classification and object detection
 
-- Easy CLI usage
-- Modular design, reusable components
-- Various architecture support
-    - 50 + infamous backbone networks
-    - Classification and Detection architecture support ( Detection models are yet to be verified, may produce bad results )
-- Integration with image augmentation library ( [Albumentations](https://albumentations.readthedocs.io/en/latest/) )
-- Integration with hyperparameter optimization library ( [Optuna](https://optuna.org/) )
-- Integration with 3rd party experiment logger ( [Comet.ml](https://www.comet.ml/site/) )
-- Graph export to [Torchscript](https://pytorch.org/docs/stable/jit.html) and [ONNX](https://onnx.ai/)
-- Visual report of model's performance and resource usage, see [this example](experiments/outputs/resnet18_softmax_cifar10/reports/resnet18_softmax_cifar10_validation_cuda:0.md)
+
+## Example
+
+1. Export and Inference example
+Train your model on any framework, export to onnx, embed metadata and run on vortex. See [this example](examples/export/export.py) for export trained model then run on vortex.
+2. Training a Classifier, Export and Benchmark using Vortex. See [this example](examples/mnasnet/mnasnet.py) for more details.
 
 ## Installation
 
 see detailed guide: https://nodefluxio.github.io/vortex/#installation
 
 ## User Guides & Documentations
 
-see [Vortex Documentation](https://nodefluxio.github.io/vortex/)
-
-## Development Pipelines
-
-![Vortex flow diagram](docs/images/vortex_development_pipeline.jpg)
-
-## Developer Guides
-
-### Run Unit Tests (with coverage)
-- run `pytest` with `htmlcov` report :
-```
-pytest tests/
-```
-### Run CLI tools Tests :
-- see [tests/cli/README.md](tests/cli/README.md)
+see [Vortex Documentation](https://nodefluxio.github.io/vortex/)
diff --git a/docs/api/exporter.rst b/docs/api/exporter.rst
@@ -0,0 +1,14 @@
+.. py:module:: vortex.development.exporter.onnx
+.. py:currentmodule:: vortex.development.exporter.onnx
+
+Exporter
+========
+
+Vortex includes onnx exporter to ease exporting model and embedding model properties.
+
+.. autoclass:: vortex.development.exporter.onnx.ONNXExporter
+
+  .. automethod:: __call__
+  .. automethod:: export
+  .. automethod:: embed_properties
+  .. automethod:: finalize_export
diff --git a/docs/api/factory.rst b/docs/api/factory.rst
diff --git a/docs/api/graph_ops.rst b/docs/api/graph_ops.rst
@@ -0,0 +1,35 @@
+.. py:module:: vortex.runtime.onnx.graph_ops
+.. py:currentmodule:: vortex.runtime.onnx.graph_ops
+
+ONNX graph ops utility
+======================
+
+Collections of utilty to manipulate onnx graph.
+
+.. autofunction:: vortex.runtime.onnx.graph_ops.embed_model_property.embed_model_property
+
+.. autofunction:: vortex.runtime.onnx.graph_ops.embed_class_names_metadata.embed_class_names_metadata
+
+.. autofunction:: vortex.runtime.onnx.graph_ops.embed_output_format_metadata.embed_output_format_metadata
+
+.. autofunction:: vortex.runtime.onnx.graph_ops.embed_metadata.embed_metadata
+
+.. autoclass:: vortex.runtime.onnx.graph_ops.embed_metadata.EmbedMetadata
+  :members:
+
+  .. automethod:: __call__
+
+.. autoclass:: vortex.runtime.onnx.graph_ops.embed_model_property.EmbedModelProperty
+  :members:
+
+  .. automethod:: __call__
+
+.. autoclass:: vortex.runtime.onnx.graph_ops.embed_class_names_metadata.EmbedClassNamesMetadata
+  :members:
+
+  .. automethod:: __call__
+
+.. autoclass:: vortex.runtime.onnx.graph_ops.embed_output_format_metadata.EmbedOutputFormatMetadata
+  :members:
+
+  .. automethod:: __call__
diff --git a/docs/api/index.rst b/docs/api/index.rst
@@ -5,6 +5,6 @@ References
     :maxdepth: 2
 
     models
-    factory
-    pipelines
+    exporter
+    graph_ops
     runtime
diff --git a/docs/api/pipelines.rst b/docs/api/pipelines.rst
diff --git a/docs/api/runtime.rst b/docs/api/runtime.rst
@@ -0,0 +1,11 @@
+.. py:module:: vortex.runtime.helper
+.. py:currentmodule:: vortex.runtime.helper
+
+Runtime Helper
+==============
+
+.. autoclass:: vortex.runtime.helper.InferenceHelper
+  :members:
+
+  .. automethod:: __call__
+
diff --git a/docs/conf.py b/docs/conf.py
@@ -105,9 +105,17 @@
 
 sphinx_gallery_conf = {
     # path to your example scripts
-    'examples_dirs': os.path.abspath('../examples/custom_models/alexnet'),
+    'examples_dirs': [
+        os.path.abspath('../examples/export'),
+        os.path.abspath('../examples/mnasnet'),
+        os.path.abspath('../examples/detr'),
+    ],
     # path to where to save gallery generated output
-    'gallery_dirs': '_build/example',
+    'gallery_dirs': [
+        '_build/examples/export',
+        '_build/examples/classification',
+        '_build/examples/detection',
+    ],
     # specify that examples should be ordered according to filename
     'within_subsection_order': FileNameSortKey,
     # directory where function granular galleries are stored

diff --git a/docs/examples/classification.rst b/docs/examples/classification.rst
@@ -0,0 +1,9 @@
+Define MNASNet model using :py:mod:`~vortex.development.networks.models.ModelBase`, then export using :py:mod:`~vortex.development.exporter.onnx.ONNXExporter`.
+
+.. raw:: html
+
+    <a href="https://colab.research.google.com/drive/184aKWgYAChwyWa9x348ep4734wj8SiUn">
+        <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+    </a>
+
+.. include:: ../_build/examples/classification/mnasnet.rst
diff --git a/docs/examples/detection.rst b/docs/examples/detection.rst
@@ -0,0 +1,7 @@
+.. raw:: html
+
+    <a href="https://colab.research.google.com/drive/1XsiShhLTR6-rt-GmQsfOXLRnrt1lW1Sa?usp">
+        <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+    </a>
+
+.. include:: ../_build/examples/detection/detr.rst
diff --git a/docs/examples/export.rst b/docs/examples/export.rst
@@ -0,0 +1,12 @@
+Export pretrained DETR to onnx, and inference using vortex runtime.
+The exporting part use default export from pytorch, ``torch.onnx.export``,
+then additional metadata is embedded to model using vortex' utility :py:mod:`~vortex.runtime.onnx.graph_ops.EmbedModelProperty`.
+Vortex provides inference and visualization helper via :py:mod:`~vortex.runtime.helper.InferenceHelper`.
+
+.. raw:: html
+
+    <a href="https://colab.research.google.com/drive/1VO2NvjcM7HIpY0SryTc3gmr64XJg05fq">
+        <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+    </a>
+
+.. include:: ../_build/examples/export/export.rst
diff --git a/docs/examples/index.rst b/docs/examples/index.rst
@@ -0,0 +1,9 @@
+Examples
+===============
+
+.. toctree::
+    :maxdepth: 1
+
+    export
+    classification
+    detection
diff --git a/docs/images/vortex_icon.png b/docs/images/vortex_icon.png
diff --git a/docs/index.rst b/docs/index.rst
@@ -1,18 +1,31 @@
 Welcome to Vortex's documentation!
 ==================================
 
-Vortex is Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc in purus 
-tempus, ullamcorper ante vitae, lobortis nulla. Nullam purus nulla, porttitor 
-eu velit eget, luctus iaculis magna. Praesent a porttitor lorem. Integer a 
-elit vel erat venenatis sollicitudin. Proin finibus nisl mi, hendrerit facilisis 
-lacus sollicitudin at. Praesent convallis, magna a fringilla suscipit, nibh metus 
-blandit diam, vel luctus ante augue vel nisi. Proin suscipit malesuada sem vitae 
-lobortis. Nullam luctus massa eget sem eleifend porta. Proin quis feugiat lacus.
-
-Morbi rutrum lorem sem, in posuere enim aliquet et. Morbi ac imperdiet enim. 
-Nullam sed tempus dolor, a faucibus justo. Nam ac faucibus mi. Nam hendrerit 
-eleifend eros eget auctor. Quisque interdum rhoncus augue. Nulla vitae imperdiet 
-nulla.
+Vortex aims to unify various task for deep learning based computer vision,
+such as classification and object detection, into single interface.
+Usually, classification and object detection has different handling mechanism
+when dealing with predicted tensor. For instance, classification task often
+predicts (possibly batched) class label and class confidence in single tensor
+while detection task requires different handling mechanism to deal with variable
+detected object hence can't use single tensor for batched output. Furthermore,
+the arrangement of values may not be the same for one model to another, for example,
+one may organize the predicted class label at the first index of prediction tensor
+while other may prefer class confidence first.
+
+To deal the such problems, we simply annotate the model with neccessary information
+about the prediction tensor. Specifically, we unify the way we take the prediction
+tensor using generic operation using numpy.
+
+The project consists of two major parts: ``development`` and ``runtime``. The ``development``
+part define our :py:mod:`~vortex.development.networks.models.ModelBase` that enforce
+additional information regarding prediction tensor to be defined. Also, an ``onnx`` exporter
+for such model is provided at :py:mod:`~vortex.development.exporter.onnx.ONNXExporter`.
+The ``runtime`` part provides class to perform inference on model with such metadata.
+Our :py:mod:`~vortex.development.networks.models.ModelBase` is derived from PyTorchLightning_,
+so we can easily define scalable model with maximum flexibility, to learn more about PyTorchLightning
+including how to train the model, please refer to https://pytorchlightning.ai/.
+
+.. _PyTorchLightning: https://github.com/PyTorchLightning/pytorch-lightning
 
 
 Contents:
@@ -23,9 +36,9 @@ Contents:
 
    installing
    quickstart
-   user_guides/index
-   modules/index
+   examples/index
    api/index
+   metadata
 
 
 

diff --git a/docs/metadata.rst b/docs/metadata.rst
@@ -0,0 +1,114 @@
+Vortex Metadata
+===============
+
+Here we describe metadata format used by vortex.
+
+1. Output Format
+----------------
+Output format **defines the slices** for each output for **single batch**.
+Our interface relies on numpy's take_ operation to extract the output
+from possibly-batched prediction tensor. Numpy take is utilized to
+take elements from an array along an axis. Arguments to numpy take
+that we use for extracting the output can be summarized in the
+following table:
+
++-----------+---------------------------------------+
+| Argument  | Description                           |
++===========+=======================================+
+| `indices` | The indices of values to extract      |
++-----------+---------------------------------------+
+| `axis`    | The axis over which values to extract |
++-----------+---------------------------------------+
+
+We also assume that the output tensor is batched, hence if the model
+is defined for single batch, one may need to unsqueeze or expand the dimension
+at the first axis.
+
+The number of output that can be defined from the output tensor is not limited.
+The output format is expected to be nested dictionary, with outer dictionary
+defines what the output is and the inner dictionary defines the arguments to
+numpy take.
+
+For example, assume we have classification model with the prediction tensor consists
+of class labels and class confidences, as follow:
+
+.. code-block:: python
+
+    array([[3 , 0.71518937],
+        [1, 0.54488318],
+        [2, 0.64589411],
+        [2, 0.891773  ]])
+
+
+where the array is 2-dimensional array, with shape of :code:`(4,2)`, with first axis is the batch index,
+and the second axis represents class labels and class confidences for the corresponding batch index.
+For instance, the prediction of the first batch is :code:`[3 , 0.71518937]`, where the predicted class
+is :code:`3` and the confidence is :code:`0.71518937`.
+
+Let's :code:`'class_label'` and :code:`'class_confidence'` represent the predicted class labels and class_confidence,
+the output format should be:
+
+.. code-block:: python
+
+    output_format = dict(
+        class_label=dict(
+            indices=[0],
+            axis=0,
+        ),
+        class_confidence=dict(
+            indices=[1],
+            axis=0,
+        )
+    )
+
+Note that the output format is defined to extract output of single batch.
+
+
+.. _take: https://numpy.org/doc/stable/reference/generated/numpy.take.html
+
+2. Class Names
+--------------
+Class Names is simply a mapping from integer to string representing a class name.
+This can be described using dictionary, for example:
+
+.. code-block:: python
+
+    class_names = dict(
+        0='cat',
+        1='dog',
+    )
+
+3. Embedding Model Metadata
+---------------------------
+To embed the described metadata above, one may use utility function
+:py:mod:`~vortex.runtime.onnx.graph_ops.embed_model_property.embed_model_property`
+which takes onnx model and model property defined as nested dictionary, for example:
+
+.. code-block:: python
+
+    output_format = dict(
+        class_label=dict(
+            indices=[0],
+            axis=0,
+        ),
+        class_confidence=dict(
+            indices=[1],
+            axis=0,
+        )
+    )
+    class_names = dict(
+        0='cat',
+        1='dog',
+    )
+    model_props = dict(
+        class_names=class_names,
+        output_format=output_format,
+    )
+
+    model : onnx.ModelProto = embed_model_property(model,model_props)
+
+
+Note that the model above is already an onnx model, and 
+:py:mod:`~vortex.runtime.onnx.graph_ops.embed_model_property.embed_model_property` also
+expects onnx model. This may be useful when one wants to use model defined on other framework.
+since most deep learning also support exporting to onnx.