General Layout Support #447

yzhliu · 2018-04-18T17:47:40Z

This PR allows NNVM to do

Replace an operator via AlterOpLayout. For x86 it is more efficient to calculate convolution in NCHW16c layout. Here is an example of how it is used: https://github.com/yzhliu/topi-intel/blob/master/e2e_general_pack/schedule_pack/avx512_conv_fwd.py#L128-L155
Note that kernel (pre-)packing is supported as well: https://github.com/dmlc/nnvm/issues/303
Infer and correct layout automatically
- Given the input data layout, or the layout of operators in the network (e.g., convolution, pooling...) InferCorrectLayout pass can infer the layout for each operator (both inputs and outputs).
- If the required input layout of an operator is different from what it receives, a __layout_transform__ operator will be inserted.
- Each operator registers a function FInferLayout. Once a model is imported, we do a InferCorrectLayout pass and store the original layouts for each operator. After AlterOpLayout, the InferCorrectLayout runs again. This time each operator will see the original layouts it inferred before, which can be used to decide whether to keep the original one. For example, softmax can still generate correct result after the input layout changes, while flatten cannot. So flatten claims it needs the original layout and triggers a layout transform, which make the network produce correct result.
- With such approach,
  - optimized layout (e.g., NCHW16c) can flow through the network as far as possible. No pack/unpack overhead for each (e.g., convolution) operator.
  - now model layout can be transparent to users. Even though a convolution neural network is trained with NHWC layout, users can still pass NCHW input as long as the input layout is specified, a layout transform happens automatically.

Moreover the convolution kernel layout becomes much clearer: #372 . Now we have kernel layout OIHW and HWIO for NCHW and NHWC respectively.

Will shoot a pull request for corresponding TVM changes.

@yidawang @kevinthesun

kevinthesun

Since this is an important change, I suggest latter we add a tutorial of how to use customized layout, especially what kinds of work and how much effort a developer need to support their own layout. From there we can continue iterating on layout related stuff.

kevinthesun · 2018-04-18T18:28:33Z

python/nnvm/frontend/mxnet.py

@@ -237,7 +247,6 @@ def _upsampling(inputs, attrs):
    'min_axis'      : _rename('min'),
    'reshape'       : _reshape,
    'sum_axis'      : _rename('sum'),
-    'UpSampling'    : _upsampling


Why removing this?

oh I was removing SSD stuff but by mistake remove this...

kevinthesun · 2018-04-18T18:46:17Z

include/nnvm/top/nn.h

+  }
+};
+
+struct MultiBoxPriorParam : public dmlc::Parameter<MultiBoxPriorParam> {


We can remove SSD related stuff from this PR. I will make a separate PR to add SSD operators and tests after TVM SSD related PR is merged.

kevinthesun · 2018-04-18T19:01:02Z

python/nnvm/compiler/build_module.py

@@ -204,8 +218,8 @@ def build(graph, target=None, shape=None, dtype="float32", params=None, target_h
        By default, llvm is used if it is enabled,
        otherwise a stackvm intepreter is used.

-    initialize : bool, optional
-        Whether to initialize variables in global dict _all_var_init.
+    layout : dict of str to str or str


Does user need to specify opt_level=3 to enable layout transform? Also how does user specify "internal" layout, if the target layout is not NCHWXc?

opt_level=3 is for AlterOpLayout. the param here is for user to specify the input layout. user do not specify internal layout, it is inferred by operators have layout, e.g., conv, pool.

kevinthesun · 2018-04-18T19:25:11Z

src/top/elemwise_op_common.h

+  Layout in, last_in, out;
+  deduce(&in, in_layouts, in_size, "input");
+  deduce(&last_in, last_in_layouts, in_size, "input (last infer pass)");
+  deduce(&out, out_layouts, out_size, "output");


Assume an elemwise_add op has two inputs with layout "NCHW16c" and "NCHW4c", will they be transformed back to "NCHW", or one of them will be transformed to another?

right transform to left. it's defined in ElemwiseBinaryKeepLeftLayout

tqchen · 2018-04-19T02:28:57Z

include/nnvm/compiler/packed_func_ext.h

+
+template<>
+struct extension_class_info<nnvm::compiler::SymbolArray> {
+  static const int code = 19;


cannot use 19, because it is taken by mxnet tvm bridge https://github.com/apache/incubator-mxnet/blob/master/include/mxnet/tensor_blob.h#L49

It seems to be quite dangerous to expose a vector of pointers here.

tqchen · 2018-04-19T02:32:51Z

include/nnvm/compiler/op_attr_types.h

-                                              std::vector<TLayoutInfo> *olayouts)>;
+using FTVMAlterOpLayout = std::function<
+  Symbol(const NodeAttrs& attrs,
+         const SymbolArray& inputs,


do we really need the input symbols, or is tinfo enough for now?

ignore this comment, I now get what is going on

tqchen · 2018-04-19T02:36:05Z

include/nnvm/compiler/op_attr_types.h

-                                              std::vector<TLayoutInfo> *ilayouts,
-                                              std::vector<TLayoutInfo> *olayouts)>;
+using FTVMAlterOpLayout = std::function<
+  Symbol(const NodeAttrs& attrs,


Can we just return the result output layout as vector of string?

ignore this

tqchen · 2018-04-19T02:45:43Z

src/pass/infer_correct_layout.cc

+ * \brief A simple layout infer pass that will
+ *        insert layout transform nodes automatically.
+ */
+nnvm::Graph InferCorrectLayout(nnvm::Graph src) {


Maybe we can just call it CorrectLayout

ok. will change.

tqchen · 2018-04-19T02:46:54Z

python/nnvm/top/symbol_array.py

+_sym_arr_get = tvm.get_global_func("nnvm.compiler._symbol_array_get")
+_sym_arr_size = tvm.get_global_func("nnvm.compiler._symbol_array_size")
+
+class SymbolArray(object):


We can just use symbol for this, because we can group multiple nodes into a single Symbol

yzhliu · 2018-04-20T21:07:39Z

include/nnvm/layout.h

+
+class Layout {
+ public:
+  using LayoutDim = char;


@tqchen
Do you want me to change the name 'dim' to 'axis' as you suggested in https://discuss.tvmlang.org/t/datalayout-structure/80/3 ?

And to my understanding, we don't have to wait for that DataLayout implemented in TVM. We can merge this PR first (so that we can move forward to having significant improvement - at least 50% speedup - for CNN) and switch to that in TVM later, right?

yes, we can merge this in first, please fix the problems I mentioned, as well as the tests

tqchen · 2018-04-24T01:27:52Z

@merrymercy Can you also do a review when you have time?

yzhliu · 2018-04-24T04:00:49Z

@tqchen The segment fault in CI looks weird. I cannot reproduce. And it happened in PR #435 as well: http://mode-gpu.cs.washington.edu:8080/blue/organizations/jenkins/dmlc%2Fnnvm/detail/PR-435/13/pipeline

tqchen · 2018-04-24T04:28:31Z

src/pass/correct_layout.cc

+      const Layout& produce = produce_ilayouts[i];
+      const Layout& request = request_ilayouts[i];
+      if (produce != request && produce.defined()) {
+        LOG(INFO) << "Insert layout transformer for " << i << "-th input of "


This seems to produce a lot of messages, maybe avoid print this, since this is expected behavior of the pass

tqchen · 2018-04-25T16:07:46Z

Thanks for improving the code through the review! this is now merged

* [RUNTIME] Enable extension type to PackedFunc. * More comments

yzhliu added 2 commits April 18, 2018 01:36

change, infer, and correct Layout

fed37f4

fix pooling layout inference

ec54d21

kevinthesun reviewed Apr 18, 2018

View reviewed changes

tqchen suggested changes Apr 19, 2018

View reviewed changes

yzhliu added 3 commits April 19, 2018 13:22

add upsampling back and remove mutlibox param

5cf39b5

remove SymbolArray

96b361b

rename _contrib_conv2d_nChwc -> _contrib_conv2d_NCHWc

9d73b0a

yzhliu commented Apr 20, 2018

View reviewed changes

fix conv bias_layout

341f9f0

yzhliu mentioned this pull request Apr 23, 2018

[TOPI] parallel schedule improve for x86 & layout_transform support apache/tvm#1130

Merged

yzhliu added 4 commits April 23, 2018 13:56

InferCorrectLayout -> CorrectLayout

dbe54ca

Merge remote-tracking branch 'origin/master' into layout.0418

0596771

update submodule tvm

63d8667

fix layout document

9e1f18d

remove object_detection import

170ea55

tqchen suggested changes Apr 24, 2018

View reviewed changes

yzhliu added 2 commits April 24, 2018 10:55

remove unnecessary log in CorrectLayout

7e905da

add missing FInferLayout. fix conv_nhwc test case

44e9241

tqchen approved these changes Apr 25, 2018

View reviewed changes

tqchen merged commit 8994059 into dmlc:master Apr 25, 2018

abergeron pushed a commit to abergeron/nnvm that referenced this pull request May 31, 2018

[RUNTIME] Enable extension type to PackedFunc. (dmlc#447)

f2ab736

* [RUNTIME] Enable extension type to PackedFunc. * More comments

abergeron pushed a commit to abergeron/nnvm that referenced this pull request May 31, 2018

General Layout Support (dmlc#447)

8f9c593

merrymercy mentioned this pull request Jun 24, 2018

[TOPI] Winograd apache/tvm#899

Closed

4 tasks

larroy pushed a commit to larroy/nnvm that referenced this pull request Feb 8, 2019

General Layout Support (dmlc#447)

9f8fcfc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

General Layout Support #447

General Layout Support #447

yzhliu commented Apr 18, 2018 •

edited

Loading

kevinthesun left a comment

kevinthesun Apr 18, 2018

yzhliu Apr 18, 2018

kevinthesun Apr 18, 2018

kevinthesun Apr 18, 2018

yzhliu Apr 18, 2018

kevinthesun Apr 18, 2018

yzhliu Apr 18, 2018

tqchen Apr 19, 2018

tqchen Apr 19, 2018

tqchen Apr 19, 2018

tqchen Apr 19, 2018

tqchen Apr 19, 2018

tqchen Apr 19, 2018

tqchen Apr 19, 2018

yzhliu Apr 20, 2018

tqchen Apr 19, 2018

yzhliu Apr 20, 2018

tqchen Apr 24, 2018 •

edited

Loading

tqchen commented Apr 24, 2018

yzhliu commented Apr 24, 2018

tqchen Apr 24, 2018

tqchen commented Apr 25, 2018

General Layout Support #447

General Layout Support #447

Conversation

yzhliu commented Apr 18, 2018 • edited Loading

kevinthesun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tqchen Apr 24, 2018 • edited Loading

Choose a reason for hiding this comment

tqchen commented Apr 24, 2018

yzhliu commented Apr 24, 2018

Choose a reason for hiding this comment

tqchen commented Apr 25, 2018

yzhliu commented Apr 18, 2018 •

edited

Loading

tqchen Apr 24, 2018 •

edited

Loading