fix: Error with `aten::div` when using truncation with Int32 tensor inputs #1442

gs-olive · 2022-11-04T22:46:32Z

Description

aten::div with truncation on integer tensor inputs currently throws an error if both inputs are integer type, as the TRT unary operations for absolute value and floor do not apply to Int32 or Bool types
For absolute value, this is a legitimate bug as aten::abs is functional for integer types
For the floor operation, aten::floor does not explicitly support integer inputs, and torch.floor() does not work with Int32 inputs by default (on 1.13.0.dev20220921+cu116). However, torch.div(..., rounding_mode="trunc") with integer tensors does return an integer value, and so the corollary Torch-TRT converter should behave similarly
Modified aten:abs converter logic to be a utility (moved file location), as the operator is used in multiple locations
Added regression test to ensure truncation divide with two integer tensors is functional

Note: The behavior of torch.floor() on Int32 types differs between 1.13.0.dev20220921+cu116 and 1.14.0.dev20221018+cu116: the former does not by default support this operation, while the latter does. This PR does not fix the general aten::floor operator for Int32 inputs, but instead fixes the aten::div truncation operator only.

Fixes #1441
Note: The issue was traced to a problem with aten::div with truncation enabled, and not aten::floor

Type of change

Bug fix (non-breaking change which fixes an issue)

Checklist:

[ x ] My code follows the style guidelines of this project (You can use the linters)
[ x ] I have performed a self-review of my own code
[ x ] I have commented my code, particularly in hard-to-understand areas and hacks
[ x ] I have made corresponding changes to the documentation
[ x ] I have added tests to verify my fix or my feature
[ x ] New and existing unit tests pass locally with my changes
[ x ] I have added the relevant labels to my PR in so that relevant reviewers are notified

gs-olive · 2022-11-04T22:49:50Z

core/conversion/converters/impl/element_wise.cpp

+                 auto abs = add_absolute_value(ctx, n, tmp_div->getOutput(0), util::node_info(n) + "_absolute_val");
+
+                 // In this case, we allow the floor unary on non-TRT Unary types, as it is needed for this
+                 // specific function. Floor applied to non-float types equates to identity
+                 nvinfer1::ILayer* floor;
+                 if ((abs->getOutput(0)->getType() == nvinfer1::DataType::kINT32) ||
+                     (abs->getOutput(0)->getType() == nvinfer1::DataType::kBOOL)) {
+                   LOG_GRAPH(
+                       "Tensor is of unsupported type " << abs->getOutput(0)->getType()
+                                                        << " for IUnaryLayer::kFLOOR. Using identity instead.");
+                   floor = ctx->net->addIdentity(*abs->getOutput(0));
+                   TORCHTRT_CHECK(floor, "Unable to create identity layer from node: " << *n);
+                 } else {
+                   floor = ctx->net->addUnary(*abs->getOutput(0), nvinfer1::UnaryOperation::kFLOOR);
+                   TORCHTRT_CHECK(floor, "Unable to create floor layer from node: " << *n);
+                 }
+                 floor->setName((util::node_info(n) + "_floor").c_str());


In this code block, both the abs and the floor operators were encountering errors when both inputs are integer types. The solution for abs was a converter utility, whereas for floor, the solution only appears here. The reasoning for this choice was that Torch support for torch.floor() applied to an Int32 type differs between 1.13.0.dev20220921+cu116 and 1.14.0.dev20221018+cu116, so it is unclear if the aten::floor converter should generally support Int32 inputs or not, currently.

narendasan · 2022-11-14T20:14:35Z

core/conversion/converters/converter_util.h

@@ -42,6 +42,12 @@ nvinfer1::ILayer* add_elementwise(
    nvinfer1::ITensor* other,
    const std::string& name);

+nvinfer1::ILayer* add_absolute_value(


rename to add_abs

narendasan · 2022-11-14T20:14:50Z

core/conversion/converters/converter_util.h

@@ -42,6 +42,12 @@ nvinfer1::ILayer* add_elementwise(
    nvinfer1::ITensor* other,
    const std::string& name);

+nvinfer1::ILayer* add_absolute_value(


Return nvinfer1::ITensor*

narendasan · 2022-11-14T20:17:39Z

core/conversion/converters/impl/element_wise.cpp

+                 nvinfer1::ILayer* floor;
+                 if ((abs->getOutput(0)->getType() == nvinfer1::DataType::kINT32) ||
+                     (abs->getOutput(0)->getType() == nvinfer1::DataType::kBOOL)) {
+                   LOG_GRAPH(


LOG_DEBUG instead of LOG_GRAPH

narendasan · 2022-11-14T20:18:44Z

core/conversion/converters/impl/element_wise.cpp

+
+                 // In this case, we allow the floor unary on non-TRT Unary types, as it is needed for this
+                 // specific function. Floor applied to non-float types equates to identity
+                 nvinfer1::ILayer* floor;


Work with ITensor* instead ideally

narendasan · 2022-11-14T20:21:37Z

core/conversion/converters/impl/element_wise.cpp

+                   floor = ctx->net->addUnary(*abs->getOutput(0), nvinfer1::UnaryOperation::kFLOOR);
+                   TORCHTRT_CHECK(floor, "Unable to create floor layer from node: " << *n);
+                 }
+                 floor->setName((util::node_info(n) + "_floor").c_str());


This needs to only be applied to the unary layer on line 342 so as not to overwrite the info from the abs on 329

gs-olive · 2022-11-14T22:54:58Z

core/conversion/converters/converter_util.cpp

+    TORCHTRT_CHECK(absolute_value_layer, "Unable to create max layer from node: " << *n);
+  }
+
+  return absolute_value_layer->getOutput(0);


Switched function schema to return the output of the absolute value layer and not the layer itself

gs-olive · 2022-11-14T22:55:46Z

core/conversion/converters/impl/element_wise.cpp

+                   LOG_DEBUG(
+                       "Tensor is of unsupported type " << abs->getType()
+                                                        << " for IUnaryLayer::kFLOOR. Using identity instead.");
+                   floor = abs;


Instead of using the identity function, the floor output tensor is just set to the absolute value result, to avoid unnecessary computation.

- `aten::div` with truncation on integer tensor inputs currently throws an error if both inputs are integer type, as the TRT unary operations for absolute value and floor do not apply to Int32 or Bool types - For absolute value, this is a legitimate bug as `aten::abs` is functional for integer types - For the floor operation, `aten::floor` does not explicitly support integer inputs, and `torch.floor()` does not work with Int32 inputs by default. However, `torch.div(..., rounding_mode="trunc")` with integer tensors does return an integer value, and so the corollary Torch-TRT converter should behave similarly - Modified `aten:abs` converter logic to be a utility, as it is used in multiple locations - Added regression test to ensure truncation divide with two integer tensors is functional - Address comments on PR - Update utility name to add_abs for conciseness - Refactor absolute value utility to return ITensor* - Update logging level for certain debug messages

narendasan

LGTM

facebook-github-bot added the cla signed label Nov 4, 2022

github-actions bot added component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: core Issues re: The core compiler component: tests Issues re: Tests labels Nov 4, 2022

github-actions bot requested review from andi4191, bowang007, narendasan and peri044 November 4, 2022 22:46

gs-olive commented Nov 4, 2022

View reviewed changes

gs-olive force-pushed the trunc_div_bugfix branch from 464fae5 to 01ee345 Compare November 7, 2022 22:47

gs-olive added the release: v1.3 Tagged to be included in v1.3 label Nov 9, 2022

gs-olive self-assigned this Nov 9, 2022

narendasan reviewed Nov 14, 2022

View reviewed changes

gs-olive force-pushed the trunc_div_bugfix branch from 046b6dd to aa9a01f Compare November 14, 2022 22:53

gs-olive commented Nov 14, 2022

View reviewed changes

gs-olive force-pushed the trunc_div_bugfix branch from aa9a01f to dac2da0 Compare November 17, 2022 01:55

narendasan approved these changes Nov 18, 2022

View reviewed changes

narendasan merged commit 3ee60b7 into pytorch:master Nov 18, 2022

gs-olive deleted the trunc_div_bugfix branch November 18, 2022 04:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Error with `aten::div` when using truncation with Int32 tensor inputs #1442

fix: Error with `aten::div` when using truncation with Int32 tensor inputs #1442

gs-olive commented Nov 4, 2022

gs-olive Nov 4, 2022

narendasan Nov 14, 2022

narendasan Nov 14, 2022

narendasan Nov 14, 2022

narendasan Nov 14, 2022

narendasan Nov 14, 2022

gs-olive Nov 14, 2022

gs-olive Nov 14, 2022

narendasan left a comment

fix: Error with aten::div when using truncation with Int32 tensor inputs #1442

fix: Error with aten::div when using truncation with Int32 tensor inputs #1442

Conversation

gs-olive commented Nov 4, 2022

Description

Type of change

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

narendasan left a comment

Choose a reason for hiding this comment

fix: Error with `aten::div` when using truncation with Int32 tensor inputs #1442

fix: Error with `aten::div` when using truncation with Int32 tensor inputs #1442