[BUG]: C++ impl for Triton inference can incorrectly split inference inputs

[dagardner-nv]

Version

23.03

Which installation method(s) does this occur on?

Docker, Conda, Source

Describe the bug.

The Triton inference stage often needs to split up the input based on the model's max batch size, which is quite often much smaller than the the number of rows in the message (pipeline_batch_size), and the input is broken up into what we call a "mini-batch".

We can also have large input fields (typically variable length fields like text) which themselves are larger than the model can accept and need to be split up into multiple inference inputs, and then we perform a reduction on the multiple outputs to produce a single output for the row.

There are currently two related bugs, the first being common:

1. We perform a partition across inputs for the same row, resulting in not all outputs being reduced.
2. If a row's input is so larger that the partitioned inputs themselves are larger than the model's max batch size.

Minimum reproducible example

The first variation of the bug occurs with the 
`scripts/validation/abp/val-abp-all.sh` script if you remove the `--truncation=True` flag from `scripts/validation/val-run-pipeline.sh`.

Relevant log output

No response

Full env printout

No response

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

Fixed in PR #667