[BUG] abp_pcap_detection example fails

[pdmack]

Describe the bug
image built from v22.06.01 hotfix tag

====Registering Pipeline====
====Registering Pipeline Complete!====
====Starting Pipeline====
====Pipeline Started====
Preprocessing rate: 0messages [00:00, ?messages/s]       ====Building Pipeline====
Inference rate: 0inf [00:00, ?inf/s]                     Added source: <from-file-0; FileSourceStage(filename=../data/abp_pcap_dump.jsonlines, iterative=False, file_type=FileTypes.Auto, repeat=1, filter_null=False, cudf_kwargs=None)>
  └─> morpheus.MessageMetaadd-class [00:00, ?add-class/s]
                                    Added stage: <deserialize-1; DeserializeStage()>
  └─ morpheus.MessageMeta -> morpheus.MultiMessage
                                                Added stage: <preprocess-anomaly-2; AbpPcapPreprocessingStage()>
  └─ morpheus.MultiMessage -> morpheus.MultiInferenceFILMessage
                                                Added stage: <monitor-3; MonitorStage(description=Preprocessing rate, smoothing=0.05, unit=messages, delayed_start=False, determine_count_fn=None)>]
  └─ morpheus.MultiInferenceFILMessage -> morpheus.MultiInferenceFILMessage
Added stage: <inference-4; TritonInferenceStage(model_name=abp-pcap-xgb, server_url=ai-engine:8001, force_convert_inputs=True, use_shared_memory=False)>
  └─ morpheus.MultiInferenceFILMessage -> morpheus.MultiResponseProbsMessage
Added stage: <monitor-5; MonitorStage(description=Inference rate, smoothing=0.05, unit=inf, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <add-class-6; AddClassificationsStage(threshold=0.5, labels=['probs'], prefix=)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <monitor-7; MonitorStage(description=Add classification rate, smoothing=0.05, unit=add-class, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <serialize-8; SerializeStage(include=None, exclude=['^ID$', '^_ts_'], fixed_columns=True)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MessageMeta
Added stage: <monitor-9; MonitorStage(description=Serialize rate, smoothing=0.05, unit=ser, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
Added stage: <to-file-10; WriteToFileStage(filename=./pcap_out.jsonlines, overwrite=True, file_type=FileTypes.Auto)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
Added stage: <monitor-11; MonitorStage(description=Write to file rate, smoothing=0.05, unit=to-file, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
====Building Pipeline Complete!====
Preprocessing rate: 50000messages [00:02, 17801.60messageError hit!
ValueError: Unsupported dtype <U36s]
Add classification rate: 0add-class [00:03, ?add-class/s]
At:ialize rate: 0ser [00:02, ?ser/s]
  cupy/_core/core.pyx(2418): cupy._core.core._array_default
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/cupy/_creation/from_data.py(76): asarray
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/cudf/core/dataframe.py(372): _setitem_tuple_arg
  /opt/conda/envs/morpheus/lib/python3.8/contextlib.py(75): inner
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/cudf/core/dataframe.py(143): __setitem__
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/messages/multi_message.py(152): set_meta
  /workspace/examples/abp_pcap_detection/abp_pcap_preprocessing.py(164): pre_process_batch

WARNING: Logging before InitGoogleLogging() is written to STDERR
E20220718 15:20:07.091929 2258049 context.cpp:125] main/preprocess-anomaly-2; rank: 0; size: 1; tid: 140320356079360: set_exception issued; issuing kill to current runnable. Exception msg: ValueError: Unsupported dtype <U36

At:
  cupy/_core/core.pyx(2418): cupy._core.core._array_default
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/cupy/_creation/from_data.py(76): asarray
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/cudf/core/dataframe.py(372): _setitem_tuple_arg
  /opt/conda/envs/morpheus/lib/python3.8/contextlib.py(75): inner
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/cudf/core/dataframe.py(143): __setitem__
  /opt/conda/envs/morpheus/lib/python3.8/site-packages/morpheus/messages/multi_message.py(152): set_meta
  /workspace/examples/abp_pcap_detection/abp_pcap_preprocessing.py(164): pre_process_batch
Preprocessing rate[Complete]: 50000messages [00:03, 17801.E20220718 15:20:07.364555 2257600 runner.cpp:190] Runner::await_join - an exception was caught while awaiting on one or more contexts/instances - rethrowing
E20220718 15:20:07.364625 2257600 instance.cpp:259] segment::Instance - an exception was caught while awaiting on one or more nodes - rethrowing
E20220718 15:20:07.364645 2257600 instance.cpp:218] pipeline::Instance - an exception was caught while awaiting on segments - rethrowing
Preprocessing rate[Complete]: 50000messages [00:02, 17801.90messages/s]
Inference rate[Complete]: 0inf [00:00, ?inf/s]
Add classification rate[Complete]: 0add-class [00:00, ?add-class/s]
Serialize rate[Complete]: 0ser [00:00, ?ser/s]
Write to file rate[Complete]: 0to-file [00:00, ?to-file/s]
====Pipeline Complete====

Steps/Code to reproduce bug

# cd examples/abp_pcap_detection/
(morpheus) root@sdk-cli-sdk1:/workspace/examples/abp_pcap_detection# python run.py \
--input_file ../data/abp_pcap_dump.jsonlines \
--output_file ./pcap_out.jsonlines \
--model_name 'abp-pcap-xgb' \
--server_url ai-engine:8001
# --server_url localhost:8001

Expected behavior
Example runs successfully.

Environment overview (please complete the following information)

• Environment location: [Bare-metal, Docker]
• Method of Morpheus install: [conda, Docker]

Environment details
https://gist.github.com/pdmack/8754bc862b26e465d5151cb6787415a5

Additional context
Possibly related to #270 #280 ?

[bsuryadevara]

@pdmack Can we close this now?

Image: morpheus:v22.09.00b-runtime

(morpheus) root@dgx04:/workspace/examples/abp_pcap_detection# python run.py \
> --input_file ../data/abp_pcap_dump.jsonlines \
> --output_file ./pcap_out.jsonlines \
> --model_name 'abp-pcap-xgb' \
> --server_url localhost:8001
====Registering Pipeline====
====Building Pipeline====
====Building Segment: linear_segment_0====
Preprocessing rate: 0 messages [00:00, ? messages/s]====Building Segment Complete!====
Preprocessing rate: 0 messages [00:00, ? messages/s]====Building Pipeline Complete!====
                                                  ====Registering Pipeline Complete!====
====Starting Pipeline====:00, ? inf/s]
====Pipeline Started==== 0 add-class [00:00, ? add-class/s]
Added source: <from-file-0; FileSourceStage(filename=../data/abp_pcap_dump.jsonlines, iterative=False, file_type=FileTypes.Auto, repeat=1, filter_null=False, cudf_kwargs=None)>
  └─> morpheus.MessageMetaile [00:00, ? to-file/s]
Added stage: <deserialize-1; DeserializeStage()>
  └─ morpheus.MessageMeta -> morpheus.MultiMessage
Added stage: <preprocess-anomaly-2; AbpPcapPreprocessingStage()>
  └─ morpheus.MultiMessage -> morpheus.MultiInferenceFILMessage
Added stage: <monitor-3; MonitorStage(description=Preprocessing rate, smoothing=0.05, unit=messages, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MultiInferenceFILMessage -> morpheus.MultiInferenceFILMessage
Added stage: <inference-4; TritonInferenceStage(model_name=abp-pcap-xgb, server_url=localhost:8001, force_convert_inputs=True, use_shared_memory=False)>
  └─ morpheus.MultiInferenceFILMessage -> morpheus.MultiResponseProbsMessage
Added stage: <monitor-5; MonitorStage(description=Inference rate, smoothing=0.05, unit=inf, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <add-class-6; AddClassificationsStage(threshold=0.5, labels=['probs'], prefix=)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <monitor-7; MonitorStage(description=Add classification rate, smoothing=0.05, unit=add-class, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <serialize-8; SerializeStage(include=[], exclude=['^ID$', '^_ts_'], fixed_columns=True)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MessageMeta
Added stage: <monitor-9; MonitorStage(description=Serialize rate, smoothing=0.05, unit=ser, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
Added stage: <to-file-10; WriteToFileStage(filename=./pcap_out.jsonlines, overwrite=True, file_type=FileTypes.Auto, include_index_col=True)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
Added stage: <monitor-11; MonitorStage(description=Write to file rate, smoothing=0.05, unit=to-file, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
Preprocessing rate[Complete]: 537241 messages [00:06, 85728.27 messages/s]
Inference rate[Complete]: 537241 inf [00:14, 38340.77 inf/s]
Add classification rate[Complete]: 537241 add-class [00:15, 35521.08 add-class/s]
Serialize rate[Complete]: 537241 ser [00:15, 35513.64 ser/s]
Write to file rate[Complete]: 537241 to-file [00:20, 26557.39 to-file/s]
====Pipeline Complete====

[mdemoret-nv]

Closing this since @bsuryadevara moved it to Done.