Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] pipeline-nlp fails between dropna and preprocess stages (22.06) #240

Closed
pdmack opened this issue Jun 29, 2022 · 0 comments · Fixed by #242
Closed

[BUG] pipeline-nlp fails between dropna and preprocess stages (22.06) #240

pdmack opened this issue Jun 29, 2022 · 0 comments · Fixed by #242
Assignees
Labels
bug Something isn't working

Comments

@pdmack
Copy link
Contributor

pdmack commented Jun 29, 2022

Describe the bug
Traditional NLP file-based smoke test fails in docker runtime image at 7684908

There's an unusual reference to "cannot convert integer from 'version https://git-lfs.github.com/spec/v1': stoi" in the stacktrace.

Configuring Pipeline via CLI
Starting pipeline via CLI... Ctrl+C to Quit
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20220629 16:01:32.859344  1126 triton_inference.cpp:248] Failed to connect to Triton at 'ai-engine:8001'. Default gRPC port of (8001) was detected but C++ InferenceClientStage uses HTTP protocol. Retrying with default HTTP port (8000)
Drop Null Attributes rate[Complete]: 93085messages [00:01, 76Line 1: cannot convert integer from 'version https://git-lfs.github.com/spec/v1': stoi
Deserialization rate: 8192messages [00:01, 6749.30messages/s]
E20220629 16:01:34.085346  1159 context.cpp:125] main/preprocess-nlp-5; rank: 0; size: 1; tid: 139774159091456: set_exception issued; issuing kill to current runnable. Exception msg: stoi: 0inf [00:00, ?inf/s]                        E20220629 16:01:34.090284   959 runner.cpp:190] Runner::await_join - an exception was caught while awaiting on one or more contexts/instances - rethrowingages [00:01, 6749.30messages/s]
E20220629 16:01:34.090341   959 instance.cpp:259] segment::Instance - an exception was caught while awaiting on one or more nodes - rethrowing
E20220629 16:01:34.090363   959 instance.cpp:218] pipeline::Instance - an exception was caught while awaiting on segments - rethrowing
Drop Null Attributes rate[Complete]: 93085messages [00:01, 76784.38messages/s]
Deserialization rate[Complete]: 93085messages [00:01, 76555.85messages/s]
Preprocessing rate[Complete]: 0messages [00:00, ?messages/s]
Inference rate[Complete]: 0inf [00:00, ?inf/s]
Classification rate[Complete]: 0messages [00:00, ?messages/s]
Serialization rate[Complete]: 0messages [00:00, ?messages/s]

Steps/Code to reproduce bug

morpheus --debug run --use_cpp=True --num_threads=8 --pipeline_batch_size=8192 --model_max_batch_size=32 --edge_buffer_size=32  pipeline-nlp --model_seq_length=256 from-file --filename=/common/data/model_data/pcap_dump.jsonlines dropna monitor --description='Drop Null Attributes rate' deserialize monitor --description='Deserialization rate' preprocess --vocab_hash_file=data/bert-base-uncased-hash.txt --truncation=True --do_lower_case=True --add_special_tokens=False monitor --description='Preprocessing rate' inf-triton --force_convert_inputs=True --model_name=sid-minibert-onnx --server_url=ai-engine:8001 --use_shared_memory=True monitor --description='Inference rate' --smoothing=0.001 --unit inf add-class monitor --description='Classification rate' serialize --exclude '^ts_' monitor --description='Serialization rate' to-file --filename=/common/data/output/sid-minibert-onnx-output.jsonlines --overwrite

Expected behavior
Completed pipeline run without errors.

Environment overview (please complete the following information)

  • Environment location: [Bare-metal, Docker]
  • Method of Morpheus install: [Docker/k8s]

Environment details
https://gist.github.com/pdmack/8f9342321523251bf9ecdd8cd349a029

Additional context
Needed by #237

@pdmack pdmack added bug Something isn't working Needs Triage Need team to review and classify labels Jun 29, 2022
@mdemoret-nv mdemoret-nv removed the Needs Triage Need team to review and classify label Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants