Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add troubleshooting for Tensor Stream Creation Failed error #10

Merged
merged 1 commit into from
Jan 22, 2024
Merged

Add troubleshooting for Tensor Stream Creation Failed error #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified public/.doctrees/environment.pickle
View file
Open in desktop
Binary file not shown.
Binary file modified public/.doctrees/troubleshooting/dev_env.doctree
View file
Open in desktop
Binary file not shown.
28 changes: 28 additions & 0 deletions public/_sources/troubleshooting/dev_env.rst.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,34 @@
Developer Environment
=====================

GXF Failure due to Tensor Stream Creation Failed
------------------------------------------------

When launching an Isaac ROS node on a Jetson accessed remotely with X11 forwarding,
X events emitted by CUDA for initializing a tensor stream are captured by the remote
monitoring machine instead of the local Jetson.

Symptom
~~~~~~~

.. code:: bash

[component_container_mt-2] 2023-09-15 14:33:11.362 ERROR /workspaces/isaac_ros-dev/src/isaac_ros_image_pipeline/isaac_ros_image_proc/gxf/tensorops/extensions/tensorops/components/TensorStream.cpp@104: tensor stream creation failed.
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/entity_warden.cpp@380: Failed to initialize component 00292 (stream)
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/core/runtime.cpp@616: Could not initialize entity 'IPOQAFVQYV_global' (E290): GXF_FAILURE
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/program.cpp@205: Failed to activate entity 00290 named IPOQAFVQYV_global: GXF_FAILURE
[component_container_mt-2] 2023-09-15 14:33:11.363 ERROR gxf/std/program.cpp@207: Deactivating...
[component_container_mt-2] 2023-09-15 14:33:11.396 ERROR gxf/core/runtime.cpp@1227: Graph activation failed with error: GXF_FAILURE
[component_container_mt-2] [ERROR] [1694759591.396604150] [left_decoder_node]: [NitrosContext] GxfGraphActivate Error: GXF_FAILURE
[component_container_mt-2] [INFO] [1694759591.396906843] [right_decoder_node]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_h264_decoder/share/isaac_ros_h264_decoder/EZXPATGEMM.yaml'
[component_container_mt-2] [ERROR] [1694759591.396922620] [left_decoder_node]: [NitrosNode] runGraphAsync Error: GXF_FAILURE
[component_container_mt-2] terminate called after throwing an instance of 'std::runtime_error'
[component_container_mt-2] what(): [NitrosNode] runGraphAsync Error: GXF_FAILURE

Solution
~~~~~~~~

Disabling X11 forwarding will prevent X events from being intercepted by the remote machine, resolving the error.

Docker ``buildx`` plugin not installed
--------------------------------------
Expand Down
17 changes: 11 additions & 6 deletions public/about/contact.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1458,31 +1458,36 @@
</li>
<li class="toctree-l1"><a class="reference internal" href="../troubleshooting/index.html">Troubleshooting</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/dev_env.html">Developer Environment</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#docker-buildx-plugin-not-installed">Docker <code class="docutils literal notranslate"><span class="pre">buildx</span></code> plugin not installed</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#gxf-failure-due-to-tensor-stream-creation-failed">GXF Failure due to Tensor Stream Creation Failed</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#symptom">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#solution">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#repository-cloned-without-pulling-git-lfs-files">Repository cloned without pulling <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code> files</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#docker-buildx-plugin-not-installed">Docker <code class="docutils literal notranslate"><span class="pre">buildx</span></code> plugin not installed</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id1">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id2">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#x11-forward-issue">X11-forward issue</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#repository-cloned-without-pulling-git-lfs-files">Repository cloned without pulling <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code> files</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id3">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id4">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#ros-2-domain-id-collision">ROS 2 Domain ID Collision</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#x11-forward-issue">X11-forward issue</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id5">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id6">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#no-opt-ros-humble-install-directory-in-docker-container">No <code class="docutils literal notranslate"><span class="pre">/opt/ros/humble/install</span></code> directory in Docker container</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#ros-2-domain-id-collision">ROS 2 Domain ID Collision</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id7">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id8">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#no-opt-ros-humble-install-directory-in-docker-container">No <code class="docutils literal notranslate"><span class="pre">/opt/ros/humble/install</span></code> directory in Docker container</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id9">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id10">Solution</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/hardware_setup.html">Hardware Setup</a><ul>
Expand Down Expand Up @@ -1680,7 +1685,7 @@ <h1>Contact<a class="headerlink" href="#contact" title="Link to this heading">

<div role="contentinfo">
<p>&#169; Copyright 2023, NVIDIA.
<span class="lastupdated">Last updated on Dec 09, 2023.
<span class="lastupdated">Last updated on Jan 22, 2024.
</span></p>
</div>

Expand Down
17 changes: 11 additions & 6 deletions public/about/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1458,31 +1458,36 @@
</li>
<li class="toctree-l1"><a class="reference internal" href="../troubleshooting/index.html">Troubleshooting</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/dev_env.html">Developer Environment</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#docker-buildx-plugin-not-installed">Docker <code class="docutils literal notranslate"><span class="pre">buildx</span></code> plugin not installed</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#gxf-failure-due-to-tensor-stream-creation-failed">GXF Failure due to Tensor Stream Creation Failed</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#symptom">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#solution">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#repository-cloned-without-pulling-git-lfs-files">Repository cloned without pulling <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code> files</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#docker-buildx-plugin-not-installed">Docker <code class="docutils literal notranslate"><span class="pre">buildx</span></code> plugin not installed</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id1">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id2">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#x11-forward-issue">X11-forward issue</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#repository-cloned-without-pulling-git-lfs-files">Repository cloned without pulling <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code> files</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id3">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id4">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#ros-2-domain-id-collision">ROS 2 Domain ID Collision</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#x11-forward-issue">X11-forward issue</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id5">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id6">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#no-opt-ros-humble-install-directory-in-docker-container">No <code class="docutils literal notranslate"><span class="pre">/opt/ros/humble/install</span></code> directory in Docker container</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#ros-2-domain-id-collision">ROS 2 Domain ID Collision</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id7">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id8">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#no-opt-ros-humble-install-directory-in-docker-container">No <code class="docutils literal notranslate"><span class="pre">/opt/ros/humble/install</span></code> directory in Docker container</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id9">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id10">Solution</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/hardware_setup.html">Hardware Setup</a><ul>
Expand Down Expand Up @@ -1685,7 +1690,7 @@ <h1>About<a class="headerlink" href="#about" title="Link to this heading"></a

<div role="contentinfo">
<p>&#169; Copyright 2023, NVIDIA.
<span class="lastupdated">Last updated on Dec 09, 2023.
<span class="lastupdated">Last updated on Jan 22, 2024.
</span></p>
</div>

Expand Down
17 changes: 11 additions & 6 deletions public/about/license.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1458,31 +1458,36 @@
</li>
<li class="toctree-l1"><a class="reference internal" href="../troubleshooting/index.html">Troubleshooting</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/dev_env.html">Developer Environment</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#docker-buildx-plugin-not-installed">Docker <code class="docutils literal notranslate"><span class="pre">buildx</span></code> plugin not installed</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#gxf-failure-due-to-tensor-stream-creation-failed">GXF Failure due to Tensor Stream Creation Failed</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#symptom">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#solution">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#repository-cloned-without-pulling-git-lfs-files">Repository cloned without pulling <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code> files</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#docker-buildx-plugin-not-installed">Docker <code class="docutils literal notranslate"><span class="pre">buildx</span></code> plugin not installed</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id1">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id2">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#x11-forward-issue">X11-forward issue</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#repository-cloned-without-pulling-git-lfs-files">Repository cloned without pulling <code class="docutils literal notranslate"><span class="pre">git-lfs</span></code> files</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id3">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id4">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#ros-2-domain-id-collision">ROS 2 Domain ID Collision</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#x11-forward-issue">X11-forward issue</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id5">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id6">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#no-opt-ros-humble-install-directory-in-docker-container">No <code class="docutils literal notranslate"><span class="pre">/opt/ros/humble/install</span></code> directory in Docker container</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#ros-2-domain-id-collision">ROS 2 Domain ID Collision</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id7">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id8">Solution</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting/dev_env.html#no-opt-ros-humble-install-directory-in-docker-container">No <code class="docutils literal notranslate"><span class="pre">/opt/ros/humble/install</span></code> directory in Docker container</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id9">Symptom</a></li>
<li class="toctree-l4"><a class="reference internal" href="../troubleshooting/dev_env.html#id10">Solution</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/hardware_setup.html">Hardware Setup</a><ul>
Expand Down Expand Up @@ -1680,7 +1685,7 @@ <h1>License<a class="headerlink" href="#license" title="Link to this heading">

<div role="contentinfo">
<p>&#169; Copyright 2023, NVIDIA.
<span class="lastupdated">Last updated on Dec 09, 2023.
<span class="lastupdated">Last updated on Jan 22, 2024.
</span></p>
</div>

Expand Down
Loading