Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot use vizplugins together with subprocesses #10

Open
mbway opened this issue Jan 8, 2024 · 0 comments
Open

Cannot use vizplugins together with subprocesses #10

mbway opened this issue Jan 8, 2024 · 0 comments

Comments

@mbway
Copy link

mbway commented Jan 8, 2024

I wanted to track CPU and memory usage of a program that uses subprocesses and found that using vizplugins crashes when the first subprocess closes.

For example:

from concurrent.futures import ProcessPoolExecutor

def multiply(a, b):
    return a * b

with ProcessPoolExecutor(max_workers=1) as e:
    f = e.submit(multiply, 1, 2)
    print(f.result())

Run with

viztracer --plugin "vizplugins --cpu_usage --memory_usage" -- example.py

Crashes with:

AssertionError: can only test a child process
  File "/usr/lib/python3.11/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.11/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/lib/python3.11/site-packages/viztracer/viztracer.py", line 303, in exit_routine
    self.stop()
  File "<venv>/lib/python3.11/site-packages/viztracer/viztracer.py", line 186, in stop
    self._plugin_manager.event("post-stop")
  File "<venv>/lib/python3.11/site-packages/viztracer/vizplugin.py", line 122, in event
    self._send_message(plugin, "event", {"when": when})
  File "<venv>/lib/python3.11/site-packages/viztracer/vizplugin.py", line 110, in _send_message
    ret = plugin.message(m_type, payload)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/lib/python3.11/site-packages/vizplugins/psutil_monitor.py", line 27, in message
    return self.stop_recording()
           ^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/lib/python3.11/site-packages/vizplugins/psutil_monitor.py", line 47, in stop_recording
    self.recordings.append(self.send_action("stop"))
                           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/lib/python3.11/site-packages/vizplugins/psutil_monitor.py", line 75, in send_action
    if not self.cpu_process.is_alive():
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: can only test a child process

I ended up creating a custom reporter that logs system-wide CPU and RAM usage to a VizCounter from a thread of the main process as a workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant