Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celery worker fails to start in development due to tty: open #681

Closed
regisb opened this issue Jun 6, 2022 · 0 comments · Fixed by #682
Closed

Celery worker fails to start in development due to tty: open #681

regisb opened this issue Jun 6, 2022 · 0 comments · Fixed by #682
Labels
bug Bugs will be investigated and fixed as quickly as possible.

Comments

@regisb
Copy link
Contributor

regisb commented Jun 6, 2022

Bug description

The lms-worker and cms-worker fail to start in development.

How to reproduce

$ tutor dev start cms-worker
cms-worker_1                 | Traceback (most recent call last):
cms-worker_1                 |   File "/openedx/venv/bin/celery", line 8, in <module>
cms-worker_1                 |     sys.exit(main())
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
cms-worker_1                 |     _main()
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
cms-worker_1                 |     cmd.execute_from_commandline(argv)
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
cms-worker_1                 |     super(CeleryCommand, self).execute_from_commandline(argv)))
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
cms-worker_1                 |     return self.handle_argv(self.prog_name, argv[1:])
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
cms-worker_1                 |     return self.execute(command, argv)
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
cms-worker_1                 |     return cls(
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
cms-worker_1                 |     *self.parse_options(prog_name, argv, command))
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
cms-worker_1                 |     self.parser = self.create_parser(prog_name, command)
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
cms-worker_1                 |     description=self._format_description(self.description),
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
cms-worker_1                 |     text.fill_paragraphs(text.dedent(description), width))
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 391, in fill
cms-worker_1                 |     return w.fill(text)
cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 363, in fill
cms-worker_1                 |     return "\n".join(self.wrap(text))
cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 354, in wrap
cms-worker_1                 |     return self._wrap_chunks(chunks)
cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
cms-worker_1                 |     raise ValueError("invalid width %r (must be > 0)" % self.width)
cms-worker_1                 | ValueError: invalid width -2 (must be > 0)

Additional context

This issue seems to be due to the tty: open option in the dev/docker-compose.yml template. It was reported upstream here: celery/celery#6302
edx-platform runs celery==4.4.7, and the issue is supposed to be fixed in 5.0.0. Indeed, the issue is fixed in Nutmeg, which runs celery==5.2.6.

@regisb regisb added the bug Bugs will be investigated and fixed as quickly as possible. label Jun 6, 2022
regisb added a commit that referenced this issue Jun 6, 2022
Celery workers failed to start in development with the following stacktrace:

    cms-worker_1                 | Traceback (most recent call last):
    cms-worker_1                 |   File "/openedx/venv/bin/celery", line 8, in <module>
    cms-worker_1                 |     sys.exit(main())
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
    cms-worker_1                 |     _main()
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
    cms-worker_1                 |     cmd.execute_from_commandline(argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
    cms-worker_1                 |     super(CeleryCommand, self).execute_from_commandline(argv)))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    cms-worker_1                 |     return self.handle_argv(self.prog_name, argv[1:])
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
    cms-worker_1                 |     return self.execute(command, argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
    cms-worker_1                 |     return cls(
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
    cms-worker_1                 |     *self.parse_options(prog_name, argv, command))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
    cms-worker_1                 |     self.parser = self.create_parser(prog_name, command)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
    cms-worker_1                 |     description=self._format_description(self.description),
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
    cms-worker_1                 |     text.fill_paragraphs(text.dedent(description), width))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 391, in fill
    cms-worker_1                 |     return w.fill(text)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 363, in fill
    cms-worker_1                 |     return "\n".join(self.wrap(text))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 354, in wrap
    cms-worker_1                 |     return self._wrap_chunks(chunks)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
    cms-worker_1                 |     raise ValueError("invalid width %r (must be > 0)" % self.width)
    cms-worker_1                 | ValueError: invalid width -2 (must be > 0)

This issue was reported upstream here: celery/celery#6302

It is caused by the `tty: true` statement, for some reason. It will be fixed in
Nutmeg, after celery is upgraded to 5.2.6.

Close #681.
regisb added a commit that referenced this issue Jun 6, 2022
Celery workers failed to start in development with the following stacktrace:

    cms-worker_1                 | Traceback (most recent call last):
    cms-worker_1                 |   File "/openedx/venv/bin/celery", line 8, in <module>
    cms-worker_1                 |     sys.exit(main())
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
    cms-worker_1                 |     _main()
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
    cms-worker_1                 |     cmd.execute_from_commandline(argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
    cms-worker_1                 |     super(CeleryCommand, self).execute_from_commandline(argv)))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    cms-worker_1                 |     return self.handle_argv(self.prog_name, argv[1:])
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
    cms-worker_1                 |     return self.execute(command, argv)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
    cms-worker_1                 |     return cls(
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
    cms-worker_1                 |     *self.parse_options(prog_name, argv, command))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
    cms-worker_1                 |     self.parser = self.create_parser(prog_name, command)
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
    cms-worker_1                 |     description=self._format_description(self.description),
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
    cms-worker_1                 |     text.fill_paragraphs(text.dedent(description), width))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/openedx/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
    cms-worker_1                 |     return sep.join(fill(p, width) for p in s.split(sep))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 391, in fill
    cms-worker_1                 |     return w.fill(text)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 363, in fill
    cms-worker_1                 |     return "\n".join(self.wrap(text))
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 354, in wrap
    cms-worker_1                 |     return self._wrap_chunks(chunks)
    cms-worker_1                 |   File "/opt/pyenv/versions/3.8.12/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
    cms-worker_1                 |     raise ValueError("invalid width %r (must be > 0)" % self.width)
    cms-worker_1                 | ValueError: invalid width -2 (must be > 0)

This issue was reported upstream here: celery/celery#6302

It is caused by the `tty: true` statement, for some reason. It will be fixed in
Nutmeg, after celery is upgraded to 5.2.6.

Close #681.
regisb added a commit that referenced this issue Jun 6, 2022
- [Fix] Crashing celery workers in development (#681). (by @regisb)
- [Fix] Fix studio logout issue. (by @Alec4r)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bugs will be investigated and fixed as quickly as possible.
Projects
Development

Successfully merging a pull request may close this issue.

1 participant