Allow -OO mode for docstring_decorator

[matthid]

What does this PR do?

Fixes

  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 987, in <module>
    class AutoConfig:
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1011, in AutoConfig
    @replace_list_option_in_docstrings()
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 966, in docstring_decorator
    lines = docstrings.split("\n")
            ^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'

Note; I found this, but I'm now using -O instead of -OO so this fix might not be exhaustive in the sense that now -OO might run into the next issue. But since I run into that one, I figured sending a fix would be worthwhile, feel free to close if you think otherwise.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

[amyeroberts]

Hi @matthid, thannks for opening a PR,

I found this, but I'm now using -O instead of -OO

Could you provide some more context here? Specifically where this change is being made and ideally a minimal code snippet which reproduces the issue?

[matthid]

Sure just use the following script:

from pytorch_lightning.cli import ArgsType

and run the script with python -OO test.py and it will fail

    from pytorch_lightning.cli import ArgsType
  File "/nix/store/16kqsy2chy51c0bgjs3cmlf861ws4hdl-python3.11-pytorch-lightning-2.1.3/lib/python3.11/site-packages/pytorch_lightning/__init__.py", line 27, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/16kqsy2chy51c0bgjs3cmlf861ws4hdl-python3.11-pytorch-lightning-2.1.3/lib/python3.11/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/nix/store/16kqsy2chy51c0bgjs3cmlf861ws4hdl-python3.11-pytorch-lightning-2.1.3/lib/python3.11/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/nix/store/16kqsy2chy51c0bgjs3cmlf861ws4hdl-python3.11-pytorch-lightning-2.1.3/lib/python3.11/site-packages/pytorch_lightning/callbacks/callback.py", line 22, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/nix/store/16kqsy2chy51c0bgjs3cmlf861ws4hdl-python3.11-pytorch-lightning-2.1.3/lib/python3.11/site-packages/pytorch_lightning/utilities/types.py", line 40, in <module>
    from torchmetrics import Metric
  File "/nix/store/8j0ykl0zmidngpshn1wj5ndp04v6q88z-python3.11-torchmetrics-1.3.1/lib/python3.11/site-packages/torchmetrics/__init__.py", line 22, in <module>
    from torchmetrics import functional  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/8j0ykl0zmidngpshn1wj5ndp04v6q88z-python3.11-torchmetrics-1.3.1/lib/python3.11/site-packages/torchmetrics/functional/__init__.py", line 121, in <module>
    from torchmetrics.functional.text._deprecated import _bleu_score as bleu_score
  File "/nix/store/8j0ykl0zmidngpshn1wj5ndp04v6q88z-python3.11-torchmetrics-1.3.1/lib/python3.11/site-packages/torchmetrics/functional/text/__init__.py", line 50, in <module>
    from torchmetrics.functional.text.bert import bert_score
  File "/nix/store/8j0ykl0zmidngpshn1wj5ndp04v6q88z-python3.11-torchmetrics-1.3.1/lib/python3.11/site-packages/torchmetrics/functional/text/bert.py", line 40, in <module>
    from transformers import AutoModel, AutoTokenizer
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1381, in __getattr__
    value = getattr(module, name)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1380, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1392, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.auto.modeling_auto because of the following error (look up to see its traceback):
'NoneType' object has no attribute 'split'

I hope this helps.

[matthid]

Even more minimal (you can see in the stacktrace):

from transformers import AutoModel, AutoTokenizer

$ python -OO test.py
Traceback (most recent call last):
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1390, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/sxr2igfkwhxbagri49b8krmcqz168sim-python3-3.11.8/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/modeling_auto.py", line 21, in <module>
    from .auto_factory import (
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 35, in <module>
    from .configuration_auto import AutoConfig, model_type_to_module_name, replace_list_option_in_docstrings
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 987, in <module>
    class AutoConfig:
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1011, in AutoConfig
    @replace_list_option_in_docstrings()
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 966, in docstring_decorator
    lines = docstrings.split("\n")
            ^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/matthid/test.py", line 1, in <module>
    from transformers import AutoModel, AutoTokenizer
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1381, in __getattr__
    value = getattr(module, name)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1380, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/rv8xdwghdad9jv2w86b8g08kan9l6ksm-python3.11-transformers-4.38.2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1392, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.auto.modeling_auto because of the following error (look up to see its traceback):
'NoneType' object has no attribute 'split'

[amyeroberts]

@matthid Could you share test.py? Without this, it's hard to tell what -00 represents here

[matthid]

The script is literally the line I showed:

from transformers import AutoModel, AutoTokenizer

-OO is the python flag for running the script:

Run the following in a shell where transformers is installed and python3 installed (venv, conda, pip install)

echo "from transformers import AutoModel, AutoTokenizer" > test.py
python3 -OO test.py

I'm not sure what else I can say.

[amyeroberts]

@matthid Sorry, my bad, I was skimming over this quickly and didn't notice the import line (and was misreading -OO as -00)

[amyeroberts]

@matthid OK, as there's other parts of the library which handle the case when __doc__ is None, I think it's OK to add this.

However, I don't think we can guarantee the whole library is going to be compatible running with the -OO flag, so use at your own risk!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.