Skip to content

Commit

Permalink
Reorder sections in running_mypy (#13738)
Browse files Browse the repository at this point in the history 
See #13681

"Mapping file paths to modules" belongs next to "Specifying code to be
checked". "How imports are found" belongs next to "How mypy handles
imports". The "other advice" section was mostly redundant.
  • Loading branch information
@hauntsaninja
hauntsaninja authored Sep 27, 2022
1 parent da43c91 commit 9783b46
Showing 1 changed file with 136 additions and 150 deletions.
286 changes: 136 additions & 150 deletions docs/source/running_mypy.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,82 @@ flags, the recommended approach is to use a
:ref:`configuration file <config-file>` instead.


.. _mapping-paths-to-modules:

Mapping file paths to modules
*****************************

One of the main ways you can tell mypy what to type check
is by providing mypy a list of paths. For example::

$ mypy file_1.py foo/file_2.py file_3.pyi some/directory

This section describes how exactly mypy maps the provided paths
to modules to type check.

- Mypy will check all paths provided that correspond to files.

- Mypy will recursively discover and check all files ending in ``.py`` or
``.pyi`` in directory paths provided, after accounting for
:option:`--exclude <mypy --exclude>`.

- For each file to be checked, mypy will attempt to associate the file (e.g.
``project/foo/bar/baz.py``) with a fully qualified module name (e.g.
``foo.bar.baz``). The directory the package is in (``project``) is then
added to mypy's module search paths.

How mypy determines fully qualified module names depends on if the options
:option:`--no-namespace-packages <mypy --no-namespace-packages>` and
:option:`--explicit-package-bases <mypy --explicit-package-bases>` are set.

1. If :option:`--no-namespace-packages <mypy --no-namespace-packages>` is set,
mypy will rely solely upon the presence of ``__init__.py[i]`` files to
determine the fully qualified module name. That is, mypy will crawl up the
directory tree for as long as it continues to find ``__init__.py`` (or
``__init__.pyi``) files.

For example, if your directory tree consists of ``pkg/subpkg/mod.py``, mypy
would require ``pkg/__init__.py`` and ``pkg/subpkg/__init__.py`` to exist in
order correctly associate ``mod.py`` with ``pkg.subpkg.mod``

2. The default case. If :option:`--namespace-packages <mypy
--no-namespace-packages>` is on, but :option:`--explicit-package-bases <mypy
--explicit-package-bases>` is off, mypy will allow for the possibility that
directories without ``__init__.py[i]`` are packages. Specifically, mypy will
look at all parent directories of the file and use the location of the
highest ``__init__.py[i]`` in the directory tree to determine the top-level
package.

For example, say your directory tree consists solely of ``pkg/__init__.py``
and ``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified
module name, mypy will look at ``pkg/__init__.py`` and conclude that the
associated module name is ``pkg.a.b.c.d.mod``.

3. You'll notice that the above case still relies on ``__init__.py``. If
you can't put an ``__init__.py`` in your top-level package, but still wish to
pass paths (as opposed to packages or modules using the ``-p`` or ``-m``
flags), :option:`--explicit-package-bases <mypy --explicit-package-bases>`
provides a solution.

With :option:`--explicit-package-bases <mypy --explicit-package-bases>`, mypy
will locate the nearest parent directory that is a member of the ``MYPYPATH``
environment variable, the :confval:`mypy_path` config or is the current
working directory. Mypy will then use the relative path to determine the
fully qualified module name.

For example, say your directory tree consists solely of
``src/namespace_pkg/mod.py``. If you run the following command, mypy
will correctly associate ``mod.py`` with ``namespace_pkg.mod``::

$ MYPYPATH=src mypy --namespace-packages --explicit-package-bases .

If you pass a file not ending in ``.py[i]``, the module name assumed is
``__main__`` (matching the behavior of the Python interpreter), unless
:option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.

Passing :option:`-v <mypy -v>` will show you the files and associated module
names that mypy will check.


How mypy handles imports
************************
Expand Down Expand Up @@ -326,6 +402,66 @@ on your system in an unconventional way.
In this case, follow the steps above on how to handle
:ref:`missing type hints in third party libraries <missing-type-hints-for-third-party-library>`.


.. _finding-imports:

How imports are found
*********************

When mypy encounters an ``import`` statement or receives module
names from the command line via the :option:`--module <mypy --module>` or :option:`--package <mypy --package>`
flags, mypy tries to find the module on the file system similar
to the way Python finds it. However, there are some differences.

First, mypy has its own search path.
This is computed from the following items:

- The ``MYPYPATH`` environment variable
(a list of directories, colon-separated on UNIX systems, semicolon-separated on Windows).
- The :confval:`mypy_path` config file option.
- The directories containing the sources given on the command line
(see :ref:`Mapping file paths to modules <mapping-paths-to-modules>`).
- The installed packages marked as safe for type checking (see
:ref:`PEP 561 support <installed-packages>`)
- The relevant directories of the
`typeshed <https://github.com/python/typeshed>`_ repo.

.. note::

You cannot point to a stub-only package (:pep:`561`) via the ``MYPYPATH``, it must be
installed (see :ref:`PEP 561 support <installed-packages>`)

Second, mypy searches for stub files in addition to regular Python files
and packages.
The rules for searching for a module ``foo`` are as follows:

- The search looks in each of the directories in the search path
(see above) until a match is found.
- If a package named ``foo`` is found (i.e. a directory
``foo`` containing an ``__init__.py`` or ``__init__.pyi`` file)
that's a match.
- If a stub file named ``foo.pyi`` is found, that's a match.
- If a Python module named ``foo.py`` is found, that's a match.

These matches are tried in order, so that if multiple matches are found
in the same directory on the search path
(e.g. a package and a Python file, or a stub file and a Python file)
the first one in the above list wins.

In particular, if a Python file and a stub file are both present in the
same directory on the search path, only the stub file is used.
(However, if the files are in different directories, the one found
in the earlier directory is used.)

Setting :confval:`mypy_path`/``MYPYPATH`` is mostly useful in the case
where you want to try running mypy against multiple distinct
sets of files that happen to share some common dependencies.

For example, if you have multiple projects that happen to be
using the same set of work-in-progress stubs, it could be
convenient to just have your ``MYPYPATH`` point to a single
directory containing the stubs.

.. _follow-imports:

Following imports
Expand Down Expand Up @@ -387,153 +523,3 @@ hard-to-debug errors.
Adjusting import following behaviour is often most useful when restricted to
specific modules. This can be accomplished by setting a per-module
:confval:`follow_imports` config option.


.. _mapping-paths-to-modules:

Mapping file paths to modules
*****************************

One of the main ways you can tell mypy what to type check
is by providing mypy a list of paths. For example::

$ mypy file_1.py foo/file_2.py file_3.pyi some/directory

This section describes how exactly mypy maps the provided paths
to modules to type check.

- Mypy will check all paths provided that correspond to files.

- Mypy will recursively discover and check all files ending in ``.py`` or
``.pyi`` in directory paths provided, after accounting for
:option:`--exclude <mypy --exclude>`.

- For each file to be checked, mypy will attempt to associate the file (e.g.
``project/foo/bar/baz.py``) with a fully qualified module name (e.g.
``foo.bar.baz``). The directory the package is in (``project``) is then
added to mypy's module search paths.

How mypy determines fully qualified module names depends on if the options
:option:`--no-namespace-packages <mypy --no-namespace-packages>` and
:option:`--explicit-package-bases <mypy --explicit-package-bases>` are set.

1. If :option:`--no-namespace-packages <mypy --no-namespace-packages>` is set,
mypy will rely solely upon the presence of ``__init__.py[i]`` files to
determine the fully qualified module name. That is, mypy will crawl up the
directory tree for as long as it continues to find ``__init__.py`` (or
``__init__.pyi``) files.

For example, if your directory tree consists of ``pkg/subpkg/mod.py``, mypy
would require ``pkg/__init__.py`` and ``pkg/subpkg/__init__.py`` to exist in
order correctly associate ``mod.py`` with ``pkg.subpkg.mod``

2. The default case. If :option:`--namespace-packages <mypy
--no-namespace-packages>` is on, but :option:`--explicit-package-bases <mypy
--explicit-package-bases>` is off, mypy will allow for the possibility that
directories without ``__init__.py[i]`` are packages. Specifically, mypy will
look at all parent directories of the file and use the location of the
highest ``__init__.py[i]`` in the directory tree to determine the top-level
package.

For example, say your directory tree consists solely of ``pkg/__init__.py``
and ``pkg/a/b/c/d/mod.py``. When determining ``mod.py``'s fully qualified
module name, mypy will look at ``pkg/__init__.py`` and conclude that the
associated module name is ``pkg.a.b.c.d.mod``.

3. You'll notice that the above case still relies on ``__init__.py``. If
you can't put an ``__init__.py`` in your top-level package, but still wish to
pass paths (as opposed to packages or modules using the ``-p`` or ``-m``
flags), :option:`--explicit-package-bases <mypy --explicit-package-bases>`
provides a solution.

With :option:`--explicit-package-bases <mypy --explicit-package-bases>`, mypy
will locate the nearest parent directory that is a member of the ``MYPYPATH``
environment variable, the :confval:`mypy_path` config or is the current
working directory. Mypy will then use the relative path to determine the
fully qualified module name.

For example, say your directory tree consists solely of
``src/namespace_pkg/mod.py``. If you run the following command, mypy
will correctly associate ``mod.py`` with ``namespace_pkg.mod``::

$ MYPYPATH=src mypy --namespace-packages --explicit-package-bases .

If you pass a file not ending in ``.py[i]``, the module name assumed is
``__main__`` (matching the behavior of the Python interpreter), unless
:option:`--scripts-are-modules <mypy --scripts-are-modules>` is passed.

Passing :option:`-v <mypy -v>` will show you the files and associated module
names that mypy will check.


.. _finding-imports:

How imports are found
*********************

When mypy encounters an ``import`` statement or receives module
names from the command line via the :option:`--module <mypy --module>` or :option:`--package <mypy --package>`
flags, mypy tries to find the module on the file system similar
to the way Python finds it. However, there are some differences.

First, mypy has its own search path.
This is computed from the following items:

- The ``MYPYPATH`` environment variable
(a list of directories, colon-separated on UNIX systems, semicolon-separated on Windows).
- The :confval:`mypy_path` config file option.
- The directories containing the sources given on the command line
(see :ref:`Mapping file paths to modules <mapping-paths-to-modules>`).
- The installed packages marked as safe for type checking (see
:ref:`PEP 561 support <installed-packages>`)
- The relevant directories of the
`typeshed <https://github.com/python/typeshed>`_ repo.

.. note::

You cannot point to a stub-only package (:pep:`561`) via the ``MYPYPATH``, it must be
installed (see :ref:`PEP 561 support <installed-packages>`)

Second, mypy searches for stub files in addition to regular Python files
and packages.
The rules for searching for a module ``foo`` are as follows:

- The search looks in each of the directories in the search path
(see above) until a match is found.
- If a package named ``foo`` is found (i.e. a directory
``foo`` containing an ``__init__.py`` or ``__init__.pyi`` file)
that's a match.
- If a stub file named ``foo.pyi`` is found, that's a match.
- If a Python module named ``foo.py`` is found, that's a match.

These matches are tried in order, so that if multiple matches are found
in the same directory on the search path
(e.g. a package and a Python file, or a stub file and a Python file)
the first one in the above list wins.

In particular, if a Python file and a stub file are both present in the
same directory on the search path, only the stub file is used.
(However, if the files are in different directories, the one found
in the earlier directory is used.)


Other advice and best practices
*******************************

There are multiple ways of telling mypy what files to type check, ranging
from passing in command line arguments to using the :confval:`files` or :confval:`mypy_path`
config file options to setting the
``MYPYPATH`` environment variable.

However, in practice, it is usually sufficient to just use either
command line arguments or the :confval:`files` config file option (the two
are largely interchangeable).

Setting :confval:`mypy_path`/``MYPYPATH`` is mostly useful in the case
where you want to try running mypy against multiple distinct
sets of files that happen to share some common dependencies.

For example, if you have multiple projects that happen to be
using the same set of work-in-progress stubs, it could be
convenient to just have your ``MYPYPATH`` point to a single
directory containing the stubs.

0 comments on commit 9783b46

Please sign in to comment.