Support third-party stub external dependencies in pytype (#9449)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

.github/workflows/tests.yml

@@ -69,6 +69,14 @@ jobs:
           cache: pip
           cache-dependency-path: requirements-tests.txt
       - run: pip install -r requirements-tests.txt
+      - name: Install external dependencies for 3rd-party stubs
+        run: |
+          DEPENDENCIES=$(python tests/get_external_stub_requirements.py)
+          if [ -n "$DEPENDENCIES" ]; then
+            echo "Installing packages: $DEPENDENCIES"
+            pip install $DEPENDENCIES
+          fi
+      - run: pip freeze --all
       - run: ./tests/pytype_test.py --print-stderr
 
   mypy:
@@ -117,7 +125,7 @@ jobs:
           python-version: "3.10"
           cache: pip
           cache-dependency-path: requirements-tests.txt
-      - name: Install 3rd-party dependencies for stubs packages
+      - name: Install external dependencies for 3rd-party stubs
         run: |
           pip install -r requirements-tests.txt
           DEPENDENCIES=$(python tests/get_external_stub_requirements.py)

CONTRIBUTING.md

@@ -46,9 +46,11 @@ virtual environment. If you're not familiar with what it is and how it works,
 please refer to this
 [documentation](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).
 
+Note that some tests require extra setup steps to install the required dependencies.
+
 ### Linux/Mac OS
 
-On Linux and Mac OS, you will be able to run the full test suite on Python 3.8,
+On Linux and Mac OS, you will be able to run the full test suite on Python
 3.9 or 3.10.
 To install the necessary requirements, run the following commands from a
 terminal window:

tests/README.md

@@ -17,7 +17,20 @@ objects at runtime.
 in the `tests` and `scripts` directories.
 
 To run the tests, follow the [setup instructions](../CONTRIBUTING.md#preparing-the-environment)
-in the `CONTRIBUTING.md` document. In particular, we recommend running with Python 3.9+.
+in the `CONTRIBUTING.md` document. In particular, you have to run with Python 3.9+.
+
+In order for `pytype_test` and `pyright_test` to work correctly, some third-party stubs
+may require extra dependencies external to typeshed to be installed in your virtual environment
+prior to running the test.
+You can list or install all of a stubs package's external dependencies using the following script:
+```bash
+(.venv3)$ python tests/get_external_stub_requirements.py <third_party_stub>  # List external dependencies for <third_party_stub>
+(.venv3)$ python tests/get_external_stub_requirements.py <third_party_stub1> <third_party_stub2>  # List external dependencies for <third_party_stub1> and <third_party_stub2>
+(.venv3)$ python tests/get_external_stub_requirements.py  # List external dependencies for all third-party stubs in typeshed
+# Install external dependencies for all third-party stubs in typeshed
+(.venv3)$ DEPENDENCIES=$(python tests/get_external_stub_requirements.py)
+(.venv3)$ if [ -n "$DEPENDENCIES" ]; then pip install $DEPENDENCIES; fi
+```
 
 ## Run all tests for a specific stub
 
@@ -78,19 +91,6 @@ checks that would typically fail on incomplete stubs (such as `Unknown` checks).
 In typeshed's CI, pyright is run with these configuration settings on a subset of
 the stubs in typeshed (including the standard library).
 
-In order for `pyright_test` to work correctly, some third-party stubs may require
-dependencies external to typeshed to be installed in your virtual environment
-prior to running the test.
-You can list or install all of a stubs package's external dependencies using the following script:
-```bash
-(.venv3)$ python tests/get_external_stub_requirements.py <third_party_stub>  # List external dependencies for <third_party_stub>
-(.venv3)$ python tests/get_external_stub_requirements.py <third_party_stub1> <third_party_stub2>  # List external dependencies for <third_party_stub1> and <third_party_stub2>
-(.venv3)$ python tests/get_external_stub_requirements.py  # List external dependencies for all third-party stubs in typeshed
-# Install external dependencies for all third-party stubs in typeshed
-(.venv3)$ DEPENDENCIES=$(python tests/get_external_stub_requirements.py)
-(.venv3)$ if [ -n "$DEPENDENCIES" ]; then pip install $DEPENDENCIES; fi
-```
-
 ## regr\_test.py
 
 This test runs mypy against the test cases for typeshed's stdlib and third-party