Rework how pyright is run in CI to allow for non-types dependencies(#9434)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

tests/README.md

@@ -78,6 +78,19 @@ checks that would typically fail on incomplete stubs (such as `Unknown` checks).
 In typeshed's CI, pyright is run with these configuration settings on a subset of
 the stubs in typeshed (including the standard library).
 
+In order for `pyright_test` to work correctly, some third-party stubs may require
+dependencies external to typeshed to be installed in your virtual environment
+prior to running the test.
+You can list or install all of a stubs package's external dependencies using the following script:
+```bash
+(.venv3)$ python tests/get_external_stub_requirements.py <third_party_stub>  # List external dependencies for <third_party_stub>
+(.venv3)$ python tests/get_external_stub_requirements.py <third_party_stub1> <third_party_stub2>  # List external dependencies for <third_party_stub1> and <third_party_stub2>
+(.venv3)$ python tests/get_external_stub_requirements.py  # List external dependencies for all third-party stubs in typeshed
+# Install external dependencies for all third-party stubs in typeshed
+(.venv3)$ DEPENDENCIES=$(python tests/get_external_stub_requirements.py)
+(.venv3)$ if [ -n "$DEPENDENCIES" ]; then pip install $DEPENDENCIES; fi
+```
+
 ## regr\_test.py
 
 This test runs mypy against the test cases for typeshed's stdlib and third-party

tests/get_external_stub_requirements.py

@@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import os
+import sys
+
+from utils import read_dependencies
+
+distributions = sys.argv[1:]
+if not distributions:
+    distributions = os.listdir("stubs")
+
+requirements = set[str]()
+for distribution in distributions:
+    requirements.update(read_dependencies(distribution).external_pkgs)
+
+for requirement in sorted(requirements):
+    print(requirement)

tests/utils.py

@@ -7,10 +7,10 @@ import re
 import subprocess
 import sys
 import venv
-from collections.abc import Mapping
+from collections.abc import Iterable, Mapping
 from functools import cache
 from pathlib import Path
-from typing import Iterable, NamedTuple
+from typing import NamedTuple
 from typing_extensions import Annotated
 
 import pathspec  # type: ignore[import]