Move test_cases to stdlib/@tests/test_cases (#11865)

tests/README.md

@@ -102,7 +102,8 @@ the stubs in typeshed (including the standard library).
 ## regr\_test.py
 
 This test runs mypy against the test cases for typeshed's stdlib and third-party
-stubs. See [the README in the `test_cases` directory](../test_cases/README.md)
+stubs. See [the REGRESSION.md document](./REGRESSION.md)
+in this directory
 for more information about what
 these test cases are for and how they work. Run `python tests/regr_test.py --help`
 for information on the various configuration options.

tests/REGRESSION.md

@@ -0,0 +1,121 @@
+## Regression tests for typeshed
+
+Regression tests for the standard library stubs can be found in the
+`stdlib/@tests/test_cases` directory. Stubs for third-party libraries that do
+have test cases can be found in `@tests/test_cases` subdirectories for each
+stubs package. For example, the test cases for `requests` can be found in the
+`stubs/requests/@tests/test_cases` directory.
+
+**Regression test cases should only be written for functions and classes which
+are known to have caused problems in the past, where the stubs are difficult to
+get right.** 100% test coverage for typeshed is neither necessary nor
+desirable, as it would lead to code duplication. Moreover, typeshed has
+multiple other mechanisms for spotting errors in the stubs.
+
+### The purpose of these tests
+
+Different test cases in this directory serve different purposes. For some stubs in
+typeshed, the type annotations are complex enough that it's useful to have
+sanity checks that test whether a type checker understands the intent of
+the annotations correctly. Examples of tests like these are
+`builtins/check_pow.py` and `asyncio/check_gather.py`.
+
+Other test cases, such as the samples for `ExitStack` in `check_contextlib.py`
+and the samples for `LogRecord` in `check_logging.py`, do not relate to
+stubs where the annotations are particularly complex, but they *do* relate to
+stubs where decisions have been taken that might be slightly unusual. These
+test cases serve a different purpose: to check that type checkers do not emit
+false-positive errors for idiomatic usage of these classes.
+
+## Running the tests
+
+To verify the test cases in this directory pass with mypy, run `python tests/regr_test.py stdlib`
+from the root of the typeshed repository. This assumes that the development
+environment has been set up as described in the [CONTRIBUTING.md](../CONTRIBUTING.md)
+document.
+
+### How the tests work
+
+The code in this directory is not intended to be directly executed. Instead,
+type checkers are run on the code, to check that typing errors are
+emitted at the correct places.
+
+Some files in this directory simply contain samples of idiomatic Python, which
+should not (if the stubs are correct) cause a type checker to emit any errors.
+
+Many test cases also make use of
+[`assert_type`](https://docs.python.org/3.11/library/typing.html#typing.assert_type),
+a function which allows us to test whether a type checker's inferred type of an
+expression is what we'd like it be.
+
+Finally, some tests make use of `# type: ignore` comments (in combination with
+mypy's
+[`--warn-unused-ignores`](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-warn-unused-ignores)
+setting and pyright's
+[`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings)
+setting) to test instances where a type checker *should* emit some kind of
+error, if the stubs are correct. Both settings are enabled by default for the entire
+subdirectory.
+
+For more information on using `assert_type` and
+`--warn-unused-ignores`/`reportUnnecessaryTypeIgnoreComment` to test type
+annotations,
+[this page](https://typing.readthedocs.io/en/latest/source/quality.html#testing-using-assert-type-and-warn-unused-ignores)
+provides a useful guide.
+
+### Naming convention
+
+Use the same top-level name for the module / package you would like to test.
+Use the `check_${thing}.py` naming pattern for individual test files.
+
+By default, test cases go into a file with the same name as the stub file, prefixed with `check_`.
+For example: `check_contextlib.py`.
+
+If that file becomes too big, we instead create a directory with files named after individual objects being tested.
+For example: `builtins/check_dict.py`.
+
+### Differences to the rest of typeshed
+
+Unlike the rest of typeshed, this directory largely contains `.py` files. This
+is because the purpose of this folder is to test the implications of typeshed
+changes for end users, who will mainly be using `.py` files rather than `.pyi`
+files.
+
+Another difference to the rest of typeshed
+(which stems from the fact that the test-case files are all `.py` files
+rather than `.pyi` files)
+is that the test cases cannot always use modern syntax for type hints.
+While we can use `from __future__ import annotations` to enable the use of
+modern typing syntax wherever possible,
+type checkers may (correctly) emit errors if PEP 604 syntax or PEP 585 syntax
+is used in a runtime context on lower versions of Python. For example:
+
+```python
+from __future__ import annotations
+
+from typing_extensions import assert_type
+
+x: str | int  # PEP 604 syntax: okay on Python >=3.7, due to __future__ annotations
+assert_type(x, str | int)  # Will fail at runtime on Python <3.10 (use typing.Union instead)
+
+y: dict[str, int]  # PEP 585 syntax: okay on Python >= 3.7, due to __future__ annotations
+assert_type(y, dict[str, int])  # Will fail at runtime on Python <3.9 (use typing.Dict instead)
+```
+
+### Version-dependent tests
+
+Some tests will only pass on mypy
+with a specific Python version passed on the command line to the `tests/regr_test.py` script.
+To mark a test-case file as being skippable on lower versions of Python,
+append `-py3*` to the filename.
+For example, if `foo` is a stdlib feature that's new in Python 3.11,
+test cases for `foo` should be put in a file named `check_foo-py311.py`.
+This means that mypy will only run the test case
+if `--python-version 3.11`, `--python-version 3.12`, etc.
+is passed on the command line to `tests/regr_test.py`,
+but it _won't_ run the test case if e.g. `--python-version 3.9`
+is passed on the command line.
+
+However, `if sys.version_info >= (3, target):` is still required for `pyright`
+in the test file itself.
+Example: [`check_exception_group-py311.py`](../stdlib/@tests/test_cases/builtins/check_exception_group-py311.py)

tests/check_typeshed_structure.py

@@ -15,6 +15,7 @@ from pathlib import Path
 from parse_metadata import read_metadata
 from utils import (
     REQS_FILE,
+    STDLIB_PATH,
     TEST_CASES_DIR,
     TESTS_DIR,
     VERSIONS_RE,
@@ -59,7 +60,8 @@ def assert_consistent_filetypes(
 
 def check_stdlib() -> None:
     """Check that the stdlib directory contains only the correct files."""
-    assert_consistent_filetypes(Path("stdlib"), kind=".pyi", allowed={"_typeshed/README.md", "VERSIONS"})
+    assert_consistent_filetypes(STDLIB_PATH, kind=".pyi", allowed={"_typeshed/README.md", "VERSIONS", TESTS_DIR})
+    check_tests_dir(tests_path("stdlib"))
 
 
 def check_stubs() -> None:
@@ -81,11 +83,13 @@ def check_stubs() -> None:
 
         tests_dir = tests_path(dist.name)
         if tests_dir.exists() and tests_dir.is_dir():
-            py_files_present = any(file.suffix == ".py" for file in tests_dir.iterdir())
-            error_message = (
-                f"Test-case files must be in an `{TESTS_DIR}/{TEST_CASES_DIR}` directory, not in the `{TESTS_DIR}` directory"
-            )
-            assert not py_files_present, error_message
+            check_tests_dir(tests_dir)
+
+
+def check_tests_dir(tests_dir: Path) -> None:
+    py_files_present = any(file.suffix == ".py" for file in tests_dir.iterdir())
+    error_message = f"Test-case files must be in an `{TESTS_DIR}/{TEST_CASES_DIR}` directory, not in the `{TESTS_DIR}` directory"
+    assert not py_files_present, error_message
 
 
 def check_distutils() -> None:
@@ -146,7 +150,7 @@ def check_versions_file() -> None:
 
 def _find_stdlib_modules() -> set[str]:
     modules = set[str]()
-    for path, _, files in os.walk("stdlib"):
+    for path, _, files in os.walk(STDLIB_PATH):
         for filename in files:
             base_module = ".".join(os.path.normpath(path).split(os.sep)[1:])
             if filename == "__init__.pyi":

tests/mypy_test.py

@@ -29,6 +29,7 @@ import tomli
 from parse_metadata import PackageDependencies, get_recursive_requirements, read_metadata
 from utils import (
     PYTHON_VERSION,
+    TESTS_DIR,
     VERSIONS_RE as VERSION_LINE_RE,
     colored,
     get_gitignore_spec,
@@ -366,7 +367,7 @@ def test_stdlib(args: TestConfig) -> TestResult:
     stdlib = Path("stdlib")
     supported_versions = parse_versions(stdlib / "VERSIONS")
     for name in os.listdir(stdlib):
-        if name == "VERSIONS" or name.startswith("."):
+        if name in ("VERSIONS", TESTS_DIR) or name.startswith("."):
             continue
         module = Path(name).stem
         module_min_version, module_max_version = supported_versions[module]

tests/utils.py

@@ -22,6 +22,7 @@ except ImportError:
 
 PYTHON_VERSION: Final = f"{sys.version_info.major}.{sys.version_info.minor}"
 
+STDLIB_PATH = Path("stdlib")
 STUBS_PATH = Path("stubs")
 
 
@@ -139,15 +140,14 @@ def distribution_info(distribution_name: str) -> DistributionTests:
 
 
 def tests_path(distribution_name: str) -> Path:
-    assert distribution_name != "stdlib"
-    return STUBS_PATH / distribution_name / TESTS_DIR
+    if distribution_name == "stdlib":
+        return STDLIB_PATH / TESTS_DIR
+    else:
+        return STUBS_PATH / distribution_name / TESTS_DIR
 
 
 def test_cases_path(distribution_name: str) -> Path:
-    if distribution_name == "stdlib":
-        return Path(TEST_CASES_DIR)
-    else:
-        return tests_path(distribution_name) / TEST_CASES_DIR
+    return tests_path(distribution_name) / TEST_CASES_DIR
 
 
 def get_all_testcase_directories() -> list[DistributionTests]: