VimPlug/typeshed
1
0
Fork 0
mirror of https://github.com/davidhalter/typeshed.git synced 2025-12-07 12:44:28 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Touchups to tests/REGRESSION.md (#12826)

Browse Source
This commit is contained in:
Alex Waygood
2024-10-16 23:43:39 +01:00
committed by GitHub
parent 2370b8b9d1
commit c9d3ca6f48
1 changed files with 42 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
tests/REGRESSION.md
View File

@@ -1,10 +1,11 @@
## Regression tests for typeshed
Regression tests for the standard library stubs can be found in the
`stdlib/@tests/test_cases` directory. Stubs for third-party libraries that do
have test cases can be found in `@tests/test_cases` subdirectories for each
stubs package. For example, the test cases for `requests` can be found in the
`stubs/requests/@tests/test_cases` directory.
`stdlib/@tests/test_cases` directory. Not all third-party-library stub
packages in typeshed have test cases, and not all of them need test cases --
but for those that do, their test cases can be found in `@tests/test_cases`
subdirectories for each stubs package. For example, the test cases for
`requests` can be found in the `stubs/requests/@tests/test_cases` directory.
**Regression test cases should only be written for functions and classes which
are known to have caused problems in the past, where the stubs are difficult to
@@ -14,14 +15,16 @@ multiple other mechanisms for spotting errors in the stubs.
### The purpose of these tests
Different test cases in this directory serve different purposes. For some stubs in
typeshed, the type annotations are complex enough that it's useful to have
Different test cases in typeshed serve different purposes. For some typeshed
stubs, the type annotations are complex enough that it's useful to have
sanity checks that test whether a type checker understands the intent of
the annotations correctly. Examples of tests like these are
`builtins/check_pow.py` and `asyncio/check_gather.py`.
`stdlib/@tests/test_cases/builtins/check_pow.py` and
`stdlib/@tests/test_cases/asyncio/check_gather.py`.
Other test cases, such as the samples for `ExitStack` in `check_contextlib.py`
and the samples for `LogRecord` in `check_logging.py`, do not relate to
Other test cases, such as the samples for `ExitStack` in
`stdlib/@tests/test_cases/check_contextlib.py` and the samples for `LogRecord`
in `stdlib/@tests/test_cases/check_logging.py`, do not relate to
stubs where the annotations are particularly complex, but they *do* relate to
stubs where decisions have been taken that might be slightly unusual. These
test cases serve a different purpose: to check that type checkers do not emit
@@ -29,19 +32,28 @@ false-positive errors for idiomatic usage of these classes.
## Running the tests
To verify the test cases in this directory pass with mypy, run `python tests/regr_test.py stdlib`
from the root of the typeshed repository. This assumes that the development
environment has been set up as described in the [CONTRIBUTING.md](../CONTRIBUTING.md)
document.
To verify the stdlib test cases pass with mypy, run
`python tests/regr_test.py stdlib` from the root of the typeshed repository.
This assumes that the development environment has been set up as described in
the [CONTRIBUTING.md](../CONTRIBUTING.md) document.
For third-party-library stubs, pass the name of the runtime library the stubs
are for. For example, to run the tests for our `requests` stubs, run
`python tests/regr_test.py requests`.
Run `python tests/regr_test.py -h` for the full range of CLI options this script
supports. There is no equivalent script for pyright currently; for pyright, the
tests are checked in CI using a GitHub Action.
### How the tests work
The code in this directory is not intended to be directly executed. Instead,
type checkers are run on the code, to check that typing errors are
emitted at the correct places.
The code in typeshed's test cases is not intended to be directly executed.
Instead, type checkers are run on the code, to check that type checker
diagnostics are emitted at the correct locations.
Some files in this directory simply contain samples of idiomatic Python, which
should not (if the stubs are correct) cause a type checker to emit any errors.
Some files in these directories simply contain samples of idiomatic Python,
which should not (if the stubs are correct) cause a type checker to emit any
diagnostics.
Many test cases also make use of
[`assert_type`](https://docs.python.org/3.11/library/typing.html#typing.assert_type),
@@ -54,8 +66,8 @@ mypy's
setting and pyright's
[`reportUnnecessaryTypeIgnoreComment`](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#type-check-diagnostics-settings)
setting) to test instances where a type checker *should* emit some kind of
error, if the stubs are correct. Both settings are enabled by default for the entire
subdirectory.
error, if the stubs are correct. Both settings are enabled by default for
all `@tests/test_cases/` subdirectories in typeshed.
For more information on using `assert_type` and
`--warn-unused-ignores`/`reportUnnecessaryTypeIgnoreComment` to test type
@@ -65,21 +77,21 @@ provides a useful guide.
### Naming convention
Use the same top-level name for the module / package you would like to test.
Use the same top-level name for the module or package you would like to test.
Use the `check_${thing}.py` naming pattern for individual test files.
By default, test cases go into a file with the same name as the stub file, prefixed with `check_`.
For example: `check_contextlib.py`.
For example: `stdlib/@tests/test_cases/check_contextlib.py`.
If that file becomes too big, we instead create a directory with files named after individual objects being tested.
For example: `builtins/check_dict.py`.
For example: `stdlib/@tests/test_cases/builtins/check_dict.py`.
### Differences to the rest of typeshed
Unlike the rest of typeshed, this directory largely contains `.py` files. This
is because the purpose of this folder is to test the implications of typeshed
changes for end users, who will mainly be using `.py` files rather than `.pyi`
files.
Unlike the rest of typeshed, the `@tests/test_cases` directories largely
contain `.py` files. This is because the purpose of these directories is to
test the implications of typeshed changes for end users, who will mainly be
using `.py` files rather than `.pyi` files.
Another difference to the rest of typeshed
(which stems from the fact that the test-case files are all `.py` files
@@ -109,7 +121,8 @@ with a specific Python version passed on the command line to the `tests/regr_tes
To mark a test-case file as being skippable on lower versions of Python,
append `-py3*` to the filename.
For example, if `foo` is a stdlib feature that's new in Python 3.11,
test cases for `foo` should be put in a file named `check_foo-py311.py`.
test cases for `foo` should be put in a file named
`stdlib/@tests/test_cases/check_foo-py311.py`.
This means that mypy will only run the test case
if `--python-version 3.11`, `--python-version 3.12`, etc.
is passed on the command line to `tests/regr_test.py`,
@@ -118,4 +131,4 @@ is passed on the command line.
However, `if sys.version_info >= (3, target):` is still required for `pyright`
in the test file itself.
Example: [`check_exception_group-py311.py`](../stdlib/@tests/test_cases/builtins/check_exception_group-py311.py)
Example: [`stdlib/@tests/test_cases/check_exception_group-py311.py`](../stdlib/@tests/test_cases/builtins/check_exception_group-py311.py)