Add infrastructure allowing for test cases for third-party stubs (#8700)

- Move the logic for running mypy on the test cases from `tests/mypy_test.py` to a separate script, `tests/regr_test.py`.
- Add the necessary logic in order to be able to have test cases for third-party stubs.
- Move logic common to `tests/mypy_test.py` and `tests/regr_test.py` into `tests/colors.py`, and rename `tests/colors.py` to `tests/utils.py`.
- Add a new check to `tests/check_consistent.py`, to enforce the use of `# pyright: reportUnnecessaryTypeIgnoreComment=true` comments in third-party test cases. These are essential if we want to have our tests against false-negatives work with pyright.
- Update the relevant documentation to account for the new test file.
- Add a new job to the `tests.yml` GitHub workflow, to run the new test in CI.
- Add a simple proof-of-concept test case for `requests`, as a regression test for #7998.

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Sebastian Rittau <srittau@rittau.biz>

.github/workflows/tests.yml

@@ -9,6 +9,7 @@ on:
   pull_request:
     paths-ignore:
       - '**/*.md'
+      - 'scripts/**'
 
 permissions:
   contents: read
@@ -65,7 +66,7 @@ jobs:
       - run: ./tests/pytype_test.py --print-stderr
 
   mypy:
-    name: Test the stubs with mypy
+    name: Run mypy against the stubs
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -80,6 +81,17 @@ jobs:
       - run: pip install -r requirements-tests.txt
       - run: ./tests/mypy_test.py --platform=${{ matrix.platform }} --python-version=${{ matrix.python-version }}
 
+  regression-tests:
+    name: Run mypy on the test cases
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+      - run: pip install -r requirements-tests.txt
+      - run: python ./tests/regr_test.py --all
+
   pyright:
     name: Test the stubs with pyright
     runs-on: ubuntu-latest