Continuing work towards #8988.
The first five commits were created using stubdefaulter on various Python versions; the following commits were all created manually by me to fix various problems. The main things this adds that weren't present in #9501 are:
- Defaults in Windows-only modules and Windows-only branches (because I'm running a Windows machine)
- Defaults in non-py311 branches
- Defaults for float parameters
- Defaults for overloads
- Add a few more hooks. These are all very fast, and I've found them useful in other projects:
- Autofixes:
- `trailing-whitespace`: fixes trailing whitespace
- `requirements-txt-fixer`: alphabetises items in `requirements.txt` files
- `end-of-file-fixer`: makes sure every file ends with a single newline character
- `mixed-line-ending`: Makes sure Windows users don't accidentally introduce CRLF line endings into a file that uses LF line endings
- None-autofixes:
- `check-yaml`: loads YAML files to validate syntax
- `check-toml`: loads TOML files to validate syntax
- `check-merge-conflict`: detects merge-conflict strings in files and blocks them from accidentally being committed
- `check-case-conflict`: checks for files with names that would conflict on a case-insensitive filesystem like MacOS HFS+ or Windows FAT; blocks them from being committed.
- Change the bot schedule to quarterly, to reduce noisy PRs
- Change the `black` language target-version to Python 3.10, synching the setting here with the changes that were made to our `pyproject.toml` file in #7538
Some fairly subtle logic was added in #9408, and I'm immediately regretting not having documented it slightly better. This PR adds some more comments to make maintaining this code easier in the future.
The approach is pretty similar to the approach I took in #9382 for regr_test.py: dynamically create virtual environments for testing stubs packages in isolation. However, since mypy_test.py tests all of typeshed's stubs in succession (and since lots of typeshed's stubs packages depend on other typeshed stubs packages), the performance issues with creating a virtual environment for testing each stubs package are even more severe than with regr_test.py.
I mitigate the performance issues associated with dynamically creating virtual environments in two ways:
- Dynamically creating venvs is mostly slow due to I/O bottlenecks. Creating the virtual environments can be sped up dramatically by creating them concurrently in a threadpool. The same goes for pip installing the requirements into the venvs -- though unfortunately, we have to use pip with --no-cache-dir, as the pip cache gets easily corrupted if you try to do concurrent reads and writes to the pip cache.
- If types-pycocotools requires numpy>=1 and types-D3DShot also requires numpy>=1 (for example), there's no need to create 2 virtual environments. The same requirements have been specified, so they can share a virtual environment between them. This means we don't have to create nearly so many virtual environments.
We're about to get a daily test failure with 63 stubtest errors (see the Ubuntu stubtest failure in #9187) due to the release of redis-py 4.3.5, and I'd rather that didn't happen.
[`atomicwrites`](https://github.com/untitaker/python-atomicwrites) is a package that's only really useful for Python 2/3 compatibility. It's now deprecated, archived, and unmaintained. Prior to #8883 a few days ago, marking the stubs as complete, the typeshed stubs hadn't received any significant updates for several years.
I propose we mark the stubs as no longer updated, and then remove them.
#8726 proposed to make these properties abstract, but it would have broken half the typed-Python ecosystem, and we've since decided that it's not worth taking action on this (https://github.com/python/typeshed/issues/2146#event-7450564547). This PR rephrases the comments to make clear that there isn't really anything actionable that can be done here.
I think stubsabot has been a really big success. Let's change it to a daily workflow, so that we don't get an avalanche of PRs every Friday, and so that we can see more quickly if a PR changing stubsabot has accidentally broken the script. (If that _does_ happen, it should be pretty easy to just revert the commit that accidentally broke stubsabot, and go back to a version that works pretty well.)
The final mypy_primer report for #8762 was an empty diff. Considering the number of issues we've had with our requests stubs over the last year, it feels like it makes sense to add a test case to make sure that it doesn't regress.
Alex Waygood