Update documentation to better reflect new directory structure. (#4976)

See #2491 for previous discussion.

Co-authored-by: Ivan Levkivskyi <ilevkivskyi@dropbox.com>

CONTRIBUTING.md

@@ -308,30 +308,19 @@ contributors.
 
 ### Stub versioning
 
-There are separate directories for `stdlib` and `third_party` stubs.
-Within those, there are separate directories for different versions of
-Python the stubs target.
+There are separate directories for `stdlib` (standard library) and `stubs`
+(all other stubs). For standard library stubs Python version support is
+given in `VERSIONS` file. Each line in this file is a module or package name
+followed by `: `, followed by the oldest *supported* Python version where
+the module is available. For third party packages, the Python version support
+(2 and/or 3 only, no finer grained version is supported) is indicated in the
+corresponding `METADATA.toml` file as `python2 = (True|False)` (defaults to
+`False`) and `python3 = (True|False)` (defaults to `True`).
 
-The directory name indicates the major version of Python that a stub targets
-and optionally the lowest minor version, with the exception of the `2and3`
-directory which applies to both Python 2 and 3.
-
-For example, stubs in the `3` directory will be applied to all versions of
-Python 3, though stubs in the `3.7` directory will only be applied to versions
-3.7 and above. However, stubs in the `2` directory will not be applied to
-Python 3.
-
-It is preferred to use a single stub in the more generic directory that
-conditionally targets specific versions when needed, as opposed
-to maintaining multiple stub files within more specific directories. Similarly,
-if the given library works on both Python 2 and Python 3, prefer to put your
-stubs in the `2and3` directory, unless the types are so different that the stubs
-become unreadable that way.
-
-You can use checks like `if sys.version_info >= (3, 8):` to denote new
-functionality introduced in a given Python version or solve type
-differences.  When doing so, only use one-tuples or two-tuples.  This is
-because:
+It is preferred to use a single stub for every module. You can use checks
+like `if sys.version_info >= (3, 8):` to denote new functionality introduced
+in a given Python version or solve type differences.  When doing so, only use
+one-tuples or two-tuples.  This is because:
 
 * mypy doesn't support more fine-grained version checks; and more
   importantly
@@ -353,10 +342,28 @@ harmless.  This is a strictly better compromise than using the latter
 two forms, which would generate false positive errors for correct use
 under Python 3.7.4.
 
+If it is not possible to generate combined stubs for all Python versions
+in a single file, you can split Python 2 and Python 3 stubs and place Python 2
+stubs into `@python2` subdirectory for corresponding distribution. Note that
+you don't need `@python2` in most cases, if your package supports Python 2,
+just put the stubs at root of the distribution directory, and put
+`python2 = True` in `METADATA.toml`.
+
 Note: in its current implementation, typeshed cannot contain stubs for
 multiple versions of the same third-party library.  Prefer to generate
 stubs for the latest version released on PyPI at the time of your
-stubbing.
+stubbing. The oldest version of the library for which the stubs are still
+applicable (i.e. reflect the actual runtime behaviour) can be indicated
+in `METADATA.toml` as `version = "x.y"`. Note that only two most significant
+version levels are supported (i.e. only single dot). When a significant change
+is made in the library, the version of the stub should be bumped (note that
+previous versions are still available on PyPI).
+
+Internal typeshed machinery will periodically build and upload modified
+third party packages to PyPI, each time this happens the least significant
+version level is incremented. For example, if `stubs/foo/METADATA.toml` has
+`version = "x.y"` the package on PyPI will be updated from `types-foo-x.y.n`
+to `types-foo-x.y.n+1`.
 
 ### What to do when a project's documentation and implementation disagree
 

README.md

@@ -23,27 +23,20 @@ Typeshed supports Python versions 2.7 and 3.6 and up.
 
 If you're just using mypy (or pytype or PyCharm), as opposed to
 developing it, you don't need to interact with the typeshed repo at
-all: a copy of typeshed is bundled with mypy.
+all: a copy of standard library part of typeshed is bundled with mypy.
+And type stubs for third party packages and modules you are using can
+be installed from PyPI. For example, if you are using `six` and `requests`,
+you can install the type stubs using
 
-When you use a checked-out clone of the mypy repo, a copy of typeshed
-should be included as a submodule, using
+    $ pip install types-six types-requests
 
-    $ git clone --recurse-submodules https://github.com/python/mypy.git
+These PyPI packages follow [PEP 561](http://www.python.org/dev/peps/pep-0561/)
+and are automatically generated by typeshed internal machinery. Also starting
+from version 0.900 mypy will provide an option to automatically install missing
+type stub packages (if found on PyPI).
 
-or
-
-    $ git clone https://github.com/python/mypy.git
-    $ cd mypy
-    $ git submodule init
-    $ git submodule update
-
-and occasionally you will have to repeat the final command (`git
-submodule update`) to pull in changes made in the upstream typeshed
-repo.
-
-PyCharm and pytype similarly include a copy of typeshed.  The one in
-pytype can be updated in the same way if you are working with the
-pytype repo.
+PyCharm, pytype etc. work in a similar way, for more details see documentation
+for the type-checking tool you are using.
 
 ## Format
 
@@ -63,16 +56,31 @@ coding style used in typeshed.
 
 ### stdlib
 
-This contains stubs for modules the Python standard library -- which
+This contains stubs for modules in the Python standard library -- which
 includes pure Python modules, dynamically loaded extension modules,
-hard-linked extension modules, and the builtins.
+hard-linked extension modules, and the builtins. `VERSIONS` file lists
+the oldest *supported* Python version where the module is available.
+In the `stdlib/@python2` subdirectory you can find Python 2 versions of
+the stub files that must be kept different for Python 2 and 3, like
+`builtins.pyi`.
 
-### third_party
+### stubs
 
 Modules that are not shipped with Python but have a type description in Python
-go into `third_party`. Since these modules can behave differently for different
-versions of Python, `third_party` has version subdirectories, just like
-`stdlib`.
+go into `stubs`. Each subdirectory there represents a PyPI distribution, and
+contains the following:
+* `METADATA.toml` that specifies oldest version of the source library for
+  which the stubs are applicable, supported Python versions (Python 3 defaults
+  to `True`, Python 2 defaults to `False`), and dependency on other type stub
+  packages.
+* Stubs (i.e. `*.pyi` files) for packages and modules that are shipped in the
+  source distribution. Similar to standard library, if the Python 2 version of
+  the stubs must be kept *separate*, it can be put in a `@python` subdirectory.
+* (Rarely) some docs specific to a given type stub package in `README` file.
+
+No other files are allowed in `stdlib` and `stubs`. When a third party stub is
+modified, an updated version of the corresponding distribution will be
+automatically uploaded to PyPI shortly (within few hours).
 
 For more information on directory structure and stub versioning, see
 [the relevant section of CONTRIBUTING.md](
@@ -87,8 +95,8 @@ following criteria is met:
 
 ## Contributing
 
-Please read [CONTRIBUTING.md](CONTRIBUTING.md) before submitting pull
-requests. If you have questions related to contributing, drop by the [typing Gitter](https://gitter.im/python/typing).
+Please read [CONTRIBUTING.md](CONTRIBUTING.md) before submitting pull requests.
+If you have questions related to contributing, drop by the [typing Gitter](https://gitter.im/python/typing).
 
 ## Running the tests