add mypy_selftest.py (#1102)

* add mypy_selftest.py
* add selftest to Travis

.travis.yml

@@ -5,6 +5,8 @@ matrix:
   include:
     - python: "3.6-dev"
       env: TEST_CMD="flake8"
+    - python: "3.6"
+      env: TEST_CMD="./tests/mypy_selftest.py"
     - python: "3.5"
       env: TEST_CMD="./tests/mypy_test.py"
     - python: "2.7"

README.md

@@ -76,7 +76,7 @@ requests.
 ## Running the tests
 
 The tests are automatically run by Travis CI on every PR and push to
-the repo.  There are two separate sets of tests: `tests/mypy_test.py`
+the repo.  There are several sets of tests: `tests/mypy_test.py`
 runs tests against [mypy](https://github.com/python/mypy/), while
 `tests/pytype_tests.py` runs tests against
 [pytype](https://github.com/google/pytype/).
@@ -87,8 +87,14 @@ imported but they don't check whether stubs match their implementation
 that each set of tests has a blacklist of modules that are not tested
 at all.  The blacklists also live in the tests directory.
 
+In addition, you can run `tests/mypy_selftest.py` to run mypy's own
+test suite using the typeshed code in your repo. This will sometimes
+catch issues with incorrectly typed stubs, but is much slower than the
+other tests.
+
 To manually run the mypy tests, you need to have Python 3.5 or higher;
 Python 3.6.1 or higher is recommended.
+
 Run:
 ```
 $ python3.6 -m venv .venv3
@@ -101,6 +107,8 @@ invoking:
 ```
 (.venv3)$ python3 tests/mypy_test.py
 ...
+(.venv3)$ python tests/mypy_selftest.py
+...
 (.venv3)$ flake8
 ...
 ```

tests/mypy_selftest.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+"""Script to run mypy's test suite against this version of typeshed."""
+
+from pathlib import Path
+import shutil
+import subprocess
+import sys
+import tempfile
+
+
+if __name__ == '__main__':
+    with tempfile.TemporaryDirectory() as tempdir:
+        dirpath = Path(tempdir)
+        subprocess.run(['git', 'clone', '--depth', '1', 'git://github.com/python/mypy',
+                        str(dirpath / 'mypy')], check=True)
+        subprocess.run([sys.executable, '-m', 'pip', 'install', '-U', '-r',
+                        str(dirpath / 'mypy/test-requirements.txt')], check=True)
+        shutil.copytree('stdlib', str(dirpath / 'mypy/typeshed/stdlib'))
+        shutil.copytree('third_party', str(dirpath / 'mypy/typeshed/third_party'))
+        try:
+            subprocess.run(['./runtests.py'], cwd=str(dirpath / 'mypy'), check=True)
+        except subprocess.CalledProcessError as e:
+            print('mypy tests failed', file=sys.stderr)
+            sys.exit(e.returncode)
+        else:
+            print('mypy tests succeeded', file=sys.stderr)
+            sys.exit(0)