QuerySet.annotate improvements (#398)

* QuerySet.annotate returns self-type. Attribute access falls back to Any.

- QuerySets that have an annotated model do not report errors during .filter() when called with invalid fields.
- QuerySets that have an annotated model return ordinary dict rather than TypedDict for .values()
- QuerySets that have an annotated model return Any rather than typed Tuple for .values_list()

* Fix .annotate so it reuses existing annotated types. Fixes error in typechecking Django testsuite.

* Fix self-typecheck error

* Fix flake8

* Fix case of .values/.values_list before .annotate.

* Extra ignores for Django 2.2 tests (false positives due to tests assuming QuerySet.first() won't return None)

Fix mypy self-check.

* More tests + more precise typing in case annotate called before values_list.

Cleanup tests.

* Test and fix annotate in combination with values/values_list with no params.

* Remove line that does nothing :)

* Formatting fixes

* Address code review

* Fix quoting in tests after mypy changed things

* Use Final

* Use typing_extensions.Final

* Fixes after ValuesQuerySet -> _ValuesQuerySet refactor. Still not passing tests yet.

* Fix inheritance of _ValuesQuerySet and remove unneeded type ignores.

This allows the test
"annotate_values_or_values_list_before_or_after_annotate_broadens_type"
to pass.

* Make it possible to annotate user code with "annotated models", using PEP 583 Annotated type.

* Add docs

* Make QuerySet[_T] an external alias to _QuerySet[_T, _T].

This currently has the drawback that error messages display the internal type _QuerySet, with both type arguments.

See also discussion on #661 and #608.

Fixes #635: QuerySet methods on Managers (like .all()) now return QuerySets rather than Managers.

Address code review by @sobolevn.

* Support passing TypedDicts to WithAnnotations

* Add an example of an error to README regarding WithAnnotations + TypedDict.

* Fix runtime behavior of ValuesQuerySet alias (you can't extend Any, for example).

Fix some edge case with from_queryset after QuerySet changed to be an
alias to _QuerySet. Can't make a minimal test case as this only occurred
on a large internal codebase.

* Fix issue when using from_queryset in some cases when having an argument with a type annotation on the QuerySet.

The mypy docstring on anal_type says not to call defer() after it.

mypy_django_plugin/main.py

@@ -10,6 +10,7 @@ from mypy.modulefinder import mypy_path
 from mypy.nodes import MypyFile, TypeInfo
 from mypy.options import Options
 from mypy.plugin import (
+    AnalyzeTypeContext,
     AttributeContext,
     ClassDefContext,
     DynamicClassDefContext,
@@ -24,7 +25,11 @@ from mypy_django_plugin.django.context import DjangoContext
 from mypy_django_plugin.lib import fullnames, helpers
 from mypy_django_plugin.transformers import fields, forms, init_create, meta, querysets, request, settings
 from mypy_django_plugin.transformers.managers import create_new_manager_class_from_from_queryset_method
-from mypy_django_plugin.transformers.models import process_model_class, set_auth_user_model_boolean_fields
+from mypy_django_plugin.transformers.models import (
+    handle_annotated_type,
+    process_model_class,
+    set_auth_user_model_boolean_fields,
+)
 
 
 def transform_model_class(ctx: ClassDefContext, django_context: DjangoContext) -> None:
@@ -230,7 +235,7 @@ class NewSemanalDjangoPlugin(Plugin):
                 related_model_module = related_model_cls.__module__
                 if related_model_module != file.fullname:
                     deps.add(self._new_dependency(related_model_module))
-        return list(deps)
+        return list(deps) + [self._new_dependency("django_stubs_ext")]  # for annotate
 
     def get_function_hook(self, fullname: str) -> Optional[Callable[[FunctionContext], MypyType]]:
         if fullname == "django.contrib.auth.get_user_model":
@@ -261,22 +266,28 @@ class NewSemanalDjangoPlugin(Plugin):
             if info and info.has_base(fullnames.FORM_MIXIN_CLASS_FULLNAME):
                 return forms.extract_proper_type_for_get_form
 
+        manager_classes = self._get_current_manager_bases()
+
         if method_name == "values":
             info = self._get_typeinfo_or_none(class_fullname)
-            if info and info.has_base(fullnames.QUERYSET_CLASS_FULLNAME):
+            if info and info.has_base(fullnames.QUERYSET_CLASS_FULLNAME) or class_fullname in manager_classes:
                 return partial(querysets.extract_proper_type_queryset_values, django_context=self.django_context)
 
         if method_name == "values_list":
             info = self._get_typeinfo_or_none(class_fullname)
-            if info and info.has_base(fullnames.QUERYSET_CLASS_FULLNAME):
+            if info and info.has_base(fullnames.QUERYSET_CLASS_FULLNAME) or class_fullname in manager_classes:
                 return partial(querysets.extract_proper_type_queryset_values_list, django_context=self.django_context)
 
+        if method_name == "annotate":
+            info = self._get_typeinfo_or_none(class_fullname)
+            if info and info.has_base(fullnames.QUERYSET_CLASS_FULLNAME) or class_fullname in manager_classes:
+                return partial(querysets.extract_proper_type_queryset_annotate, django_context=self.django_context)
+
         if method_name == "get_field":
             info = self._get_typeinfo_or_none(class_fullname)
             if info and info.has_base(fullnames.OPTIONS_CLASS_FULLNAME):
                 return partial(meta.return_proper_field_type_from_get_field, django_context=self.django_context)
 
-        manager_classes = self._get_current_manager_bases()
         if class_fullname in manager_classes and method_name == "create":
             return partial(init_create.redefine_and_typecheck_model_create, django_context=self.django_context)
         if class_fullname in manager_classes and method_name in {"filter", "get", "exclude"}:
@@ -314,6 +325,14 @@ class NewSemanalDjangoPlugin(Plugin):
             return partial(set_auth_user_model_boolean_fields, django_context=self.django_context)
         return None
 
+    def get_type_analyze_hook(self, fullname: str) -> Optional[Callable[[AnalyzeTypeContext], MypyType]]:
+        if fullname in (
+            "typing.Annotated",
+            "typing_extensions.Annotated",
+            "django_stubs_ext.annotations.WithAnnotations",
+        ):
+            return partial(handle_annotated_type, django_context=self.django_context)
+
     def get_dynamic_class_hook(self, fullname: str) -> Optional[Callable[[DynamicClassDefContext], None]]:
         if fullname.endswith("from_queryset"):
             class_name, _, _ = fullname.rpartition(".")