This removes the _infer_type_vars util in favour of a polymorphic
implementation, removing the conditional checks on the type of
the annotation instance.
While for the moment this creates some circular imports, further
refactoring to follow should be able to remove those.