Fix a nasty issue in the tokenizer. Fixes #836.

At the same time there was a related issue of not cleaning up newlines properly.
2017-01-24 00:50:37 +01:00
parent 741993a738
commit 09779c88aa
4 changed files with 29 additions and 13 deletions
--- a/jedi/parser/tokenize.py
+++ b/jedi/parser/tokenize.py
@@ -259,8 +259,7 @@ def generate_tokens(readline, use_exact_op_types=False):
                    # line is an error token.
                    txt = line[pos:]
                yield TokenInfo(ERRORTOKEN, txt, (lnum, pos), prefix)
-                pos += 1
-                continue
+                break

            prefix = additional_prefix + pseudomatch.group(1)
            additional_prefix = ''