Rework the parser so we can use arbitrary start nodes of the syntax.

This also includes a rework for error recovery in the parser. This is now just possible for file_input parsing, which means for full files.
Includes also a refactoring of the tokenizer. No more do we have to add an additional newline, because it now works correctly (removes certain confusion.
This commit is contained in:
Dave Halter
2015-12-20 22:21:47 +01:00
parent 9a93d599da
commit c4906e0e3f
22 changed files with 246 additions and 198 deletions

View File

@@ -1,11 +1,11 @@
from jedi._compatibility import u
from jedi.parser import Parser, load_grammar
from jedi.parser import ParserWithRecovery, load_grammar
def test_basic_parsing():
def compare(string):
"""Generates the AST object and then regenerates the code."""
assert Parser(load_grammar(), string).module.get_code() == string
assert ParserWithRecovery(load_grammar(), string).module.get_code() == string
compare(u('\na #pass\n'))
compare(u('wblabla* 1\t\n'))