Dave Halter
94bd48bae1
Fix tokenizer: Dedents before error tokens are properly done, now.
2019-01-06 19:26:49 +01:00
Dave Halter
5da51720cd
Fix tokenizer: Dedents should only happen after newlines
2019-01-03 11:44:17 +01:00
Dave Halter
e05d7fd59f
Error recovery should not match the whole line in case of an invalid token, fixes #40
2018-07-03 01:31:32 +02:00
Dave Halter
8407894b25
Fix python 2 tests
2018-06-27 00:15:00 +02:00
Dave Halter
03de9cebb8
Introduce TokenTypes
2018-06-24 16:24:09 +02:00
Aaron Meurer
34154d05a0
Don't mutate the standard library token.tok_name dictionary ( #42 )
...
* Don't mutate the standard library token.tok_name dictionary
Fixes #41 .
* More robust test that tok_name isn't mutated
This test now works in Python 2.7, and actually tests something in Python 3.7,
and it's better anyway because it tests the whole dictionary instead of just
one token.
* Fix test_tok_name_copied in Python 3.7 and PyPy
Apparently Python 3.7 adds N_TOKENS to the tok_name dictionary, and PyPy
doesn't have NT_OFFSET in it.
2018-06-08 18:46:16 +02:00
Dave Halter
f20106d88e
Fix a prefix issue with error leafs.
2018-04-22 19:28:30 +02:00
Dave Halter
7c7f4f4e54
Fix a test
2018-04-05 00:45:23 +02:00
Dave Halter
b944fb9145
Fix a test around rb/br string literals.
2017-09-02 14:09:15 +02:00
Dave Halter
88c3c0567a
Rename TokenInfo to PythonToken.
2017-08-21 09:33:41 +02:00
Dave Halter
ab027885c7
Refactor splitlines -> split_lines.
2017-08-15 19:54:21 +02:00
Dave Halter
6876e1b7c0
Use ERROR_DEDENT instead of ERRORTOKEN for wrong dedents.
2017-07-19 20:15:06 +02:00
Dave Halter
78c371f73a
Refactor the version info to use a tuple, always.
2017-07-19 09:09:33 +02:00
Dave Halter
e731eecdd8
Make the tokenizer version independent.
2017-07-11 23:29:44 +02:00
Dave Halter
5b5e4a0616
Generate error tokens in the tokenizer if the indentation is dedented wrongly.
2017-07-09 10:44:24 +02:00
Dave Halter
c28ec5f182
Move tokenize.py and token.py to the python folder.
2017-07-09 00:06:14 +02:00
Dave Halter
adc3ec268a
Fix a 27 test.
2017-05-26 14:14:08 -04:00
Dave Halter
976c5151af
Rename source_tokens to tokenize and generate_tokens to tokenize_lines.
2017-05-26 12:01:35 -04:00
Dave Halter
f2c257d7dd
Move the utility function parse to the __init__ script.
2017-05-24 00:08:58 -04:00
Dave Halter
2d8d3cc157
Whitespace.
2017-05-17 14:16:36 -04:00
Dave Halter
b8005a4908
Remove a unittest.TestCase usage and replace it with pytest tests.
2017-05-17 14:15:57 -04:00
Dave Halter
a1675dfe2e
Change the last few tests to remove jedi completely and now tox -e py35 passes.
2017-05-15 15:19:41 -04:00
Dave Halter
206dfd113b
Use jedi less in the tests.
2017-05-15 13:53:30 -04:00
Dave Halter
7445c303e3
A test suite imported from jedi.
2017-05-11 09:43:37 -04:00
