Dave Halter
95f353a15f
Merge branch 'rm-2.6' of https://github.com/hugovk/parso
2020-01-05 23:50:20 +01:00
yangyang
b90e5cd758
fixed #86 and #87
2019-12-05 19:22:58 +08:00
Hugo
f6bdba65c0
Drop support for EOL Python 2.6
2019-08-08 14:56:27 +03:00
Dave Halter
0a5b5f3346
Fix name tokenizing for Python 2
2019-07-13 15:34:23 +02:00
Dave Halter
99dd4a84d4
Merge branch 'master' of github.com:davidhalter/parso
2019-07-12 21:35:06 +02:00
Dave Halter
9501b0bde0
Fixed name tokenizing issues for tamil characters, fixes davidhalter/jedi#1368
2019-07-12 21:31:49 +02:00
Benjamin Woodruff
ad57a51800
Fix line continuation characters inside f-strings
...
Line continuation characters are valid inside of strings, but weren't
handled correctly in certain cases with f-strings, due to some small
tokenizer bugs.
This pull request to address those issues, and adds tests to validate
the new logic.
2019-07-12 21:20:00 +02:00
Dave Halter
60e4591837
Fix: End detection for strings was mostly wrong, fixes #51
2019-01-23 10:13:25 +01:00
Dave Halter
ef56debb78
Fix f-string escapes, fixes #48
...
The tokenizer was not detecting backslash escapes for f-string endings properly
2019-01-22 22:20:32 +01:00
Dave Halter
9cc8178998
Fix tokenizer: backslashes sometimes led to newline token generation
2019-01-14 09:59:16 +01:00
Dave Halter
d7171ae927
Fix tokenizer: Carriage returns after backslashes were not properly handled
2019-01-14 01:49:09 +01:00
Dave Halter
7ae1efe5c7
Fix tokenizer: Form feeds and multiline docstrings didn't work together
2019-01-13 23:16:09 +01:00
Dave Halter
dd1761da96
Fix tokenizer: Closing parentheses in the wrong place should not lead to strange behavior
2019-01-13 14:52:33 +01:00
Dave Halter
94bd48bae1
Fix tokenizer: Dedents before error tokens are properly done, now.
2019-01-06 19:26:49 +01:00
Dave Halter
5da51720cd
Fix tokenizer: Dedents should only happen after newlines
2019-01-03 11:44:17 +01:00
Dave Halter
e05d7fd59f
Error recovery should not match the whole line in case of an invalid token, fixes #40
2018-07-03 01:31:32 +02:00
Dave Halter
8407894b25
Fix python 2 tests
2018-06-27 00:15:00 +02:00
Dave Halter
03de9cebb8
Introduce TokenTypes
2018-06-24 16:24:09 +02:00
Aaron Meurer
34154d05a0
Don't mutate the standard library token.tok_name dictionary ( #42 )
...
* Don't mutate the standard library token.tok_name dictionary
Fixes #41 .
* More robust test that tok_name isn't mutated
This test now works in Python 2.7, and actually tests something in Python 3.7,
and it's better anyway because it tests the whole dictionary instead of just
one token.
* Fix test_tok_name_copied in Python 3.7 and PyPy
Apparently Python 3.7 adds N_TOKENS to the tok_name dictionary, and PyPy
doesn't have NT_OFFSET in it.
2018-06-08 18:46:16 +02:00
Dave Halter
f20106d88e
Fix a prefix issue with error leafs.
2018-04-22 19:28:30 +02:00
Dave Halter
7c7f4f4e54
Fix a test
2018-04-05 00:45:23 +02:00
Dave Halter
b944fb9145
Fix a test around rb/br string literals.
2017-09-02 14:09:15 +02:00
Dave Halter
88c3c0567a
Rename TokenInfo to PythonToken.
2017-08-21 09:33:41 +02:00
Dave Halter
ab027885c7
Refactor splitlines -> split_lines.
2017-08-15 19:54:21 +02:00
Dave Halter
6876e1b7c0
Use ERROR_DEDENT instead of ERRORTOKEN for wrong dedents.
2017-07-19 20:15:06 +02:00
Dave Halter
78c371f73a
Refactor the version info to use a tuple, always.
2017-07-19 09:09:33 +02:00
Dave Halter
e731eecdd8
Make the tokenizer version independent.
2017-07-11 23:29:44 +02:00
Dave Halter
5b5e4a0616
Generate error tokens in the tokenizer if the indentation is dedented wrongly.
2017-07-09 10:44:24 +02:00
Dave Halter
c28ec5f182
Move tokenize.py and token.py to the python folder.
2017-07-09 00:06:14 +02:00
Dave Halter
adc3ec268a
Fix a 27 test.
2017-05-26 14:14:08 -04:00
Dave Halter
976c5151af
Rename source_tokens to tokenize and generate_tokens to tokenize_lines.
2017-05-26 12:01:35 -04:00
Dave Halter
f2c257d7dd
Move the utility function parse to the __init__ script.
2017-05-24 00:08:58 -04:00
Dave Halter
2d8d3cc157
Whitespace.
2017-05-17 14:16:36 -04:00
Dave Halter
b8005a4908
Remove a unittest.TestCase usage and replace it with pytest tests.
2017-05-17 14:15:57 -04:00
Dave Halter
a1675dfe2e
Change the last few tests to remove jedi completely and now tox -e py35 passes.
2017-05-15 15:19:41 -04:00
Dave Halter
206dfd113b
Use jedi less in the tests.
2017-05-15 13:53:30 -04:00
Dave Halter
7445c303e3
A test suite imported from jedi.
2017-05-11 09:43:37 -04:00
