Dave Halter
e8653a49ff
Make is_definition work on setitem modifications, see #66
2019-09-04 09:52:55 +02:00
Dave Halter
e723b3e74b
Refactor the ambiguity tests a bit, see #70
2019-07-13 20:15:56 +02:00
Benjamin Woodruff
0032bae041
Make pgen2's grammar ambiguity detection handle more cases
...
Under the old implementation,
```
outer: A [inner] B C
inner: B C [inner]
```
wouldn't get detected as the ambiguous grammar that it is, whereas
```
outer: A rest
rest: [inner] B C
inner: B C [inner]
```
would.
This would manifest itself as non-determinism in the DFA state
generation. See the discussion #62 on for a full explanation.
This modifies the ambiguity detection to work on a broader class of
issues, so it should now hopefully detect all cases where the given
grammar is ambiguous.
At some point, we could extend this logic to allow developers to
optionally set precedence of grammar productions, which could resolve
ambiguities, but that's not a strict requirement for parsing python.
2019-07-13 20:04:32 +02:00
Dave Halter
0a5b5f3346
Fix name tokenizing for Python 2
2019-07-13 15:34:23 +02:00
Dave Halter
99dd4a84d4
Merge branch 'master' of github.com:davidhalter/parso
2019-07-12 21:35:06 +02:00
Dave Halter
9501b0bde0
Fixed name tokenizing issues for tamil characters, fixes davidhalter/jedi#1368
2019-07-12 21:31:49 +02:00
Benjamin Woodruff
ad57a51800
Fix line continuation characters inside f-strings
...
Line continuation characters are valid inside of strings, but weren't
handled correctly in certain cases with f-strings, due to some small
tokenizer bugs.
This pull request to address those issues, and adds tests to validate
the new logic.
2019-07-12 21:20:00 +02:00
Dave Halter
ecbe2b9926
Add positional only arguments to grammar
2019-06-09 21:15:03 +02:00
Dave Halter
a7aa23a7f0
Parse named expressions
2019-06-02 23:34:37 +02:00
Dave Halter
5430415d44
Change a test, because it doesn't really matter
...
The test had changed behavior for Python 3.8, a syntax error of:
SyntaxError: unexpected EOF while parsing
instead of
SyntaxError: invalid syntax
2019-06-02 22:54:45 +02:00
Dave Halter
6cdd47fe2b
f-string syntax in Python 3.8 was enhanced
...
See e.g. https://twitter.com/raymondh/status/1135253771846471680
2019-06-02 22:48:47 +02:00
Dave Halter
917b4421f3
Fix fstring format spec parsing, fixes #74
2019-06-02 15:18:42 +02:00
prim
93ddf5322a
parse long number notation ( #72 )
...
* parse long number notation
* parse long number notation
2019-06-02 11:14:15 +02:00
Dave Halter
b2ab64d8f9
Fix Python 3.8 error issues
2019-04-05 18:30:48 +02:00
Dave Halter
dfe7fba08e
continue in finally is no longer an error
2019-04-05 16:17:30 +02:00
Dave Halter
3e2956264c
Add FileIO to make it possible to cache e.g. files from zip files
2019-03-25 00:48:59 +01:00
Dave Halter
3f6fc8a5ad
Fix an f-string tokenizer issue
2019-02-13 00:17:37 +01:00
Dave Halter
58850f8bfa
Rename a test
2019-02-06 09:51:46 +01:00
Dave Halter
d38a60278e
Remove some unused code
2019-02-06 09:50:27 +01:00
Dave Halter
6c65aea47d
Fix working with async functions in the diff parser, fixes #56
2019-02-06 09:31:46 +01:00
Dave Halter
0d37ff865c
Fix bytes/fstring mixing when using iter_errors, fixes #57 .
2019-02-06 01:28:47 +01:00
Dave Halter
bb2855897b
Escape a backslash properly
2019-01-24 00:31:26 +01:00
Dave Halter
32a83b932a
Fix get_start_pos_of_prefix
2019-01-24 00:00:06 +01:00
Dave Halter
60e4591837
Fix: End detection for strings was mostly wrong, fixes #51
2019-01-23 10:13:25 +01:00
Dave Halter
ef56debb78
Fix f-string escapes, fixes #48
...
The tokenizer was not detecting backslash escapes for f-string endings properly
2019-01-22 22:20:32 +01:00
Dave Halter
ce8b531175
Fix diff parser: The previous fix was a bit off
2019-01-20 19:03:45 +01:00
Dave Halter
069c08883a
Change fuzzer: Add ways to not always use correct parse input
2019-01-20 18:18:13 +01:00
Dave Halter
0da0a8655a
Fix diff parser: issue with opening brackets
2019-01-20 00:41:11 +01:00
Dave Halter
3d890c3a00
Async doesn't work in 3.4
2019-01-19 12:59:35 +01:00
Dave Halter
956ea55048
Skip some tests for Python2.6 and Python3.3
2019-01-19 12:08:39 +01:00
Dave Halter
0bd17bee2c
Fix diff parser: DEDENT as error leaves should also be ignored and reparsed
2019-01-18 18:41:08 +01:00
Dave Halter
f3015efb2d
Fix diff parser: error dedents in between nodes should be ignored for now when copying
2019-01-18 02:43:12 +01:00
Dave Halter
197391dc53
Fix diff parser: Don't copy error nodes/leaves in the beginning, leads to strange issues in some cases
2019-01-17 23:48:00 +01:00
Dave Halter
32321a74b1
Diff fuzzer: Create a check to see if the errors make sense.
2019-01-17 22:07:59 +01:00
Dave Halter
52d01685ba
Fix diff parser: Don't copy DEDENT tokens at the beginning
2019-01-17 21:31:13 +01:00
Dave Halter
e591b929eb
Fix diff parser: Skip last leaves for last line offset leaves
2019-01-17 00:15:38 +01:00
Dave Halter
dac4c445a7
Fix indentation error tokens
2019-01-16 23:21:31 +01:00
Dave Halter
20fd32b45d
Fix diff parser: Avoid side effects for prefix
2019-01-14 21:37:19 +01:00
Dave Halter
9cc8178998
Fix tokenizer: backslashes sometimes led to newline token generation
2019-01-14 09:59:16 +01:00
Dave Halter
1e25445176
Make lines easier copyable in the fuzzer
2019-01-14 01:50:39 +01:00
Dave Halter
d7171ae927
Fix tokenizer: Carriage returns after backslashes were not properly handled
2019-01-14 01:49:09 +01:00
Dave Halter
d3d28480ed
Fix in diff parser: prefix calculation was wrong when copying nodes
2019-01-14 01:00:17 +01:00
Dave Halter
564be7882e
Replace --print-diff with --print-code
2019-01-14 00:20:49 +01:00
Dave Halter
76c5754b76
Fix diff parser generation for empty files
2019-01-13 23:38:35 +01:00
Dave Halter
55247a5a2c
Docopt should not be needed for tests
2019-01-13 23:24:17 +01:00
Dave Halter
7ae1efe5c7
Fix tokenizer: Form feeds and multiline docstrings didn't work together
2019-01-13 23:16:09 +01:00
Dave Halter
01dba7f8ce
Fix diff parser: Need to calculate the prefix for the diff tokenizer better
2019-01-13 22:38:53 +01:00
Dave Halter
a7e24a37e7
Fix newline endings and a few parser/copy counts
2019-01-13 19:55:18 +01:00
Dave Halter
f80d9de7a0
Feature: The diff parser fuzzer is now able to use random Python fragments
...
This hopefully leads to the fuzzer finding more and faster issues in the diff
parser.
2019-01-13 16:00:36 +01:00
Dave Halter
dd1761da96
Fix tokenizer: Closing parentheses in the wrong place should not lead to strange behavior
2019-01-13 14:52:33 +01:00
