e20f2069ba
Move the grammar to a fitting file.
2018-06-26 10:20:05 +02:00
4cf198285a
Move things out of the grammar class
2018-06-26 10:15:31 +02:00
30cf491b4f
Move the Grammar to the pgen module
2018-06-26 10:08:44 +02:00
c1675da0cb
Make nonterminal_to_dfas public
2018-06-26 09:56:49 +02:00
7b7b66eb3c
Get rid of the first_terminal variable in the grammar generator
2018-06-26 09:48:13 +02:00
5d46c3e18b
Trying to reduce the amount of variables used in first sets
2018-06-26 01:04:22 +02:00
e9fde82512
Remove the overlapcheck, it's probably not needed anymore
2018-06-26 01:00:06 +02:00
a46ecbb499
Fix an ambiguity issue
...
Unfortunately had to refactor most of the transition generation
2018-06-26 00:58:19 +02:00
da5aa8a2ab
Better detection of ambiguities
2018-06-25 01:56:02 +02:00
43d4a8a834
Don't use a function that doesn't work
2018-06-24 23:45:18 +02:00
309033ae2d
Work with token types whenever possible
2018-06-24 17:59:32 +02:00
2a9d8632fe
Remove label caching
2018-06-24 17:56:13 +02:00
530a324643
Remove labels
2018-06-24 17:55:22 +02:00
71003bc20e
Just use caching instead of strange transitions
2018-06-24 17:29:03 +02:00
c5d141bf60
Make some more things faster
2018-06-24 16:43:28 +02:00
e958b241c7
Use some tokenize names directly
2018-06-24 16:39:48 +02:00
34ab35558f
Remove a lot of the old token code
2018-06-24 16:31:58 +02:00
03de9cebb8
Introduce TokenTypes
2018-06-24 16:24:09 +02:00
6098d89150
Add PythonTokens to get rid of a lot of the token module eventually
2018-06-24 13:03:07 +02:00
ff4358cd97
Merge remote-tracking branch 'origin/master' into pgen
2018-06-24 11:49:34 +02:00
b5378e4602
Use token.OP and use reserved words
...
This change breaks the tokenizer backwards compatibility a bit. Details of
operators is now part of the parser and not the tokenizer anymore. The parser
does this anyway, so we don't need the complexity in the tokenizer.
2018-06-24 11:28:23 +02:00
33e321a539
Don't set the root node before it's not actually defined
2018-06-22 13:04:00 +02:00
a890ddd6cc
Remove make_first
2018-06-22 12:54:27 +02:00
1362d4f05d
Remove more unused grammar stuff
2018-06-22 12:53:38 +02:00
532aef2342
Remove nonterminal2number and number2nonterminal, they are no longer used
2018-06-22 12:52:44 +02:00
878b4b2d3b
Use nonterminals instead of numbers if possible
2018-06-22 12:47:02 +02:00
87299335c4
Remove more unused code
2018-06-22 12:36:48 +02:00
4f0e9c0fd7
Remove old dfas and states from the parser generator
2018-06-22 11:47:18 +02:00
67ca091631
delete a lot of the old parser code
2018-06-22 11:44:42 +02:00
4e5ba02dbb
Fix the final issues of the new parser
2018-06-22 11:38:34 +02:00
a85f544901
Fix all tests except diff tests. Mostly error recovery fixes
2018-06-22 11:12:10 +02:00
9e8066c6fd
Fix a lot of the old error recovery
2018-06-22 09:56:58 +02:00
68eab72229
Some slight changes to error recovery
2018-06-22 01:59:39 +02:00
d9264609f2
Get quite a bit of the error recovery working
2018-06-22 01:56:29 +02:00
79c7e0b59d
Use the new parser. Error recovery is not yet working
2018-06-22 00:38:18 +02:00
f03a87b876
Actually parse some first things with the new approach
2018-06-21 23:56:34 +02:00
2a082d69df
Add better reprs
2018-06-21 22:13:27 +02:00
e6fc739670
Get most things ready for plans
2018-06-21 21:32:05 +02:00
12e11b3d16
Remove a while loop that is not necessary
2018-06-21 21:14:14 +02:00
cc8038966b
isfinal -> is_final
2018-06-21 21:11:46 +02:00
31aecf2d35
Calculate the first plans in a very messy way
2018-06-21 21:10:28 +02:00
d8554d86d1
A lot of new code to hopefully transition to a better parsing mechanism in the future
2018-06-21 18:17:32 +02:00
d691bf0fd1
Some minor changes
2018-06-20 09:42:21 +02:00
5712ffb5ca
Introduce a label cache that is currently not used
2018-06-18 20:14:29 +02:00
55d6a69aad
Some more renames
2018-06-18 01:14:09 +02:00
453471eeb6
Move some ParserGenerator stuff into the Grammar class
2018-06-18 00:15:31 +02:00
a06c3a3129
name -> nonterminal
2018-06-17 23:10:27 +02:00
73ce57428b
Try to completely remove the word symbol and use nonterminal
...
The ones that we could not remove are in grammar.py, because that's the public documented API.
2018-06-17 18:30:20 +02:00
640f544af9
One instance of symbol -> terminal
2018-06-17 18:12:13 +02:00
b6cbf306d7
Use the name first_terminals instead of first
2018-06-17 18:09:12 +02:00
Dave Halter