 Dave Halter
|
d9bf9be35e
|
small refactorings of the tokenizer
|
2014-02-21 17:45:56 +01:00 |
|
|
Dave Halter
|
efba0cd80c
|
deleted a big part of the generate_tokens code that doesn't seem to be needed anymore
|
2014-02-21 16:38:15 +01:00 |
|
|
Dave Halter
|
73057d4176
|
removed the INDENT usages that were left in the parser code
|
2014-02-21 15:39:14 +01:00 |
|
|
Dave Halter
|
fe02088dd7
|
restructure user_context reverse tokenizer to remove INDENT tokens in the future
|
2014-02-21 15:30:46 +01:00 |
|
|
Dave Halter
|
9c3b7b9c46
|
get rid of more variables
|
2014-02-21 10:44:33 +01:00 |
|
|
Dave Halter
|
184e90a3a0
|
removed some line checks from tokenizer
|
2014-02-21 00:13:23 +01:00 |
|
|
Dave Halter
|
a69eee5876
|
error tokens only exist in the end in a tokenizer
|
2014-02-21 00:09:49 +01:00 |
|
|
Dave Halter
|
3232ae5b0c
|
removed parentheses counting from generate_tokens
|
2014-02-20 18:45:22 +01:00 |
|
|
Dave Halter
|
d7033726fd
|
tokenize removed NL/NEWLINE ambiguity
|
2014-02-20 01:52:30 +01:00 |
|
|
Dave Halter
|
c26ae3c00d
|
ignore dedents in tokenizer
|
2014-02-20 01:21:20 +01:00 |
|
|
Dave Halter
|
3e9b72b636
|
created a PushBackTokenizer specifically for the parser
|
2014-02-20 01:17:19 +01:00 |
|
|
Dave Halter
|
c8d6fbb0a1
|
temporary changes for the tokenizer issues
|
2014-02-20 00:43:42 +01:00 |
|
|
Dave Halter
|
1b6df4602d
|
replace parser tokenizer offset with line offset
|
2014-02-17 23:02:04 +01:00 |
|
|
Dave Halter
|
6952596117
|
implement an offset in generate_tokens
|
2014-02-17 10:08:32 +01:00 |
|
|
Dave Halter
|
bb111daf91
|
removed line from tokenizer
|
2014-02-16 15:28:18 +01:00 |
|
|
Dave Halter
|
2e65fbb00f
|
rename parser.tokenizer -> parser.tokenize
|
2014-01-05 13:38:14 +01:00 |
|