Dave Halter
|
18dc92f85f
|
removed a few old/unnecessary tokenize definitions
|
2014-04-28 18:33:40 +02:00 |
|
Dave Halter
|
a49c624154
|
tokenize corrections, add unicode literals, because they had been removed from Python 3.2 (reintroduced in 3.3)
|
2014-04-22 15:17:48 +02:00 |
|
Dave Halter
|
bb6874bc7c
|
fix for problems with incomplete one liner string literals, after a start of an incomplete string literal the whole line should be seen as an error token
|
2014-04-19 13:56:29 +02:00 |
|
Dave Halter
|
2e12eb7861
|
start with the integration of an Operator class to make way for precedences
|
2014-02-26 14:44:51 +01:00 |
|
Dave Halter
|
e152939791
|
remove encoding stuff from tokenizer - encoding is always unicode
|
2014-02-26 12:55:32 +01:00 |
|
Dave Halter
|
40be00826e
|
clean up tokenize
|
2014-02-25 17:17:33 +01:00 |
|
Dave Halter
|
761c28ef00
|
remove __getitem__ from Token
|
2014-02-25 17:03:56 +01:00 |
|
Dave Halter
|
18e985a961
|
TokenInfo -> Token
|
2014-02-25 16:44:48 +01:00 |
|
Dave Halter
|
2db26abf72
|
start and end don't exst anymore in parser.token.Token, it's now start_pos/end_pos as everywhere else
|
2014-02-25 16:34:27 +01:00 |
|
Dave Halter
|
aea2c4620f
|
more unicode switches in the parser
|
2014-02-25 14:27:50 +01:00 |
|
Dave Halter
|
f4f79317fe
|
start uniting tokenize.TokenInfo and token.Token
|
2014-02-25 13:54:18 +01:00 |
|
Dave Halter
|
5b84f0b27f
|
remove end_pos stuff from tokenizer, the tokens can do that themselves
|
2014-02-25 13:29:27 +01:00 |
|
Dave Halter
|
3a23c80ae5
|
prepare for eventual? tokenizer end_pos replacement.
|
2014-02-25 11:59:10 +01:00 |
|
Dave Halter
|
246118f851
|
start using @ganwell's new token class (modified in some ways) as the main token class - hope to gain a little bit of memory/cpu/pickling performance
|
2014-02-25 02:06:26 +01:00 |
|
Dave Halter
|
9943bb6205
|
remove some old parameters from Parser and FastTokenizer
|
2014-02-24 11:24:54 +01:00 |
|
Dave Halter
|
7db090a48a
|
moved NoErrorTokenizer to fast.FastTokenizer
|
2014-02-24 11:05:31 +01:00 |
|
Dave Halter
|
553ff66c8b
|
remove last_previous from NoErrorTokenizer
|
2014-02-23 12:51:05 +01:00 |
|
Dave Halter
|
c5fcebde82
|
changed _compatibility.utf8 -> 'u' and removed a lot of the issues with the now enforced unicode source input of the parser
|
2014-02-23 11:29:00 +01:00 |
|
Dave Halter
|
9c2fcfffd7
|
StringIo is now always used as part of the io library
|
2014-02-21 17:52:09 +01:00 |
|
Dave Halter
|
d9bf9be35e
|
small refactorings of the tokenizer
|
2014-02-21 17:45:56 +01:00 |
|
Dave Halter
|
efba0cd80c
|
deleted a big part of the generate_tokens code that doesn't seem to be needed anymore
|
2014-02-21 16:38:15 +01:00 |
|
Dave Halter
|
73057d4176
|
removed the INDENT usages that were left in the parser code
|
2014-02-21 15:39:14 +01:00 |
|
Dave Halter
|
fe02088dd7
|
restructure user_context reverse tokenizer to remove INDENT tokens in the future
|
2014-02-21 15:30:46 +01:00 |
|
Dave Halter
|
9c3b7b9c46
|
get rid of more variables
|
2014-02-21 10:44:33 +01:00 |
|
Dave Halter
|
184e90a3a0
|
removed some line checks from tokenizer
|
2014-02-21 00:13:23 +01:00 |
|
Dave Halter
|
a69eee5876
|
error tokens only exist in the end in a tokenizer
|
2014-02-21 00:09:49 +01:00 |
|
Dave Halter
|
3232ae5b0c
|
removed parentheses counting from generate_tokens
|
2014-02-20 18:45:22 +01:00 |
|
Dave Halter
|
d7033726fd
|
tokenize removed NL/NEWLINE ambiguity
|
2014-02-20 01:52:30 +01:00 |
|
Dave Halter
|
c26ae3c00d
|
ignore dedents in tokenizer
|
2014-02-20 01:21:20 +01:00 |
|
Dave Halter
|
3e9b72b636
|
created a PushBackTokenizer specifically for the parser
|
2014-02-20 01:17:19 +01:00 |
|
Dave Halter
|
c8d6fbb0a1
|
temporary changes for the tokenizer issues
|
2014-02-20 00:43:42 +01:00 |
|
Dave Halter
|
1b6df4602d
|
replace parser tokenizer offset with line offset
|
2014-02-17 23:02:04 +01:00 |
|
Dave Halter
|
6952596117
|
implement an offset in generate_tokens
|
2014-02-17 10:08:32 +01:00 |
|
Dave Halter
|
bb111daf91
|
removed line from tokenizer
|
2014-02-16 15:28:18 +01:00 |
|
Dave Halter
|
2e65fbb00f
|
rename parser.tokenizer -> parser.tokenize
|
2014-01-05 13:38:14 +01:00 |
|