69 Commits

Author	SHA1	Message	Date
Dave Halter	553ff66c8b	remove last_previous from NoErrorTokenizer	2014-02-23 12:51:05 +01:00
Dave Halter	c5fcebde82	changed _compatibility.utf8 -> 'u' and removed a lot of the issues with the now enforced unicode source input of the parser	2014-02-23 11:29:00 +01:00
Dave Halter	9c2fcfffd7	StringIo is now always used as part of the io library	2014-02-21 17:52:09 +01:00
Dave Halter	d9bf9be35e	small refactorings of the tokenizer	2014-02-21 17:45:56 +01:00
Dave Halter	efba0cd80c	deleted a big part of the generate_tokens code that doesn't seem to be needed anymore	2014-02-21 16:38:15 +01:00
Dave Halter	73057d4176	removed the INDENT usages that were left in the parser code	2014-02-21 15:39:14 +01:00
Dave Halter	fe02088dd7	restructure user_context reverse tokenizer to remove INDENT tokens in the future	2014-02-21 15:30:46 +01:00
Dave Halter	9c3b7b9c46	get rid of more variables	2014-02-21 10:44:33 +01:00
Dave Halter	184e90a3a0	removed some line checks from tokenizer	2014-02-21 00:13:23 +01:00
Dave Halter	a69eee5876	error tokens only exist in the end in a tokenizer	2014-02-21 00:09:49 +01:00
Dave Halter	3232ae5b0c	removed parentheses counting from generate_tokens	2014-02-20 18:45:22 +01:00
Dave Halter	d7033726fd	tokenize removed NL/NEWLINE ambiguity	2014-02-20 01:52:30 +01:00
Dave Halter	c26ae3c00d	ignore dedents in tokenizer	2014-02-20 01:21:20 +01:00
Dave Halter	3e9b72b636	created a PushBackTokenizer specifically for the parser	2014-02-20 01:17:19 +01:00
Dave Halter	c8d6fbb0a1	temporary changes for the tokenizer issues	2014-02-20 00:43:42 +01:00
Dave Halter	1b6df4602d	replace parser tokenizer offset with line offset	2014-02-17 23:02:04 +01:00
Dave Halter	6952596117	implement an offset in generate_tokens	2014-02-17 10:08:32 +01:00
Dave Halter	bb111daf91	removed line from tokenizer	2014-02-16 15:28:18 +01:00
Dave Halter	2e65fbb00f	rename parser.tokenizer -> parser.tokenize	2014-01-05 13:38:14 +01:00