11 Commits

Author	SHA1	Message	Date
Dave Halter	184e90a3a0	removed some line checks from tokenizer	2014-02-21 00:13:23 +01:00
Dave Halter	a69eee5876	error tokens only exist in the end in a tokenizer	2014-02-21 00:09:49 +01:00
Dave Halter	3232ae5b0c	removed parentheses counting from generate_tokens	2014-02-20 18:45:22 +01:00
Dave Halter	d7033726fd	tokenize removed NL/NEWLINE ambiguity	2014-02-20 01:52:30 +01:00
Dave Halter	c26ae3c00d	ignore dedents in tokenizer	2014-02-20 01:21:20 +01:00
Dave Halter	3e9b72b636	created a PushBackTokenizer specifically for the parser	2014-02-20 01:17:19 +01:00
Dave Halter	c8d6fbb0a1	temporary changes for the tokenizer issues	2014-02-20 00:43:42 +01:00
Dave Halter	1b6df4602d	replace parser tokenizer offset with line offset	2014-02-17 23:02:04 +01:00
Dave Halter	6952596117	implement an offset in generate_tokens	2014-02-17 10:08:32 +01:00
Dave Halter	bb111daf91	removed line from tokenizer	2014-02-16 15:28:18 +01:00
Dave Halter	2e65fbb00f	rename parser.tokenizer -> parser.tokenize	2014-01-05 13:38:14 +01:00