5 Commits

Author	SHA1	Message	Date
David Halter	0d385563a5	tokenizer: removed encoding	2013-04-20 09:28:32 +04:30
David Halter	6b2619a844	tokenizer removed unused functions	2013-04-20 09:23:27 +04:30
David Halter	06da7ddeec	tokenizer py27 support	2013-04-20 09:21:22 +04:30
David Halter	3b0f4b87cf	removed unnecessary stuff for the jedi tokenizer	2013-04-19 23:59:33 +04:30
David Halter	a28bc7195f	added tokenizer of the standard library to jedi, because in the future it shall replace tokenize	2013-04-19 23:50:28 +04:30