Commit Graph

83 Commits

Author SHA1 Message Date
Dave Halter
ce96af5e04 Fix an issue with open parentheses and function definitions right after. The fast parser should behave like the normal one and just ignore the open brackets. 2015-02-19 11:02:11 +01:00
Dave Halter
0e73bf7d80 Account for code parts that were not parsed in the fast parser. 2015-02-19 01:42:13 +01:00
Dave Halter
39bf9f426b Handle backslash escaping. 2015-02-18 17:32:34 +01:00
Dave Halter
595da50ab8 The fast parser splitting now also checks for parentheses levels, because without that, sometimes we split in very strange positions, while ignoring others. 2015-02-18 13:49:03 +01:00
Dave Halter
38e26892f2 The fast parser doesn't work with open parentheses properly, document that. 2015-02-18 12:50:26 +01:00
Dave Halter
cefd76e5d1 Testing open parentheses in the fast parser. 2015-02-17 17:26:00 +01:00
Dave Halter
506d602795 Fix multi line param issues in the fast parser. 2015-02-17 15:24:49 +01:00
Dave Halter
7663703989 Fix issues with multi line for loops in the fast parser. 2015-02-17 14:57:00 +01:00
Dave Halter
e1c28d2c3f variables starting with 'class' and 'def' should not slow down the parser, changed the check to 'class ' and 'def '. 2015-02-16 10:08:22 +01:00
Dave Halter
3680784234 Add another for in one line test for the fast parser. 2015-02-15 20:28:59 +01:00
Dave Halter
a3b32729a7 Test for an issue with for loops and a statement on the same line. (fast parser) 2015-02-14 16:27:04 +01:00
Dave Halter
07156b427c Fix some compatibilty issues in the test suite for Python 2.7. 2015-02-09 14:15:25 +01:00
Dave Halter
2d9c644ab6 Fixed some minor mocking differences in Python 2 and 3. 2015-02-05 01:25:53 +01:00
Dave Halter
b57ee880af Remove assertEqual from tokenize tests, we can do it with just assert, py.test converts all of that automatically. 2015-02-05 00:48:40 +01:00
Dave Halter
c6b818c504 Changed a tokenize test to match the current intended behavior of the tokenizer. 2015-02-05 00:43:25 +01:00
Dave Halter
dce952aec6 Fix an issue with omited dedents in the parser. 2015-02-05 00:11:12 +01:00
Dave Halter
66dfa59286 Fix some endmarker prefix issues in the fast parser. 2015-02-03 22:09:55 +01:00
Dave Halter
6cdfecb541 Fix a number of issues in the fast parser around functions with only one statement (no suite) and wrong indentations). 2015-02-02 15:03:57 +01:00
Dave Halter
e913872192 Merged the tokenize is_identifier changes. 2015-02-01 20:32:01 +01:00
Dave Halter
bc118e8047 Simplify the fast parser tokenizer more. Now it is more readable and less buggy (+bugfixes). 2015-01-31 20:09:44 +01:00
Dave Halter
1826f432c8 Fix an issue in the fast parser splitting. 2015-01-30 15:17:38 +01:00
Dave Halter
413da3b790 Remove the line_offset calculation. We can now also remove it from tokenize. With the position_modifier we have enough tools to change a position, we don't need to do that in tokenize.py. 2015-01-29 17:57:01 +01:00
Dave Halter
a221eee02c Fix more issues in the fast parser. 2015-01-29 15:38:38 +01:00
Dave Halter
0a537c05c4 Fix an issue with Function/Flow combination in the fast parser. 2015-01-29 02:24:11 +01:00
Dave Halter
dde0e9c7c6 Fix for loop issues in the fast parser. 2015-01-29 01:36:16 +01:00
Dave Halter
e412694fa2 Fix issues with flows in the fast parser. 2015-01-28 17:06:18 +01:00
Dave Halter
d0589430bb FastModule should inherit from SubModule, because it has almost all the same properties. 2015-01-28 14:59:00 +01:00
Dave Halter
6ec89e6785 Fix issues with flows. 2015-01-28 13:03:57 +01:00
Dave Halter
5e8f8f7a8d Fix issues with error correction / newline correction. 2015-01-27 12:24:54 +01:00
Dave Halter
4a07f97f10 Reenable a few get_code tests. 2015-01-27 01:19:09 +01:00
Dave Halter
88a3e25814 Fix newline stuff for empty parsers. 2015-01-27 01:15:39 +01:00
Dave Halter
39e869d146 Test added newline module end_pos as well. 2015-01-26 22:02:11 +01:00
Dave Halter
07c60d7ff6 Fix DEDENT issues in _remove_newline. 2015-01-26 21:17:50 +01:00
Dave Halter
61e2bba380 Tests and implementation to remove the last newline again in the parser tree, to be able to exactly reproduce the parser input. 2015-01-26 21:07:14 +01:00
Dave Halter
a8943b8a80 Get the position modifiers right. 2015-01-24 20:42:28 +01:00
Dave Halter
446f5b9018 Fix issues with the right count of parsers used. 2015-01-24 20:19:03 +01:00
Dave Halter
4d6afd3c99 Fix fast parser tests. 2015-01-24 00:06:16 +01:00
Savor d'Isavano
c3c07c4ec2 Fixed issue #526. 2015-01-16 18:45:34 +08:00
Dave Halter
e6b9111749 Python 2.7 compatibility. 2015-01-13 02:12:49 +01:00
Dave Halter
cc64265187 Grammar modifications so that the Python2.7 grammar looks more like the Python 3.4 grammar. 2015-01-13 01:05:13 +01:00
Dave Halter
ef72f4fb6c Test the new error correction feature. 2015-01-12 01:27:25 +01:00
Dave Halter
6d58fed0e8 Remove get_defined_names in favor of names_dict in the parser tree. 2015-01-08 13:38:03 +01:00
Dave Halter
eaace104dd Replace the tokenizer's output with a tuple (switching back from a Token class). 2014-12-16 00:10:07 +01:00
Dave Halter
d6595ad020 Fixed more parser tests. 2014-12-08 00:36:09 +01:00
Dave Halter
fe8a99dfd5 Fix the parser tests. 2014-12-08 00:32:38 +01:00
Dave Halter
34c9422749 The parser tests should also give the parser a grammar. 2014-12-08 00:22:33 +01:00
Dave Halter
fe1d7b7030 Replace the old tokenizer tests with the refactored attributes. 2014-12-07 14:19:21 +01:00
Dave Halter
02cb1fef95 Rename test_tokenizer to test_tokenize. 2014-11-26 16:16:58 +01:00
Dave Halter
cc1098b93c Fix a few tokenize tests and merge them back together. 2014-11-26 16:09:28 +01:00
Dave Halter
f43c371467 Merge @joel-wright's whitespace tokenizer branch. Thanks! 2014-11-26 15:56:11 +01:00