Dave Halter
c7a74e6d1c
Make the tokenizer a generator.
2017-03-09 18:53:09 +01:00
Dave Halter
ee98eab64c
Merge branch 'dev' of github.com:davidhalter/jedi into dev
2017-03-07 19:01:42 +01:00
Dave Halter
05e05252fa
Make python -m jedi.parser.tokenize possible for debugging purposes.
2017-03-07 18:31:12 +01:00
Matthias Bussonnier
0fb386d7e2
Make sure error token set the new_line flag when necessary
...
Should solve #855
2017-02-24 00:05:38 +01:00
Dave Halter
4918fb49f5
Implement binary search for get_leaf_for_position. This makes it a lot faster.
2017-01-25 22:27:36 +01:00
Dave Halter
09779c88aa
Fix a nasty issue in the tokenizer. Fixes #836 .
...
At the same time there was a related issue of not cleaning up newlines properly.
2017-01-24 00:50:37 +01:00
Dave Halter
8d2ec6556e
Fix a Python 2.7 issue.
2017-01-23 00:36:57 +01:00
Dave Halter
7300f3e7ef
Fix issues with Python 3.6's f strings and underscores in numbers.
2017-01-08 19:39:14 +01:00
Dave Halter
00a9f1ec0a
Update the tokenizer to include f literals and underscores. Need tests still.
2017-01-08 16:03:45 +01:00
Dave Halter
9b85d5517f
Fix more issues in the diff parser.
2016-09-27 00:29:11 +02:00
Dave Halter
2563746810
Fix issues with octals in Python 2 (and possibly 3). Fixes #559 .
2016-07-17 22:36:26 +02:00
Dave Halter
927534a8d5
Strange unicode characters are error tokens. Fixes #587 .
2016-07-13 08:53:08 +02:00
Dave Halter
746a513ef9
Return a TokenInfo object in the tokenizer instead of a tuple (it's a namedtuple now.)
2016-06-27 09:14:11 +02:00
Dave Halter
969100e471
Move the parsing away from user_context to api.py.
2016-06-27 08:48:36 +02:00
Dave Halter
82c76fa689
Merge with the linter branch (especially the changes of pep484.
2016-06-07 13:51:25 +02:00
Dave Halter
e60c06b691
In every case of a new line the tokenize position should be correct.
2016-06-04 00:34:00 +02:00
Dave Halter
d4a10929e2
Starting to create a way of how context sensitive completions can be made.
...
This involves playing heavily with the parser pgen2. We use its stack to check for all possible tokens/keywords.
2016-05-23 18:11:44 +02:00
Claude
7077d0b762
Using python 3.7-like parser, instead of python 3.5 magic.
...
See https://github.com/davidhalter/jedi/pull/691#issuecomment-182815864
Revert "Update tokenizer to adhere to PEP492 magic"
This reverts commit 65187930bd
2016-02-11 19:14:31 +01:00
Claude
65187930bd
Update tokenizer to adhere to PEP492 magic
2016-02-09 21:07:18 +01:00
Claude
0ed149070a
add python 3.5 '@' operator to tokenizer
2016-02-09 17:13:25 +01:00
Dave Halter
2a691eefff
Including pgen2 tests from the cpython repo.
2015-12-26 03:15:09 +01:00
Dave Halter
ab5d0ed72b
Starting with Python 3.4 from is not a token that always is a "new" statement.
2015-12-26 02:47:22 +01:00
Dave Halter
c15551ccc1
Errortokens should also make the parser fail in the normal parser.
2015-12-20 23:11:52 +01:00
Dave Halter
c4906e0e3f
Rework the parser so we can use arbitrary start nodes of the syntax.
...
This also includes a rework for error recovery in the parser. This is now just possible for file_input parsing, which means for full files.
Includes also a refactoring of the tokenizer. No more do we have to add an additional newline, because it now works correctly (removes certain confusion.
2015-12-20 22:25:41 +01:00
Dave Halter
66557903ae
\\\r\n is as possible as \\\n.
2015-04-28 18:53:14 +02:00
Dave Halter
b6ebb2f8bf
Fixed issues with last positions in the tokenizer, which was messed up a little bit a few commits ago.
2015-04-27 21:42:40 +02:00
Dave Halter
0a96083fde
Fix ur'' literals.
2015-04-27 19:21:41 +02:00
Dave Halter
902482568e
The tokenize endmarker should really be the maximum position possible. Caused matplotlib to fail. Fixes davidhalter/jedi-vim#377 .
2015-04-27 19:01:45 +02:00
farhad
32081bd156
Merge branch 'dev' into unicode_tokenize_fix2
...
Conflicts:
AUTHORS.txt
2015-03-06 12:14:38 +04:00
farhad
3747b009bf
fix tokenization of code containing unicode strings
2015-03-06 09:11:35 +04:00
Dave Halter
a91e240c8b
ALWAYS_BREAK_TOKEN -> ALWAYS_BREAK_TOKENS
2015-02-23 14:10:29 +01:00
Dave Halter
3ec96b25cc
Issue with backslashes again in the fast parser.
2015-02-21 18:07:21 +01:00
Dave Halter
39bf9f426b
Handle backslash escaping.
2015-02-18 17:32:34 +01:00
Dave Halter
c689573b0b
Removed the line_offset from tokenize, we have better ways to modify positions, now.
2015-02-05 14:00:58 +01:00
Dave Halter
e913872192
Merged the tokenize is_identifier changes.
2015-02-01 20:32:01 +01:00
Dave Halter
a3cdec819e
Fix the prefix in tokenize, which was the wrong way around.
2015-01-29 17:10:00 +01:00
Dave Halter
a221eee02c
Fix more issues in the fast parser.
2015-01-29 15:38:38 +01:00
Dave Halter
dde0e9c7c6
Fix for loop issues in the fast parser.
2015-01-29 01:36:16 +01:00
Dave Halter
4d6afd3c99
Fix fast parser tests.
2015-01-24 00:06:16 +01:00
Savor d'Isavano
c3c07c4ec2
Fixed issue #526 .
2015-01-16 18:45:34 +08:00
Dave Halter
b2e54ca1eb
The tokenizer now includes all newlines and comments in its prefix.
2014-12-17 20:11:42 +01:00
Dave Halter
e53e211325
Python 2 compatibility in fake module.
2014-12-16 02:07:20 +01:00
Dave Halter
d9d3740c92
Trying to replace the old pgen2 token module with a token module more tightly coupled to the standard library.
2014-12-16 01:52:15 +01:00
Dave Halter
eaace104dd
Replace the tokenizer's output with a tuple (switching back from a Token class).
2014-12-16 00:10:07 +01:00
Dave Halter
2c684906e3
Working with dedents in error recovery.
2014-11-28 21:33:40 +01:00
Dave Halter
31600b9552
classes and functions are new statements and should never get removed by the error recovery.
2014-11-28 02:44:34 +01:00
Dave Halter
128dbd34b6
Check parentheses level in tokenizer.
2014-11-28 02:14:38 +01:00
Dave Halter
e1d6511f2f
Trying to move the indent/dedent logic back into the tokenizer.
2014-11-28 02:04:04 +01:00
Dave Halter
97516eb26b
The new tokenizer is more or less working now. Indents are calculated as they should
2014-11-27 16:03:58 +01:00
Dave Halter
c7862925f5
Small tokenizer changes & tokens now have a prefix attribute instead of preceeding_whitespace.
2014-11-27 01:10:45 +01:00
