Dave Halter
2252271bf5
fix unicode issues with python2.7
2014-02-25 12:54:06 +01:00
Dave Halter
3a23c80ae5
prepare for eventual? tokenizer end_pos replacement.
2014-02-25 11:59:10 +01:00
Dave Halter
246118f851
start using @ganwell's new token class (modified in some ways) as the main token class - hope to gain a little bit of memory/cpu/pickling performance
2014-02-25 02:06:26 +01:00
Dave Halter
e2a6d1dd43
remove offset param for Parser
2014-02-25 01:31:24 +01:00
Dave Halter
44d560c53a
finally removed self-contained iterator from Parser - after knowing this since almost the beginning.
2014-02-25 01:26:19 +01:00
Dave Halter
a7a8a73a2c
removed Parser._start_pos as well
2014-02-25 01:20:55 +01:00
Dave Halter
add8259d7e
successfully removed end_pos from parser
2014-02-25 01:05:00 +01:00
Dave Halter
936c7dfde4
protect start_pos/end_pos in parser to hopefully remove them soon
2014-02-25 00:27:39 +01:00
Dave Halter
ef8d3633dd
use the module end_pos in fast parser
2014-02-25 00:18:24 +01:00
Dave Halter
a5a6e9ac12
parser: remove current
2014-02-24 20:43:00 +01:00
Dave Halter
50f8b8bf0c
start using tokens all the way through in the parser
2014-02-24 20:35:36 +01:00
Dave Halter
6058e8b9c3
remove unnecessary checks
2014-02-24 11:40:05 +01:00
Dave Halter
9943bb6205
remove some old parameters from Parser and FastTokenizer
2014-02-24 11:24:54 +01:00
Dave Halter
7db090a48a
moved NoErrorTokenizer to fast.FastTokenizer
2014-02-24 11:05:31 +01:00
Dave Halter
9257062910
remove NoErrorTokenizer dependency for all but the fast parsers
2014-02-24 10:31:15 +01:00
Dave Halter
553ff66c8b
remove last_previous from NoErrorTokenizer
2014-02-23 12:51:05 +01:00
Dave Halter
8e847f4982
fix python 2.7 issues. the parser now only takes unicode inputs
2014-02-23 11:44:32 +01:00
Dave Halter
c5fcebde82
changed _compatibility.utf8 -> 'u' and removed a lot of the issues with the now enforced unicode source input of the parser
2014-02-23 11:29:00 +01:00
Dave Halter
5478e50f8b
Merge branch 'dev' of github.com:davidhalter/jedi into dev
2014-02-21 17:52:44 +01:00
Dave Halter
9c2fcfffd7
StringIo is now always used as part of the io library
2014-02-21 17:52:09 +01:00
Dave Halter
d9bf9be35e
small refactorings of the tokenizer
2014-02-21 17:45:56 +01:00
Dave Halter
efba0cd80c
deleted a big part of the generate_tokens code that doesn't seem to be needed anymore
2014-02-21 16:38:15 +01:00
Dave Halter
73057d4176
removed the INDENT usages that were left in the parser code
2014-02-21 15:39:14 +01:00
Dave Halter
fe02088dd7
restructure user_context reverse tokenizer to remove INDENT tokens in the future
2014-02-21 15:30:46 +01:00
Dave Halter
9c3b7b9c46
get rid of more variables
2014-02-21 10:44:33 +01:00
Dave Halter
184e90a3a0
removed some line checks from tokenizer
2014-02-21 00:13:23 +01:00
Dave Halter
a69eee5876
error tokens only exist in the end in a tokenizer
2014-02-21 00:09:49 +01:00
Dave Halter
3232ae5b0c
removed parentheses counting from generate_tokens
2014-02-20 18:45:22 +01:00
Dave Halter
7e651684ff
fix problems with debug mode if not correctly initialized.
2014-02-20 14:17:05 +01:00
Dave Halter
d7033726fd
tokenize removed NL/NEWLINE ambiguity
2014-02-20 01:52:30 +01:00
Dave Halter
c26ae3c00d
ignore dedents in tokenizer
2014-02-20 01:21:20 +01:00
Dave Halter
3e9b72b636
created a PushBackTokenizer specifically for the parser
2014-02-20 01:17:19 +01:00
Dave Halter
c8d6fbb0a1
temporary changes for the tokenizer issues
2014-02-20 00:43:42 +01:00
Dave Halter
000e929e3c
Merge pull request #375 from ColinDuquesnoy/dev
...
Fix import of compiled module with python3.
2014-02-19 22:11:27 +04:30
ColinDuquesnoy
9d7ecae27c
Add precompiled extension modules
2014-02-19 08:14:51 +01:00
ColinDuquesnoy
2c97d01bd0
Move extensions dir into test
2014-02-19 08:10:43 +01:00
ColinDuquesnoy
f808dbbd28
Change sys.path for the test to succeed.
...
Tested locally with a python3 extension module (in
/extensions/compiled33).
Also tested that reverting a75773cf9f make
the test fail.
2014-02-18 22:14:43 +01:00
ColinDuquesnoy
7977d57169
Add test_compiled
...
Should pass on travis since there is no precompiled modules yet
2014-02-18 20:31:06 +01:00
ColinDuquesnoy
7aed62cb99
Add compiled modules sources + readme
2014-02-18 20:12:30 +01:00
ColinDuquesnoy
5a706265bc
Move dummy pyc to dummy package
2014-02-18 19:18:17 +01:00
ColinDuquesnoy
18a31dcbf1
Fix read mode for pyc files
...
Now all test should pass on travis.
2014-02-18 19:00:55 +01:00
ColinDuquesnoy
ea43efc9d8
Fix pyc test for python3
...
To import pyc modules, we must move them out of the __pycache__
directory and rename them to remove ".cpython-%s%d".
This should still faild with python3 (UnicodeDecodeError)
2014-02-18 19:00:55 +01:00
ColinDuquesnoy
d80caa7108
Add test pyc
...
Should succeed for python2 and faild for python3
2014-02-18 17:36:01 +01:00
Dave Halter
1b6df4602d
replace parser tokenizer offset with line offset
2014-02-17 23:02:04 +01:00
Dave Halter
6952596117
implement an offset in generate_tokens
2014-02-17 10:08:32 +01:00
ColinDuquesnoy
a75773cf9f
Fix import of compiled module with python3.
...
This at least fix #331
2014-02-17 09:08:30 +01:00
Dave Halter
bb111daf91
removed line from tokenizer
2014-02-16 15:28:18 +01:00
Dave Halter
22928dbcd0
simplified some crazy iterator stuff within parser
2014-02-16 15:04:57 +01:00
Dave Halter
82f27569b2
little changes to clean up the code (flake8)
2014-02-14 12:53:22 +01:00
Dave Halter
d704743422
more consistent __slots__ usage in the parser -> 20% memory decrease for wx
2014-02-14 00:56:42 +01:00