Dave Halter
26cce4d078
Add the grammar as an argument to saving the parser.
...
This makes collisions of different grammars when loading from the cache impossible.
2017-03-22 18:32:49 +01:00
Dave Halter
53b4e78a9b
Make some stuff private in the pgen parser API.
2017-03-20 08:36:11 +01:00
Dave Halter
a83b43ccfd
Merge branch 'dev' of github.com:davidhalter/jedi into dev
2017-03-15 19:12:51 +01:00
Dave Halter
0effd348e8
Add a note about the grammar creation.
2017-03-15 18:18:06 +01:00
Matthias Bussonnier
9a851165ad
Lookup optimisation
...
avoid a couple of dynamic lookup in the core of the parsing loop.
Performance improvement will be minimal, but adding a little many time
can be consequent.
2017-03-14 23:55:03 +01:00
Dave Halter
97fc3bc23c
Refactored the parser calls. Now it's possible to use jedi.parser.python.parse to quickly parse something.
2017-03-14 00:38:58 +01:00
Dave Halter
9b5e6d16da
Improved grammar loading API.
2017-03-13 20:33:29 +01:00
Dave Halter
c7a74e6d1c
Make the tokenizer a generator.
2017-03-09 18:53:09 +01:00
Dave Halter
6853bd70f4
Adding a token in pgen should have the same signature that the tokenizer uses.
2016-06-14 18:09:31 +02:00
Dave Halter
01ddacfec4
Generalize the tuple order of tokens.
2016-06-14 08:57:38 +02:00
Dave Halter
6b63b9cf54
Further fixes for failing tests.
2016-06-04 00:59:17 +02:00
Dave Halter
ad8d730a57
More test fixes.
2016-05-31 01:12:07 +02:00
Dave Halter
daa68b66ad
Fix a few issues caused by the refactoring.
2016-05-29 19:49:35 +02:00
Dave Halter
d4a10929e2
Starting to create a way of how context sensitive completions can be made.
...
This involves playing heavily with the parser pgen2. We use its stack to check for all possible tokens/keywords.
2016-05-23 18:11:44 +02:00
Dave Halter
36a135c347
pgen2: Don't overwrite type
2016-05-21 16:08:12 +02:00
Dave Halter
a373e34229
The parser without error recovery raises an error now if he's not able to parse something.
2015-12-25 18:53:05 +01:00
Dave Halter
c4906e0e3f
Rework the parser so we can use arbitrary start nodes of the syntax.
...
This also includes a rework for error recovery in the parser. This is now just possible for file_input parsing, which means for full files.
Includes also a refactoring of the tokenizer. No more do we have to add an additional newline, because it now works correctly (removes certain confusion.
2015-12-20 22:25:41 +01:00
Dave Halter
10df0f933f
Remove the strange check in the parser to always create expr_stmt and file_input.
2015-03-05 15:30:07 +01:00
Dave Halter
b489019f5b
Most integration tests (except 2) pass if we don't always make the use of an ExprStmt.
2015-03-05 01:55:25 +01:00
Dave Halter
aea38ca9aa
Remove the classify function in the parser. This could make Jedi a tiny bit faster.
2015-03-04 17:15:33 +01:00
Dave Halter
9c2e73d460
Add syntax errors to the parser.
2015-03-04 17:12:51 +01:00
Dave Halter
3ec96b25cc
Issue with backslashes again in the fast parser.
2015-02-21 18:07:21 +01:00
Dave Halter
a095f8d9e0
Replace some isinstance checks in the parser tree with .type checks.
2015-02-09 12:27:29 +01:00
Dave Halter
4d6afd3c99
Fix fast parser tests.
2015-01-24 00:06:16 +01:00
Dave Halter
fd1cb86765
Now able to remove both tokenize and token from pgen2.
2014-12-16 02:00:33 +01:00
Dave Halter
d9d3740c92
Trying to replace the old pgen2 token module with a token module more tightly coupled to the standard library.
2014-12-16 01:52:15 +01:00
Dave Halter
680fdd574b
Remove some old unused tokenize stuff.
2014-12-15 17:44:40 +01:00
Dave Halter
955f125c0d
Trying to remove token from pgen2.
2014-12-15 17:36:15 +01:00
Dave Halter
491b4ad76d
Pgen2 license amendments.
2014-12-15 17:29:32 +01:00
Dave Halter
b911a39fb4
The driver file is now empty.
2014-12-15 17:27:27 +01:00
Dave Halter
55a6dbc8a2
Remove the old driver code of pgen2.
2014-12-15 17:18:01 +01:00
Dave Halter
2c684906e3
Working with dedents in error recovery.
2014-11-28 21:33:40 +01:00
Dave Halter
31600b9552
classes and functions are new statements and should never get removed by the error recovery.
2014-11-28 02:44:34 +01:00
Dave Halter
e1d6511f2f
Trying to move the indent/dedent logic back into the tokenizer.
2014-11-28 02:04:04 +01:00
Dave Halter
97516eb26b
The new tokenizer is more or less working now. Indents are calculated as they should
2014-11-27 16:03:58 +01:00
Dave Halter
427056a22d
Change the pgen2 parser and its driver so that it can be accessed easily from the outside. This is a minor change and will allow Jedis tokenizer to work with pgen2.
2014-11-26 15:38:53 +01:00
Dave Halter
5d82b11f59
First implementation to be ready to complete corrupt imports. Working ok.
2014-11-25 19:35:27 +01:00
Dave Halter
9549c2b389
Last few small changes to the parser. Now beginning to work on the tests again.
2014-11-05 17:39:56 +01:00
Dave Halter
70bc6642d8
symbols on the stack are now defined with a smaller tuple.
2014-11-05 10:27:24 +01:00
Dave Halter
da5dd0efa1
simplify the parser stack.
2014-11-05 00:35:58 +01:00
Dave Halter
61df804e6e
context is not needed for nodes.
2014-11-04 19:49:39 +01:00
Dave Halter
3b4a8dcd7e
convert: run away from tuples.
2014-11-04 19:26:33 +01:00
Dave Halter
3518123afa
Moved the convert function away from pytree
2014-11-04 19:09:58 +01:00
Dave Halter
f3e4bf9ed1
Splitting up the convert function into leaves and nodes.
2014-11-04 18:54:25 +01:00
Dave Halter
d483d50284
Small optimizations to make parsing faster.
2014-11-04 17:23:16 +01:00
Dave Halter
1ff4713848
Move error recovery function.
2014-11-02 17:24:06 +01:00
Dave Halter
1bb0eccc86
Simple error recovery in the parser: For now just discard everything that is not a suite or file_input, if we detect an error.
2014-10-30 01:40:22 +01:00
Dave Halter
e9f4c60e49
Use used_names not in pgen2, but only in our parser.
2014-10-22 15:50:02 +02:00
Dave Halter
fb2ef5a7a0
Start using arrays.
2014-10-21 11:05:12 +02:00
Dave Halter
718f43431c
Introduce error recovery for the parser: At the moment just recover from broken statements.
2014-10-21 09:57:22 +02:00
