\\\r\n is as possible as \\\n.

run buildout detection only once per buildout script

.gitignore

@@ -10,3 +10,5 @@
 /docs/_build/
 /dist/
 jedi.egg-info/
+record.json
+/.cache/

.travis.yml

@@ -4,10 +4,15 @@ env:
     - TOXENV=py27
     - TOXENV=py32
     - TOXENV=py33
+    - TOXENV=py34
+    - TOXENV=pypy
     - TOXENV=cov
+    - TOXENV=sith
 matrix:
     allow_failures:
         - env: TOXENV=cov
+        - env: TOXENV=sith
+        - env: TOXENV=pypy
 install:
     - pip install --quiet --use-mirrors tox
 script:

AUTHORS.txt

@@ -1,14 +1,37 @@
 Main Authors
 ============
 
-David Halter (@davidhalter)
-Takafumi Arakaki (@tkf)
+David Halter (@davidhalter) <davidhalter88@gmail.com>
+Takafumi Arakaki (@tkf) <aka.tkf@gmail.com>
 
 Code Contributors
 =================
 
-Danilo Bargen (@dbrgn)
+Danilo Bargen (@dbrgn) <mail@dbrgn.ch>
+Laurens Van Houtven (@lvh) <_@lvh.cc>
+Aldo Stracquadanio (@Astrac) <aldo.strac@gmail.com>
+Jean-Louis Fuchs (@ganwell) <ganwell@fangorn.ch>
 tek (@tek)
-Yasha Borevich (@jjay)
-Aaron Griffin
+Yasha Borevich (@jjay) <j.borevich@gmail.com>
+Aaron Griffin <aaronmgriffin@gmail.com>
 andviro (@andviro)
+Mike Gilbert (@floppym) <floppym@gentoo.org>
+Aaron Meurer (@asmeurer) <asmeurer@gmail.com>
+Lubos Trilety <ltrilety@redhat.com>
+Akinori Hattori (@hattya) <hattya@gmail.com>
+srusskih (@srusskih)
+Steven Silvester (@blink1073)
+Colin Duquesnoy (@ColinDuquesnoy) <colin.duquesnoy@gmail.com>
+Jorgen Schaefer (@jorgenschaefer) <contact@jorgenschaefer.de>
+Fredrik Bergroth (@fbergroth)
+Mathias Fußenegger (@mfussenegger)
+Syohei Yoshida (@syohex) <syohex@gmail.com>
+ppalucky (@ppalucky)
+immerrr (@immerrr) immerrr@gmail.com
+Albertas Agejevas (@alga)
+Savor d'Isavano (@KenetJervet) <newelevenken@163.com>
+Phillip Berndt (@phillipberndt) <phillip.berndt@gmail.com>
+Ian Lee (@IanLee1521) <IanLee1521@gmail.com>
+Farkhad Khatamov (@hatamov) <comsgn@gmail.com>
+
+Note: (@user) means a github user name.

CHANGELOG.rst

@@ -3,13 +3,54 @@
 Changelog
 ---------
 
+0.9.0 (2015-04-10)
+++++++++++++++++++
+
+- Integrated the parser of 2to3. This will make refactoring possible. It will
+  also be possible to check for error messages (like compiling an AST would give)
+  in the future.
+- With the new parser, the evaluation also completely changed. It's now simpler
+  and more readable.
+- Completely rewritten REPL completion.
+- Added ``jedi.names``, a command to do static analysis. Thanks to that
+  sourcegraph guys for sponsoring this!
+- Alpha version of the linter.
+
+
+0.8.1 (2014-07-23)
++++++++++++++++++++
+
+- Bugfix release, the last release forgot to include files that improve
+  autocompletion for builtin libraries. Fixed.
+
+0.8.0 (2014-05-05)
++++++++++++++++++++
+
+- Memory Consumption for compiled modules (e.g. builtins, sys) has been reduced
+  drastically. Loading times are down as well (it takes basically as long as an
+  import).
+- REPL completion is starting to become usable.
+- Various small API changes. Generally this release focuses on stability and
+  refactoring of internal APIs.
+- Introducing operator precedence, which makes calculating correct Array
+  indices and ``__getattr__`` strings possible.
+
+0.7.0 (2013-08-09)
+++++++++++++++++++
+
+- Switched from LGPL to MIT license.
+- Added an Interpreter class to the API to make autocompletion in REPL
+  possible.
+- Added autocompletion support for namespace packages.
+- Add sith.py, a new random testing method.
+
 0.6.0 (2013-05-14)
 ++++++++++++++++++
 
-* much faster parser with builtin part caching
-* a test suite, thanks @tkf
+- Much faster parser with builtin part caching.
+- A test suite, thanks @tkf.
 
 0.5 versions (2012)
 +++++++++++++++++++
 
-* Initial development
+- Initial development.

CONTRIBUTING.md

@@ -10,4 +10,19 @@ My **master** branch is a 100% stable (should be). I only push to it after I am
 certain that things are working out. Many people are using Jedi directly from
 the github master branch.
 
-Please use Pep8 to style your code.
+**Try to use the PEP8 style guide.**
+
+
+Changing Issues to Pull Requests (Github)
+-----------------------------------------
+
+If you have have previously filed a GitHub issue and want to contribute code
+that addresses that issue, we prefer it if you use
+[hub](https://github.com/github/hub) to convert your existing issue to a pull
+request. To do that, first push the changes to a separate branch in your fork
+and then issue the following command:
+
+    hub pull-request -b davidhalter:dev -i <issue-number> -h <your-github-username>:<your-branch-name>
+
+It's no strict requirement though, if you don't have hub installed or prefer to
+use the web interface, then feel free to post a traditional pull request.

LICENSE.txt

@@ -1,170 +1,21 @@
-Licensed under the GNU LGPL v3 or later.
-Copyright (C) 2012 David Halter <davidhalter88@gmail.com>.
+The MIT License (MIT)
 
-===============================================================================
+Copyright (c) <2013> <David Halter and others, see AUTHORS.txt>
 
-                   GNU LESSER GENERAL PUBLIC LICENSE
-                       Version 3, 29 June 2007
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
- Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
 
-
-  This version of the GNU Lesser General Public License incorporates
-the terms and conditions of version 3 of the GNU General Public
-License, supplemented by the additional permissions listed below.
-
-  0. Additional Definitions.
-
-  As used herein, "this License" refers to version 3 of the GNU Lesser
-General Public License, and the "GNU GPL" refers to version 3 of the GNU
-General Public License.
-
-  "The Library" refers to a covered work governed by this License,
-other than an Application or a Combined Work as defined below.
-
-  An "Application" is any work that makes use of an interface provided
-by the Library, but which is not otherwise based on the Library.
-Defining a subclass of a class defined by the Library is deemed a mode
-of using an interface provided by the Library.
-
-  A "Combined Work" is a work produced by combining or linking an
-Application with the Library.  The particular version of the Library
-with which the Combined Work was made is also called the "Linked
-Version".
-
-  The "Minimal Corresponding Source" for a Combined Work means the
-Corresponding Source for the Combined Work, excluding any source code
-for portions of the Combined Work that, considered in isolation, are
-based on the Application, and not on the Linked Version.
-
-  The "Corresponding Application Code" for a Combined Work means the
-object code and/or source code for the Application, including any data
-and utility programs needed for reproducing the Combined Work from the
-Application, but excluding the System Libraries of the Combined Work.
-
-  1. Exception to Section 3 of the GNU GPL.
-
-  You may convey a covered work under sections 3 and 4 of this License
-without being bound by section 3 of the GNU GPL.
-
-  2. Conveying Modified Versions.
-
-  If you modify a copy of the Library, and, in your modifications, a
-facility refers to a function or data to be supplied by an Application
-that uses the facility (other than as an argument passed when the
-facility is invoked), then you may convey a copy of the modified
-version:
-
-   a) under this License, provided that you make a good faith effort to
-   ensure that, in the event an Application does not supply the
-   function or data, the facility still operates, and performs
-   whatever part of its purpose remains meaningful, or
-
-   b) under the GNU GPL, with none of the additional permissions of
-   this License applicable to that copy.
-
-  3. Object Code Incorporating Material from Library Header Files.
-
-  The object code form of an Application may incorporate material from
-a header file that is part of the Library.  You may convey such object
-code under terms of your choice, provided that, if the incorporated
-material is not limited to numerical parameters, data structure
-layouts and accessors, or small macros, inline functions and templates
-(ten or fewer lines in length), you do both of the following:
-
-   a) Give prominent notice with each copy of the object code that the
-   Library is used in it and that the Library and its use are
-   covered by this License.
-
-   b) Accompany the object code with a copy of the GNU GPL and this license
-   document.
-
-  4. Combined Works.
-
-  You may convey a Combined Work under terms of your choice that,
-taken together, effectively do not restrict modification of the
-portions of the Library contained in the Combined Work and reverse
-engineering for debugging such modifications, if you also do each of
-the following:
-
-   a) Give prominent notice with each copy of the Combined Work that
-   the Library is used in it and that the Library and its use are
-   covered by this License.
-
-   b) Accompany the Combined Work with a copy of the GNU GPL and this license
-   document.
-
-   c) For a Combined Work that displays copyright notices during
-   execution, include the copyright notice for the Library among
-   these notices, as well as a reference directing the user to the
-   copies of the GNU GPL and this license document.
-
-   d) Do one of the following:
-
-       0) Convey the Minimal Corresponding Source under the terms of this
-       License, and the Corresponding Application Code in a form
-       suitable for, and under terms that permit, the user to
-       recombine or relink the Application with a modified version of
-       the Linked Version to produce a modified Combined Work, in the
-       manner specified by section 6 of the GNU GPL for conveying
-       Corresponding Source.
-
-       1) Use a suitable shared library mechanism for linking with the
-       Library.  A suitable mechanism is one that (a) uses at run time
-       a copy of the Library already present on the user's computer
-       system, and (b) will operate properly with a modified version
-       of the Library that is interface-compatible with the Linked
-       Version.
-
-   e) Provide Installation Information, but only if you would otherwise
-   be required to provide such information under section 6 of the
-   GNU GPL, and only to the extent that such information is
-   necessary to install and execute a modified version of the
-   Combined Work produced by recombining or relinking the
-   Application with a modified version of the Linked Version. (If
-   you use option 4d0, the Installation Information must accompany
-   the Minimal Corresponding Source and Corresponding Application
-   Code. If you use option 4d1, you must provide the Installation
-   Information in the manner specified by section 6 of the GNU GPL
-   for conveying Corresponding Source.)
-
-  5. Combined Libraries.
-
-  You may place library facilities that are a work based on the
-Library side by side in a single library together with other library
-facilities that are not Applications and are not covered by this
-License, and convey such a combined library under terms of your
-choice, if you do both of the following:
-
-   a) Accompany the combined library with a copy of the same work based
-   on the Library, uncombined with any other library facilities,
-   conveyed under the terms of this License.
-
-   b) Give prominent notice with the combined library that part of it
-   is a work based on the Library, and explaining where to find the
-   accompanying uncombined form of the same work.
-
-  6. Revised Versions of the GNU Lesser General Public License.
-
-  The Free Software Foundation may publish revised and/or new versions
-of the GNU Lesser General Public License from time to time. Such new
-versions will be similar in spirit to the present version, but may
-differ in detail to address new problems or concerns.
-
-  Each version is given a distinguishing version number. If the
-Library as you received it specifies that a certain numbered version
-of the GNU Lesser General Public License "or any later version"
-applies to it, you have the option of following the terms and
-conditions either of that published version or of any later version
-published by the Free Software Foundation. If the Library as you
-received it does not specify a version number of the GNU Lesser
-General Public License, you may choose any version of the GNU Lesser
-General Public License ever published by the Free Software Foundation.
-
-  If the Library as you received it specifies that a proxy can decide
-whether future versions of the GNU Lesser General Public License shall
-apply, that proxy's public statement of acceptance of any version is
-permanent authorization for you to choose that version for the
-Library.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

MANIFEST.in

@@ -2,5 +2,13 @@ include README.rst
 include CHANGELOG.rst
 include LICENSE.txt
 include AUTHORS.txt
-include jedi/mixin/*.pym
+include .coveragerc
+include sith.py
+include conftest.py
+include pytest.ini
+include tox.ini
+include jedi/evaluate/compiled/fake/*.pym
+include jedi/parser/grammar*.txt
+recursive-include test *
+recursive-include docs *
 recursive-exclude * *.pyc

README.rst

@@ -1,6 +1,6 @@
-###################################################
-Jedi - an awesome autocompletion library for Python
-###################################################
+###################################################################
+Jedi - an awesome autocompletion/static analysis library for Python
+###################################################################
 
 .. image:: https://secure.travis-ci.org/davidhalter/jedi.png?branch=master
     :target: http://travis-ci.org/davidhalter/jedi
@@ -11,28 +11,41 @@ Jedi - an awesome autocompletion library for Python
     :alt: Coverage Status
 
 
-Jedi is an autocompletion tool for Python that can be used in IDEs/editors.
-Jedi works. Jedi is fast. It understands all of the basic Python syntax
-elements including many builtin functions.
+*If you have specific questions, please add an issue or ask on* `stackoverflow
+<https://stackoverflow.com>`_ *with the label* ``python-jedi``.
 
-Additionaly, Jedi suports two different goto functions and has support for
-renaming as well as Pydoc support and some other IDE features.
+
+Jedi is a static analysis tool for Python that can be used in IDEs/editors. Its
+historic focus is autocompletion, but does static analysis for now as well.
+Jedi is fast and is very well tested. It understands Python on a deeper level
+than all other static analysis frameworks for Python.
+
+Jedi has support for two different goto functions. It's possible to search for
+related names and to list all names in a Python file and infer them. Jedi
+understands docstrings and you can use Jedi autocompletion in your REPL as
+well.
 
 Jedi uses a very simple API to connect with IDE's. There's a reference
 implementation as a `VIM-Plugin <https://github.com/davidhalter/jedi-vim>`_,
-which uses Jedi's autocompletion.  I encourage you to use Jedi in your IDEs.
-It's really easy. If there are any problems (also with licensing), just contact
-me.
+which uses Jedi's autocompletion.  We encourage you to use Jedi in your IDEs.
+It's really easy.
 
-Jedi can be used with the following plugins/software:
+Jedi can currently be used with the following editors:
 
-- `VIM-Plugin <https://github.com/davidhalter/jedi-vim>`_
-- `Emacs-Plugin <https://github.com/tkf/emacs-jedi>`_
-- `Sublime-Plugin <https://github.com/svaiter/SublimeJEDI>`_
-- `wdb (web debugger) <https://github.com/Kozea/wdb>`_
+- Vim (jedi-vim_, YouCompleteMe_)
+- Emacs (Jedi.el_, elpy_, anaconda-mode_, ycmd_)
+- Sublime Text (SublimeJEDI_ [ST2 + ST3], anaconda_ [only ST3])
+- SynWrite_
+- TextMate_ (Not sure if it's actually working)
+- Kate_ version 4.13+ supports it natively, you have to enable it, though. [`proof
+  <https://projects.kde.org/projects/kde/applications/kate/repository/show?rev=KDE%2F4.13>`_]
+
+And it powers the following projects:
+
+- wdb_ - Web Debugger
 
 
-Here are some pictures:
+Here are some pictures taken from jedi-vim_:
 
 .. image:: https://github.com/davidhalter/jedi/raw/master/docs/_screenshots/screenshot_complete.png
 
@@ -44,15 +57,15 @@ Display of function/class bodies, docstrings.
 
 .. image:: https://github.com/davidhalter/jedi/raw/master/docs/_screenshots/screenshot_pydoc.png
 
-Pydoc support (with highlighting, Shift+k).
+Pydoc support (Shift+k).
 
 There is also support for goto and renaming.
 
 Get the latest version from `github <https://github.com/davidhalter/jedi>`_
 (master branch should always be kind of stable/working).
 
-Docs are available at `https://jedi.readthedocs.org/
-<https://jedi.readthedocs.org/>`_. Pull requests with documentation
+Docs are available at `https://jedi.readthedocs.org/en/latest/
+<https://jedi.readthedocs.org/en/latest/>`_. Pull requests with documentation
 enhancements and/or fixes are awesome and most welcome. Jedi uses `semantic
 versioning <http://semver.org/>`_.
 
@@ -74,23 +87,61 @@ Feature Support and Caveats
 ===========================
 
 Jedi really understands your Python code. For a comprehensive list what Jedi
-can do, see: `Features
+understands, see: `Features
 <https://jedi.readthedocs.org/en/latest/docs/features.html>`_. A list of
 caveats can be found on the same page.
 
-You can run Jedi on cPython 2.6, 2.7, 3.2 or 3.3, but it should also
+You can run Jedi on cPython 2.6, 2.7, 3.2, 3.3 or 3.4, but it should also
 understand/parse code older than those versions.
 
 Tips on how to use Jedi efficiently can be found `here
 <https://jedi.readthedocs.org/en/latest/docs/recipes.html>`_.
 
+API
+---
 
-API for IDEs
-============
+You can find the documentation for the `API here <https://jedi.readthedocs.org/en/latest/docs/plugin-api.html>`_.
 
-It's very easy to create an editor plugin that uses Jedi. See `Plugin API
-<https://jedi.readthedocs.org/en/latest/docs/plugin-api.html>`_ for more
-information.
+
+Autocompletion / Goto / Pydoc
+-----------------------------
+
+Please check the API for a good explanation. There are the following commands:
+
+- ``jedi.Script.goto_assignments``
+- ``jedi.Script.completions``
+- ``jedi.Script.usages``
+
+The returned objects are very powerful and really all you might need.
+
+
+Autocompletion in your REPL (IPython, etc.)
+-------------------------------------------
+
+It's possible to have Jedi autocompletion in REPL modes - `example video <https://vimeo.com/122332037>`_.
+This means that IPython and others are `supported
+<https://jedi.readthedocs.org/en/latest/docs/usage.html#tab-completion-in-the-python-shell>`_.
+
+
+Static Analysis / Linter
+------------------------
+
+To do all forms of static analysis, please try to use ``jedi.names``. It will
+return a list of names that you can use to infer types and so on.
+
+Linting is another thing that is going to be part of Jedi. For now you can try
+an alpha version ``python -m jedi linter``. The API might change though and
+it's still buggy. It's Jedi's goal to be smarter than classic linter and
+understand ``AttributeError`` and other code issues.
+
+
+Refactoring
+-----------
+
+Jedi would in theory support refactoring, but we have never publicized it,
+because it's not production ready. If you're interested in helping out here,
+let me know. With the latest parser changes, it should be very easy to actually
+make it work.
 
 
 Development
@@ -121,3 +172,17 @@ Tests are also run automatically on `Travis CI
 
 For more detailed information visit the `testing documentation
 <https://jedi.readthedocs.org/en/latest/docs/testing.html>`_
+
+
+.. _jedi-vim: https://github.com/davidhalter/jedi-vim
+.. _youcompleteme: http://valloric.github.io/YouCompleteMe/
+.. _Jedi.el: https://github.com/tkf/emacs-jedi
+.. _elpy: https://github.com/jorgenschaefer/elpy
+.. _anaconda-mode: https://github.com/proofit404/anaconda-mode
+.. _ycmd: https://github.com/abingham/emacs-ycmd
+.. _sublimejedi: https://github.com/srusskih/SublimeJEDI
+.. _anaconda: https://github.com/DamnWidget/anaconda
+.. _SynWrite: http://uvviewsoft.com/synjedi/
+.. _wdb: https://github.com/Kozea/wdb
+.. _TextMate: https://github.com/lawrenceakka/python-jedi.tmbundle
+.. _Kate: http://kate-editor.org

conftest.py

@@ -21,12 +21,27 @@ jedi_cache_directory_orig = None
 jedi_cache_directory_temp = None
 
 
+def pytest_addoption(parser):
+    parser.addoption("--jedi-debug", "-D", action='store_true',
+                     help="Enables Jedi's debug output.")
+
+    parser.addoption("--warning-is-error", action='store_true',
+                     help="Warnings are treated as errors.")
+
+
 def pytest_configure(config):
     global jedi_cache_directory_orig, jedi_cache_directory_temp
     jedi_cache_directory_orig = jedi.settings.cache_directory
     jedi_cache_directory_temp = tempfile.mkdtemp(prefix='jedi-test-')
     jedi.settings.cache_directory = jedi_cache_directory_temp
 
+    if config.option.jedi_debug:
+        jedi.set_debug_function()
+
+    if config.option.warning_is_error:
+        import warnings
+        warnings.simplefilter("error")
+
 
 def pytest_unconfigure(config):
     global jedi_cache_directory_orig, jedi_cache_directory_temp

docs/_static/logo-src.txt

@@ -0,0 +1,3 @@
+The source of the logo is a photoshop file hosted here:
+
+https://dl.dropboxusercontent.com/u/170011615/Jedi12_Logo.psd.xz

docs/conf.py

@@ -11,13 +11,14 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import sys, os, datetime
+import sys
+import os
+import datetime
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 sys.path.insert(0, os.path.abspath('..'))
-sys.path.insert(0, os.path.abspath('../jedi'))
 sys.path.append(os.path.abspath('_themes'))
 
 # -- General configuration -----------------------------------------------------
@@ -28,7 +29,7 @@ sys.path.append(os.path.abspath('_themes'))
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode', 'sphinx.ext.todo',
-              'sphinx.ext.inheritance_diagram']
+              'sphinx.ext.intersphinx', 'sphinx.ext.inheritance_diagram']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -47,18 +48,16 @@ project = u'Jedi'
 copyright = u'2012 - {today.year}, Jedi contributors'.format(today=datetime.date.today())
 
 import jedi
-_path = os.path.dirname(os.path.abspath(__file__))
-_version_strs = [str(x) for x in jedi.__version__]
+from jedi.utils import version_info
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-
-version = '.'.join(_version_strs[:2])
+version = '.'.join(str(x) for x in version_info()[:2])
 # The full version, including alpha/beta/rc tags.
-release = '.'.join(_version_strs)
+release = jedi.__version__
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -141,12 +140,12 @@ html_static_path = ['_static']
 # Custom sidebar templates, maps document names to template names.
 html_sidebars = {
     '**': [
-#        'sidebarlogo.html',
+        'sidebarlogo.html',
         'localtoc.html',
-#        'relations.html',
+        #'relations.html',
         'ghbuttons.html',
-#        'sourcelink.html',
-#        'searchbox.html'
+        #'sourcelink.html',
+        #'searchbox.html'
     ]
 }
 
@@ -189,21 +188,21 @@ htmlhelp_basename = 'Jedidoc'
 # -- Options for LaTeX output --------------------------------------------------
 
 latex_elements = {
-# The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
+    # The paper size ('letterpaper' or 'a4paper').
+    #'papersize': 'letterpaper',
 
-# The font size ('10pt', '11pt' or '12pt').
-#'pointsize': '10pt',
+    # The font size ('10pt', '11pt' or '12pt').
+    #'pointsize': '10pt',
 
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
+    # Additional stuff for the LaTeX preamble.
+    #'preamble': '',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title, author, documentclass [howto/manual]).
 latex_documents = [
-  ('index', 'Jedi.tex', u'Jedi Documentation',
-   u'Jedi contributors', 'manual'),
+    ('index', 'Jedi.tex', u'Jedi Documentation',
+     u'Jedi contributors', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -246,9 +245,9 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-  ('index', 'Jedi', u'Jedi Documentation',
-   u'Jedi contributors', 'Jedi', 'Awesome Python autocompletion library.',
-   'Miscellaneous'),
+    ('index', 'Jedi', u'Jedi Documentation',
+     u'Jedi contributors', 'Jedi', 'Awesome Python autocompletion library.',
+     'Miscellaneous'),
 ]
 
 # Documents to append as an appendix to all manuals.
@@ -266,6 +265,27 @@ todo_include_todos = False
 
 # -- Options for autodoc module ------------------------------------------------
 
+autoclass_content = 'both'
 autodoc_member_order = 'bysource'
 autodoc_default_flags = []
 #autodoc_default_flags = ['members', 'undoc-members']
+
+
+# -- Options for intersphinx module --------------------------------------------
+
+intersphinx_mapping = {
+    'http://docs.python.org/': None,
+}
+
+
+def skip_deprecated(app, what, name, obj, skip, options):
+    """
+    All attributes containing a deprecated note shouldn't be documented
+    anymore. This makes it even clearer that they are not supported anymore.
+    """
+    doc = obj.__doc__
+    return skip or doc and '.. deprecated::' in doc
+
+
+def setup(app):
+    app.connect('autodoc-skip-member', skip_deprecated)

docs/docs/development.rst

@@ -44,25 +44,27 @@ The Jedi Core
 
 The core of Jedi consists of three parts:
 
-- :ref:`Parser <parsing>`
+- :ref:`Parser <parser>`
 - :ref:`Python code evaluation <evaluate>`
 - :ref:`API <dev-api>`
 
 Most people are probably interested in :ref:`code evaluation <evaluate>`,
 because that's where all the magic happens. I need to introduce the :ref:`parser
-<parsing>` first, because :mod:`evaluate` uses it extensively.
+<parser>` first, because :mod:`jedi.evaluate` uses it extensively.
 
-.. _parsing:
+.. _parser:
 
-Parser (parsing.py)
-~~~~~~~~~~~~~~~~~~~
+Parser (parser/__init__.py)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. automodule:: parsing
+.. automodule:: jedi.parser
 
-Parser Representation (parser_representation.py)
+Parser Representation (parser/representation.py)
 ++++++++++++++++++++++++++++++++++++++++++++++++
 
-.. automodule:: parsing_representation
+.. automodule:: jedi.parser.representation
+
+Class inheritance diagram:
 
 .. inheritance-diagram::
    SubModule
@@ -82,15 +84,15 @@ Parser Representation (parser_representation.py)
 
 .. _evaluate:
 
-Evaluation of python code (evaluate.py)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Evaluation of python code (evaluate/__init__.py)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. automodule:: evaluate
+.. automodule:: jedi.evaluate
 
-Evaluation Representation (evaluate_representation.py)
+Evaluation Representation (evaluate/representation.py)
 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
-.. automodule:: evaluate_representation
+.. automodule:: jedi.evaluate.representation
 
 .. inheritance-diagram::
    Executable
@@ -98,11 +100,18 @@ Evaluation Representation (evaluate_representation.py)
    InstanceElement
    Class
    Function
-   Execution
-   Generator
-   Array
+   FunctionExecution
    :parts: 1
 
+
+.. _name_resolution:
+
+Name resolution (evaluate/finder.py)
+++++++++++++++++++++++++++++++++++++
+
+.. automodule:: jedi.evaluate.finder
+
+
 .. _dev-api:
 
 API (api.py and api_classes.py)
@@ -115,7 +124,6 @@ conservative with the API.  I'd rather not add new API features if they are not
 necessary, because it's much harder to deprecate stuff than to add it later.
 
 
-
 .. _core-extensions:
 
 Core Extensions
@@ -123,7 +131,8 @@ Core Extensions
 
 Core Extensions is a summary of the following topics:
 
-- :ref:`Dynamic Arrays & Function Parameters <dynamic>`
+- :ref:`Iterables & Dynamic Arrays <iterables>`
+- :ref:`Dynamic Parameters <dynamic>`
 - :ref:`Fast Parser <fast_parser>`
 - :ref:`Docstrings <docstrings>`
 - :ref:`Refactoring <refactoring>`
@@ -132,35 +141,45 @@ These topics are very important to understand what Jedi additionally does, but
 they could be removed from Jedi and Jedi would still work. But slower and
 without some features.
 
-.. _dynamic:
+.. _iterables:
 
-Dynamic Arrays & Function Parameters (dynamic.py)
+Iterables & Dynamic Arrays (evaluate/iterable.py)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. automodule:: dynamic
+To understand Python on a deeper level, |jedi| needs to understand some of the
+dynamic features of Python, however this probably the most complicated part:
+
+.. automodule:: jedi.evaluate.iterable
+
+
+.. _dynamic:
+
+Parameter completion (evaluate/dynamic.py)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: jedi.evaluate.dynamic
 
 
 .. _fast_parser:
 
-Fast Parser (fast_parser.py)
+Fast Parser (parser/fast.py)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. automodule:: fast_parser
+.. automodule:: jedi.parser.fast
 
 .. _docstrings:
 
-Docstrings (docstrings.py)
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Docstrings (evaluate/docstrings.py)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. automodule:: docstrings
+.. automodule:: jedi.evaluate.docstrings
 
 .. _refactoring:
 
-Refactoring (refactoring.py)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. automodule:: refactoring
+Refactoring (evaluate/refactoring.py)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+.. automodule:: jedi.refactoring
 
 
 .. _imports-modules:
@@ -173,33 +192,27 @@ Imports & Modules
 - :ref:`Builtin Modules <builtin>`
 - :ref:`Imports <imports>`
 
-.. _modules:
-
-Modules (modules.py)
-~~~~~~~~~~~~~~~~~~~~
-
-.. automodule:: modules
 
 .. _builtin:
 
-Builtin Modules (builtin.py)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Compiled Modules (evaluate/compiled.py)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: jedi.evaluate.compiled
 
-.. automodule:: builtin
 
 .. _imports:
 
-Imports (imports.py)
-~~~~~~~~~~~~~~~~~~~~
-
-.. automodule:: imports
+Imports (evaluate/imports.py)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+.. automodule:: jedi.evaluate.imports
 
 
 .. _caching-recursions:
 
 Caching & Recursions
-----------------------
+--------------------
 
 
 - :ref:`Caching <cache>`
@@ -210,15 +223,14 @@ Caching & Recursions
 Caching (cache.py)
 ~~~~~~~~~~~~~~~~~~
 
-.. automodule:: cache
+.. automodule:: jedi.cache
 
 .. _recursion:
 
 Recursions (recursion.py)
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. automodule:: recursion
-
+.. automodule:: jedi.evaluate.recursion
 
 
 .. _dev-helpers:
@@ -226,12 +238,11 @@ Recursions (recursion.py)
 Helper Modules
 ---------------
 
-There are some helper modules: `common.py`, `helpers.py`, `debug.py`,
-`keywords.py`, which I won't describe further. Some functions are inheritely
-important and central for Jedi to work, but not important to understand how
-Jedi works.
+Most other modules are not really central to how Jedi works. They all contain
+relevant code, but you if you understand the modules above, you pretty much
+understand Jedi.
 
 Python 2/3 compatibility (_compatibility.py)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. automodule:: _compatibility
+.. automodule:: jedi._compatibility

docs/docs/features.rst

@@ -3,7 +3,18 @@
 Features and Caveats
 ====================
 
-|jedi| supports many of the widely used Python features:
+Jedi obviously supports autocompletion. It's also possible to get it working in
+(:ref:`your REPL (IPython, etc.) <repl-completion>`).
+
+Static analysis is also possible by using the command ``jedi.names``.
+
+The Jedi Linter is currently in an alpha version and can be tested by calling
+``python -m jedi linter``.
+
+Jedi would in theory support refactoring, but we have never publicized it,
+because it's not production ready. If you're interested in helping out here,
+let me know. With the latest parser changes, it should be very easy to actually
+make it work.
 
 
 General Features
@@ -13,13 +24,15 @@ General Features
 - ignores syntax errors and wrong indentation
 - can deal with complex module / function / class structures
 - virtualenv support
-- can infer function arguments from sphinx and epydoc docstrings (:ref:`type
-  hinting <type-hinting>`)
+- can infer function arguments from sphinx, epydoc and basic numpydoc docstrings
+  (:ref:`type hinting <type-hinting>`)
 
 
 Supported Python Features
 -------------------------
 
+|jedi| supports many of the widely used Python features:
+
 - builtins
 - multiple returns or yields
 - tuple assignments / array indexing / dictionary indexing
@@ -40,6 +53,8 @@ Supported Python Features
   case, that doesn't work with |jedi|)
 - simple/usual ``sys.path`` modifications
 - ``isinstance`` checks for if/while/assert
+- namespace packages (includes ``pkgutil`` and ``pkg_resources`` namespaces)
+- Django / Flask / Buildout support
 
 
 Unsupported Features
@@ -55,7 +70,7 @@ Will probably never be implemented:
 - metaclasses (how could an auto-completion ever support this)
 - ``setattr()``, ``__import__()``
 - writing to some dicts: ``globals()``, ``locals()``, ``object.__dict__``
-- evaluating ``if`` / ``while``
+- evaluating ``if`` / ``while`` / ``del``
 
 
 Caveats
@@ -64,8 +79,8 @@ Caveats
 **Malformed Syntax**
 
 Syntax errors and other strange stuff may lead to undefined behaviour of the
-completion. |jedi| is **NOT** a Python compiler, that tries to correct you. It is
-a tool that wants to help you. But **YOU** have to know Python, not |jedi|.
+completion. |jedi| is **NOT** a Python compiler, that tries to correct you. It
+is a tool that wants to help you. But **YOU** have to know Python, not |jedi|.
 
 **Legacy Python 2 Features**
 
@@ -75,23 +90,131 @@ older Python 2 features have been left out:
 
 - Classes: Always Python 3 like, therefore all classes inherit from ``object``.
 - Generators: No ``next()`` method. The ``__next__()`` method is used instead.
-- Exceptions are only looked at in the form of ``Exception as e``, no comma!
 
 **Slow Performance**
 
-Importing ``numpy`` can be quite slow sometimes, as well as loading the builtins
-the first time. If you want to speed things up, you could write import hooks in
-|jedi|, which preload stuff. However, once loaded, this is not a problem anymore.
-The same is true for huge modules like ``PySide``, ``wx``, etc.
+Importing ``numpy`` can be quite slow sometimes, as well as loading the
+builtins the first time. If you want to speed things up, you could write import
+hooks in |jedi|, which preload stuff. However, once loaded, this is not a
+problem anymore.  The same is true for huge modules like ``PySide``, ``wx``,
+etc.
 
 **Security**
 
-Security is an important issue for |jedi|. Therefore no Python code is executed.
-As long as you write pure python, everything is evaluated statically. But: If
-you use builtin modules (``c_builtin``) there is no other option than to execute
-those modules. However: Execute isn't that critical (as e.g. in pythoncomplete,
-which used to execute *every* import!), because it means one import and no more.
-So basically the only dangerous thing is using the import itself. If your
-``c_builtin`` uses some strange initializations, it might be dangerous. But if
-it does you're screwed anyways, because eventualy you're going to execute your
-code, which executes the import.
+Security is an important issue for |jedi|. Therefore no Python code is
+executed.  As long as you write pure python, everything is evaluated
+statically. But: If you use builtin modules (``c_builtin``) there is no other
+option than to execute those modules. However: Execute isn't that critical (as
+e.g. in pythoncomplete, which used to execute *every* import!), because it
+means one import and no more.  So basically the only dangerous thing is using
+the import itself. If your ``c_builtin`` uses some strange initializations, it
+might be dangerous. But if it does you're screwed anyways, because eventualy
+you're going to execute your code, which executes the import.
+
+
+Recipes
+-------
+
+Here are some tips on how to use |jedi| efficiently.
+
+
+.. _type-hinting:
+
+Type Hinting
+~~~~~~~~~~~~
+
+If |jedi| cannot detect the type of a function argument correctly (due to the
+dynamic nature of Python), you can help it by hinting the type using
+one of the following docstring syntax styles:
+
+**Sphinx style**
+
+http://sphinx-doc.org/domains.html#info-field-lists
+
+::
+
+    def myfunction(node, foo):
+        """Do something with a ``node``.
+
+        :type node: ProgramNode
+        :param str foo: foo parameter description
+
+        """
+        node.| # complete here
+
+**Epydoc**
+
+http://epydoc.sourceforge.net/manual-fields.html
+
+::
+
+    def myfunction(node):
+        """Do something with a ``node``.
+
+        @type node: ProgramNode
+
+        """
+        node.| # complete here
+
+**Numpydoc**
+
+https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
+
+In order to support the numpydoc format, you need to install the `numpydoc
+<https://pypi.python.org/pypi/numpydoc>`__ package.
+
+::
+
+    def foo(var1, var2, long_var_name='hi'):
+        r"""A one-line summary that does not use variable names or the
+        function name.
+
+        ...
+
+        Parameters
+        ----------
+        var1 : array_like
+            Array_like means all those objects -- lists, nested lists,
+            etc. -- that can be converted to an array. We can also
+            refer to variables like `var1`.
+        var2 : int
+            The type above can either refer to an actual Python type
+            (e.g. ``int``), or describe the type of the variable in more
+            detail, e.g. ``(N,) ndarray`` or ``array_like``.
+        long_variable_name : {'hi', 'ho'}, optional
+            Choices in brackets, default first when optional.
+
+        ...
+
+        """
+        var2.| # complete here
+
+A little history
+----------------
+
+The Star Wars Jedi are awesome. My Jedi software tries to imitate a little bit
+of the precognition the Jedi have. There's even an awesome `scene
+<http://www.youtube.com/watch?v=5BDO3pyavOY>`_ of Monty Python Jedis :-).
+
+But actually the name hasn't so much to do with Star Wars. It's part of my
+second name.
+
+After I explained Guido van Rossum, how some parts of my auto-completion work,
+he said (we drank a beer or two):
+
+    *"Oh, that worries me..."*
+
+When it's finished, I hope he'll like it :-)
+
+I actually started Jedi, because there were no good solutions available for VIM.
+Most auto-completions just didn't work well. The only good solution was PyCharm.
+But I like my good old VIM. Rope was never really intended to be an
+auto-completion (and also I really hate project folders for my Python scripts).
+It's more of a refactoring suite. So I decided to do my own version of a
+completion, which would execute non-dangerous code. But I soon realized, that
+this wouldn't work. So I built an extremely recursive thing which understands
+many of Python's key features.
+
+By the way, I really tried to program it as understandable as possible. But I
+think understanding it might need quite some time, because of its recursive
+nature.

docs/docs/history.rst

@@ -1,31 +0,0 @@
-.. include:: ../global.rst
-
-A little history
-================
-
-The Star Wars Jedi are awesome. My Jedi software tries to imitate a little bit
-of the precognition the Jedi have. There's even an awesome `scene
-<http://www.youtube.com/watch?v=5BDO3pyavOY>`_ of Monty Python Jedis :-).
-
-But actually the name hasn't so much to do with Star Wars. It's part of my
-second name.
-
-After I explained Guido van Rossum, how some parts of my auto-completion work,
-he said (we drank a beer or two):
-
-    *"Oh, that worries me..."*
-
-When it's finished, I hope he'll like it :-)
-
-I actually started Jedi, because there were no good solutions available for VIM.
-Most auto-completions just didn't work well. The only good solution was PyCharm.
-But I like my good old VIM. Rope was never really intended to be an
-auto-completion (and also I really hate project folders for my Python scripts).
-It's more of a refactoring suite. So I decided to do my own version of a
-completion, which would execute non-dangerous code. But I soon realized, that
-this wouldn't work. So I built an extremely recursive thing which understands
-many of Python's key features.
-
-By the way, I really tried to program it as understandable as possible. But I
-think understanding it might need quite some time, because of its recursive
-nature.

docs/docs/installation.rst

@@ -30,22 +30,31 @@ System-wide installation via a package manager
 Arch Linux
 ~~~~~~~~~~
 
-You can install jedi directly from AUR: `python-jedi at AUR
-<https://aur.archlinux.org/packages/python-jedi/>`__.
+You can install |jedi| directly from official Arch Linux packages:
 
-(There is also a packaged version of the vim plugin available: `vim-jedi at AUR
-<https://aur.archlinux.org/packages/vim-jedi/>`__.)
+- `python-jedi <https://www.archlinux.org/packages/community/any/python-jedi/>`__
+  (Python 3)
+- `python2-jedi <https://www.archlinux.org/packages/community/any/python2-jedi/>`__
+  (Python 2)
+
+The specified Python version just refers to the *runtime environment* for
+|jedi|. Use the Python 2 version if you're running vim (or whatever editor you
+use) under Python 2. Otherwise, use the Python 3 version. But whatever version
+you choose, both are able to complete both Python 2 and 3 *code*.
+
+(There is also a packaged version of the vim plugin available: `vim-jedi at
+Arch Linux<https://www.archlinux.org/packages/community/any/vim-jedi/>`__.)
 
 Debian
 ~~~~~~
 
-Debian packages are available as `experimental packages
-<http://packages.debian.org/experimental/python-jedi>`__.
+Debian packages are available in the `unstable repository
+<http://packages.debian.org/search?keywords=python%20jedi>`__.
 
 Others
 ~~~~~~
 
-We are in the discussion of adding Jedi to the Fedora repositories.
+We are in the discussion of adding |jedi| to the Fedora repositories.
 
 
 Manual installation from a downloaded package
@@ -53,7 +62,7 @@ Manual installation from a downloaded package
 
 If you prefer not to use an automated package installer, you can `download
 <https://github.com/davidhalter/jedi/archive/master.zip>`__ a current copy of
-*Jedi* and install it manually.
+|jedi| and install it manually.
 
 To install it, navigate to the directory containing `setup.py` on your console
 and type::

docs/docs/plugin-api-classes.rst

@@ -0,0 +1,10 @@
+.. include:: ../global.rst
+
+.. _plugin-api-classes:
+
+API Return Classes
+------------------
+
+.. automodule:: jedi.api.classes
+    :members:
+    :undoc-members:

docs/docs/plugin-api.rst

@@ -8,8 +8,20 @@ The Plugin API
 Note: This documentation is for Plugin developers, who want to improve their
 editors/IDE autocompletion 
 
-If you want to use |jedi|, you first need to
-``import jedi``. You then have direct access to the :class:`.Script`.
+If you want to use |jedi|, you first need to ``import jedi``.  You then have
+direct access to the :class:`.Script`. You can then call the functions
+documented here. These functions return :ref:`API classes
+<plugin-api-classes>`.
+
+
+Deprecations
+------------
+
+The deprecation process is as follows:
+
+1. A deprecation is announced in the next major/minor release.
+2. We wait either at least a year & at least two minor releases until we remove
+   the deprecated functionality.
 
 
 API documentation
@@ -18,24 +30,11 @@ API documentation
 API Interface
 ~~~~~~~~~~~~~
 
-.. automodule:: api
+.. automodule:: jedi.api
     :members:
     :undoc-members:
 
 
-
-API Return Classes
-~~~~~~~~~~~~~~~~~~
-
-.. automodule:: api_classes
-    :members:
-    :undoc-members:
-
-Settings Module
-~~~~~~~~~~~~~~~
-
-.. automodule:: settings
-
 Examples
 --------
 
@@ -73,10 +72,10 @@ Definitions / Goto:
     ... inception()'''
     >>> script = jedi.Script(source, 8, 1, '')
     >>>
-    >>> script.goto()
+    >>> script.goto_assignments()
     [<Definition inception=my_list[2]>]
     >>>
-    >>> script.get_definition()
+    >>> script.goto_definitions()
     [<Definition def my_func>]
 
 Related names:

docs/docs/recipes.rst

@@ -1,44 +0,0 @@
-.. include:: ../global.rst
-
-Recipes
-=======
-
-Here are some tips on how to use |jedi| efficiently.
-
-
-.. _type-hinting:
-
-Type Hinting
-------------
-
-If |jedi| cannot detect the type of a function argument correctly (due to the
-dynamic nature of Python), you can help it by hinting the type using
-Sphinx-style info field lists or Epydoc docstrings.
-
-**Sphinx style**
-
-http://sphinx-doc.org/domains.html#info-field-lists
-
-::
-
-    def myfunction(node):
-        """Do something with a ``node``.
-
-        :type node: ProgramNode
-
-        """
-        node.| # complete here
-
-**Epydoc**
-
-http://epydoc.sourceforge.net/manual-fields.html
-
-::
-
-    def myfunction(node):
-        """Do something with a ``node``.
-
-        @param node: ProgramNode
-
-        """
-        node.| # complete here

docs/docs/settings.rst

@@ -0,0 +1,6 @@
+.. include:: ../global.rst
+
+Settings
+========
+
+.. automodule:: jedi.settings

docs/docs/testing.rst

@@ -23,6 +23,9 @@ You want to add a test for |jedi|? Great! We love that. Normally you should
 write your tests as :ref:`Blackbox Tests <blackbox>`. Most tests would
 fit right in there.
 
+For specific API testing we're using simple unit tests, with a focus on a
+simple and readable testing structure.
+
 .. _blackbox:
 
 Blackbox Tests (run.py)
@@ -30,12 +33,8 @@ Blackbox Tests (run.py)
 
 .. automodule:: test.run
 
-Regression Tests (test_regression.py)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. automodule:: test.test_regression
-
 Refactoring Tests (refactor.py)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. automodule:: test.refactor
+

docs/docs/usage.rst

@@ -0,0 +1,88 @@
+.. include:: ../global.rst
+
+End User Usage
+==============
+
+If you are a not an IDE Developer, the odds are that you just want to use
+|jedi| as a browser plugin or in the shell. Yes that's :ref:`also possible
+<repl-completion>`!
+
+|jedi| is relatively young and can be used in a variety of Plugins and
+Software. If your Editor/IDE is not among them, recommend |jedi| to your IDE
+developers.
+
+
+.. _editor-plugins:
+
+Editor Plugins
+--------------
+
+Vim:
+
+- jedi-vim_
+- YouCompleteMe_
+
+Emacs:
+
+- Jedi.el_
+- elpy_
+- anaconda-mode_
+
+Sublime Text 2/3:
+
+- SublimeJEDI_ (ST2 & ST3)
+- anaconda_ (only ST3)
+
+SynWrite:
+
+- SynJedi_
+
+TextMate:
+
+- Textmate_ (Not sure if it's actually working)
+
+Kate:
+
+- Kate_ version 4.13+ `supports it natively
+  <https://projects.kde.org/projects/kde/applications/kate/repository/entry/addons/kate/pate/src/plugins/python_autocomplete_jedi.py?rev=KDE%2F4.13>`__,
+  you have to enable it, though.
+
+
+.. _other-software:
+
+Other Software Using Jedi
+-------------------------
+
+- wdb_ - Web Debugger
+
+
+.. _repl-completion:
+
+Tab completion in the Python Shell
+----------------------------------
+
+There are two different options how you can use Jedi autocompletion in
+your Python interpreter. One with your custom ``$HOME/.pythonrc.py`` file
+and one that uses ``PYTHONSTARTUP``.
+
+Using ``PYTHONSTARTUP``
+~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: jedi.replstartup
+
+Using a custom ``$HOME/.pythonrc.py``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autofunction:: jedi.utils.setup_readline
+
+.. _jedi-vim: https://github.com/davidhalter/jedi-vim
+.. _youcompleteme: http://valloric.github.io/YouCompleteMe/
+.. _Jedi.el: https://github.com/tkf/emacs-jedi
+.. _elpy: https://github.com/jorgenschaefer/elpy
+.. _anaconda-mode: https://github.com/proofit404/anaconda-mode
+.. _sublimejedi: https://github.com/srusskih/SublimeJEDI
+.. _anaconda: https://github.com/DamnWidget/anaconda
+.. _SynJedi: http://uvviewsoft.com/synjedi/
+.. _wdb: https://github.com/Kozea/wdb
+.. _TextMate: https://github.com/lawrenceakka/python-jedi.tmbundle
+.. _kate: http://kate-editor.org/

docs/index.rst

@@ -1,7 +1,7 @@
 .. include global.rst
 
-Jedi - an awesome autocompletion library for Python
-===================================================
+Jedi - an awesome autocompletion/static analysis library for Python
+===================================================================
 
 Release v\ |release|. (:doc:`Installation <docs/installation>`)
 
@@ -18,13 +18,14 @@ Docs
 ----
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
+   docs/usage
    docs/installation
    docs/features
-   docs/recipes
    docs/plugin-api
-   docs/history
+   docs/plugin-api-classes
+   docs/settings
    docs/development
    docs/testing
 
@@ -37,21 +38,3 @@ Resources
 - `Source Code on Github <https://github.com/davidhalter/jedi>`_
 - `Travis Testing <https://travis-ci.org/davidhalter/jedi>`_
 - `Python Package Index <http://pypi.python.org/pypi/jedi/>`_
-
-
-.. _editor-plugins:
-
-Editor Plugins
---------------
-
-- `Vim <http://github.com/davidhalter/jedi-vim>`_
-- `Emacs <https://github.com/tkf/emacs-jedi>`_
-- `Sublime Text 2 <https://github.com/svaiter/SublimeJEDI>`_
-
-
-.. _other-software:
-
-Other Software Using Jedi
--------------------------
-
-- `wdb <https://github.com/Kozea/wdb>`_

jedi/__init__.py

@@ -1,16 +1,18 @@
 """
-Jedi is an autocompletion tool for Python that can be used in IDEs/editors.
-Jedi works. Jedi is fast. It understands all of the basic Python syntax
-elements including many builtin functions.
+Jedi is a static analysis tool for Python that can be used in IDEs/editors. Its
+historic focus is autocompletion, but does static analysis for now as well.
+Jedi is fast and is very well tested. It understands Python on a deeper level
+than all other static analysis frameworks for Python.
 
-Additionaly, Jedi suports two different goto functions and has support for
-renaming as well as Pydoc support and some other IDE features.
+Jedi has support for two different goto functions. It's possible to search for
+related names and to list all names in a Python file and infer them. Jedi
+understands docstrings and you can use Jedi autocompletion in your REPL as
+well.
 
 Jedi uses a very simple API to connect with IDE's. There's a reference
-implementation as a `VIM-Plugin <http://github.com/davidhalter/jedi-vim>`_,
-which uses Jedi's autocompletion.  I encourage you to use Jedi in your IDEs.
-It's really easy. If there are any problems (also with licensing), just contact
-me.
+implementation as a `VIM-Plugin <https://github.com/davidhalter/jedi-vim>`_,
+which uses Jedi's autocompletion.  We encourage you to use Jedi in your IDEs.
+It's really easy.
 
 To give you a simple example how you can use the Jedi library, here is an
 example for the autocompletion feature:
@@ -34,16 +36,8 @@ As you see Jedi is pretty simple and allows you to concentrate on writing a
 good text editor, while still having very good IDE features for Python.
 """
 
-__version__ = 0, 6, 0
+__version__ = '0.9.0'
 
-import sys
-
-# python imports are hell sometimes. Especially the combination of relative
-# imports and circular imports... Just avoid it:
-sys.path.insert(0, __path__[0])
-
-from .api import Script, NotFoundError, set_debug_function, _quick_complete, \
-                 preload_module
-from . import settings
-
-sys.path.pop(0)
+from jedi.api import Script, Interpreter, NotFoundError, set_debug_function
+from jedi.api import preload_module, defined_names, names
+from jedi import settings

jedi/__main__.py

@@ -0,0 +1,43 @@
+from sys import argv
+from os.path import join, dirname, abspath, isdir
+
+
+if len(argv) == 2 and argv[1] == 'repl':
+    # don't want to use __main__ only for repl yet, maybe we want to use it for
+    # something else. So just use the keyword ``repl`` for now.
+    print(join(dirname(abspath(__file__)), 'api', 'replstartup.py'))
+elif len(argv) > 1 and argv[1] == 'linter':
+    """
+    This is a pre-alpha API. You're not supposed to use it at all, except for
+    testing. It will very likely change.
+    """
+    import jedi
+    import sys
+
+    if '--debug' in sys.argv:
+        jedi.set_debug_function()
+
+    for path in sys.argv[2:]:
+        if path.startswith('--'):
+            continue
+        if isdir(path):
+            import fnmatch
+            import os
+
+            paths = []
+            for root, dirnames, filenames in os.walk(path):
+                for filename in fnmatch.filter(filenames, '*.py'):
+                    paths.append(os.path.join(root, filename))
+        else:
+            paths = [path]
+
+        try:
+            for path in paths:
+                for error in jedi.Script(path=path)._analysis():
+                    print(error)
+        except Exception:
+            if '--pdb' in sys.argv:
+                import pdb
+                pdb.post_mortem()
+            else:
+                raise

jedi/_compatibility.py

@@ -1,91 +1,75 @@
 """
 To ensure compatibility from Python ``2.6`` - ``3.3``, a module has been
-created. Clearly there is huge need to use conforming syntax. But many changes
-(e.g. ``property``, ``hasattr`` in ``2.5``) can be rewritten in pure python.
+created. Clearly there is huge need to use conforming syntax.
 """
 import sys
 import imp
 import os
+import re
 try:
     import importlib
-except:
+except ImportError:
     pass
 
-is_py3k = sys.hexversion >= 0x03000000
-is_py33 = sys.hexversion >= 0x03030000
+is_py3 = sys.version_info[0] >= 3
+is_py33 = is_py3 and sys.version_info.minor >= 3
+is_py26 = not is_py3 and sys.version_info[1] < 7
 
 
 def find_module_py33(string, path=None):
-    mod_info = (None, None, None)
-    loader = None
-    if path is not None:
-        # Check for the module in the specidied path
-        loader = importlib.machinery.PathFinder.find_module(string, path)
-    else:
-        # Check for the module in sys.path
-        loader = importlib.machinery.PathFinder.find_module(string, sys.path)
-        if loader is None:
-            # Fallback to find builtins
+    loader = importlib.machinery.PathFinder.find_module(string, path)
+
+    if loader is None and path is None:  # Fallback to find builtins
+        try:
             loader = importlib.find_loader(string)
+        except ValueError as e:
+            # See #491. Importlib might raise a ValueError, to avoid this, we
+            # just raise an ImportError to fix the issue.
+            raise ImportError("Originally ValueError: " + e.message)
 
     if loader is None:
-        raise ImportError
+        raise ImportError("Couldn't find a loader for {0}".format(string))
 
     try:
-        if (loader.is_package(string)):
-            mod_info = (None, os.path.dirname(loader.path), True)
+        is_package = loader.is_package(string)
+        if is_package:
+            module_path = os.path.dirname(loader.path)
+            module_file = None
         else:
-            filename = loader.get_filename(string)
-            if filename and os.path.exists(filename):
-                mod_info = (open(filename, 'U'), filename, False)
-            else:
-                mod_info = (None, filename, False)
+            module_path = loader.get_filename(string)
+            module_file = open(module_path, 'rb')
     except AttributeError:
-        mod_info = (None, loader.load_module(string).__name__, False)
+        # ExtensionLoader has not attribute get_filename, instead it has a
+        # path attribute that we can use to retrieve the module path
+        try:
+            module_path = loader.path
+            module_file = open(loader.path, 'rb')
+        except AttributeError:
+            module_path = string
+            module_file = None
+        finally:
+            is_package = False
 
-    return mod_info
+    return module_file, module_path, is_package
 
 
 def find_module_pre_py33(string, path=None):
-    mod_info = None
-    if path is None:
-        mod_info = imp.find_module(string)
-    else:
-        mod_info = imp.find_module(string, path)
-
-    return (mod_info[0], mod_info[1], mod_info[2][2] == imp.PKG_DIRECTORY)
+    module_file, module_path, description = imp.find_module(string, path)
+    module_type = description[2]
+    return module_file, module_path, module_type is imp.PKG_DIRECTORY
 
 
-def find_module(string, path=None):
-    """Provides information about a module.
+find_module = find_module_py33 if is_py33 else find_module_pre_py33
+find_module.__doc__ = """
+Provides information about a module.
 
-    This function isolates the differences in importing libraries introduced with
-    python 3.3 on; it gets a module name and optionally a path. It will return a
-    tuple containin an open file for the module (if not builtin), the filename
-    or the name of the module if it is a builtin one and a boolean indicating
-    if the module is contained in a package."""
-    if is_py33:
-        return find_module_py33(string, path)
-    else:
-        return find_module_pre_py33(string, path)
+This function isolates the differences in importing libraries introduced with
+python 3.3 on; it gets a module name and optionally a path. It will return a
+tuple containin an open file for the module (if not builtin), the filename
+or the name of the module if it is a builtin one and a boolean indicating
+if the module is contained in a package.
+"""
 
-# next was defined in python 2.6, in python 3 obj.next won't be possible
-# anymore
-try:
-    next = next
-except NameError:
-    _raiseStopIteration = object()
-
-    def next(iterator, default=_raiseStopIteration):
-        if not hasattr(iterator, 'next'):
-            raise TypeError("not an iterator")
-        try:
-            return iterator.next()
-        except StopIteration:
-            if default is _raiseStopIteration:
-                raise
-            else:
-                return default
 
 # unicode function
 try:
@@ -93,17 +77,17 @@ try:
 except NameError:
     unicode = str
 
-if is_py3k:
-    utf8 = lambda s: s
+if is_py3:
+    u = lambda s: s
 else:
-    utf8 = lambda s: s.decode('utf-8')
+    u = lambda s: s.decode('utf-8')
 
-utf8.__doc__ = """
+u.__doc__ = """
 Decode a raw string into unicode object.  Do nothing in Python 3.
 """
 
 # exec function
-if is_py3k:
+if is_py3:
     def exec_function(source, global_map):
         exec(source, global_map)
 else:
@@ -111,7 +95,7 @@ else:
                         exec source in global_map """, 'blub', 'exec'))
 
 # re-raise function
-if is_py3k:
+if is_py3:
     def reraise(exception, traceback):
         raise exception.with_traceback(traceback)
 else:
@@ -129,24 +113,6 @@ Usage::
 
 """
 
-# StringIO (Python 2.5 has no io module), so use io only for py3k
-try:
-    from StringIO import StringIO
-except ImportError:
-    from io import StringIO
-
-# hasattr function used because python
-if is_py3k:
-    hasattr = hasattr
-else:
-    def hasattr(obj, name):
-        try:
-            getattr(obj, name)
-            return True
-        except AttributeError:
-            return False
-
-
 class Python3Method(object):
     def __init__(self, func):
         self.func = func
@@ -157,12 +123,6 @@ class Python3Method(object):
         else:
             return lambda *args, **kwargs: self.func(obj, *args, **kwargs)
 
-try:
-    # the python3 way
-    from functools import reduce
-except ImportError:
-    reduce = reduce
-
 
 def use_metaclass(meta, *bases):
     """ Create a class with a metaclass. """
@@ -170,10 +130,6 @@ def use_metaclass(meta, *bases):
         bases = (object,)
     return meta("HackClass", bases, {})
 
-try:
-    from functools import reduce  # Python 3
-except ImportError:
-    reduce = reduce
 
 try:
     encoding = sys.stdout.encoding
@@ -181,3 +137,69 @@ try:
         encoding = 'utf-8'
 except AttributeError:
     encoding = 'ascii'
+
+
+def u(string):
+    """Cast to unicode DAMMIT!
+    Written because Python2 repr always implicitly casts to a string, so we
+    have to cast back to a unicode (and we now that we always deal with valid
+    unicode, because we check that in the beginning).
+    """
+    if is_py3:
+        return str(string)
+    elif not isinstance(string, unicode):
+        return unicode(str(string), 'UTF-8')
+    return string
+
+try:
+    import builtins  # module name in python 3
+except ImportError:
+    import __builtin__ as builtins
+
+
+import ast
+
+
+def literal_eval(string):
+    # py3.0, py3.1 and py32 don't support unicode literals. Support those, I
+    # don't want to write two versions of the tokenizer.
+    if is_py3 and sys.version_info.minor < 3:
+        if re.match('[uU][\'"]', string):
+            string = string[1:]
+    return ast.literal_eval(string)
+
+
+try:
+    from itertools import zip_longest
+except ImportError:
+    from itertools import izip_longest as zip_longest  # Python 2
+
+
+def no_unicode_pprint(dct):
+    """
+    Python 2/3 dict __repr__ may be different, because of unicode differens
+    (with or without a `u` prefix). Normally in doctests we could use `pprint`
+    to sort dicts and check for equality, but here we have to write a separate
+    function to do that.
+    """
+    import pprint
+    s = pprint.pformat(dct)
+    print(re.sub("u'", "'", s))
+
+
+def utf8_repr(func):
+    """
+    ``__repr__`` methods in Python 2 don't allow unicode objects to be
+    returned. Therefore cast them to utf-8 bytes in this decorator.
+    """
+    def wrapper(self):
+        result = func(self)
+        if isinstance(result, unicode):
+            return result.encode('utf-8')
+        else:
+            return result
+
+    if is_py3:
+        return func
+    else:
+        return wrapper

jedi/api.py

@@ -1,569 +0,0 @@
-"""
-The API basically only provides one class. You can create a :class:`Script` and
-use its methods.
-
-Additionally you can add a debug function with :func:`set_debug_function` and
-catch :exc:`NotFoundError` which is being raised if your completion is not
-possible.
-"""
-from __future__ import with_statement
-
-import re
-import os
-import warnings
-
-from jedi import parsing
-from jedi import parsing_representation as pr
-from jedi import debug
-from jedi import settings
-from jedi import helpers
-from jedi import common
-from jedi import cache
-from jedi import modules
-from jedi._compatibility import next, unicode
-import evaluate
-import keywords
-import api_classes
-import evaluate_representation as er
-import dynamic
-import imports
-import builtin
-
-
-class NotFoundError(Exception):
-    """A custom error to avoid catching the wrong exceptions."""
-    pass
-
-
-class Script(object):
-    """
-    A Script is the base for completions, goto or whatever you want to do with
-    |jedi|.
-
-    :param source: The source code of the current file, separated by newlines.
-    :type source: str
-    :param line: The line to perform actions on (starting with 1).
-    :type line: int
-    :param col: The column of the cursor (starting with 0).
-    :type col: int
-    :param source_path: The path of the file in the file system, or ``''`` if
-        it hasn't been saved yet.
-    :type source_path: str or None
-    :param source_encoding: The encoding of ``source``, if it is not a
-        ``unicode`` object (default ``'utf-8'``).
-    :type source_encoding: str
-    """
-    def __init__(self, source, line, column, source_path,
-                                 source_encoding='utf-8'):
-        api_classes._clear_caches()
-        debug.reset_time()
-        self.source = modules.source_to_unicode(source, source_encoding)
-        self.pos = line, column
-        self._module = modules.ModuleWithCursor(source_path,
-                                        source=self.source, position=self.pos)
-        self._source_path = source_path
-        self.source_path = None if source_path is None \
-                                    else os.path.abspath(source_path)
-        debug.speed('init')
-
-    def __repr__(self):
-        return '<%s: %s>' % (self.__class__.__name__, repr(self._source_path))
-
-    @property
-    def _parser(self):
-        """ lazy parser."""
-        return self._module.parser
-
-    @api_classes._clear_caches_after_call
-    def completions(self):
-        """
-        Return :class:`api_classes.Completion` objects. Those objects contain
-        information about the completions, more than just names.
-
-        :return: Completion objects, sorted by name and __ comes last.
-        :rtype: list of :class:`api_classes.Completion`
-        """
-        debug.speed('completions start')
-        path = self._module.get_path_until_cursor()
-        if re.search('^\.|\.\.$', path):
-            return []
-        path, dot, like = self._get_completion_parts(path)
-        completion_line = self._module.get_line(self.pos[0])[:self.pos[1]]
-
-        try:
-            scopes = list(self._prepare_goto(path, True))
-        except NotFoundError:
-            scopes = []
-            scope_generator = evaluate.get_names_of_scope(
-                                            self._parser.user_scope, self.pos)
-            completions = []
-            for scope, name_list in scope_generator:
-                for c in name_list:
-                    completions.append((c, scope))
-        else:
-            completions = []
-            debug.dbg('possible scopes', scopes)
-            for s in scopes:
-                if s.isinstance(er.Function):
-                    names = s.get_magic_method_names()
-                else:
-                    if isinstance(s, imports.ImportPath):
-                        if like == 'import':
-                            if not completion_line.endswith('import import'):
-                                continue
-                        a = s.import_stmt.alias
-                        if a and a.start_pos <= self.pos <= a.end_pos:
-                            continue
-                        names = s.get_defined_names(on_import_stmt=True)
-                    else:
-                        names = s.get_defined_names()
-
-                for c in names:
-                    completions.append((c, s))
-
-        if not dot:  # named params have no dots
-            for call_def in self.call_signatures():
-                if not call_def.module.is_builtin():
-                    for p in call_def.params:
-                        completions.append((p.get_name(), p))
-
-            # Do the completion if there is no path before and no import stmt.
-            u = self._parser.user_stmt
-            bs = builtin.Builtin.scope
-            if isinstance(u, pr.Import):
-                if (u.relative_count > 0 or u.from_ns) and not re.search(
-                            r'(,|from)\s*$|import\s+$', completion_line):
-                    completions += ((k, bs) for k
-                                            in keywords.get_keywords('import'))
-
-            if not path and not isinstance(u, pr.Import):
-                # add keywords
-                completions += ((k, bs) for k in keywords.get_keywords(
-                                                                    all=True))
-
-        needs_dot = not dot and path
-
-        comps = []
-        comp_dct = {}
-        for c, s in set(completions):
-            n = c.names[-1]
-            if settings.case_insensitive_completion \
-                    and n.lower().startswith(like.lower()) \
-                    or n.startswith(like):
-                if not evaluate.filter_private_variable(s,
-                                                    self._parser.user_stmt, n):
-                    new = api_classes.Completion(c, needs_dot,
-                                                    len(like), s)
-                    k = (new.name, new.complete)  # key
-                    if k in comp_dct and settings.no_completion_duplicates:
-                        comp_dct[k]._same_name_completions.append(new)
-                    else:
-                        comp_dct[k] = new
-                        comps.append(new)
-
-        debug.speed('completions end')
-
-        return sorted(comps, key=lambda x: (x.name.startswith('__'),
-                                            x.name.startswith('_'),
-                                            x.name.lower()))
-
-    def _prepare_goto(self, goto_path, is_like_search=False):
-        """
-        Base for completions/goto. Basically it returns the resolved scopes
-        under cursor.
-        """
-        debug.dbg('start: %s in %s' % (goto_path, self._parser.user_scope))
-
-        user_stmt = self._parser.user_stmt
-        debug.speed('parsed')
-        if not user_stmt and len(goto_path.split('\n')) > 1:
-            # If the user_stmt is not defined and the goto_path is multi line,
-            # something's strange. Most probably the backwards tokenizer
-            # matched to much.
-            return []
-
-        if isinstance(user_stmt, pr.Import):
-            scopes = [self._get_on_import_stmt(is_like_search)[0]]
-        else:
-            # just parse one statement, take it and evaluate it
-            stmt = self._get_under_cursor_stmt(goto_path)
-            scopes = evaluate.follow_statement(stmt)
-        return scopes
-
-    def _get_under_cursor_stmt(self, cursor_txt):
-        offset = self.pos[0] - 1, self.pos[1]
-        r = parsing.Parser(cursor_txt, no_docstr=True, offset=offset)
-        try:
-            stmt = r.module.statements[0]
-        except IndexError:
-            raise NotFoundError()
-        stmt.parent = self._parser.user_scope
-        return stmt
-
-    def complete(self):
-        """
-        .. deprecated:: 0.6.0
-           Use :attr:`.completions` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use completions instead.", DeprecationWarning)
-        return self.completions()
-
-    def goto(self):
-        """
-        .. deprecated:: 0.6.0
-           Use :attr:`.goto_assignments` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use goto_assignments instead.", DeprecationWarning)
-        return self.goto_assignments()
-
-    def definition(self):
-        """
-        .. deprecated:: 0.6.0
-           Use :attr:`.goto_definitions` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use goto_definitions instead.", DeprecationWarning)
-        return self.goto_definitions()
-
-    def get_definition(self):
-        """
-        .. deprecated:: 0.5.0
-           Use :attr:`.goto_definitions` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use goto_definitions instead.", DeprecationWarning)
-        return self.goto_definitions()
-
-    def related_names(self):
-        """
-        .. deprecated:: 0.6.0
-           Use :attr:`.usages` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use usages instead.", DeprecationWarning)
-        return self.usages()
-
-    def get_in_function_call(self):
-        """
-        .. deprecated:: 0.6.0
-           Use :attr:`.call_signatures` instead.
-        .. todo:: Remove!
-        """
-        return self.function_definition()
-
-    def function_definition(self):
-        """
-        .. deprecated:: 0.6.0
-           Use :attr:`.call_signatures` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use line instead.", DeprecationWarning)
-        sig = self.call_signatures()
-        return sig[0] if sig else None
-
-    @api_classes._clear_caches_after_call
-    def goto_definitions(self):
-        """
-        Return the definitions of a the path under the cursor.  goto function!
-        This follows complicated paths and returns the end, not the first
-        definition. The big difference between :meth:`goto_assignments` and
-        :meth:`goto_definitions` is that :meth:`goto_assignments` doesn't
-        follow imports and statements. Multiple objects may be returned,
-        because Python itself is a dynamic language, which means depending on
-        an option you can have two different versions of a function.
-
-        :rtype: list of :class:`api_classes.Definition`
-        """
-        def resolve_import_paths(scopes):
-            for s in scopes.copy():
-                if isinstance(s, imports.ImportPath):
-                    scopes.remove(s)
-                    scopes.update(resolve_import_paths(set(s.follow())))
-            return scopes
-
-        goto_path = self._module.get_path_under_cursor()
-
-        context = self._module.get_context()
-        scopes = set()
-        lower_priority_operators = ('()', '(', ',')
-        """Operators that could hide callee."""
-        if next(context) in ('class', 'def'):
-            scopes = set([self._module.parser.user_scope])
-        elif not goto_path:
-            op = self._module.get_operator_under_cursor()
-            if op and op not in lower_priority_operators:
-                scopes = set([keywords.get_operator(op, self.pos)])
-
-        # Fetch definition of callee
-        if not goto_path:
-            (call, _) = self._func_call_and_param_index()
-            if call is not None:
-                while call.next is not None:
-                    call = call.next
-                # reset cursor position:
-                (row, col) = call.name.end_pos
-                self.pos = (row, max(col - 1, 0))
-                self._module = modules.ModuleWithCursor(
-                    self._source_path,
-                    source=self.source,
-                    position=self.pos)
-                # then try to find the path again
-                goto_path = self._module.get_path_under_cursor()
-
-        if not scopes:
-            if goto_path:
-                scopes = set(self._prepare_goto(goto_path))
-            elif op in lower_priority_operators:
-                scopes = set([keywords.get_operator(op, self.pos)])
-
-        scopes = resolve_import_paths(scopes)
-
-        # add keywords
-        scopes |= keywords.get_keywords(string=goto_path, pos=self.pos)
-
-        d = set([api_classes.Definition(s) for s in scopes
-                    if not isinstance(s, imports.ImportPath._GlobalNamespace)])
-        return self._sorted_defs(d)
-
-    @api_classes._clear_caches_after_call
-    def goto_assignments(self):
-        """
-        Return the first definition found. Imports and statements aren't
-        followed. Multiple objects may be returned, because Python itself is a
-        dynamic language, which means depending on an option you can have two
-        different versions of a function.
-
-        :rtype: list of :class:`api_classes.Definition`
-        """
-        d = [api_classes.Definition(d) for d in set(self._goto()[0])]
-        return self._sorted_defs(d)
-
-    def _goto(self, add_import_name=False):
-        """
-        Used for goto_assignments and usages.
-
-        :param add_import_name: TODO add description
-        """
-        def follow_inexistent_imports(defs):
-            """ Imports can be generated, e.g. following
-            `multiprocessing.dummy` generates an import dummy in the
-            multiprocessing module. The Import doesn't exist -> follow.
-            """
-            definitions = set(defs)
-            for d in defs:
-                if isinstance(d.parent, pr.Import) \
-                                        and d.start_pos == (0, 0):
-                    i = imports.ImportPath(d.parent).follow(is_goto=True)
-                    definitions.remove(d)
-                    definitions |= follow_inexistent_imports(i)
-            return definitions
-
-        goto_path = self._module.get_path_under_cursor()
-        context = self._module.get_context()
-        user_stmt = self._parser.user_stmt
-        if next(context) in ('class', 'def'):
-            user_scope = self._parser.user_scope
-            definitions = set([user_scope.name])
-            search_name = unicode(user_scope.name)
-        elif isinstance(user_stmt, pr.Import):
-            s, name_part = self._get_on_import_stmt()
-            try:
-                definitions = [s.follow(is_goto=True)[0]]
-            except IndexError:
-                definitions = []
-            search_name = unicode(name_part)
-
-            if add_import_name:
-                import_name = user_stmt.get_defined_names()
-                # imports have only one name
-                if name_part == import_name[0].names[-1]:
-                    definitions.append(import_name[0])
-        else:
-            stmt = self._get_under_cursor_stmt(goto_path)
-            defs, search_name = evaluate.goto(stmt)
-            definitions = follow_inexistent_imports(defs)
-            if isinstance(user_stmt, pr.Statement):
-                if user_stmt.get_commands()[0].start_pos > self.pos:
-                    # The cursor must be after the start, otherwise the
-                    # statement is just an assignee.
-                    definitions = [user_stmt]
-        return definitions, search_name
-
-    @api_classes._clear_caches_after_call
-    def usages(self, additional_module_paths=()):
-        """
-        Return :class:`api_classes.Usage` objects, which contain all
-        names that point to the definition of the name under the cursor. This
-        is very useful for refactoring (renaming), or to show all usages of a
-        variable.
-
-        .. todo:: Implement additional_module_paths
-
-        :rtype: list of :class:`api_classes.Usage`
-        """
-        user_stmt = self._parser.user_stmt
-        definitions, search_name = self._goto(add_import_name=True)
-        if isinstance(user_stmt, pr.Statement) \
-                    and self.pos < user_stmt.get_commands()[0].start_pos:
-            # the search_name might be before `=`
-            definitions = [v for v in user_stmt.set_vars
-                                if unicode(v.names[-1]) == search_name]
-        if not isinstance(user_stmt, pr.Import):
-            # import case is looked at with add_import_name option
-            definitions = dynamic.usages_add_import_modules(definitions,
-                                                                search_name)
-
-        module = set([d.get_parent_until() for d in definitions])
-        module.add(self._parser.module)
-        names = dynamic.usages(definitions, search_name, module)
-
-        for d in set(definitions):
-            if isinstance(d, pr.Module):
-                names.append(api_classes.Usage(d, d))
-            else:
-                names.append(api_classes.Usage(d.names[-1], d))
-
-        return self._sorted_defs(set(names))
-
-    @api_classes._clear_caches_after_call
-    def call_signatures(self):
-        """
-        Return the function object of the call you're currently in.
-
-        E.g. if the cursor is here::
-
-            abs(# <-- cursor is here
-
-        This would return the ``abs`` function. On the other hand::
-
-            abs()# <-- cursor is here
-
-        This would return ``None``.
-
-        :rtype: :class:`api_classes.CallDef`
-        """
-
-        call, index = self._func_call_and_param_index()
-        if call is None:
-            return []
-
-        user_stmt = self._parser.user_stmt
-        with common.scale_speed_settings(settings.scale_function_definition):
-            _callable = lambda: evaluate.follow_call(call)
-            origins = cache.cache_function_definition(_callable, user_stmt)
-        debug.speed('func_call followed')
-
-        return [api_classes.CallDef(o, index, call) for o in origins]
-
-    def _func_call_and_param_index(self):
-        debug.speed('func_call start')
-        call, index = None, 0
-        if call is None:
-            user_stmt = self._parser.user_stmt
-            if user_stmt is not None and isinstance(user_stmt, pr.Statement):
-                call, index, _ = helpers.search_function_definition(
-                                                        user_stmt, self.pos)
-        debug.speed('func_call parsed')
-        return call, index
-
-    def _get_on_import_stmt(self, is_like_search=False):
-        """ Resolve the user statement, if it is an import. Only resolve the
-        parts until the user position. """
-        user_stmt = self._parser.user_stmt
-        import_names = user_stmt.get_all_import_names()
-        kill_count = -1
-        cur_name_part = None
-        for i in import_names:
-            if user_stmt.alias == i:
-                continue
-            for name_part in i.names:
-                if name_part.end_pos >= self.pos:
-                    if not cur_name_part:
-                        cur_name_part = name_part
-                    kill_count += 1
-
-        i = imports.ImportPath(user_stmt, is_like_search,
-                                kill_count=kill_count, direct_resolve=True)
-        return i, cur_name_part
-
-    def _get_completion_parts(self, path):
-        """
-        Returns the parts for the completion
-        :return: tuple - (path, dot, like)
-        """
-        match = re.match(r'^(.*?)(\.|)(\w?[\w\d]*)$', path, flags=re.S)
-        return match.groups()
-
-    @staticmethod
-    def _sorted_defs(d):
-        # Note: `or ''` below is required because `module_path` could be
-        #       None and you can't compare None and str in Python 3.
-        return sorted(d, key=lambda x: (x.module_path or '', x.start_pos))
-
-
-def defined_names(source, source_path=None, source_encoding='utf-8'):
-    """
-    Get all definitions in `source` sorted by its position.
-
-    This functions can be used for listing functions, classes and
-    data defined in a file.  This can be useful if you want to list
-    them in "sidebar".  Each element in the returned list also has
-    `defined_names` method which can be used to get sub-definitions
-    (e.g., methods in class).
-
-    :rtype: list of api_classes.Definition
-    """
-    parser = parsing.Parser(
-        modules.source_to_unicode(source, source_encoding),
-        module_path=source_path,
-    )
-    return api_classes._defined_names(parser.scope)
-
-
-def preload_module(*modules):
-    """
-    Preloading modules tells Jedi to load a module now, instead of lazy parsing
-    of modules. Usful for IDEs, to control which modules to load on startup.
-
-    :param modules: different module names, list of string.
-    """
-    for m in modules:
-        s = "import %s as x; x." % m
-        Script(s, 1, len(s), None).complete()
-
-
-def set_debug_function(func_cb=debug.print_to_stdout, warnings=True,
-                                            notices=True, speed=True):
-    """
-    Define a callback debug function to get all the debug messages.
-
-    :param func_cb: The callback function for debug messages, with n params.
-    """
-    debug.debug_function = func_cb
-    debug.enable_warning = warnings
-    debug.enable_notice = notices
-    debug.enable_speed = speed
-
-
-def _quick_complete(source):
-    """
-    Convenience function to complete a source string at the end.
-
-    Example:
-
-    >>> _quick_complete('''
-    ... import datetime
-    ... datetime.da''')                                 #doctest: +ELLIPSIS
-    [<Completion: date>, <Completion: datetime>, ...]
-
-    :param source: The source code to be completed.
-    :type source: string
-    :return: Completion objects as returned by :meth:`complete`.
-    :rtype: list of :class:`api_classes.Completion`
-    """
-    lines = re.sub(r'[\n\r\s]*$', '', source).splitlines()
-    pos = len(lines), len(lines[-1])
-    script = Script(source, pos[0], pos[1], '')
-    return script.completions()

jedi/api/__init__.py

@@ -0,0 +1,709 @@
+"""
+The API basically only provides one class. You can create a :class:`Script` and
+use its methods.
+
+Additionally you can add a debug function with :func:`set_debug_function`.
+
+.. warning:: Please, note that Jedi is **not thread safe**.
+"""
+import re
+import os
+import warnings
+import sys
+from itertools import chain
+
+from jedi._compatibility import unicode, builtins
+from jedi.parser import Parser, load_grammar
+from jedi.parser.tokenize import source_tokens
+from jedi.parser import tree
+from jedi.parser.user_context import UserContext, UserContextParser
+from jedi import debug
+from jedi import settings
+from jedi import common
+from jedi import cache
+from jedi.api import keywords
+from jedi.api import classes
+from jedi.api import interpreter
+from jedi.api import usages
+from jedi.api import helpers
+from jedi.evaluate import Evaluator
+from jedi.evaluate import representation as er
+from jedi.evaluate import compiled
+from jedi.evaluate import imports
+from jedi.evaluate.cache import memoize_default
+from jedi.evaluate.helpers import FakeName, get_module_names
+from jedi.evaluate.finder import global_names_dict_generator, filter_definition_names
+from jedi.evaluate import analysis
+
+# Jedi uses lots and lots of recursion. By setting this a little bit higher, we
+# can remove some "maximum recursion depth" errors.
+sys.setrecursionlimit(2000)
+
+
+class NotFoundError(Exception):
+    """A custom error to avoid catching the wrong exceptions.
+
+    .. deprecated:: 0.9.0
+       Not in use anymore, Jedi just returns no goto result if you're not on a
+       valid name.
+    .. todo:: Remove!
+    """
+
+
+class Script(object):
+    """
+    A Script is the base for completions, goto or whatever you want to do with
+    |jedi|.
+
+    You can either use the ``source`` parameter or ``path`` to read a file.
+    Usually you're going to want to use both of them (in an editor).
+
+    :param source: The source code of the current file, separated by newlines.
+    :type source: str
+    :param line: The line to perform actions on (starting with 1).
+    :type line: int
+    :param col: The column of the cursor (starting with 0).
+    :type col: int
+    :param path: The path of the file in the file system, or ``''`` if
+        it hasn't been saved yet.
+    :type path: str or None
+    :param encoding: The encoding of ``source``, if it is not a
+        ``unicode`` object (default ``'utf-8'``).
+    :type encoding: str
+    :param source_encoding: The encoding of ``source``, if it is not a
+        ``unicode`` object (default ``'utf-8'``).
+    :type encoding: str
+    """
+    def __init__(self, source=None, line=None, column=None, path=None,
+                 encoding='utf-8', source_path=None, source_encoding=None):
+        if source_path is not None:
+            warnings.warn("Use path instead of source_path.", DeprecationWarning)
+            path = source_path
+        if source_encoding is not None:
+            warnings.warn("Use encoding instead of source_encoding.", DeprecationWarning)
+            encoding = source_encoding
+
+        self._orig_path = path
+        self.path = None if path is None else os.path.abspath(path)
+
+        if source is None:
+            with open(path) as f:
+                source = f.read()
+
+        self.source = common.source_to_unicode(source, encoding)
+        lines = common.splitlines(self.source)
+        line = max(len(lines), 1) if line is None else line
+        if not (0 < line <= len(lines)):
+            raise ValueError('`line` parameter is not in a valid range.')
+
+        line_len = len(lines[line - 1])
+        column = line_len if column is None else column
+        if not (0 <= column <= line_len):
+            raise ValueError('`column` parameter is not in a valid range.')
+        self._pos = line, column
+
+        cache.clear_time_caches()
+        debug.reset_time()
+        self._grammar = load_grammar('grammar%s.%s' % sys.version_info[:2])
+        self._user_context = UserContext(self.source, self._pos)
+        self._parser = UserContextParser(self._grammar, self.source, path,
+                                         self._pos, self._user_context,
+                                         self._parsed_callback)
+        self._evaluator = Evaluator(self._grammar)
+        debug.speed('init')
+
+    def _parsed_callback(self, parser):
+        module = self._evaluator.wrap(parser.module)
+        imports.add_module(self._evaluator, unicode(module.name), module)
+
+    @property
+    def source_path(self):
+        """
+        .. deprecated:: 0.7.0
+           Use :attr:`.path` instead.
+        .. todo:: Remove!
+        """
+        warnings.warn("Use path instead of source_path.", DeprecationWarning)
+        return self.path
+
+    def __repr__(self):
+        return '<%s: %s>' % (self.__class__.__name__, repr(self._orig_path))
+
+    def completions(self):
+        """
+        Return :class:`classes.Completion` objects. Those objects contain
+        information about the completions, more than just names.
+
+        :return: Completion objects, sorted by name and __ comes last.
+        :rtype: list of :class:`classes.Completion`
+        """
+        def get_completions(user_stmt, bs):
+            # TODO this closure is ugly. it also doesn't work with
+            # simple_complete (used for Interpreter), somehow redo.
+            module = self._evaluator.wrap(self._parser.module())
+            names, level, only_modules, unfinished_dotted = \
+                helpers.check_error_statements(module, self._pos)
+            completion_names = []
+            if names is not None:
+                imp_names = tuple(str(n) for n in names if n.end_pos < self._pos)
+                i = imports.Importer(self._evaluator, imp_names, module, level)
+                completion_names = i.completion_names(self._evaluator, only_modules)
+
+            # TODO this paragraph is necessary, but not sure it works.
+            context = self._user_context.get_context()
+            if not next(context).startswith('.'):  # skip the path
+                if next(context) == 'from':
+                    # completion is just "import" if before stands from ..
+                    if unfinished_dotted:
+                        return completion_names
+                    else:
+                        return keywords.keyword_names('import')
+
+            if isinstance(user_stmt, tree.Import):
+                module = self._parser.module()
+                completion_names += imports.completion_names(self._evaluator,
+                                                             user_stmt, self._pos)
+                return completion_names
+
+            if names is None and not isinstance(user_stmt, tree.Import):
+                if not path and not dot:
+                    # add keywords
+                    completion_names += keywords.keyword_names(all=True)
+                    # TODO delete? We should search for valid parser
+                    # transformations.
+                completion_names += self._simple_complete(path, dot, like)
+            return completion_names
+
+        debug.speed('completions start')
+        path = self._user_context.get_path_until_cursor()
+        # Dots following an int are not the start of a completion but a float
+        # literal.
+        if re.search(r'^\d\.$', path):
+            return []
+        path, dot, like = helpers.completion_parts(path)
+
+        user_stmt = self._parser.user_stmt_with_whitespace()
+
+        b = compiled.builtin
+        completion_names = get_completions(user_stmt, b)
+
+        if not dot:
+            # add named params
+            for call_sig in self.call_signatures():
+                # Allow protected access, because it's a public API.
+                module = call_sig._name.get_parent_until()
+                # Compiled modules typically don't allow keyword arguments.
+                if not isinstance(module, compiled.CompiledObject):
+                    for p in call_sig.params:
+                        # Allow access on _definition here, because it's a
+                        # public API and we don't want to make the internal
+                        # Name object public.
+                        if p._definition.stars == 0:  # no *args/**kwargs
+                            completion_names.append(p._name)
+
+        needs_dot = not dot and path
+
+        comps = []
+        comp_dct = {}
+        for c in set(completion_names):
+            n = str(c)
+            if settings.case_insensitive_completion \
+                    and n.lower().startswith(like.lower()) \
+                    or n.startswith(like):
+                if isinstance(c.parent, (tree.Function, tree.Class)):
+                    # TODO I think this is a hack. It should be an
+                    #   er.Function/er.Class before that.
+                    c = self._evaluator.wrap(c.parent).name
+                new = classes.Completion(self._evaluator, c, needs_dot, len(like))
+                k = (new.name, new.complete)  # key
+                if k in comp_dct and settings.no_completion_duplicates:
+                    comp_dct[k]._same_name_completions.append(new)
+                else:
+                    comp_dct[k] = new
+                    comps.append(new)
+
+        debug.speed('completions end')
+
+        return sorted(comps, key=lambda x: (x.name.startswith('__'),
+                                            x.name.startswith('_'),
+                                            x.name.lower()))
+
+    def _simple_complete(self, path, dot, like):
+        if not path and not dot:
+            scope = self._parser.user_scope()
+            if not scope.is_scope():  # Might be a flow (if/while/etc).
+                scope = scope.get_parent_scope()
+            names_dicts = global_names_dict_generator(
+                self._evaluator,
+                self._evaluator.wrap(scope),
+                self._pos
+            )
+            completion_names = []
+            for names_dict, pos in names_dicts:
+                names = list(chain.from_iterable(names_dict.values()))
+                if not names:
+                    continue
+                completion_names += filter_definition_names(names, self._parser.user_stmt(), pos)
+        elif self._get_under_cursor_stmt(path) is None:
+            return []
+        else:
+            scopes = list(self._prepare_goto(path, True))
+            completion_names = []
+            debug.dbg('possible completion scopes: %s', scopes)
+            for s in scopes:
+                names = []
+                for names_dict in s.names_dicts(search_global=False):
+                    names += chain.from_iterable(names_dict.values())
+
+                completion_names += filter_definition_names(names, self._parser.user_stmt())
+        return completion_names
+
+    def _prepare_goto(self, goto_path, is_completion=False):
+        """
+        Base for completions/goto. Basically it returns the resolved scopes
+        under cursor.
+        """
+        debug.dbg('start: %s in %s', goto_path, self._parser.user_scope())
+
+        user_stmt = self._parser.user_stmt_with_whitespace()
+        if not user_stmt and len(goto_path.split('\n')) > 1:
+            # If the user_stmt is not defined and the goto_path is multi line,
+            # something's strange. Most probably the backwards tokenizer
+            # matched to much.
+            return []
+
+        if isinstance(user_stmt, tree.Import):
+            i, _ = helpers.get_on_import_stmt(self._evaluator, self._user_context,
+                                              user_stmt, is_completion)
+            if i is None:
+                return []
+            scopes = [i]
+        else:
+            # just parse one statement, take it and evaluate it
+            eval_stmt = self._get_under_cursor_stmt(goto_path)
+            if eval_stmt is None:
+                return []
+
+            module = self._evaluator.wrap(self._parser.module())
+            names, level, _, _ = helpers.check_error_statements(module, self._pos)
+            if names:
+                names = [str(n) for n in names]
+                i = imports.Importer(self._evaluator, names, module, level)
+                return i.follow()
+
+            scopes = self._evaluator.eval_element(eval_stmt)
+
+        return scopes
+
+    @memoize_default()
+    def _get_under_cursor_stmt(self, cursor_txt, start_pos=None):
+        tokenizer = source_tokens(cursor_txt)
+        r = Parser(self._grammar, cursor_txt, tokenizer=tokenizer)
+        try:
+            # Take the last statement available that is not an endmarker.
+            # And because it's a simple_stmt, we need to get the first child.
+            stmt = r.module.children[-2].children[0]
+        except (AttributeError, IndexError):
+            return None
+
+        user_stmt = self._parser.user_stmt()
+        if user_stmt is None:
+            # Set the start_pos to a pseudo position, that doesn't exist but
+            # works perfectly well (for both completions in docstrings and
+            # statements).
+            pos = start_pos or self._pos
+        else:
+            pos = user_stmt.start_pos
+
+        stmt.move(pos[0] - 1, pos[1])  # Moving the offset.
+        stmt.parent = self._parser.user_scope()
+        return stmt
+
+    def goto_definitions(self):
+        """
+        Return the definitions of a the path under the cursor.  goto function!
+        This follows complicated paths and returns the end, not the first
+        definition. The big difference between :meth:`goto_assignments` and
+        :meth:`goto_definitions` is that :meth:`goto_assignments` doesn't
+        follow imports and statements. Multiple objects may be returned,
+        because Python itself is a dynamic language, which means depending on
+        an option you can have two different versions of a function.
+
+        :rtype: list of :class:`classes.Definition`
+        """
+        def resolve_import_paths(scopes):
+            for s in scopes.copy():
+                if isinstance(s, imports.ImportWrapper):
+                    scopes.remove(s)
+                    scopes.update(resolve_import_paths(set(s.follow())))
+            return scopes
+
+        goto_path = self._user_context.get_path_under_cursor()
+        context = self._user_context.get_context()
+        definitions = set()
+        if next(context) in ('class', 'def'):
+            definitions = set([self._evaluator.wrap(self._parser.user_scope())])
+        else:
+            # Fetch definition of callee, if there's no path otherwise.
+            if not goto_path:
+                definitions = set(signature._definition
+                                  for signature in self.call_signatures())
+
+        if re.match('\w[\w\d_]*$', goto_path) and not definitions:
+            user_stmt = self._parser.user_stmt()
+            if user_stmt is not None and user_stmt.type == 'expr_stmt':
+                for name in user_stmt.get_defined_names():
+                    if name.start_pos <= self._pos <= name.end_pos:
+                        # TODO scaning for a name and then using it should be
+                        # the default.
+                        definitions = set(self._evaluator.goto_definition(name))
+
+        if not definitions and goto_path:
+            definitions = set(self._prepare_goto(goto_path))
+
+        definitions = resolve_import_paths(definitions)
+        names = [s.name for s in definitions]
+        defs = [classes.Definition(self._evaluator, name) for name in names]
+        return helpers.sorted_definitions(set(defs))
+
+    def goto_assignments(self):
+        """
+        Return the first definition found. Imports and statements aren't
+        followed. Multiple objects may be returned, because Python itself is a
+        dynamic language, which means depending on an option you can have two
+        different versions of a function.
+
+        :rtype: list of :class:`classes.Definition`
+        """
+        results = self._goto()
+        d = [classes.Definition(self._evaluator, d) for d in set(results)]
+        return helpers.sorted_definitions(d)
+
+    def _goto(self, add_import_name=False):
+        """
+        Used for goto_assignments and usages.
+
+        :param add_import_name: Add the the name (if import) to the result.
+        """
+        def follow_inexistent_imports(defs):
+            """ Imports can be generated, e.g. following
+            `multiprocessing.dummy` generates an import dummy in the
+            multiprocessing module. The Import doesn't exist -> follow.
+            """
+            definitions = set(defs)
+            for d in defs:
+                if isinstance(d.parent, tree.Import) \
+                        and d.start_pos == (0, 0):
+                    i = imports.ImportWrapper(self._evaluator, d.parent).follow(is_goto=True)
+                    definitions.remove(d)
+                    definitions |= follow_inexistent_imports(i)
+            return definitions
+
+        goto_path = self._user_context.get_path_under_cursor()
+        context = self._user_context.get_context()
+        user_stmt = self._parser.user_stmt()
+        user_scope = self._parser.user_scope()
+
+        stmt = self._get_under_cursor_stmt(goto_path)
+        if stmt is None:
+            return []
+
+        if user_scope is None:
+            last_name = None
+        else:
+            # Try to use the parser if possible.
+            last_name = user_scope.name_for_position(self._pos)
+
+        if last_name is None:
+            last_name = stmt
+            while not isinstance(last_name, tree.Name):
+                try:
+                    last_name = last_name.children[-1]
+                except AttributeError:
+                    # Doesn't have a name in it.
+                    return []
+
+        if next(context) in ('class', 'def'):
+            # The cursor is on a class/function name.
+            user_scope = self._parser.user_scope()
+            definitions = set([user_scope.name])
+        elif isinstance(user_stmt, tree.Import):
+            s, name = helpers.get_on_import_stmt(self._evaluator,
+                                                 self._user_context, user_stmt)
+
+            definitions = self._evaluator.goto(name)
+        else:
+            # The Evaluator.goto function checks for definitions, but since we
+            # use a reverse tokenizer, we have new name_part objects, so we
+            # have to check the user_stmt here for positions.
+            if isinstance(user_stmt, tree.ExprStmt) \
+                    and isinstance(last_name.parent, tree.ExprStmt):
+                for name in user_stmt.get_defined_names():
+                    if name.start_pos <= self._pos <= name.end_pos:
+                        return [name]
+
+            defs = self._evaluator.goto(last_name)
+            definitions = follow_inexistent_imports(defs)
+        return definitions
+
+    def usages(self, additional_module_paths=()):
+        """
+        Return :class:`classes.Definition` objects, which contain all
+        names that point to the definition of the name under the cursor. This
+        is very useful for refactoring (renaming), or to show all usages of a
+        variable.
+
+        .. todo:: Implement additional_module_paths
+
+        :rtype: list of :class:`classes.Definition`
+        """
+        temp, settings.dynamic_flow_information = \
+            settings.dynamic_flow_information, False
+        try:
+            user_stmt = self._parser.user_stmt()
+            definitions = self._goto(add_import_name=True)
+            if not definitions and isinstance(user_stmt, tree.Import):
+                # For not defined imports (goto doesn't find something, we take
+                # the name as a definition. This is enough, because every name
+                # points to it.
+                name = user_stmt.name_for_position(self._pos)
+                if name is None:
+                    # Must be syntax
+                    return []
+                definitions = [name]
+
+            if not definitions:
+                # Without a definition for a name we cannot find references.
+                return []
+
+            if not isinstance(user_stmt, tree.Import):
+                # import case is looked at with add_import_name option
+                definitions = usages.usages_add_import_modules(self._evaluator,
+                                                               definitions)
+
+            module = set([d.get_parent_until() for d in definitions])
+            module.add(self._parser.module())
+            names = usages.usages(self._evaluator, definitions, module)
+
+            for d in set(definitions):
+                names.append(classes.Definition(self._evaluator, d))
+        finally:
+            settings.dynamic_flow_information = temp
+
+        return helpers.sorted_definitions(set(names))
+
+    def call_signatures(self):
+        """
+        Return the function object of the call you're currently in.
+
+        E.g. if the cursor is here::
+
+            abs(# <-- cursor is here
+
+        This would return the ``abs`` function. On the other hand::
+
+            abs()# <-- cursor is here
+
+        This would return ``None``.
+
+        :rtype: list of :class:`classes.CallSignature`
+        """
+        call_txt, call_index, key_name, start_pos = self._user_context.call_signature()
+        if call_txt is None:
+            return []
+
+        stmt = self._get_under_cursor_stmt(call_txt, start_pos)
+        if stmt is None:
+            return []
+
+        with common.scale_speed_settings(settings.scale_call_signatures):
+            origins = cache.cache_call_signatures(self._evaluator, stmt,
+                                                  self.source, self._pos)
+        debug.speed('func_call followed')
+
+        return [classes.CallSignature(self._evaluator, o.name, stmt, call_index, key_name)
+                for o in origins if hasattr(o, 'py__call__')]
+
+    def _analysis(self):
+        def check_types(types):
+            for typ in types:
+                try:
+                    f = typ.iter_content
+                except AttributeError:
+                    pass
+                else:
+                    check_types(f())
+
+        #statements = set(chain(*self._parser.module().used_names.values()))
+        nodes, imp_names, decorated_funcs = \
+            analysis.get_module_statements(self._parser.module())
+        # Sort the statements so that the results are reproducible.
+        for n in imp_names:
+            imports.ImportWrapper(self._evaluator, n).follow()
+        for node in sorted(nodes, key=lambda obj: obj.start_pos):
+            check_types(self._evaluator.eval_element(node))
+
+        for dec_func in decorated_funcs:
+            er.Function(self._evaluator, dec_func).get_decorated_func()
+
+        ana = [a for a in self._evaluator.analysis if self.path == a.path]
+        return sorted(set(ana), key=lambda x: x.line)
+
+
+class Interpreter(Script):
+    """
+    Jedi API for Python REPLs.
+
+    In addition to completion of simple attribute access, Jedi
+    supports code completion based on static code analysis.
+    Jedi can complete attributes of object which is not initialized
+    yet.
+
+    >>> from os.path import join
+    >>> namespace = locals()
+    >>> script = Interpreter('join().up', [namespace])
+    >>> print(script.completions()[0].name)
+    upper
+    """
+
+    def __init__(self, source, namespaces, **kwds):
+        """
+        Parse `source` and mixin interpreted Python objects from `namespaces`.
+
+        :type source: str
+        :arg  source: Code to parse.
+        :type namespaces: list of dict
+        :arg  namespaces: a list of namespace dictionaries such as the one
+                          returned by :func:`locals`.
+
+        Other optional arguments are same as the ones for :class:`Script`.
+        If `line` and `column` are None, they are assumed be at the end of
+        `source`.
+        """
+        if type(namespaces) is not list or len(namespaces) == 0 or \
+           any([type(x) is not dict for x in namespaces]):
+            raise TypeError("namespaces must be a non-empty list of dict")
+
+        super(Interpreter, self).__init__(source, **kwds)
+        self.namespaces = namespaces
+
+        # Don't use the fast parser, because it does crazy stuff that we don't
+        # need in our very simple and small code here (that is always
+        # changing).
+        self._parser = UserContextParser(self._grammar, self.source,
+                                         self._orig_path, self._pos,
+                                         self._user_context, self._parsed_callback,
+                                         use_fast_parser=False)
+        interpreter.add_namespaces_to_parser(self._evaluator, namespaces,
+                                             self._parser.module())
+
+    def _simple_complete(self, path, dot, like):
+        user_stmt = self._parser.user_stmt_with_whitespace()
+        is_simple_path = not path or re.search('^[\w][\w\d.]*$', path)
+        if isinstance(user_stmt, tree.Import) or not is_simple_path:
+            return super(Interpreter, self)._simple_complete(path, dot, like)
+        else:
+            class NamespaceModule(object):
+                def __getattr__(_, name):
+                    for n in self.namespaces:
+                        try:
+                            return n[name]
+                        except KeyError:
+                            pass
+                    raise AttributeError()
+
+                def __dir__(_):
+                    gen = (n.keys() for n in self.namespaces)
+                    return list(set(chain.from_iterable(gen)))
+
+            paths = path.split('.') if path else []
+
+            namespaces = (NamespaceModule(), builtins)
+            for p in paths:
+                old, namespaces = namespaces, []
+                for n in old:
+                    try:
+                        namespaces.append(getattr(n, p))
+                    except Exception:
+                        pass
+
+            completion_names = []
+            for namespace in namespaces:
+                for name in dir(namespace):
+                    if name.lower().startswith(like.lower()):
+                        scope = self._parser.module()
+                        n = FakeName(name, scope)
+                        completion_names.append(n)
+            return completion_names
+
+
+def defined_names(source, path=None, encoding='utf-8'):
+    """
+    Get all definitions in `source` sorted by its position.
+
+    This functions can be used for listing functions, classes and
+    data defined in a file.  This can be useful if you want to list
+    them in "sidebar".  Each element in the returned list also has
+    `defined_names` method which can be used to get sub-definitions
+    (e.g., methods in class).
+
+    :rtype: list of classes.Definition
+
+    .. deprecated:: 0.9.0
+       Use :func:`names` instead.
+    .. todo:: Remove!
+    """
+    warnings.warn("Use call_signatures instead.", DeprecationWarning)
+    return names(source, path, encoding)
+
+
+def names(source=None, path=None, encoding='utf-8', all_scopes=False,
+          definitions=True, references=False):
+    """
+    Returns a list of `Definition` objects, containing name parts.
+    This means you can call ``Definition.goto_assignments()`` and get the
+    reference of a name.
+    The parameters are the same as in :py:class:`Script`, except or the
+    following ones:
+
+    :param all_scopes: If True lists the names of all scopes instead of only
+        the module namespace.
+    :param definitions: If True lists the names that have been defined by a
+        class, function or a statement (``a = b`` returns ``a``).
+    :param references: If True lists all the names that are not listed by
+        ``definitions=True``. E.g. ``a = b`` returns ``b``.
+    """
+    def def_ref_filter(_def):
+        is_def = _def.is_definition()
+        return definitions and is_def or references and not is_def
+
+    # Set line/column to a random position, because they don't matter.
+    script = Script(source, line=1, column=0, path=path, encoding=encoding)
+    defs = [classes.Definition(script._evaluator, name_part)
+            for name_part in get_module_names(script._parser.module(), all_scopes)]
+    return sorted(filter(def_ref_filter, defs), key=lambda x: (x.line, x.column))
+
+
+def preload_module(*modules):
+    """
+    Preloading modules tells Jedi to load a module now, instead of lazy parsing
+    of modules. Usful for IDEs, to control which modules to load on startup.
+
+    :param modules: different module names, list of string.
+    """
+    for m in modules:
+        s = "import %s as x; x." % m
+        Script(s, 1, len(s), None).completions()
+
+
+def set_debug_function(func_cb=debug.print_to_stdout, warnings=True,
+                       notices=True, speed=True):
+    """
+    Define a callback debug function to get all the debug messages.
+
+    :param func_cb: The callback function for debug messages, with n params.
+    """
+    debug.debug_function = func_cb
+    debug.enable_warning = warnings
+    debug.enable_notice = notices
+    debug.enable_speed = speed

jedi/api/classes.py

@@ -0,0 +1,735 @@
+"""
+The :mod:`jedi.api.classes` module contains the return classes of the API.
+These classes are the much bigger part of the whole API, because they contain
+the interesting information about completion and goto operations.
+"""
+import warnings
+from itertools import chain
+import re
+
+from jedi._compatibility import unicode, use_metaclass
+from jedi import settings
+from jedi import common
+from jedi.parser import tree
+from jedi.evaluate.cache import memoize_default, CachedMetaClass
+from jedi.evaluate import representation as er
+from jedi.evaluate import iterable
+from jedi.evaluate import imports
+from jedi.evaluate import compiled
+from jedi.api import keywords
+from jedi.evaluate.finder import filter_definition_names
+
+
+def defined_names(evaluator, scope):
+    """
+    List sub-definitions (e.g., methods in class).
+
+    :type scope: Scope
+    :rtype: list of Definition
+    """
+    dct = scope.names_dict
+    names = list(chain.from_iterable(dct.values()))
+    names = filter_definition_names(names, scope)
+    return [Definition(evaluator, d) for d in sorted(names, key=lambda s: s.start_pos)]
+
+
+class BaseDefinition(object):
+    _mapping = {
+        'posixpath': 'os.path',
+        'riscospath': 'os.path',
+        'ntpath': 'os.path',
+        'os2emxpath': 'os.path',
+        'macpath': 'os.path',
+        'genericpath': 'os.path',
+        'posix': 'os',
+        '_io': 'io',
+        '_functools': 'functools',
+        '_sqlite3': 'sqlite3',
+        '__builtin__': '',
+        'builtins': '',
+    }
+
+    _tuple_mapping = dict((tuple(k.split('.')), v) for (k, v) in {
+        'argparse._ActionsContainer': 'argparse.ArgumentParser',
+        '_sre.SRE_Match': 're.MatchObject',
+        '_sre.SRE_Pattern': 're.RegexObject',
+    }.items())
+
+    def __init__(self, evaluator, name):
+        self._evaluator = evaluator
+        self._name = name
+        """
+        An instance of :class:`jedi.parser.reprsentation.Name` subclass.
+        """
+        self._definition = evaluator.wrap(self._name.get_definition())
+        self.is_keyword = isinstance(self._definition, keywords.Keyword)
+
+        # generate a path to the definition
+        self._module = name.get_parent_until()
+        if self.in_builtin_module():
+            self.module_path = None
+        else:
+            self.module_path = self._module.path
+            """Shows the file path of a module. e.g. ``/usr/lib/python2.7/os.py``"""
+
+    @property
+    def name(self):
+        """
+        Name of variable/function/class/module.
+
+        For example, for ``x = None`` it returns ``'x'``.
+
+        :rtype: str or None
+        """
+        return unicode(self._name)
+
+    @property
+    def start_pos(self):
+        """
+        .. deprecated:: 0.7.0
+           Use :attr:`.line` and :attr:`.column` instead.
+        .. todo:: Remove!
+        """
+        warnings.warn("Use line/column instead.", DeprecationWarning)
+        return self._name.start_pos
+
+    @property
+    def type(self):
+        """
+        The type of the definition.
+
+        Here is an example of the value of this attribute.  Let's consider
+        the following source.  As what is in ``variable`` is unambiguous
+        to Jedi, :meth:`jedi.Script.goto_definitions` should return a list of
+        definition for ``sys``, ``f``, ``C`` and ``x``.
+
+        >>> from jedi import Script
+        >>> source = '''
+        ... import keyword
+        ...
+        ... class C:
+        ...     pass
+        ...
+        ... class D:
+        ...     pass
+        ...
+        ... x = D()
+        ...
+        ... def f():
+        ...     pass
+        ...
+        ... for variable in [keyword, f, C, x]:
+        ...     variable'''
+
+        >>> script = Script(source)
+        >>> defs = script.goto_definitions()
+
+        Before showing what is in ``defs``, let's sort it by :attr:`line`
+        so that it is easy to relate the result to the source code.
+
+        >>> defs = sorted(defs, key=lambda d: d.line)
+        >>> defs                           # doctest: +NORMALIZE_WHITESPACE
+        [<Definition module keyword>, <Definition class C>,
+         <Definition class D>, <Definition def f>]
+
+        Finally, here is what you can get from :attr:`type`:
+
+        >>> defs[0].type
+        'module'
+        >>> defs[1].type
+        'class'
+        >>> defs[2].type
+        'instance'
+        >>> defs[3].type
+        'function'
+
+        """
+        stripped = self._definition
+        if isinstance(stripped, er.InstanceElement):
+            stripped = stripped.var
+
+        if isinstance(stripped, compiled.CompiledObject):
+            return stripped.api_type()
+        elif isinstance(stripped, iterable.Array):
+            return 'instance'
+        elif isinstance(stripped, tree.Import):
+            return 'import'
+
+        string = type(stripped).__name__.lower().replace('wrapper', '')
+        if string == 'exprstmt':
+            return 'statement'
+        else:
+            return string
+
+    def _path(self):
+        """The path to a module/class/function definition."""
+        path = []
+        par = self._definition
+        while par is not None:
+            if isinstance(par, tree.Import):
+                path += imports.ImportWrapper(self._evaluator, self._name).import_path
+                break
+            try:
+                name = par.name
+            except AttributeError:
+                pass
+            else:
+                if isinstance(par, er.ModuleWrapper):
+                    # TODO just make the path dotted from the beginning, we
+                    # shouldn't really split here.
+                    path[0:0] = par.py__name__().split('.')
+                    break
+                else:
+                    path.insert(0, unicode(name))
+            par = par.parent
+        return path
+
+    @property
+    def module_name(self):
+        """
+        The module name.
+
+        >>> from jedi import Script
+        >>> source = 'import json'
+        >>> script = Script(source, path='example.py')
+        >>> d = script.goto_definitions()[0]
+        >>> print(d.module_name)                       # doctest: +ELLIPSIS
+        json
+        """
+        return str(self._module.name)
+
+    def in_builtin_module(self):
+        """Whether this is a builtin module."""
+        return isinstance(self._module, compiled.CompiledObject)
+
+    @property
+    def line(self):
+        """The line where the definition occurs (starting with 1)."""
+        if self.in_builtin_module():
+            return None
+        return self._name.start_pos[0]
+
+    @property
+    def column(self):
+        """The column where the definition occurs (starting with 0)."""
+        if self.in_builtin_module():
+            return None
+        return self._name.start_pos[1]
+
+    def docstring(self, raw=False):
+        r"""
+        Return a document string for this completion object.
+
+        Example:
+
+        >>> from jedi import Script
+        >>> source = '''\
+        ... def f(a, b=1):
+        ...     "Document for function f."
+        ... '''
+        >>> script = Script(source, 1, len('def f'), 'example.py')
+        >>> doc = script.goto_definitions()[0].docstring()
+        >>> print(doc)
+        f(a, b=1)
+        <BLANKLINE>
+        Document for function f.
+
+        Notice that useful extra information is added to the actual
+        docstring.  For function, it is call signature.  If you need
+        actual docstring, use ``raw=True`` instead.
+
+        >>> print(script.goto_definitions()[0].docstring(raw=True))
+        Document for function f.
+
+        """
+        if raw:
+            return _Help(self._definition).raw()
+        else:
+            return _Help(self._definition).full()
+
+    @property
+    def doc(self):
+        """
+        .. deprecated:: 0.8.0
+           Use :meth:`.docstring` instead.
+        .. todo:: Remove!
+        """
+        warnings.warn("Use docstring() instead.", DeprecationWarning)
+        return self.docstring()
+
+    @property
+    def raw_doc(self):
+        """
+        .. deprecated:: 0.8.0
+           Use :meth:`.docstring` instead.
+        .. todo:: Remove!
+        """
+        warnings.warn("Use docstring() instead.", DeprecationWarning)
+        return self.docstring(raw=True)
+
+    @property
+    def description(self):
+        """A textual description of the object."""
+        return unicode(self._name)
+
+    @property
+    def full_name(self):
+        """
+        Dot-separated path of this object.
+
+        It is in the form of ``<module>[.<submodule>[...]][.<object>]``.
+        It is useful when you want to look up Python manual of the
+        object at hand.
+
+        Example:
+
+        >>> from jedi import Script
+        >>> source = '''
+        ... import os
+        ... os.path.join'''
+        >>> script = Script(source, 3, len('os.path.join'), 'example.py')
+        >>> print(script.goto_definitions()[0].full_name)
+        os.path.join
+
+        Notice that it correctly returns ``'os.path.join'`` instead of
+        (for example) ``'posixpath.join'``.
+
+        """
+        path = [unicode(p) for p in self._path()]
+        # TODO add further checks, the mapping should only occur on stdlib.
+        if not path:
+            return None  # for keywords the path is empty
+
+        with common.ignored(KeyError):
+            path[0] = self._mapping[path[0]]
+        for key, repl in self._tuple_mapping.items():
+            if tuple(path[:len(key)]) == key:
+                path = [repl] + path[len(key):]
+
+        return '.'.join(path if path[0] else path[1:])
+
+    def goto_assignments(self):
+        defs = self._evaluator.goto(self._name)
+        return [Definition(self._evaluator, d) for d in defs]
+
+    @memoize_default()
+    def _follow_statements_imports(self):
+        """
+        Follow both statements and imports, as far as possible.
+        """
+        if self._definition.isinstance(tree.ExprStmt):
+            return self._evaluator.eval_statement(self._definition)
+        elif self._definition.isinstance(tree.Import):
+            return imports.ImportWrapper(self._evaluator, self._name).follow()
+        else:
+            return [self._definition]
+
+    @property
+    @memoize_default()
+    def params(self):
+        """
+        Raises an ``AttributeError``if the definition is not callable.
+        Otherwise returns a list of `Definition` that represents the params.
+        """
+        followed = self._follow_statements_imports()
+        if not followed or not hasattr(followed[0], 'py__call__'):
+            raise AttributeError()
+        followed = followed[0]  # only check the first one.
+
+        if followed.type == 'funcdef':
+            if isinstance(followed, er.InstanceElement):
+                params = followed.params[1:]
+            else:
+                params = followed.params
+        elif followed.isinstance(er.compiled.CompiledObject):
+            params = followed.params
+        else:
+            try:
+                sub = followed.get_subscope_by_name('__init__')
+                params = sub.params[1:]  # ignore self
+            except KeyError:
+                return []
+        return [_Param(self._evaluator, p.name) for p in params]
+
+    def parent(self):
+        scope = self._definition.get_parent_scope()
+        scope = self._evaluator.wrap(scope)
+        return Definition(self._evaluator, scope.name)
+
+    def __repr__(self):
+        return "<%s %s>" % (type(self).__name__, self.description)
+
+
+class Completion(BaseDefinition):
+    """
+    `Completion` objects are returned from :meth:`api.Script.completions`. They
+    provide additional information about a completion.
+    """
+    def __init__(self, evaluator, name, needs_dot, like_name_length):
+        super(Completion, self).__init__(evaluator, name)
+
+        self._needs_dot = needs_dot
+        self._like_name_length = like_name_length
+
+        # Completion objects with the same Completion name (which means
+        # duplicate items in the completion)
+        self._same_name_completions = []
+
+    def _complete(self, like_name):
+        dot = '.' if self._needs_dot else ''
+        append = ''
+        if settings.add_bracket_after_function \
+                and self.type == 'Function':
+            append = '('
+
+        if settings.add_dot_after_module:
+            if isinstance(self._definition, tree.Module):
+                append += '.'
+        if isinstance(self._definition, tree.Param):
+            append += '='
+
+        name = str(self._name)
+        if like_name:
+            name = name[self._like_name_length:]
+        return dot + name + append
+
+    @property
+    def complete(self):
+        """
+        Return the rest of the word, e.g. completing ``isinstance``::
+
+            isinstan# <-- Cursor is here
+
+        would return the string 'ce'. It also adds additional stuff, depending
+        on your `settings.py`.
+        """
+        return self._complete(True)
+
+    @property
+    def name_with_symbols(self):
+        """
+        Similar to :attr:`name`, but like :attr:`name`
+        returns also the symbols, for example::
+
+            list()
+
+        would return ``.append`` and others (which means it adds a dot).
+        """
+        return self._complete(False)
+
+    @property
+    def description(self):
+        """Provide a description of the completion object."""
+        if self._definition is None:
+            return ''
+        t = self.type
+        if t == 'statement' or t == 'import':
+            desc = self._definition.get_code()
+        else:
+            desc = '.'.join(unicode(p) for p in self._path())
+
+        line = '' if self.in_builtin_module else '@%s' % self.line
+        return '%s: %s%s' % (t, desc, line)
+
+    def __repr__(self):
+        return '<%s: %s>' % (type(self).__name__, self._name)
+
+    def docstring(self, raw=False, fast=True):
+        """
+        :param fast: Don't follow imports that are only one level deep like
+            ``import foo``, but follow ``from foo import bar``. This makes
+            sense for speed reasons. Completing `import a` is slow if you use
+            the ``foo.docstring(fast=False)`` on every object, because it
+            parses all libraries starting with ``a``.
+        """
+        definition = self._definition
+        if isinstance(definition, tree.Import):
+            i = imports.ImportWrapper(self._evaluator, self._name)
+            if len(i.import_path) > 1 or not fast:
+                followed = self._follow_statements_imports()
+                if followed:
+                    # TODO: Use all of the followed objects as input to Documentation.
+                    definition = followed[0]
+
+        if raw:
+            return _Help(definition).raw()
+        else:
+            return _Help(definition).full()
+
+    @property
+    def type(self):
+        """
+        The type of the completion objects. Follows imports. For a further
+        description, look at :attr:`jedi.api.classes.BaseDefinition.type`.
+        """
+        if isinstance(self._definition, tree.Import):
+            i = imports.ImportWrapper(self._evaluator, self._name)
+            if len(i.import_path) <= 1:
+                return 'module'
+
+            followed = self.follow_definition()
+            if followed:
+                # Caveat: Only follows the first one, ignore the other ones.
+                # This is ok, since people are almost never interested in
+                # variations.
+                return followed[0].type
+        return super(Completion, self).type
+
+    @memoize_default()
+    def _follow_statements_imports(self):
+        # imports completion is very complicated and needs to be treated
+        # separately in Completion.
+        definition = self._definition
+        if definition.isinstance(tree.Import):
+            i = imports.ImportWrapper(self._evaluator, self._name)
+            return i.follow()
+        return super(Completion, self)._follow_statements_imports()
+
+    @memoize_default()
+    def follow_definition(self):
+        """
+        Return the original definitions. I strongly recommend not using it for
+        your completions, because it might slow down |jedi|. If you want to
+        read only a few objects (<=20), it might be useful, especially to get
+        the original docstrings. The basic problem of this function is that it
+        follows all results. This means with 1000 completions (e.g.  numpy),
+        it's just PITA-slow.
+        """
+        defs = self._follow_statements_imports()
+        return [Definition(self._evaluator, d.name) for d in defs]
+
+
+class Definition(use_metaclass(CachedMetaClass, BaseDefinition)):
+    """
+    *Definition* objects are returned from :meth:`api.Script.goto_assignments`
+    or :meth:`api.Script.goto_definitions`.
+    """
+    def __init__(self, evaluator, definition):
+        super(Definition, self).__init__(evaluator, definition)
+
+    @property
+    def description(self):
+        """
+        A description of the :class:`.Definition` object, which is heavily used
+        in testing. e.g. for ``isinstance`` it returns ``def isinstance``.
+
+        Example:
+
+        >>> from jedi import Script
+        >>> source = '''
+        ... def f():
+        ...     pass
+        ...
+        ... class C:
+        ...     pass
+        ...
+        ... variable = f if random.choice([0,1]) else C'''
+        >>> script = Script(source, column=3)  # line is maximum by default
+        >>> defs = script.goto_definitions()
+        >>> defs = sorted(defs, key=lambda d: d.line)
+        >>> defs
+        [<Definition def f>, <Definition class C>]
+        >>> str(defs[0].description)  # strip literals in python2
+        'def f'
+        >>> str(defs[1].description)
+        'class C'
+
+        """
+        d = self._definition
+        if isinstance(d, er.InstanceElement):
+            d = d.var
+
+        if isinstance(d, compiled.CompiledObject):
+            typ = d.api_type()
+            if typ == 'instance':
+                typ = 'class'  # The description should be similar to Py objects.
+            d = typ + ' ' + d.name.get_code()
+        elif isinstance(d, iterable.Array):
+            d = 'class ' + d.type
+        elif isinstance(d, (tree.Class, er.Class, er.Instance)):
+            d = 'class ' + unicode(d.name)
+        elif isinstance(d, (er.Function, tree.Function)):
+            d = 'def ' + unicode(d.name)
+        elif isinstance(d, tree.Module):
+            # only show module name
+            d = 'module %s' % self.module_name
+        elif isinstance(d, tree.Param):
+            d = d.get_code().strip()
+            if d.endswith(','):
+                d = d[:-1]  # Remove the comma.
+        else:  # ExprStmt
+            try:
+                first_leaf = d.first_leaf()
+            except AttributeError:
+                # `d` is already a Leaf (Name).
+                first_leaf = d
+            # Remove the prefix, because that's not what we want for get_code
+            # here.
+            old, first_leaf.prefix = first_leaf.prefix, ''
+            try:
+                d = d.get_code()
+            finally:
+                first_leaf.prefix = old
+        # Delete comments:
+        d = re.sub('#[^\n]+\n', ' ', d)
+        # Delete multi spaces/newlines
+        return re.sub('\s+', ' ', d).strip()
+
+    @property
+    def desc_with_module(self):
+        """
+        In addition to the definition, also return the module.
+
+        .. warning:: Don't use this function yet, its behaviour may change. If
+            you really need it, talk to me.
+
+        .. todo:: Add full path. This function is should return a
+            `module.class.function` path.
+        """
+        position = '' if self.in_builtin_module else '@%s' % (self.line)
+        return "%s:%s%s" % (self.module_name, self.description, position)
+
+    @memoize_default()
+    def defined_names(self):
+        """
+        List sub-definitions (e.g., methods in class).
+
+        :rtype: list of Definition
+        """
+        defs = self._follow_statements_imports()
+        # For now we don't want base classes or evaluate decorators.
+        defs = [d.base if isinstance(d, (er.Class, er.Function)) else d for d in defs]
+        iterable = (defined_names(self._evaluator, d) for d in defs)
+        iterable = list(iterable)
+        return list(chain.from_iterable(iterable))
+
+    def is_definition(self):
+        """
+        Returns True, if defined as a name in a statement, function or class.
+        Returns False, if it's a reference to such a definition.
+        """
+        return self._name.is_definition()
+
+    def __eq__(self, other):
+        return self._name.start_pos == other._name.start_pos \
+            and self.module_path == other.module_path \
+            and self.name == other.name \
+            and self._evaluator == other._evaluator
+
+    def __ne__(self, other):
+        return not self.__eq__(other)
+
+    def __hash__(self):
+        return hash((self._name.start_pos, self.module_path, self.name, self._evaluator))
+
+
+class CallSignature(Definition):
+    """
+    `CallSignature` objects is the return value of `Script.function_definition`.
+    It knows what functions you are currently in. e.g. `isinstance(` would
+    return the `isinstance` function. without `(` it would return nothing.
+    """
+    def __init__(self, evaluator, executable_name, call_stmt, index, key_name):
+        super(CallSignature, self).__init__(evaluator, executable_name)
+        self._index = index
+        self._key_name = key_name
+        self._call_stmt = call_stmt
+
+    @property
+    def index(self):
+        """
+        The Param index of the current call.
+        Returns None if the index cannot be found in the curent call.
+        """
+        if self._key_name is not None:
+            for i, param in enumerate(self.params):
+                if self._key_name == param.name:
+                    return i
+            if self.params and self.params[-1]._name.get_definition().stars == 2:
+                return i
+            else:
+                return None
+
+        if self._index >= len(self.params):
+
+            for i, param in enumerate(self.params):
+                # *args case
+                if param._name.get_definition().stars == 1:
+                    return i
+            return None
+        return self._index
+
+    @property
+    def bracket_start(self):
+        """
+        The indent of the bracket that is responsible for the last function
+        call.
+        """
+        return self._call_stmt.end_pos
+
+    @property
+    def call_name(self):
+        """
+        .. deprecated:: 0.8.0
+           Use :attr:`.name` instead.
+        .. todo:: Remove!
+
+        The name (e.g. 'isinstance') as a string.
+        """
+        warnings.warn("Use name instead.", DeprecationWarning)
+        return unicode(self.name)
+
+    @property
+    def module(self):
+        """
+        .. deprecated:: 0.8.0
+           Use :attr:`.module_name` for the module name.
+        .. todo:: Remove!
+        """
+        return self._executable.get_parent_until()
+
+    def __repr__(self):
+        return '<%s: %s index %s>' % (type(self).__name__, self._name,
+                                      self.index)
+
+
+class _Param(Definition):
+    """
+    Just here for backwards compatibility.
+    """
+    def get_code(self):
+        """
+        .. deprecated:: 0.8.0
+           Use :attr:`.description` and :attr:`.name` instead.
+        .. todo:: Remove!
+
+        A function to get the whole code of the param.
+        """
+        warnings.warn("Use description instead.", DeprecationWarning)
+        return self.description
+
+
+class _Help(object):
+    """
+    Temporary implementation, will be used as `Script.help() or something in
+    the future.
+    """
+    def __init__(self, definition):
+        self._name = definition
+
+    def full(self):
+        try:
+            return self._name.doc
+        except AttributeError:
+            return self.raw()
+
+    def raw(self):
+        """
+        The raw docstring ``__doc__`` for any object.
+
+        See :attr:`doc` for example.
+        """
+        try:
+            return self._name.raw_doc
+        except AttributeError:
+            return ''

jedi/api/helpers.py

@@ -0,0 +1,78 @@
+"""
+Helpers for the API
+"""
+import re
+
+from jedi.parser import tree as pt
+from jedi.evaluate import imports
+
+
+def completion_parts(path_until_cursor):
+    """
+    Returns the parts for the completion
+    :return: tuple - (path, dot, like)
+    """
+    match = re.match(r'^(.*?)(\.|)(\w?[\w\d]*)$', path_until_cursor, flags=re.S)
+    return match.groups()
+
+
+def sorted_definitions(defs):
+    # Note: `or ''` below is required because `module_path` could be
+    return sorted(defs, key=lambda x: (x.module_path or '', x.line or 0, x.column or 0))
+
+
+def get_on_import_stmt(evaluator, user_context, user_stmt, is_like_search=False):
+    """
+    Resolve the user statement, if it is an import. Only resolve the
+    parts until the user position.
+    """
+    name = user_stmt.name_for_position(user_context.position)
+    if name is None:
+        return None, None
+
+    i = imports.ImportWrapper(evaluator, name)
+    return i, name
+
+
+def check_error_statements(module, pos):
+    for error_statement in module.error_statement_stacks:
+        if error_statement.first_type in ('import_from', 'import_name') \
+                and error_statement.first_pos < pos <= error_statement.next_start_pos:
+            return importer_from_error_statement(error_statement, pos)
+    return None, 0, False, False
+
+
+def importer_from_error_statement(error_statement, pos):
+    def check_dotted(children):
+        for name in children[::2]:
+            if name.start_pos <= pos:
+                yield name
+
+    names = []
+    level = 0
+    only_modules = True
+    unfinished_dotted = False
+    for typ, nodes in error_statement.stack:
+        if typ == 'dotted_name':
+            names += check_dotted(nodes)
+            if nodes[-1] == '.':
+                # An unfinished dotted_name
+                unfinished_dotted = True
+        elif typ == 'import_name':
+            if nodes[0].start_pos <= pos <= nodes[0].end_pos:
+                # We are on the import.
+                return None, 0, False, False
+        elif typ == 'import_from':
+            for node in nodes:
+                if node.start_pos >= pos:
+                    break
+                elif isinstance(node, pt.Node) and node.type == 'dotted_name':
+                    names += check_dotted(node.children)
+                elif node in ('.', '...'):
+                    level += len(node.value)
+                elif isinstance(node, pt.Name):
+                    names.append(node)
+                elif node == 'import':
+                    only_modules = False
+
+    return names, level, only_modules, unfinished_dotted

jedi/api/interpreter.py

@@ -0,0 +1,109 @@
+"""
+TODO Some parts of this module are still not well documented.
+"""
+import inspect
+import re
+
+from jedi._compatibility import builtins
+from jedi import debug
+from jedi.common import source_to_unicode
+from jedi.cache import underscore_memoization
+from jedi.evaluate import compiled
+from jedi.evaluate.compiled.fake import get_module
+from jedi.parser import tree as pt
+from jedi.parser import load_grammar
+from jedi.parser.fast import FastParser
+from jedi.evaluate import helpers
+from jedi.evaluate import iterable
+from jedi.evaluate import representation as er
+
+
+def add_namespaces_to_parser(evaluator, namespaces, parser_module):
+    for namespace in namespaces:
+        for key, value in namespace.items():
+            # Name lookups in an ast tree work by checking names_dict.
+            # Therefore we just add fake names to that and we're done.
+            arr = parser_module.names_dict.setdefault(key, [])
+            arr.append(LazyName(evaluator, parser_module, key, value))
+
+
+class LazyName(helpers.FakeName):
+    def __init__(self, evaluator, module, name, value):
+        super(LazyName, self).__init__(name)
+        self._module = module
+        self._evaluator = evaluator
+        self._value = value
+        self._name = name
+
+    def is_definition(self):
+        return True
+
+    @property
+    @underscore_memoization
+    def parent(self):
+        """
+        Creating fake statements for the interpreter.
+        """
+        obj = self._value
+        parser_path = []
+        if inspect.ismodule(obj):
+            module = obj
+        else:
+            names = []
+            try:
+                o = obj.__objclass__
+                names.append(obj.__name__)
+                obj = o
+            except AttributeError:
+                pass
+
+            try:
+                module_name = obj.__module__
+                names.insert(0, obj.__name__)
+            except AttributeError:
+                # Unfortunately in some cases like `int` there's no __module__
+                module = builtins
+            else:
+                # TODO this import is wrong. Yields x for x.y.z instead of z
+                module = __import__(module_name)
+            parser_path = names
+        raw_module = get_module(self._value)
+
+        found = []
+        try:
+            path = module.__file__
+        except AttributeError:
+            pass
+        else:
+            path = re.sub('c$', '', path)
+            if path.endswith('.py'):
+                # cut the `c` from `.pyc`
+                with open(path) as f:
+                    source = source_to_unicode(f.read())
+                mod = FastParser(load_grammar(), source, path[:-1]).module
+                if parser_path:
+                    assert len(parser_path) == 1
+                    found = self._evaluator.find_types(mod, parser_path[0], search_global=True)
+                else:
+                    found = [self._evaluator.wrap(mod)]
+
+                if not found:
+                    debug.warning('Possibly an interpreter lookup for Python code failed %s',
+                                  parser_path)
+
+        if not found:
+            evaluated = compiled.CompiledObject(obj)
+            if evaluated == builtins:
+                # The builtins module is special and always cached.
+                evaluated = compiled.builtin
+            found = [evaluated]
+
+        content = iterable.AlreadyEvaluated(found)
+        stmt = pt.ExprStmt([self, pt.Operator(pt.zero_position_modifier,
+                                              '=', (0, 0), ''), content])
+        stmt.parent = self._module
+        return stmt
+
+    @parent.setter
+    def parent(self, value):
+        """Needed because the super class tries to set parent."""

jedi/api/keywords.py

@@ -1,11 +1,10 @@
-from __future__ import with_statement
-
 import pydoc
 import keyword
 
-from jedi._compatibility import is_py3k
+from jedi._compatibility import is_py3
 from jedi import common
-import builtin
+from jedi.evaluate import compiled
+from jedi.evaluate.helpers import FakeName
 
 try:
     from pydoc_data import topics as pydoc_topics
@@ -13,13 +12,13 @@ except ImportError:
     # Python 2.6
     import pydoc_topics
 
-if is_py3k:
+if is_py3:
     keys = keyword.kwlist
 else:
     keys = keyword.kwlist + ['None', 'False', 'True']
 
 
-def get_keywords(string='', pos=(0, 0), all=False):
+def keywords(string='', pos=(0, 0), all=False):
     if all:
         return set([Keyword(k, pos) for k in keys])
     if string in keys:
@@ -27,15 +26,19 @@ def get_keywords(string='', pos=(0, 0), all=False):
     return set()
 
 
+def keyword_names(*args, **kwargs):
+    return [k.name for k in keywords(*args, **kwargs)]
+
+
 def get_operator(string, pos):
     return Keyword(string, pos)
 
 
 class Keyword(object):
     def __init__(self, name, pos):
-        self.name = name
+        self.name = FakeName(name, self, pos)
         self.start_pos = pos
-        self.parent = builtin.Builtin.scope
+        self.parent = compiled.builtin
 
     def get_parent_until(self):
         return self.parent

jedi/api/replstartup.py

@@ -0,0 +1,27 @@
+"""
+To use Jedi completion in Python interpreter, add the following in your shell
+setup (e.g., ``.bashrc``)::
+
+    export PYTHONSTARTUP="$(python -m jedi repl)"
+
+Then you will be able to use Jedi completer in your Python interpreter::
+
+    $ python
+    Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
+    [GCC 4.6.1] on linux2
+    Type "help", "copyright", "credits" or "license" for more information.
+    >>> import os
+    >>> os.path.join().split().in<TAB>                     # doctest: +SKIP
+    os.path.join().split().index   os.path.join().split().insert
+
+"""
+import jedi.utils
+from jedi import __version__ as __jedi_version__
+
+print('REPL completion using Jedi %s' % __jedi_version__)
+jedi.utils.setup_readline()
+
+del jedi
+
+# Note: try not to do many things here, as it will contaminate global
+# namespace of the interpreter.

jedi/api/usages.py

@@ -0,0 +1,49 @@
+from jedi._compatibility import unicode
+from jedi.api import classes
+from jedi.parser import tree
+from jedi.evaluate import imports
+
+
+def usages(evaluator, definition_names, mods):
+    """
+    :param definitions: list of Name
+    """
+    def compare_array(definitions):
+        """ `definitions` are being compared by module/start_pos, because
+        sometimes the id's of the objects change (e.g. executions).
+        """
+        result = []
+        for d in definitions:
+            module = d.get_parent_until()
+            result.append((module, d.start_pos))
+        return result
+
+    search_name = unicode(list(definition_names)[0])
+    compare_definitions = compare_array(definition_names)
+    mods |= set([d.get_parent_until() for d in definition_names])
+    definitions = []
+    for m in imports.get_modules_containing_name(evaluator, mods, search_name):
+        try:
+            check_names = m.used_names[search_name]
+        except KeyError:
+            continue
+        for name in check_names:
+
+            result = evaluator.goto(name)
+            if [c for c in compare_array(result) if c in compare_definitions]:
+                definitions.append(classes.Definition(evaluator, name))
+                # Previous definitions might be imports, so include them
+                # (because goto might return that import name).
+                compare_definitions += compare_array([name])
+    return definitions
+
+
+def usages_add_import_modules(evaluator, definitions):
+    """ Adds the modules of the imports """
+    new = set()
+    for d in definitions:
+        imp_or_stmt = d.get_definition()
+        if isinstance(imp_or_stmt, tree.Import):
+            s = imports.ImportWrapper(evaluator, d)
+            new |= set(s.follow(is_goto=True))
+    return set(definitions) | new

jedi/api_classes.py

@@ -1,594 +0,0 @@
-"""
-The :mod:`api_classes` module contains the return classes of the API. These
-classes are the much bigger part of the whole API, because they contain the
-interesting information about completion and goto operations.
-"""
-from __future__ import with_statement
-
-import warnings
-import functools
-
-from jedi._compatibility import unicode, next
-from jedi import settings
-from jedi import common
-from jedi import parsing_representation as pr
-from jedi import cache
-import keywords
-import recursion
-import dynamic
-import evaluate
-import imports
-import evaluate_representation as er
-
-
-def _clear_caches():
-    """
-    Clear all caches of this and related modules. The only cache that will not
-    be deleted is the module cache.
-    """
-    cache.clear_caches()
-    dynamic.search_param_cache.clear()
-    recursion.ExecutionRecursionDecorator.reset()
-
-    evaluate.follow_statement.reset()
-
-    imports.imports_processed = 0
-
-
-def _clear_caches_after_call(func):
-    """
-    Clear caches just before returning a value.
-    """
-    @functools.wraps(func)
-    def wrapper(*args, **kwds):
-        result = func(*args, **kwds)
-        _clear_caches()
-        return result
-    return wrapper
-
-
-class BaseDefinition(object):
-    _mapping = {
-        'posixpath': 'os.path',
-        'riscospath': 'os.path',
-        'ntpath': 'os.path',
-        'os2emxpath': 'os.path',
-        'macpath': 'os.path',
-        'genericpath': 'os.path',
-        'posix': 'os',
-        '_io': 'io',
-        '_functools': 'functools',
-        '_sqlite3': 'sqlite3',
-        '__builtin__': '',
-        'builtins': '',
-    }
-
-    _tuple_mapping = dict((tuple(k.split('.')), v) for (k, v) in {
-        'argparse._ActionsContainer': 'argparse.ArgumentParser',
-        '_sre.SRE_Match': 're.MatchObject',
-        '_sre.SRE_Pattern': 're.RegexObject',
-    }.items())
-
-    def __init__(self, definition, start_pos):
-        self.start_pos = start_pos
-        self._definition = definition
-        """
-        An instance of :class:`jedi.parsing_representation.Base` subclass.
-        """
-        self.is_keyword = isinstance(definition, keywords.Keyword)
-
-        # generate a path to the definition
-        self._module = definition.get_parent_until()
-        self.module_path = self._module.path
-
-    @property
-    def type(self):
-        """
-        The type of the definition.
-
-        Here is an example of the value of this attribute.  Let's consider
-        the following source.  As what is in ``variable`` is unambiguous
-        to Jedi, :meth:`api.Script.definition` should return a list of
-        definition for ``sys``, ``f``, ``C`` and ``x``.
-
-        >>> from jedi import Script
-        >>> source = '''
-        ... import sys
-        ...
-        ... class C:
-        ...     pass
-        ...
-        ... class D:
-        ...     pass
-        ...
-        ... x = D()
-        ...
-        ... def f():
-        ...     pass
-        ...
-        ... variable = sys or f or C or x'''
-        >>> script = Script(source, len(source.splitlines()), 3, 'example.py')
-        >>> defs = script.definition()
-
-        Before showing what is in ``defs``, let's sort it by :attr:`line`
-        so that it is easy to relate the result to the source code.
-
-        >>> defs = sorted(defs, key=lambda d: d.line)
-        >>> defs                           # doctest: +NORMALIZE_WHITESPACE
-        [<Definition module sys>, <Definition class C>,
-         <Definition class D>, <Definition def f>]
-
-        Finally, here is what you can get from :attr:`type`:
-
-        >>> defs[0].type
-        'module'
-        >>> defs[1].type
-        'class'
-        >>> defs[2].type
-        'instance'
-        >>> defs[3].type
-        'function'
-
-        """
-        # generate the type
-        stripped = self._definition
-        if isinstance(self._definition, er.InstanceElement):
-            stripped = self._definition.var
-        if isinstance(stripped, pr.Name):
-            stripped = stripped.parent
-        return type(stripped).__name__.lower()
-
-    @property
-    def path(self):
-        """The module path."""
-        path = []
-        if not isinstance(self._definition, keywords.Keyword):
-            par = self._definition
-            while par is not None:
-                with common.ignored(AttributeError):
-                    path.insert(0, par.name)
-                par = par.parent
-        return path
-
-    @property
-    def module_name(self):
-        """
-        The module name.
-
-        >>> from jedi import Script
-        >>> source = 'import datetime'
-        >>> script = Script(source, 1, len(source), 'example.py')
-        >>> d = script.definition()[0]
-        >>> print(d.module_name)                       # doctest: +ELLIPSIS
-        datetime
-        """
-        return str(self._module.name)
-
-    def in_builtin_module(self):
-        """Whether this is a builtin module."""
-        return not (self.module_path is None or
-                    self.module_path.endswith('.py'))
-
-    @property
-    def line_nr(self):
-        """
-        .. deprecated:: 0.5.0
-           Use :attr:`.line` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use line instead.", DeprecationWarning)
-        return self.line
-
-    @property
-    def line(self):
-        """The line where the definition occurs (starting with 1)."""
-        return self.start_pos[0]
-
-    @property
-    def column(self):
-        """The column where the definition occurs (starting with 0)."""
-        return self.start_pos[1]
-
-    @property
-    def doc(self):
-        r"""
-        Return a document string for this completion object.
-
-        Example:
-
-        >>> from jedi import Script
-        >>> source = '''\
-        ... def f(a, b=1):
-        ...     "Document for function f."
-        ... '''
-        >>> script = Script(source, 1, len('def f'), 'example.py')
-        >>> d = script.definition()[0]
-        >>> print(d.doc)
-        f(a, b = 1)
-        <BLANKLINE>
-        Document for function f.
-
-        Notice that useful extra information is added to the actual
-        docstring.  For function, it is call signature.  If you need
-        actual docstring, use :attr:`raw_doc` instead.
-
-        >>> print(d.raw_doc)
-        Document for function f.
-
-        """
-        try:
-            return self._definition.doc
-        except AttributeError:
-            return self.raw_doc
-
-    @property
-    def raw_doc(self):
-        """
-        The raw docstring ``__doc__`` for any object.
-
-        See :attr:`doc` for example.
-        """
-        try:
-            return unicode(self._definition.docstr)
-        except AttributeError:
-            return ''
-
-    @property
-    def description(self):
-        """
-        A textual description of the object.
-
-        Example:
-
-        >>> from jedi import Script
-        >>> source = '''
-        ... def f():
-        ...     pass
-        ...
-        ... class C:
-        ...     pass
-        ...
-        ... variable = f or C'''
-        >>> script = Script(source, len(source.splitlines()), 3, 'example.py')
-        >>> defs = script.definition()                      # doctest: +SKIP
-        >>> defs = sorted(defs, key=lambda d: d.line)       # doctest: +SKIP
-        >>> defs                                            # doctest: +SKIP
-        [<Definition def f>, <Definition class C>]
-        >>> defs[0].description                             # doctest: +SKIP
-        'def f'
-        >>> defs[1].description                             # doctest: +SKIP
-        'class C'
-
-        """
-        return unicode(self._definition)
-
-    @property
-    def full_name(self):
-        """
-        Dot-separated path of this object.
-
-        It is in the form of ``<module>[.<submodule>[...]][.<object>]``.
-        It is useful when you want to look up Python manual of the
-        object at hand.
-
-        Example:
-
-        >>> from jedi import Script
-        >>> source = '''
-        ... import os
-        ... os.path.join'''
-        >>> script = Script(source, 3, len('os.path.join'), 'example.py')
-        >>> print(script.definition()[0].full_name)
-        os.path.join
-
-        Notice that it correctly returns ``'os.path.join'`` instead of
-        (for example) ``'posixpath.join'``.
-
-        """
-        path = [unicode(p) for p in self.path]
-        # TODO add further checks, the mapping should only occur on stdlib.
-        if not path:
-            return None  # for keywords the path is empty
-
-        with common.ignored(KeyError):
-            path[0] = self._mapping[path[0]]
-        for key, repl in self._tuple_mapping.items():
-            if tuple(path[:len(key)]) == key:
-                path = [repl] + path[len(key):]
-
-        return '.'.join(path if path[0] else path[1:])
-
-    def __repr__(self):
-        return "<%s %s>" % (type(self).__name__, self.description)
-
-
-class Completion(BaseDefinition):
-    """
-    `Completion` objects are returned from :meth:`api.Script.completions`. They
-    provide additional information about a completion.
-    """
-    def __init__(self, name, needs_dot, like_name_length, base):
-        super(Completion, self).__init__(name.parent, name.start_pos)
-
-        self._name = name
-        self._needs_dot = needs_dot
-        self._like_name_length = like_name_length
-        self._base = base
-
-        # Completion objects with the same Completion name (which means
-        # duplicate items in the completion)
-        self._same_name_completions = []
-
-        self._followed_definitions = None
-
-    @property
-    def complete(self):
-        """
-        Return the rest of the word, e.g. completing ``isinstance``::
-
-            isinstan# <-- Cursor is here
-
-        would return the string 'ce'. It also adds additional stuff, depending
-        on your `settings.py`.
-        """
-        dot = '.' if self._needs_dot else ''
-        append = ''
-        if settings.add_bracket_after_function \
-                    and self.type == 'Function':
-            append = '('
-
-        if settings.add_dot_after_module:
-            if isinstance(self._base, pr.Module):
-                append += '.'
-        if isinstance(self._base, pr.Param):
-            append += '='
-        return dot + self._name.names[-1][self._like_name_length:] + append
-
-    @property
-    def name(self):
-        """
-        Similar to :meth:`Completion.complete`, but return the whole word, for
-        example::
-
-            isinstan
-
-        would return 'isinstance'.
-        """
-        return unicode(self._name.names[-1])
-
-    @property
-    def word(self):
-        """
-        .. deprecated:: 0.6.0
-           Use :attr:`.name` instead.
-        .. todo:: Remove!
-        """
-        warnings.warn("Use name instead.", DeprecationWarning)
-        return self.name
-
-
-    @property
-    def description(self):
-        """
-        Provide a description of the completion object.
-
-        .. todo:: return value is just __repr__ of some objects, improve!
-        """
-        parent = self._name.parent
-        if parent is None:
-            return ''
-        t = self.type
-        if t == 'Statement' or t == 'Import':
-            desc = self._definition.get_code(False)
-        else:
-            desc = '.'.join(unicode(p) for p in self.path)
-
-        line = '' if self.in_builtin_module else '@%s' % self.line
-        return '%s: %s%s' % (t, desc, line)
-
-    def follow_definition(self):
-        """
-        Return the original definitions. I strongly recommend not using it for
-        your completions, because it might slow down |jedi|. If you want to
-        read only a few objects (<=20), it might be useful, especially to get
-        the original docstrings. The basic problem of this function is that it
-        follows all results. This means with 1000 completions (e.g.  numpy),
-        it's just PITA-slow.
-        """
-        if self._followed_definitions is None:
-            if self._definition.isinstance(pr.Statement):
-                defs = evaluate.follow_statement(self._definition)
-            elif self._definition.isinstance(pr.Import):
-                defs = imports.strip_imports([self._definition])
-            else:
-                return [self]
-
-            self._followed_definitions = \
-                            [BaseDefinition(d, d.start_pos) for d in defs]
-            _clear_caches()
-
-        return self._followed_definitions
-
-    def __repr__(self):
-        return '<%s: %s>' % (type(self).__name__, self._name)
-
-
-class Definition(BaseDefinition):
-    """
-    *Definition* objects are returned from :meth:`api.Script.goto_assignments`
-    or :meth:`api.Script.goto_definitions`.
-    """
-    def __init__(self, definition):
-        super(Definition, self).__init__(definition, definition.start_pos)
-
-    @property
-    def name(self):
-        """
-        Name of variable/function/class/module.
-
-        For example, for ``x = None`` it returns ``'x'``.
-
-        :rtype: str or None
-        """
-        d = self._definition
-        if isinstance(d, er.InstanceElement):
-            d = d.var
-
-        if isinstance(d, pr.Name):
-            return d.names[-1] if d.names else None
-        elif isinstance(d, er.Array):
-            return unicode(d.type)
-        elif isinstance(d, (pr.Class, er.Class, er.Instance,
-                            er.Function, pr.Function)):
-            return unicode(d.name)
-        elif isinstance(d, pr.Module):
-            return self.module_name
-        elif isinstance(d, pr.Import):
-            try:
-                return d.get_defined_names()[0].names[-1]
-            except (AttributeError, IndexError):
-                return None
-        elif isinstance(d, pr.Statement):
-            try:
-                return d.assignment_details[0][1].values[0][0].name.names[-1]
-            except IndexError:
-                return None
-        return None
-
-    @property
-    def description(self):
-        """
-        A description of the :class:`.Definition` object, which is heavily used
-        in testing. e.g. for ``isinstance`` it returns ``def isinstance``.
-        """
-        d = self._definition
-        if isinstance(d, er.InstanceElement):
-            d = d.var
-        if isinstance(d, pr.Name):
-            d = d.parent
-
-        if isinstance(d, er.Array):
-            d = 'class ' + d.type
-        elif isinstance(d, (pr.Class, er.Class, er.Instance)):
-            d = 'class ' + unicode(d.name)
-        elif isinstance(d, (er.Function, pr.Function)):
-            d = 'def ' + unicode(d.name)
-        elif isinstance(d, pr.Module):
-            # only show module name
-            d = 'module %s' % self.module_name
-        elif self.is_keyword:
-            d = 'keyword %s' % d.name
-        else:
-            d = d.get_code().replace('\n', '')
-        return d
-
-    @property
-    def desc_with_module(self):
-        """
-        In addition to the definition, also return the module.
-
-        .. warning:: Don't use this function yet, its behaviour may change. If
-            you really need it, talk to me.
-
-        .. todo:: Add full path. This function is should return a
-            `module.class.function` path.
-        """
-        if self.module_path.endswith('.py') \
-                    and not isinstance(self._definition, pr.Module):
-            position = '@%s' % (self.line)
-        else:
-            # is a builtin or module
-            position = ''
-        return "%s:%s%s" % (self.module_name, self.description, position)
-
-    def defined_names(self):
-        """
-        List sub-definitions (e.g., methods in class).
-
-        :rtype: list of Definition
-        """
-        d = self._definition
-        if isinstance(d, er.InstanceElement):
-            d = d.var
-        if isinstance(d, pr.Name):
-            d = d.parent
-        return _defined_names(d)
-
-
-def _defined_names(scope):
-    """
-    List sub-definitions (e.g., methods in class).
-
-    :type scope: Scope
-    :rtype: list of Definition
-    """
-    pair = next(evaluate.get_names_of_scope(
-        scope, star_search=False, include_builtin=False), None)
-    names = pair[1] if pair else []
-    return [Definition(d) for d in sorted(names, key=lambda s: s.start_pos)]
-
-
-class Usage(BaseDefinition):
-    """TODO: document this"""
-    def __init__(self, name_part, scope):
-        super(Usage, self).__init__(scope, name_part.start_pos)
-        self.text = unicode(name_part)
-        self.end_pos = name_part.end_pos
-
-    @property
-    def description(self):
-        return "%s@%s,%s" % (self.text, self.start_pos[0], self.start_pos[1])
-
-    def __eq__(self, other):
-        return self.start_pos == other.start_pos \
-            and self.module_path == other.module_path
-
-    def __hash__(self):
-        return hash((self.start_pos, self.module_path))
-
-
-class CallDef(object):
-    """
-    `CallDef` objects is the return value of `Script.function_definition`.
-    It knows what functions you are currently in. e.g. `isinstance(` would
-    return the `isinstance` function. without `(` it would return nothing.
-    """
-    def __init__(self, executable, index, call):
-        self._executable = executable
-        self.index = index
-        self._call = call
-
-    @property
-    def params(self):
-        if self._executable.isinstance(er.Function):
-            if isinstance(self._executable, er.InstanceElement):
-                return self._executable.params[1:]
-            return self._executable.params
-        else:
-            try:
-                sub = self._executable.get_subscope_by_name('__init__')
-                return sub.params[1:]  # ignore self
-            except KeyError:
-                return []
-
-    @property
-    def bracket_start(self):
-        """ The indent of the bracket that is responsible for the last function
-        call. """
-        c = self._call
-        while c.next is not None:
-            c = c.next
-        return c.name.end_pos
-
-    @property
-    def call_name(self):
-        """ The name (e.g. 'isinstance') as a string. """
-        return unicode(self._executable.name)
-
-    @property
-    def module(self):
-        return self._executable.get_parent_until()
-
-    def __repr__(self):
-        return '<%s: %s index %s>' % (type(self).__name__, self._executable,
-                                    self.index)

jedi/builtin.py

@@ -1,449 +0,0 @@
-"""
-A big part of the Python standard libraries are unfortunately not only written
-in Python. The process works like this:
-
-- ``BuiltinModule`` imports the builtin module (e.g. ``sys``)
-- then ``BuiltinModule`` generates code with the docstrings of builtin
-  functions.
-- The :mod:`parsing` parser processes the generated code.
-
-This is possible, because many builtin functions supply docstrings, for example
-the method ``list.index`` has the following attribute ``__doc__``:
-
-    L.index(value, [start, [stop]]) -> integer -- return first index of value.
-        Raises ValueError if the value is not present.
-
-`PEP 257 <http://www.python.org/dev/peps/pep-0257/#one-line-docstrings>`_
-teaches how docstrings should look like for C functions.
-
-Additionally there's a ``Builtin``  instance in this module, to make it
-possible to access functions like ``list`` and ``int`` directly, the same way
-|jedi| access other functions.
-"""
-
-from __future__ import with_statement
-from jedi._compatibility import exec_function, is_py3k
-
-import re
-import sys
-import os
-if is_py3k:
-    import io
-import types
-import inspect
-
-from jedi import common
-from jedi import debug
-from jedi import parsing
-from jedi import modules
-import evaluate
-
-
-class BuiltinModule(modules.CachedModule):
-    """
-    This module is a parser for all builtin modules, which are programmed in
-    C/C++. It should also work on third party modules.
-    It can be instantiated with either a path or a name of the module. The path
-    is important for third party modules.
-
-    :param name: The name of the module.
-    :param path: The path of the module.
-    :param sys_path: The sys.path, which is can be customizable.
-    """
-
-    map_types = {
-        'floating point number': '0.0',
-        'string': '""',
-        'str': '""',
-        'character': '"a"',
-        'integer': '0',
-        'int': '0',
-        'dictionary': '{}',
-        'list': '[]',
-        'file object': 'file("")',
-        # TODO things like dbg: ('not working', 'tuple of integers')
-    }
-
-    if is_py3k:
-        map_types['file object'] = 'import io; return io.TextIOWrapper()'
-
-    def __init__(self, path=None, name=None, sys_path=None):
-        if sys_path is None:
-            sys_path = modules.get_sys_path()
-        if not name:
-            name = os.path.basename(path)
-            name = name.rpartition('.')[0]  # cut file type (normally .so)
-        super(BuiltinModule, self).__init__(path=path, name=name)
-
-        self.sys_path = list(sys_path)
-        self._module = None
-
-    @property
-    def module(self):
-        def load_module(name, path):
-            if path:
-                self.sys_path.insert(0, path)
-
-            temp, sys.path = sys.path, self.sys_path
-            content = {}
-            try:
-                exec_function('import %s as module' % name, content)
-                self._module = content['module']
-            except AttributeError:
-                # use sys.modules, because you cannot access some modules
-                # directly. -> #59
-                self._module = sys.modules[name]
-            sys.path = temp
-
-            if path:
-                self.sys_path.pop(0)
-
-        # module might already be defined
-        if not self._module:
-            path = self.path
-            name = self.name
-            if self.path:
-
-                dot_path = []
-                p = self.path
-                # search for the builtin with the correct path
-                while p and p not in sys.path:
-                    p, sep, mod = p.rpartition(os.path.sep)
-                    dot_path.append(mod.partition('.')[0])
-                if p:
-                    name = ".".join(reversed(dot_path))
-                    path = p
-                else:
-                    path = os.path.dirname(self.path)
-
-            load_module(name, path)
-        return self._module
-
-    def _get_source(self):
-        """ Override this abstract method """
-        return _generate_code(self.module, self._load_mixins())
-
-    def _load_mixins(self):
-        """
-        Load functions that are mixed in to the standard library.
-        E.g. builtins are written in C (binaries), but my autocompletion only
-        understands Python code. By mixing in Python code, the autocompletion
-        should work much better for builtins.
-        """
-        regex = r'^(def|class)\s+([\w\d]+)'
-
-        def process_code(code, depth=0):
-            funcs = {}
-            matches = list(re.finditer(regex, code, re.MULTILINE))
-            positions = [m.start() for m in matches]
-            for i, pos in enumerate(positions):
-                try:
-                    code_block = code[pos:positions[i + 1]]
-                except IndexError:
-                    code_block = code[pos:len(code)]
-                structure_name = matches[i].group(1)
-                name = matches[i].group(2)
-                if structure_name == 'def':
-                    funcs[name] = code_block
-                elif structure_name == 'class':
-                    if depth > 0:
-                        raise NotImplementedError()
-
-                    # remove class line
-                    c = re.sub(r'^[^\n]+', '', code_block)
-                    # remove whitespace
-                    c = re.compile(r'^[ ]{4}', re.MULTILINE).sub('', c)
-
-                    funcs[name] = process_code(c)
-                else:
-                    raise NotImplementedError()
-            return funcs
-
-        try:
-            name = self.name
-            # sometimes there are stupid endings like `_sqlite3.cpython-32mu`
-            name = re.sub(r'\..*', '', name)
-
-            if name == '__builtin__' and not is_py3k:
-                name = 'builtins'
-            path = os.path.dirname(os.path.abspath(__file__))
-            with open(os.path.sep.join([path, 'mixin', name]) + '.pym') as f:
-                s = f.read()
-        except IOError:
-            return {}
-        else:
-            mixin_dct = process_code(s)
-            if is_py3k and self.name == Builtin.name:
-                # in the case of Py3k xrange is now range
-                mixin_dct['range'] = mixin_dct['xrange']
-            return mixin_dct
-
-
-def _generate_code(scope, mixin_funcs={}, depth=0):
-    """
-    Generate a string, which uses python syntax as an input to the Parser.
-    """
-    def get_doc(obj, indent=False):
-        doc = inspect.getdoc(obj)
-        if doc:
-            doc = ('r"""\n%s\n"""\n' % doc)
-            if indent:
-                doc = common.indent_block(doc)
-            return doc
-        return ''
-
-    def is_in_base_classes(cls, name, comparison):
-        """ Base classes may contain the exact same object """
-        if name in mixin_funcs:
-            return False
-        try:
-            mro = cls.mro()
-        except TypeError:
-            # this happens, if cls == type
-            return False
-        for base in mro[1:]:
-            try:
-                attr = getattr(base, name)
-            except AttributeError:
-                continue
-            if attr == comparison:
-                return True
-        return False
-
-    def get_scope_objects(names):
-        """
-        Looks for the names defined with dir() in an objects and divides
-        them into different object types.
-        """
-        classes = {}
-        funcs = {}
-        stmts = {}
-        members = {}
-        for n in names:
-            try:
-                # this has a builtin_function_or_method
-                exe = getattr(scope, n)
-            except AttributeError:
-                # happens e.g. in properties of
-                # PyQt4.QtGui.QStyleOptionComboBox.currentText
-                # -> just set it to None
-                members[n] = None
-            else:
-                if inspect.isclass(scope):
-                    if is_in_base_classes(scope, n, exe):
-                        continue
-                if inspect.isbuiltin(exe) or inspect.ismethod(exe) \
-                            or inspect.ismethoddescriptor(exe):
-                    funcs[n] = exe
-                elif inspect.isclass(exe) or inspect.ismodule(exe):
-                    classes[n] = exe
-                elif inspect.ismemberdescriptor(exe):
-                    members[n] = exe
-                else:
-                    stmts[n] = exe
-        return classes, funcs, stmts, members
-
-    code = ''
-    if inspect.ismodule(scope):  # generate comment where the code's from.
-        try:
-            path = scope.__file__
-        except AttributeError:
-            path = '?'
-        code += '# Generated module %s from %s\n' % (scope.__name__, path)
-
-    code += get_doc(scope)
-
-    names = set(dir(scope)) - set(['__file__', '__name__', '__doc__',
-                                            '__path__', '__package__']) \
-                            | set(['mro'])
-
-    classes, funcs, stmts, members = get_scope_objects(names)
-
-    # classes
-    for name, cl in classes.items():
-        bases = (c.__name__ for c in cl.__bases__) if inspect.isclass(cl) \
-                                                   else []
-        code += 'class %s(%s):\n' % (name, ','.join(bases))
-        if depth == 0:
-            try:
-                mixin = mixin_funcs[name]
-            except KeyError:
-                mixin = {}
-            cl_code = _generate_code(cl, mixin, depth + 1)
-            code += common.indent_block(cl_code)
-        code += '\n'
-
-    # functions
-    for name, func in funcs.items():
-        params, ret = _parse_function_doc(func)
-        if depth > 0:
-            params = 'self, ' + params
-        doc_str = get_doc(func, indent=True)
-        try:
-            mixin = mixin_funcs[name]
-        except KeyError:
-            # normal code generation
-            code += 'def %s(%s):\n' % (name, params)
-            code += doc_str
-            code += common.indent_block('%s\n\n' % ret)
-        else:
-            # generation of code with mixins
-            # the parser only supports basic functions with a newline after
-            # the double dots
-            # find doc_str place
-            try:
-                pos = re.search(r'\):\s*\n', mixin).end()
-            except TypeError:
-                # pypy uses a different reversed builtin
-                if name == 'reversed':
-                    mixin = 'def reversed(sequence):\n' \
-                            '    for i in self.__sequence: yield i'
-                    pos = 24
-                else:
-                    debug.warning('mixin trouble in pypy: %s', name)
-                    raise
-            if pos is None:
-                raise Exception("Builtin function not parsed correctly")
-            code += mixin[:pos] + doc_str + mixin[pos:]
-
-    # class members (functions) properties?
-    for name, func in members.items():
-        # recursion problem in properties TODO remove
-        if name in ['fget', 'fset', 'fdel']:
-            continue
-        ret = 'pass'
-        code += '@property\ndef %s(self):\n' % (name)
-        code += common.indent_block(get_doc(func) + '%s\n\n' % ret)
-
-    # variables
-    for name, value in stmts.items():
-        if is_py3k:
-            file_type = io.TextIOWrapper
-        else:
-            file_type = types.FileType
-        if type(value) == file_type:
-            value = 'open()'
-        elif name == 'None':
-            value = ''
-        elif type(value).__name__ in ['int', 'bool', 'float',
-                                      'dict', 'list', 'tuple']:
-            value = repr(value)
-        else:
-            # get the type, if the type is not simple.
-            mod = type(value).__module__
-            value = type(value).__name__ + '()'
-            if mod != '__builtin__':
-                value = '%s.%s' % (mod, value)
-        code += '%s = %s\n' % (name, value)
-
-    if depth == 0:
-        #with open('writeout.py', 'w') as f:
-        #    f.write(code)
-        #import sys
-        #sys.stdout.write(code)
-        #exit()
-        pass
-    return code
-
-
-def _parse_function_doc(func):
-    """
-    Takes a function and returns the params and return value as a tuple.
-    This is nothing more than a docstring parser.
-    """
-    # TODO: things like utime(path, (atime, mtime)) and a(b [, b]) -> None
-    doc = inspect.getdoc(func)
-
-    # get full string, parse round parentheses: def func(a, (b,c))
-    try:
-        count = 0
-        debug.dbg(func, func.__name__, doc)
-        start = doc.index('(')
-        for i, s in enumerate(doc[start:]):
-            if s == '(':
-                count += 1
-            elif s == ')':
-                count -= 1
-            if count == 0:
-                end = start + i
-                break
-        param_str = doc[start + 1:end]
-
-        # remove square brackets, that show an optional param ( = None)
-        def change_options(m):
-            args = m.group(1).split(',')
-            for i, a in enumerate(args):
-                if a and '=' not in a:
-                    args[i] += '=None'
-            return ','.join(args)
-        while True:
-            param_str, changes = re.subn(r' ?\[([^\[\]]+)\]',
-                                            change_options, param_str)
-            if changes == 0:
-                break
-    except (ValueError, AttributeError):
-        debug.dbg('no brackets found - no param')
-        end = 0
-        param_str = ''
-
-    param_str = param_str.replace('-', '_')  # see: isinstance.__doc__
-
-    if doc is not None:
-        r = re.search('-[>-]* ', doc[end:end + 7])
-    if doc is None or r is None:
-        ret = 'pass'
-    else:
-        index = end + r.end()
-        # get result type, which can contain newlines
-        pattern = re.compile(r'(,\n|[^\n-])+')
-        ret_str = pattern.match(doc, index).group(0).strip()
-        # New object -> object()
-        ret_str = re.sub(r'[nN]ew (.*)', r'\1()', ret_str)
-
-        ret = BuiltinModule.map_types.get(ret_str, ret_str)
-        if ret == ret_str and ret not in ['None', 'object', 'tuple', 'set']:
-            debug.dbg('not working', ret_str)
-        if ret != 'pass':
-            ret = ('return ' if 'return' not in ret else '') + ret
-    return param_str, ret
-
-
-class Builtin(object):
-    """ The builtin scope / module """
-    # Python 3 compatibility
-    if is_py3k:
-        name = 'builtins'
-    else:
-        name = '__builtin__'
-
-    _builtin = None
-
-    @property
-    def builtin(self):
-        if self._builtin is None:
-            self._builtin = BuiltinModule(name=self.name)
-        return self._builtin
-
-    @property
-    def scope(self):
-        return self.builtin.parser.module
-
-    @property
-    def magic_function_scope(self):
-        try:
-            return self._magic_function_scope
-        except AttributeError:
-            # depth = 1 because this is not a module
-            class Container(object):
-                FunctionType = types.FunctionType
-            source = _generate_code(Container, depth=0)
-            parser = parsing.Parser(source, None)
-            module = parser.module
-            module.parent = self.scope
-            typ = evaluate.follow_path(iter(['FunctionType']), module, module)
-
-            s = self._magic_function_scope = typ.pop()
-            return s
-
-
-Builtin = Builtin()

jedi/cache.py

@@ -3,11 +3,8 @@ This caching is very important for speed and memory optimizations. There's
 nothing really spectacular, just some decorators. The following cache types are
 available:
 
-- module caching (`load_module` and `save_module`), which uses pickle and is
+- module caching (`load_parser` and `save_parser`), which uses pickle and is
   really important to assure low load times of modules like ``numpy``.
-- the popular ``memoize_default`` works like a typical memoize and returns the
-  default otherwise.
-- ``CachedMetaClass`` uses ``memoize_default`` to do the same with classes.
 - ``time_cache`` can be used to cache something for just a limited time span,
   which can be useful if there's user interaction and the user cannot react
   faster than a certain time.
@@ -16,29 +13,25 @@ This module is one of the reasons why |jedi| is not thread-safe. As you can see
 there are global variables, which are holding the cache information. Some of
 these variables are being cleaned after every API usage.
 """
-from __future__ import with_statement
-
 import time
 import os
 import sys
 import json
 import hashlib
+import gc
+import inspect
+import shutil
+import re
 try:
     import cPickle as pickle
-except:
+except ImportError:
     import pickle
-import shutil
 
 from jedi import settings
 from jedi import common
 from jedi import debug
 
-# memoize caches will be deleted after every action
-memoize_caches = []
-
-time_caches = []
-
-star_import_cache = {}
+_time_caches = {}
 
 # for fast_parser, should not be deleted
 parser_cache = {}
@@ -52,27 +45,22 @@ class ParserCacheItem(object):
         self.change_time = change_time
 
 
-def clear_caches(delete_all=False):
+def clear_time_caches(delete_all=False):
     """ Jedi caches many things, that should be completed after each completion
     finishes.
 
     :param delete_all: Deletes also the cache that is normally not deleted,
         like parser cache, which is important for faster parsing.
     """
-    global memoize_caches, time_caches
-
-    # memorize_caches must never be deleted, because the dicts will get lost in
-    # the wrappers.
-    for m in memoize_caches:
-        m.clear()
+    global _time_caches
 
     if delete_all:
-        time_caches = []
-        star_import_cache.clear()
+        for cache in _time_caches.values():
+            cache.clear()
         parser_cache.clear()
     else:
         # normally just kill the expired entries, not all
-        for tc in time_caches:
+        for tc in _time_caches.values():
             # check time_cache for expired entries
             for key, (t, value) in list(tc.items()):
                 if t < time.time():
@@ -80,59 +68,29 @@ def clear_caches(delete_all=False):
                     del tc[key]
 
 
-def memoize_default(default=None, cache=memoize_caches):
-    """ This is a typical memoization decorator, BUT there is one difference:
-    To prevent recursion it sets defaults.
-
-    Preventing recursion is in this case the much bigger use than speed. I
-    don't think, that there is a big speed difference, but there are many cases
-    where recursion could happen (think about a = b; b = a).
-    """
-    def func(function):
-        memo = {}
-        cache.append(memo)
-
-        def wrapper(*args, **kwargs):
-            key = (args, frozenset(kwargs.items()))
-            if key in memo:
-                return memo[key]
-            else:
-                memo[key] = default
-                rv = function(*args, **kwargs)
-                memo[key] = rv
-                return rv
-        return wrapper
-    return func
-
-
-class CachedMetaClass(type):
-    """ This is basically almost the same than the decorator above, it just
-    caches class initializations. I haven't found any other way, so I do it
-    with meta classes.
-    """
-    @memoize_default()
-    def __call__(self, *args, **kwargs):
-        return super(CachedMetaClass, self).__call__(*args, **kwargs)
-
-
 def time_cache(time_add_setting):
-    """ This decorator works as follows: Call it with a setting and after that
+    """
+    s
+    This decorator works as follows: Call it with a setting and after that
     use the function with a callable that returns the key.
     But: This function is only called if the key is not available. After a
     certain amount of time (`time_add_setting`) the cache is invalid.
     """
     def _temp(key_func):
         dct = {}
-        time_caches.append(dct)
+        _time_caches[time_add_setting] = dct
 
-        def wrapper(optional_callable, *args, **kwargs):
-            key = key_func(*args, **kwargs)
-            value = None
-            if key in dct:
+        def wrapper(*args, **kwargs):
+            generator = key_func(*args, **kwargs)
+            key = next(generator)
+            try:
                 expiry, value = dct[key]
                 if expiry > time.time():
                     return value
-            value = optional_callable()
+            except KeyError:
+                pass
+
+            value = next(generator)
             time_add = getattr(settings, time_add_setting)
             if key is not None:
                 dct[key] = time.time() + time_add, value
@@ -141,85 +99,133 @@ def time_cache(time_add_setting):
     return _temp
 
 
-@time_cache("function_definition_validity")
-def cache_function_definition(stmt):
-    module_path = stmt.get_parent_until().path
-    return None if module_path is None else (module_path, stmt.start_pos)
+@time_cache("call_signatures_validity")
+def cache_call_signatures(evaluator, call, source, user_pos):
+    """This function calculates the cache key."""
+    index = user_pos[0] - 1
+    lines = common.splitlines(source)
+
+    before_cursor = lines[index][:user_pos[1]]
+    other_lines = lines[call.start_pos[0]:index]
+    whole = '\n'.join(other_lines + [before_cursor])
+    before_bracket = re.match(r'.*\(', whole, re.DOTALL)
+
+    module_path = call.get_parent_until().path
+    yield None if module_path is None else (module_path, before_bracket, call.start_pos)
+    yield evaluator.eval_element(call)
 
 
-def cache_star_import(func):
-    def wrapper(scope, *args, **kwargs):
-        with common.ignored(KeyError):
-            mods = star_import_cache[scope]
-            if mods[0] + settings.star_import_cache_validity > time.time():
-                return mods[1]
-        # cache is too old and therefore invalid or not available
-        invalidate_star_import_cache(scope)
-        mods = func(scope, *args, **kwargs)
-        star_import_cache[scope] = time.time(), mods
+def underscore_memoization(func):
+    """
+    Decorator for methods::
+
+        class A(object):
+            def x(self):
+                if self._x:
+                    self._x = 10
+                return self._x
+
+    Becomes::
+
+        class A(object):
+            @underscore_memoization
+            def x(self):
+                return 10
+
+    A now has an attribute ``_x`` written by this decorator.
+    """
+    name = '_' + func.__name__
+
+    def wrapper(self):
+        try:
+            return getattr(self, name)
+        except AttributeError:
+            result = func(self)
+            if inspect.isgenerator(result):
+                result = list(result)
+            setattr(self, name, result)
+            return result
 
-        return mods
     return wrapper
 
 
-def invalidate_star_import_cache(module, only_main=False):
+def memoize_method(method):
+    """A normal memoize function."""
+    def wrapper(self, *args, **kwargs):
+        dct = self.__dict__.setdefault('_memoize_method_dct', {})
+        key = (args, frozenset(kwargs.items()))
+        try:
+            return dct[key]
+        except KeyError:
+            result = method(self, *args, **kwargs)
+            dct[key] = result
+            return result
+    return wrapper
+
+
+def cache_star_import(func):
+    @time_cache("star_import_cache_validity")
+    def wrapper(self):
+        yield self.base  # The cache key
+        yield func(self)
+    return wrapper
+
+
+def _invalidate_star_import_cache_module(module, only_main=False):
     """ Important if some new modules are being reparsed """
-    with common.ignored(KeyError):
-        t, mods = star_import_cache[module]
-
-        del star_import_cache[module]
-
-        for m in mods:
-            invalidate_star_import_cache(m, only_main=True)
-
-    if not only_main:
-        # We need a list here because otherwise the list is being changed
-        # during the iteration in py3k: iteritems -> items.
-        for key, (t, mods) in list(star_import_cache.items()):
-            if module in mods:
-                invalidate_star_import_cache(key)
+    try:
+        t, modules = _time_caches['star_import_cache_validity'][module]
+    except KeyError:
+        pass
+    else:
+        del _time_caches['star_import_cache_validity'][module]
 
 
-def load_module(path, name):
+def invalidate_star_import_cache(path):
+    """On success returns True."""
+    try:
+        parser_cache_item = parser_cache[path]
+    except KeyError:
+        pass
+    else:
+        _invalidate_star_import_cache_module(parser_cache_item.parser.module)
+
+
+def load_parser(path):
     """
     Returns the module or None, if it fails.
     """
-    if path is None and name is None:
-        return None
-
-    tim = os.path.getmtime(path) if path else None
-    n = name if path is None else path
+    p_time = os.path.getmtime(path) if path else None
     try:
-        parser_cache_item = parser_cache[n]
-        if not path or tim <= parser_cache_item.change_time:
+        parser_cache_item = parser_cache[path]
+        if not path or p_time <= parser_cache_item.change_time:
             return parser_cache_item.parser
         else:
             # In case there is already a module cached and this module
             # has to be reparsed, we also need to invalidate the import
             # caches.
-            invalidate_star_import_cache(parser_cache_item.parser.module)
+            _invalidate_star_import_cache_module(parser_cache_item.parser.module)
     except KeyError:
         if settings.use_filesystem_cache:
-            return ModulePickling.load_module(n, tim)
+            return ParserPickling.load_parser(path, p_time)
 
 
-def save_module(path, name, parser, pickling=True):
+def save_parser(path, parser, pickling=True):
     try:
-        p_time = None if not path else os.path.getmtime(path)
+        p_time = None if path is None else os.path.getmtime(path)
     except OSError:
         p_time = None
         pickling = False
 
-    n = name if path is None else path
     item = ParserCacheItem(parser, p_time)
-    parser_cache[n] = item
+    parser_cache[path] = item
     if settings.use_filesystem_cache and pickling:
-        ModulePickling.save_module(n, item)
+        ParserPickling.save_parser(path, item)
 
 
-class _ModulePickling(object):
+class ParserPickling(object):
 
-    version = 3
+    version = 24
     """
     Version number (integer) for file system cache.
 
@@ -245,7 +251,7 @@ class _ModulePickling(object):
         .. todo:: Detect interpreter (e.g., PyPy).
         """
 
-    def load_module(self, path, original_changed_time):
+    def load_parser(self, path, original_changed_time):
         try:
             pickle_changed_time = self._index[path]
         except KeyError:
@@ -256,13 +262,17 @@ class _ModulePickling(object):
             return None
 
         with open(self._get_hashed_path(path), 'rb') as f:
-            parser_cache_item = pickle.load(f)
+            try:
+                gc.disable()
+                parser_cache_item = pickle.load(f)
+            finally:
+                gc.enable()
 
-        debug.dbg('pickle loaded', path)
+        debug.dbg('pickle loaded: %s', path)
         parser_cache[path] = parser_cache_item
         return parser_cache_item.parser
 
-    def save_module(self, path, parser_cache_item):
+    def save_parser(self, path, parser_cache_item):
         self.__index = None
         try:
             files = self._index
@@ -282,12 +292,12 @@ class _ModulePickling(object):
             try:
                 with open(self._get_path('index.json')) as f:
                     data = json.load(f)
-            except IOError:
+            except (IOError, ValueError):
                 self.__index = {}
             else:
                 # 0 means version is not defined (= always delete cache):
                 if data.get('version', 0) != self.version:
-                    self.delete_cache()
+                    self.clear_cache()
                     self.__index = {}
                 else:
                     self.__index = data['index']
@@ -306,7 +316,7 @@ class _ModulePickling(object):
             json.dump(data, f)
         self.__index = None
 
-    def delete_cache(self):
+    def clear_cache(self):
         shutil.rmtree(self._cache_directory())
 
     def _get_hashed_path(self, path):
@@ -323,4 +333,4 @@ class _ModulePickling(object):
 
 
 # is a singleton
-ModulePickling = _ModulePickling()
+ParserPickling = ParserPickling()

jedi/common.py

@@ -2,20 +2,12 @@
 import sys
 import contextlib
 import functools
-import tokenizer as tokenize
+import re
+from ast import literal_eval
 
-from jedi._compatibility import next, reraise
+from jedi._compatibility import unicode, reraise
 from jedi import settings
 
-FLOWS = ['if', 'else', 'elif', 'while', 'with', 'try', 'except', 'finally']
-
-
-class MultiLevelStopIteration(Exception):
-    """
-    StopIteration's get catched pretty easy by for loops, let errors propagate.
-    """
-    pass
-
 
 class UncaughtAttributeError(Exception):
     """
@@ -30,7 +22,11 @@ class UncaughtAttributeError(Exception):
     """
 
 
-def rethrow_uncaught(func):
+def safe_property(func):
+    return property(reraise_uncaught(func))
+
+
+def reraise_uncaught(func):
     """
     Re-throw uncaught `AttributeError`.
 
@@ -84,115 +80,21 @@ class PushBackIterator(object):
         return self.current
 
 
-class NoErrorTokenizer(object):
-    def __init__(self, readline, offset=(0, 0), is_fast_parser=False):
-        self.readline = readline
-        self.gen = tokenize.generate_tokens(readline)
-        self.offset = offset
-        self.closed = False
-        self.is_first = True
-        self.push_backs = []
-
-        # fast parser options
-        self.is_fast_parser = is_fast_parser
-        self.current = self.previous = [None, None, (0, 0), (0, 0), '']
-        self.in_flow = False
-        self.new_indent = False
-        self.parser_indent = self.old_parser_indent = 0
-        self.is_decorator = False
-        self.first_stmt = True
-
-    def push_last_back(self):
-        self.push_backs.append(self.current)
-
-    def next(self):
-        """ Python 2 Compatibility """
-        return self.__next__()
-
-    def __next__(self):
-        if self.closed:
-            raise MultiLevelStopIteration()
-        if self.push_backs:
-            return self.push_backs.pop(0)
-
-        self.last_previous = self.previous
-        self.previous = self.current
-        self.current = next(self.gen)
-        c = list(self.current)
-
-        if c[0] == tokenize.ENDMARKER:
-            self.current = self.previous
-            self.previous = self.last_previous
-            raise MultiLevelStopIteration()
-
-        # this is exactly the same check as in fast_parser, but this time with
-        # tokenize and therefore precise.
-        breaks = ['def', 'class', '@']
-
-        if self.is_first:
-            c[2] = self.offset[0] + c[2][0], self.offset[1] + c[2][1]
-            c[3] = self.offset[0] + c[3][0], self.offset[1] + c[3][1]
-            self.is_first = False
-        else:
-            c[2] = self.offset[0] + c[2][0], c[2][1]
-            c[3] = self.offset[0] + c[3][0], c[3][1]
-        self.current = c
-
-        def close():
-            if not self.first_stmt:
-                self.closed = True
-                raise MultiLevelStopIteration()
-        # ignore indents/comments
-        if self.is_fast_parser \
-                and self.previous[0] in (tokenize.INDENT, tokenize.NL, None,
-                                         tokenize.NEWLINE, tokenize.DEDENT) \
-                and c[0] not in (tokenize.COMMENT, tokenize.INDENT,
-                             tokenize.NL, tokenize.NEWLINE, tokenize.DEDENT):
-            #print c, tokenize.tok_name[c[0]]
-
-            tok = c[1]
-            indent = c[2][1]
-            if indent < self.parser_indent:  # -> dedent
-                self.parser_indent = indent
-                self.new_indent = False
-                if not self.in_flow or indent < self.old_parser_indent:
-                    close()
-                self.in_flow = False
-            elif self.new_indent:
-                self.parser_indent = indent
-                self.new_indent = False
-
-            if not self.in_flow:
-                if tok in FLOWS or tok in breaks:
-                    self.in_flow = tok in FLOWS
-                    if not self.is_decorator and not self.in_flow:
-                        close()
-                    self.is_decorator = '@' == tok
-                    if not self.is_decorator:
-                        self.old_parser_indent = self.parser_indent
-                        self.parser_indent += 1  # new scope: must be higher
-                        self.new_indent = True
-
-            if tok != '@':
-                if self.first_stmt and not self.new_indent:
-                    self.parser_indent = indent
-                self.first_stmt = False
-        return c
-
-
 @contextlib.contextmanager
 def scale_speed_settings(factor):
     a = settings.max_executions
     b = settings.max_until_execution_unique
     settings.max_executions *= factor
     settings.max_until_execution_unique *= factor
-    yield
-    settings.max_executions = a
-    settings.max_until_execution_unique = b
+    try:
+        yield
+    finally:
+        settings.max_executions = a
+        settings.max_until_execution_unique = b
 
 
 def indent_block(text, indention='    '):
-    """ This function indents a text block with a default of four spaces """
+    """This function indents a text block with a default of four spaces."""
     temp = ''
     while text and text[-1] == '\n':
         temp += text[-1]
@@ -203,9 +105,50 @@ def indent_block(text, indention='    '):
 
 @contextlib.contextmanager
 def ignored(*exceptions):
-    """Context manager that ignores all of the specified exceptions. This will
-    be in the standard library starting with Python 3.4."""
+    """
+    Context manager that ignores all of the specified exceptions. This will
+    be in the standard library starting with Python 3.4.
+    """
     try:
         yield
     except exceptions:
         pass
+
+
+def source_to_unicode(source, encoding=None):
+    def detect_encoding():
+        """
+        For the implementation of encoding definitions in Python, look at:
+        - http://www.python.org/dev/peps/pep-0263/
+        - http://docs.python.org/2/reference/lexical_analysis.html#encoding-declarations
+        """
+        byte_mark = literal_eval(r"b'\xef\xbb\xbf'")
+        if source.startswith(byte_mark):
+            # UTF-8 byte-order mark
+            return 'utf-8'
+
+        first_two_lines = re.match(r'(?:[^\n]*\n){0,2}', str(source)).group(0)
+        possible_encoding = re.search(r"coding[=:]\s*([-\w.]+)",
+                                      first_two_lines)
+        if possible_encoding:
+            return possible_encoding.group(1)
+        else:
+            # the default if nothing else has been set -> PEP 263
+            return encoding if encoding is not None else 'iso-8859-1'
+
+    if isinstance(source, unicode):
+        # only cast str/bytes
+        return source
+
+    # cast to unicode by default
+    return unicode(source, detect_encoding(), 'replace')
+
+
+def splitlines(string):
+    """
+    A splitlines for Python code. In contrast to Python's ``str.splitlines``,
+    looks at form feeds and other special characters as normal text. Just
+    splits ``\n`` and ``\r\n``.
+    Also different: Returns ``['']`` for an empty string input.
+    """
+    return re.split('\n|\r\n', string)

jedi/debug.py

@@ -1,10 +1,20 @@
+from jedi._compatibility import encoding, is_py3, u
 import inspect
+import os
 import time
 
 try:
-    # Use colorama for nicer console output.
-    from colorama import Fore, init
-    init()
+    if os.name == 'nt':
+        # does not work on Windows, as pyreadline and colorama interfere
+        raise ImportError
+    else:
+        # Use colorama for nicer console output.
+        from colorama import Fore, init
+        from colorama import initialise
+        # pytest resets the stream at the end - causes troubles. Since after
+        # every output the stream is reset automatically we don't need this.
+        initialise.atexit_done = True
+        init()
 except ImportError:
     class Fore(object):
         RED = ''
@@ -22,33 +32,51 @@ enable_notice = False
 
 # callback, interface: level, str
 debug_function = None
-ignored_modules = ['parsing', 'builtin', 'jedi.builtin', 'jedi.parsing']
+ignored_modules = ['jedi.evaluate.builtin', 'jedi.parser']
+_debug_indent = -1
+_start_time = time.time()
 
 
 def reset_time():
-    global start_time
-    start_time = time.time()
+    global _start_time, _debug_indent
+    _start_time = time.time()
+    _debug_indent = -1
 
 
-def dbg(*args):
+def increase_indent(func):
+    """Decorator for makin """
+    def wrapper(*args, **kwargs):
+        global _debug_indent
+        _debug_indent += 1
+        try:
+            result = func(*args, **kwargs)
+        finally:
+            _debug_indent -= 1
+        return result
+    return wrapper
+
+
+def dbg(message, *args):
     """ Looks at the stack, to see if a debug message should be printed. """
     if debug_function and enable_notice:
         frm = inspect.stack()[1]
         mod = inspect.getmodule(frm[0])
         if not (mod.__name__ in ignored_modules):
-            debug_function(NOTICE, 'dbg: ' + ', '.join(str(a) for a in args))
+            i = ' ' * _debug_indent
+            debug_function(NOTICE, i + 'dbg: ' + message % tuple(u(repr(a)) for a in args))
 
 
-def warning(*args):
+def warning(message, *args):
     if debug_function and enable_warning:
-        debug_function(WARNING, 'warning: ' + ', '.join(str(a) for a in args))
+        i = ' ' * _debug_indent
+        debug_function(WARNING, i + 'warning: ' + message % tuple(u(repr(a)) for a in args))
 
 
 def speed(name):
     if debug_function and enable_speed:
-        global start_time
         now = time.time()
-        debug_function(SPEED, 'speed: ' + '%s %s' % (name, now - start_time))
+        i = ' ' * _debug_indent
+        debug_function(SPEED, i + 'speed: ' + '%s %s' % (name, now - _start_time))
 
 
 def print_to_stdout(level, str_out):
@@ -59,7 +87,9 @@ def print_to_stdout(level, str_out):
         col = Fore.RED
     else:
         col = Fore.YELLOW
+    if not is_py3:
+        str_out = str_out.encode(encoding, 'replace')
     print(col + str_out + Fore.RESET)
 
 
-#debug_function = print_to_stdout
+# debug_function = print_to_stdout

jedi/docstrings.py

@@ -1,127 +0,0 @@
-"""
-Docstrings are another source of information for functions and classes.
-:mod:`dynamic` tries to find all executions of functions, while the docstring
-parsing is much easier. There are two different types of docstrings that |jedi|
-understands:
-
-- `Sphinx <http://sphinx-doc.org/markup/desc.html#info-field-lists>`_
-- `Epydoc <http://epydoc.sourceforge.net/manual-fields.html>`_
-
-For example, the sphinx annotation ``:type foo: str`` clearly states that the
-type of ``foo`` is ``str``.
-
-As an addition to parameter searching, this module also provides return
-annotations.
-"""
-
-import re
-
-from jedi import cache
-from jedi import parsing
-import evaluate
-import evaluate_representation as er
-
-DOCSTRING_PARAM_PATTERNS = [
-    r'\s*:type\s+%s:\s*([^\n]+)',  # Sphinx
-    r'\s*@type\s+%s:\s*([^\n]+)',  # Epydoc
-]
-
-DOCSTRING_RETURN_PATTERNS = [
-        re.compile(r'\s*:rtype:\s*([^\n]+)', re.M),  # Sphinx
-        re.compile(r'\s*@rtype:\s*([^\n]+)', re.M),  # Epydoc
-]
-
-REST_ROLE_PATTERN = re.compile(r':[^`]+:`([^`]+)`')
-
-
-@cache.memoize_default()
-def follow_param(param):
-    func = param.parent_function
-    #print func, param, param.parent_function
-    param_str = _search_param_in_docstr(func.docstr, str(param.get_name()))
-    user_position = (1, 0)
-
-    if param_str is not None:
-
-        # Try to import module part in dotted name.
-        # (e.g., 'threading' in 'threading.Thread').
-        if '.' in param_str:
-            param_str = 'import %s\n%s' % (
-                param_str.rsplit('.', 1)[0],
-                param_str)
-            user_position = (2, 0)
-
-        p = parsing.Parser(param_str, None, user_position,
-                                  no_docstr=True)
-        return evaluate.follow_statement(p.user_stmt)
-    return []
-
-
-def _search_param_in_docstr(docstr, param_str):
-    """
-    Search `docstr` for a type of `param_str`.
-
-    >>> _search_param_in_docstr(':type param: int', 'param')
-    'int'
-    >>> _search_param_in_docstr('@type param: int', 'param')
-    'int'
-    >>> _search_param_in_docstr(
-    ...   ':type param: :class:`threading.Thread`', 'param')
-    'threading.Thread'
-    >>> _search_param_in_docstr('no document', 'param') is None
-    True
-
-    """
-    # look at #40 to see definitions of those params
-    patterns = [re.compile(p % re.escape(param_str))
-                for p in DOCSTRING_PARAM_PATTERNS]
-    for pattern in patterns:
-        match = pattern.search(docstr)
-        if match:
-            return _strip_rest_role(match.group(1))
-
-    return None
-
-
-def _strip_rest_role(type_str):
-    """
-    Strip off the part looks like a ReST role in `type_str`.
-
-    >>> _strip_rest_role(':class:`ClassName`')  # strip off :class:
-    'ClassName'
-    >>> _strip_rest_role(':py:obj:`module.Object`')  # works with domain
-    'module.Object'
-    >>> _strip_rest_role('ClassName')  # do nothing when not ReST role
-    'ClassName'
-
-    See also:
-    http://sphinx-doc.org/domains.html#cross-referencing-python-objects
-
-    """
-    match = REST_ROLE_PATTERN.match(type_str)
-    if match:
-        return match.group(1)
-    else:
-        return type_str
-
-
-def find_return_types(func):
-    def search_return_in_docstr(code):
-        for p in DOCSTRING_RETURN_PATTERNS:
-            match = p.search(code)
-            if match:
-                return match.group(1)
-
-    if isinstance(func, er.InstanceElement):
-        func = func.var
-
-    if isinstance(func, er.Function):
-        func = func.base_func
-
-    type_str = search_return_in_docstr(func.docstr)
-    if not type_str:
-        return []
-
-    p = parsing.Parser(type_str, None, (1, 0), no_docstr=True)
-    p.user_stmt.parent = func
-    return list(evaluate.follow_statement(p.user_stmt))

jedi/dynamic.py

@@ -1,551 +0,0 @@
-"""
-To understand Python on a deeper level, |jedi| needs to understand some of the
-dynamic features of Python, however this probably the most complicated part:
-
-- Array modifications (e.g. ``list.append``)
-- Parameter completion in functions
-- Flow checks (e.g. ``if isinstance(a, str)`` -> a is a str)
-
-Array modifications
-*******************
-
-If the content of an array (``set``/``list``) is wanted somewhere, the current
-module will be checked for appearances of ``arr.append``, ``arr.insert``, etc.
-If the ``arr`` name points to an actual array, the content will be added
-
-This can be really cpu intensive, as you can imagine. Because |jedi| has to
-follow **every** ``append``. However this works pretty good, because in *slow*
-cases, the recursion detector and other settings will stop this process.
-
-It is important to note that:
-
-1. Array modfications work only in the current module
-2. Only Array additions are being checked, ``list.pop``, etc. is being ignored.
-
-Parameter completion
-********************
-
-One of the really important features of |jedi| is to have an option to
-understand code like this::
-
-    def foo(bar):
-        bar. # completion here
-    foo(1)
-
-There's no doubt wheter bar is an ``int`` or not, but if there's also a call
-like ``foo('str')``, what would happen? Well, we'll just show both. Because
-that's what a human would expect.
-
-It works as follows:
-
-- A param is being encountered
-- search for function calls named ``foo``
-- execute these calls and check the injected params. This work with a
-  ``ParamListener``.
-
-Flow checks
-***********
-
-Flow checks are not really mature. There's only a check for ``isinstance``.  It
-would check whether a flow has the form of ``if isinstance(a, type_or_tuple)``.
-Unfortunately every other thing is being ignored (e.g. a == '' would be easy to
-check for -> a is a string). There's big potential in these checks.
-"""
-from __future__ import with_statement
-
-import os
-
-from jedi import cache
-from jedi import parsing_representation as pr
-from jedi import modules
-from jedi import settings
-from jedi import common
-from jedi import debug
-from jedi import fast_parser
-import api_classes
-import evaluate
-import imports
-import evaluate_representation as er
-
-# This is something like the sys.path, but only for searching params. It means
-# that this is the order in which Jedi searches params.
-search_param_modules = ['.']
-search_param_cache = {}
-
-
-def get_directory_modules_for_name(mods, name):
-    """
-    Search a name in the directories of modules.
-    """
-    def check_python_file(path):
-        try:
-            return cache.parser_cache[path].parser.module
-        except KeyError:
-            try:
-                return check_fs(path)
-            except IOError:
-                return None
-
-    def check_fs(path):
-        with open(path) as f:
-            source = modules.source_to_unicode(f.read())
-            if name in source:
-                return modules.Module(path, source).parser.module
-
-    # skip non python modules
-    mods = set(m for m in mods if m.path is None or m.path.endswith('.py'))
-    mod_paths = set()
-    for m in mods:
-        mod_paths.add(m.path)
-        yield m
-
-    if settings.dynamic_params_for_other_modules:
-        paths = set(settings.additional_dynamic_modules)
-        for p in mod_paths:
-            if p is not None:
-                d = os.path.dirname(p)
-                for entry in os.listdir(d):
-                    if entry not in mod_paths:
-                        if entry.endswith('.py'):
-                            paths.add(d + os.path.sep + entry)
-
-        for p in paths:
-            c = check_python_file(p)
-            if c is not None and c not in mods:
-                yield c
-
-
-def search_param_memoize(func):
-    """
-    Is only good for search params memoize, respectively the closure,
-    because it just caches the input, not the func, like normal memoize does.
-    """
-    def wrapper(*args, **kwargs):
-        key = (args, frozenset(kwargs.items()))
-        if key in search_param_cache:
-            return search_param_cache[key]
-        else:
-            rv = func(*args, **kwargs)
-            search_param_cache[key] = rv
-            return rv
-    return wrapper
-
-
-class ParamListener(object):
-    """
-    This listener is used to get the params for a function.
-    """
-    def __init__(self):
-        self.param_possibilities = []
-
-    def execute(self, params):
-        self.param_possibilities.append(params)
-
-
-@cache.memoize_default([])
-def search_params(param):
-    """
-    This is a dynamic search for params. If you try to complete a type:
-    >>> def func(foo):
-    >>>     # here is the completion
-    >>>     foo
-    >>> func(1)
-    >>> func("")
-
-    It is not known what the type is, because it cannot be guessed with
-    recursive madness. Therefore one has to analyse the statements that are
-    calling the function, as well as analyzing the incoming params.
-    """
-    if not settings.dynamic_params:
-        return []
-
-    def get_params_for_module(module):
-        """
-        Returns the values of a param, or an empty array.
-        """
-        @search_param_memoize
-        def get_posibilities(module, func_name):
-            try:
-                possible_stmts = module.used_names[func_name]
-            except KeyError:
-                return []
-
-            for stmt in possible_stmts:
-                if not isinstance(stmt, pr.Import):
-                    calls = _scan_statement(stmt, func_name)
-                    for c in calls:
-                        # no execution means that params cannot be set
-                        call_path = c.generate_call_path()
-                        pos = c.start_pos
-                        scope = stmt.parent
-                        evaluate.follow_call_path(call_path, scope, pos)
-            return listener.param_possibilities
-
-        result = []
-        for params in get_posibilities(module, func_name):
-            for p in params:
-                if str(p) == param_name:
-                    result += evaluate.follow_statement(p.parent)
-        return result
-
-    func = param.get_parent_until(pr.Function)
-    current_module = param.get_parent_until()
-    func_name = str(func.name)
-    if func_name == '__init__' and isinstance(func.parent, pr.Class):
-        func_name = str(func.parent.name)
-
-    # get the param name
-    if param.assignment_details:
-        # first assignment details, others would be a syntax error
-        commands, op = param.assignment_details[0]
-    else:
-        commands = param.get_commands()
-    offset = 1 if commands[0] in ['*', '**'] else 0
-    param_name = str(commands[offset].name)
-
-    # add the listener
-    listener = ParamListener()
-    func.listeners.add(listener)
-
-    result = []
-    # This is like backtracking: Get the first possible result.
-    for mod in get_directory_modules_for_name([current_module], func_name):
-        result = get_params_for_module(mod)
-        if result:
-            break
-
-    # cleanup: remove the listener; important: should not stick.
-    func.listeners.remove(listener)
-
-    return result
-
-
-def check_array_additions(array):
-    """ Just a mapper function for the internal _check_array_additions """
-    if not pr.Array.is_type(array._array, pr.Array.LIST, pr.Array.SET):
-        # TODO also check for dict updates
-        return []
-
-    is_list = array._array.type == 'list'
-    current_module = array._array.get_parent_until()
-    res = _check_array_additions(array, current_module, is_list)
-    return res
-
-
-def _scan_statement(stmt, search_name, assignment_details=False):
-    """ Returns the function Call that match search_name in an Array. """
-    def scan_array(arr, search_name):
-        result = []
-        if arr.type == pr.Array.DICT:
-            for key_stmt, value_stmt in arr.items():
-                result += _scan_statement(key_stmt, search_name)
-                result += _scan_statement(value_stmt, search_name)
-        else:
-            for stmt in arr:
-                result += _scan_statement(stmt, search_name)
-        return result
-
-    check = list(stmt.get_commands())
-    if assignment_details:
-        for commands, op in stmt.assignment_details:
-            check += commands
-
-    result = []
-    for c in check:
-        if isinstance(c, pr.Array):
-            result += scan_array(c, search_name)
-        elif isinstance(c, pr.Call):
-            s_new = c
-            while s_new is not None:
-                n = s_new.name
-                if isinstance(n, pr.Name) and search_name in n.names:
-                    result.append(c)
-
-                if s_new.execution is not None:
-                    result += scan_array(s_new.execution, search_name)
-                s_new = s_new.next
-
-    return result
-
-
-@cache.memoize_default([])
-def _check_array_additions(compare_array, module, is_list):
-    """
-    Checks if a `pr.Array` has "add" statements:
-    >>> a = [""]
-    >>> a.append(1)
-    """
-    if not settings.dynamic_array_additions or module.is_builtin():
-        return []
-
-    def check_calls(calls, add_name):
-        """
-        Calls are processed here. The part before the call is searched and
-        compared with the original Array.
-        """
-        result = []
-        for c in calls:
-            call_path = list(c.generate_call_path())
-            separate_index = call_path.index(add_name)
-            if add_name == call_path[-1] or separate_index == 0:
-                # this means that there is no execution -> [].append
-                # or the keyword is at the start -> append()
-                continue
-            backtrack_path = iter(call_path[:separate_index])
-
-            position = c.start_pos
-            scope = c.get_parent_until(pr.IsScope)
-
-            found = evaluate.follow_call_path(backtrack_path, scope, position)
-            if not compare_array in found:
-                continue
-
-            params = call_path[separate_index + 1]
-            if not params.values:
-                continue  # no params: just ignore it
-            if add_name in ['append', 'add']:
-                for param in params:
-                    result += evaluate.follow_statement(param)
-            elif add_name in ['insert']:
-                try:
-                    second_param = params[1]
-                except IndexError:
-                    continue
-                else:
-                    result += evaluate.follow_statement(second_param)
-            elif add_name in ['extend', 'update']:
-                for param in params:
-                    iterators = evaluate.follow_statement(param)
-                result += evaluate.get_iterator_types(iterators)
-        return result
-
-    def get_execution_parent(element, *stop_classes):
-        """ Used to get an Instance/Execution parent """
-        if isinstance(element, er.Array):
-            stmt = element._array.parent
-        else:
-            # is an Instance with an ArrayInstance inside
-            stmt = element.var_args[0].var_args.parent
-        if isinstance(stmt, er.InstanceElement):
-            stop_classes = list(stop_classes) + [er.Function]
-        return stmt.get_parent_until(stop_classes)
-
-    temp_param_add = settings.dynamic_params_for_other_modules
-    settings.dynamic_params_for_other_modules = False
-
-    search_names = ['append', 'extend', 'insert'] if is_list else \
-                                                            ['add', 'update']
-    comp_arr_parent = get_execution_parent(compare_array, er.Execution)
-
-    possible_stmts = []
-    res = []
-    for n in search_names:
-        try:
-            possible_stmts += module.used_names[n]
-        except KeyError:
-            continue
-        for stmt in possible_stmts:
-            # Check if the original scope is an execution. If it is, one
-            # can search for the same statement, that is in the module
-            # dict. Executions are somewhat special in jedi, since they
-            # literally copy the contents of a function.
-            if isinstance(comp_arr_parent, er.Execution):
-                stmt = comp_arr_parent. \
-                                get_statement_for_position(stmt.start_pos)
-                if stmt is None:
-                    continue
-            # InstanceElements are special, because they don't get copied,
-            # but have this wrapper around them.
-            if isinstance(comp_arr_parent, er.InstanceElement):
-                stmt = er.InstanceElement(comp_arr_parent.instance, stmt)
-
-            if evaluate.follow_statement.push_stmt(stmt):
-                # check recursion
-                continue
-            res += check_calls(_scan_statement(stmt, n), n)
-            evaluate.follow_statement.pop_stmt()
-    # reset settings
-    settings.dynamic_params_for_other_modules = temp_param_add
-    return res
-
-
-def check_array_instances(instance):
-    """Used for set() and list() instances."""
-    if not settings.dynamic_arrays_instances:
-        return instance.var_args
-    ai = ArrayInstance(instance)
-    return [ai]
-
-
-class ArrayInstance(pr.Base):
-    """
-    Used for the usage of set() and list().
-    This is definitely a hack, but a good one :-)
-    It makes it possible to use set/list conversions.
-    """
-    def __init__(self, instance):
-        self.instance = instance
-        self.var_args = instance.var_args
-
-    def iter_content(self):
-        """
-        The index is here just ignored, because of all the appends, etc.
-        lists/sets are too complicated too handle that.
-        """
-        items = []
-        for stmt in self.var_args:
-            for typ in evaluate.follow_statement(stmt):
-                if isinstance(typ, er.Instance) and len(typ.var_args):
-                    array = typ.var_args[0]
-                    if isinstance(array, ArrayInstance):
-                        # prevent recursions
-                        # TODO compare Modules
-                        if self.var_args.start_pos != array.var_args.start_pos:
-                            items += array.iter_content()
-                        else:
-                            debug.warning('ArrayInstance recursion', self.var_args)
-                        continue
-                items += evaluate.get_iterator_types([typ])
-
-        # TODO check if exclusion of tuple is a problem here.
-        if isinstance(self.var_args, tuple) or self.var_args.parent is None:
-            return []  # generated var_args should not be checked for arrays
-
-        module = self.var_args.get_parent_until()
-        is_list = str(self.instance.name) == 'list'
-        items += _check_array_additions(self.instance, module, is_list)
-        return items
-
-
-def usages(definitions, search_name, mods):
-    def compare_array(definitions):
-        """ `definitions` are being compared by module/start_pos, because
-        sometimes the id's of the objects change (e.g. executions).
-        """
-        result = []
-        for d in definitions:
-            module = d.get_parent_until()
-            result.append((module, d.start_pos))
-        return result
-
-    def check_call(call):
-        result = []
-        follow = []  # There might be multiple search_name's in one call_path
-        call_path = list(call.generate_call_path())
-        for i, name in enumerate(call_path):
-            # name is `pr.NamePart`.
-            if name == search_name:
-                follow.append(call_path[:i + 1])
-
-        for f in follow:
-            follow_res, search = evaluate.goto(call.parent, f)
-            follow_res = usages_add_import_modules(follow_res, search)
-
-            compare_follow_res = compare_array(follow_res)
-            # compare to see if they match
-            if any(r in compare_definitions for r in compare_follow_res):
-                scope = call.parent
-                result.append(api_classes.Usage(search, scope))
-
-        return result
-
-    if not definitions:
-        return set()
-
-    compare_definitions = compare_array(definitions)
-    mods |= set([d.get_parent_until() for d in definitions])
-    names = []
-    for m in get_directory_modules_for_name(mods, search_name):
-        try:
-            stmts = m.used_names[search_name]
-        except KeyError:
-            continue
-        for stmt in stmts:
-            if isinstance(stmt, pr.Import):
-                count = 0
-                imps = []
-                for i in stmt.get_all_import_names():
-                    for name_part in i.names:
-                        count += 1
-                        if name_part == search_name:
-                            imps.append((count, name_part))
-
-                for used_count, name_part in imps:
-                    i = imports.ImportPath(stmt, kill_count=count - used_count,
-                                                        direct_resolve=True)
-                    f = i.follow(is_goto=True)
-                    if set(f) & set(definitions):
-                        names.append(api_classes.Usage(name_part, stmt))
-            else:
-                for call in _scan_statement(stmt, search_name,
-                                            assignment_details=True):
-                    names += check_call(call)
-    return names
-
-
-def usages_add_import_modules(definitions, search_name):
-    """ Adds the modules of the imports """
-    new = set()
-    for d in definitions:
-        if isinstance(d.parent, pr.Import):
-            s = imports.ImportPath(d.parent, direct_resolve=True)
-            with common.ignored(IndexError):
-                new.add(s.follow(is_goto=True)[0])
-    return set(definitions) | new
-
-
-def check_flow_information(flow, search_name, pos):
-    """ Try to find out the type of a variable just with the information that
-    is given by the flows: e.g. It is also responsible for assert checks.::
-
-        if isinstance(k, str):
-            k.  # <- completion here
-
-    ensures that `k` is a string.
-    """
-    result = []
-    if isinstance(flow, (pr.Scope, fast_parser.Module)) and not result:
-        for ass in reversed(flow.asserts):
-            if pos is None or ass.start_pos > pos:
-                continue
-            result = check_statement_information(ass, search_name)
-            if result:
-                break
-
-    if isinstance(flow, pr.Flow) and not result:
-        if flow.command in ['if', 'while'] and len(flow.inputs) == 1:
-            result = check_statement_information(flow.inputs[0], search_name)
-    return result
-
-
-def check_statement_information(stmt, search_name):
-    try:
-        commands = stmt.get_commands()
-        # this might be removed if we analyze and, etc
-        assert len(commands) == 1
-        call = commands[0]
-        assert type(call) == pr.Call and str(call.name) == 'isinstance'
-        assert bool(call.execution)
-
-        # isinstance check
-        isinst = call.execution.values
-        assert len(isinst) == 2  # has two params
-        obj, classes = [statement.get_commands() for statement in isinst]
-        assert len(obj) == 1
-        assert len(classes) == 1
-        assert isinstance(obj[0], pr.Call)
-        # names fit?
-        assert str(obj[0].name) == search_name
-        assert isinstance(classes[0], pr.Call)  # can be type or tuple
-    except AssertionError:
-        return []
-
-    result = []
-    for c in evaluate.follow_call(classes[0]):
-        if isinstance(c, er.Array):
-            result += c.get_index_types()
-        else:
-            result.append(c)
-    for i, c in enumerate(result):
-        result[i] = er.Instance(c)
-    return result

jedi/evaluate.py

@@ -1,796 +0,0 @@
-"""
-Evaluation of Python code in |jedi| is based on three assumptions:
-
-* Code is recursive (to weaken this assumption, the :mod:`dynamic` module
-  exists).
-* No magic is being used:
-
-  - metaclasses
-  - ``setattr()`` / ``__import__()``
-  - writing to ``globals()``, ``locals()``, ``object.__dict__``
-* The programmer is not a total dick, e.g. like `this
-  <https://github.com/davidhalter/jedi/issues/24>`_ :-)
-
-That said, there's mainly one entry point in this script: ``follow_statement``.
-This is where autocompletion starts. Everything you want to complete is either
-a ``Statement`` or some special name like ``class``, which is easy to complete.
-
-Therefore you need to understand what follows after ``follow_statement``. Let's
-make an example::
-
-    import datetime
-    datetime.date.toda# <-- cursor here
-
-First of all, this module doesn't care about completion. It really just cares
-about ``datetime.date``. At the end of the procedure ``follow_statement`` will
-return the ``datetime`` class.
-
-To *visualize* this (simplified):
-
-- ``follow_statement`` - ``<Statement: datetime.date>``
-
-    - Unpacking of the statement into ``[[<Call: datetime.date>]]``
-- ``follow_call_list``, calls ``follow_call`` with ``<Call: datetime.date>``
-- ``follow_call`` - searches the ``datetime`` name within the module.
-
-This is exactly where it starts to get complicated. Now recursions start to
-kick in. The statement has not been resolved fully, but now we need to resolve
-the datetime import. So it continues
-
-- follow import, which happens in the :mod:`imports` module.
-- now the same ``follow_call`` as above calls ``follow_paths`` to follow the
-  second part of the statement ``date``.
-- After ``follow_paths`` returns with the desired ``datetime.date`` class, the
-  result is being returned and the recursion finishes.
-
-Now what would happen if we wanted ``datetime.date.foo.bar``? Just two more
-calls to ``follow_paths`` (which calls itself with a recursion). What if the
-import would contain another Statement like this::
-
-    from foo import bar
-    Date = bar.baz
-
-Well... You get it. Just another ``follow_statement`` recursion. It's really
-easy. Just that Python is not that easy sometimes. To understand tuple
-assignments and different class scopes, a lot more code had to be written.  Yet
-we're still not talking about Descriptors and Nested List Comprehensions, just
-the simple stuff.
-
-So if you want to change something, write a test and then just change what you
-want. This module has been tested by about 600 tests. Don't be afraid to break
-something. The tests are good enough.
-
-I need to mention now that this recursive approach is really good because it
-only *evaluates* what needs to be *evaluated*. All the statements and modules
-that are not used are just being ignored. It's a little bit similar to the
-backtracking algorithm.
-
-
-.. todo:: nonlocal statement, needed or can be ignored? (py3k)
-"""
-from __future__ import with_statement
-
-import sys
-import itertools
-
-from jedi._compatibility import next, hasattr, is_py3k, unicode, reraise
-from jedi import common
-from jedi import cache
-from jedi import parsing_representation as pr
-from jedi import debug
-import evaluate_representation as er
-import recursion
-import docstrings
-import builtin
-import imports
-import dynamic
-
-
-def get_defined_names_for_position(scope, position=None, start_scope=None):
-    """
-    Return filtered version of ``scope.get_defined_names()``.
-
-    This function basically does what :meth:`scope.get_defined_names
-    <parsing_representation.Scope.get_defined_names>` does.
-
-    - If `position` is given, delete all names defined after `position`.
-    - For special objects like instances, `position` is ignored and all
-      names are returned.
-
-    :type     scope: :class:`parsing_representation.IsScope`
-    :param    scope: Scope in which names are searched.
-    :param position: the position as a line/column tuple, default is infinity.
-    """
-    names = scope.get_defined_names()
-    # Instances have special rules, always return all the possible completions,
-    # because class variables are always valid and the `self.` variables, too.
-    if (not position or isinstance(scope, (er.Array, er.Instance))
-                or start_scope != scope
-                and isinstance(start_scope, (pr.Function, er.Execution))):
-        return names
-    names_new = []
-    for n in names:
-        if n.start_pos[0] is not None and n.start_pos < position:
-            names_new.append(n)
-    return names_new
-
-
-def get_names_of_scope(scope, position=None, star_search=True,
-                                                        include_builtin=True):
-    """
-    Get all completions (names) possible for the current scope.
-    The star search option is only here to provide an optimization. Otherwise
-    the whole thing would probably start a little recursive madness.
-
-    This function is used to include names from outer scopes.  For example,
-    when the current scope is function:
-
-    >>> from jedi.parsing import Parser
-    >>> parser = Parser('''
-    ... x = ['a', 'b', 'c']
-    ... def func():
-    ...     y = None
-    ... ''')
-    >>> scope = parser.module.subscopes[0]
-    >>> scope
-    <Function: func@3-5>
-
-    `get_names_of_scope` is a generator.  First it yields names from
-    most inner scope.
-
-    >>> pairs = list(get_names_of_scope(scope))
-    >>> pairs[0]
-    (<Function: func@3-5>, [<Name: y@4,4>])
-
-    Then it yield the names from one level outer scope.  For this
-    example, this is the most outer scope.
-
-    >>> pairs[1]
-    (<SubModule: None@1-5>, [<Name: x@2,0>, <Name: func@3,4>])
-
-    Finally, it yields names from builtin, if `include_builtin` is
-    true (default).
-
-    >>> pairs[2]                                        #doctest: +ELLIPSIS
-    (<Module: ...builtin...>, [<Name: ...>, ...])
-
-    :rtype: [(pr.Scope, [pr.Name])]
-    :return: Return an generator that yields a pair of scope and names.
-    """
-    in_func_scope = scope
-    non_flow = scope.get_parent_until(pr.Flow, reverse=True)
-    while scope:
-        if isinstance(scope, pr.SubModule) and scope.parent:
-            # we don't want submodules to report if we have modules.
-            scope = scope.parent
-            continue
-        # `pr.Class` is used, because the parent is never `Class`.
-        # Ignore the Flows, because the classes and functions care for that.
-        # InstanceElement of Class is ignored, if it is not the start scope.
-        if not (scope != non_flow and scope.isinstance(pr.Class)
-                    or scope.isinstance(pr.Flow)
-                    or scope.isinstance(er.Instance)
-                        and non_flow.isinstance(er.Function)
-                    ):
-            try:
-                if isinstance(scope, er.Instance):
-                    for g in scope.scope_generator():
-                        yield g
-                else:
-                    yield scope, get_defined_names_for_position(scope,
-                                                    position, in_func_scope)
-            except StopIteration:
-                reraise(common.MultiLevelStopIteration, sys.exc_info()[2])
-        if scope.isinstance(pr.ForFlow) and scope.is_list_comp:
-            # is a list comprehension
-            yield scope, scope.get_set_vars(is_internal_call=True)
-
-        scope = scope.parent
-        # This is used, because subscopes (Flow scopes) would distort the
-        # results.
-        if scope and scope.isinstance(er.Function, pr.Function, er.Execution):
-            in_func_scope = scope
-
-    # Add star imports.
-    if star_search:
-        for s in imports.remove_star_imports(non_flow.get_parent_until()):
-            for g in get_names_of_scope(s, star_search=False):
-                yield g
-
-        # Add builtins to the global scope.
-        if include_builtin:
-            builtin_scope = builtin.Builtin.scope
-            yield builtin_scope, builtin_scope.get_defined_names()
-
-
-def find_name(scope, name_str, position=None, search_global=False,
-                                                        is_goto=False):
-    """
-    This is the search function. The most important part to debug.
-    `remove_statements` and `filter_statements` really are the core part of
-    this completion.
-
-    :param position: Position of the last statement -> tuple of line, column
-    :return: List of Names. Their parents are the scopes, they are defined in.
-    :rtype: list
-    """
-    def remove_statements(result):
-        """
-        This is the part where statements are being stripped.
-
-        Due to lazy evaluation, statements like a = func; b = a; b() have to be
-        evaluated.
-        """
-        res_new = []
-        for r in result:
-            add = []
-            if r.isinstance(pr.Statement):
-                check_instance = None
-                if isinstance(r, er.InstanceElement) and r.is_class_var:
-                    check_instance = r.instance
-                    r = r.var
-
-                # Global variables handling.
-                if r.is_global():
-                    for token_name in r.token_list[1:]:
-                        if isinstance(token_name, pr.Name):
-                            add = find_name(r.parent, str(token_name))
-                else:
-                    # generated objects are used within executions, but these
-                    # objects are in functions, and we have to dynamically
-                    # execute first.
-                    if isinstance(r, pr.Param):
-                        func = r.parent
-                        # Instances are typically faked, if the instance is not
-                        # called from outside. Here we check it for __init__
-                        # functions and return.
-                        if isinstance(func, er.InstanceElement) \
-                                and func.instance.is_generated \
-                                and hasattr(func, 'name') \
-                                and str(func.name) == '__init__' \
-                                and r.position_nr > 0:  # 0 would be self
-                            r = func.var.params[r.position_nr]
-
-                        # add docstring knowledge
-                        doc_params = docstrings.follow_param(r)
-                        if doc_params:
-                            res_new += doc_params
-                            continue
-
-                        if not r.is_generated:
-                            res_new += dynamic.search_params(r)
-                            if not r.assignment_details:
-                                # this means that there are no default params,
-                                # so just ignore it.
-                                continue
-
-                    if r.docstr:
-                        res_new.append(r)
-
-                    scopes = follow_statement(r, seek_name=name_str)
-                    add += remove_statements(scopes)
-
-                if check_instance is not None:
-                    # class renames
-                    add = [er.InstanceElement(check_instance, a, True)
-                                if isinstance(a, (er.Function, pr.Function))
-                                else a for a in add]
-                res_new += add
-            else:
-                if isinstance(r, pr.Class):
-                    r = er.Class(r)
-                elif isinstance(r, pr.Function):
-                    r = er.Function(r)
-                if r.isinstance(er.Function):
-                    try:
-                        r = r.get_decorated_func()
-                    except er.DecoratorNotFound:
-                        continue
-                res_new.append(r)
-        debug.dbg('sfn remove, new: %s, old: %s' % (res_new, result))
-        return res_new
-
-    def filter_name(scope_generator):
-        """
-        Filters all variables of a scope (which are defined in the
-        `scope_generator`), until the name fits.
-        """
-        def handle_for_loops(loop):
-            # Take the first statement (for has always only
-            # one, remember `in`). And follow it.
-            if not loop.inputs:
-                return []
-            result = get_iterator_types(follow_statement(loop.inputs[0]))
-            if len(loop.set_vars) > 1:
-                commands = loop.set_stmt.get_commands()
-                # loops with loop.set_vars > 0 only have one command
-                result = assign_tuples(commands[0], result, name_str)
-            return result
-
-        def process(name):
-            """
-            Returns the parent of a name, which means the element which stands
-            behind a name.
-            """
-            result = []
-            no_break_scope = False
-            par = name.parent
-            exc = pr.Class, pr.Function
-            until = lambda: par.parent.parent.get_parent_until(exc)
-
-            if par.isinstance(pr.Flow):
-                if par.command == 'for':
-                    result += handle_for_loops(par)
-                else:
-                    debug.warning('Flow: Why are you here? %s' % par.command)
-            elif par.isinstance(pr.Param) \
-                    and par.parent is not None \
-                    and isinstance(until(), pr.Class) \
-                    and par.position_nr == 0:
-                # This is where self gets added - this happens at another
-                # place, if the var_args are clear. But sometimes the class is
-                # not known. Therefore add a new instance for self. Otherwise
-                # take the existing.
-                if isinstance(scope, er.InstanceElement):
-                    inst = scope.instance
-                else:
-                    inst = er.Instance(er.Class(until()))
-                    inst.is_generated = True
-                result.append(inst)
-            elif par.isinstance(pr.Statement):
-                def is_execution(calls):
-                    for c in calls:
-                        if c.isinstance(pr.Array):
-                            if is_execution(c):
-                                return True
-                        elif c.isinstance(pr.Call):
-                            # Compare start_pos, because names may be different
-                            # because of executions.
-                            if c.name.start_pos == name.start_pos \
-                                                            and c.execution:
-                                return True
-                    return False
-
-                is_exe = False
-                for assignee, op in par.assignment_details:
-                    is_exe |= is_execution(assignee)
-
-                if is_exe:
-                    # filter array[3] = ...
-                    # TODO check executions for dict contents
-                    pass
-                else:
-                    details = par.assignment_details
-                    if details and details[0][1] != '=':
-                        no_break_scope = True
-
-                    # TODO this makes self variables non-breakable. wanted?
-                    if isinstance(name, er.InstanceElement) \
-                                                and not name.is_class_var:
-                        no_break_scope = True
-
-                    result.append(par)
-            else:
-                result.append(par)
-            return result, no_break_scope
-
-        flow_scope = scope
-        result = []
-        # compare func uses the tuple of line/indent = line/column
-        comparison_func = lambda name: (name.start_pos)
-
-        for nscope, name_list in scope_generator:
-            break_scopes = []
-            # here is the position stuff happening (sorting of variables)
-            for name in sorted(name_list, key=comparison_func, reverse=True):
-                p = name.parent.parent if name.parent else None
-                if isinstance(p, er.InstanceElement) \
-                            and isinstance(p.var, pr.Class):
-                    p = p.var
-                if name_str == name.get_code() and p not in break_scopes:
-                    r, no_break_scope = process(name)
-                    if is_goto:
-                        if r:
-                            # Directly assign the name, but there has to be a
-                            # result.
-                            result.append(name)
-                    else:
-                        result += r
-                    # for comparison we need the raw class
-                    s = nscope.base if isinstance(nscope, er.Class) else nscope
-                    # this means that a definition was found and is not e.g.
-                    # in if/else.
-                    if result and not no_break_scope:
-                        if not name.parent or p == s:
-                            break
-                        break_scopes.append(p)
-
-            while flow_scope:
-                # TODO check if result is in scope -> no evaluation necessary
-                n = dynamic.check_flow_information(flow_scope, name_str,
-                                                                    position)
-                if n:
-                    result = n
-                    break
-
-                if result:
-                    break
-                if flow_scope == nscope:
-                    break
-                flow_scope = flow_scope.parent
-            flow_scope = nscope
-            if result:
-                break
-
-        if not result and isinstance(nscope, er.Instance):
-            # __getattr__ / __getattribute__
-            result += check_getattr(nscope, name_str)
-        debug.dbg('sfn filter "%s" in (%s-%s): %s@%s' % (name_str, scope,
-                                                nscope, result, position))
-        return result
-
-    def descriptor_check(result):
-        """Processes descriptors"""
-        res_new = []
-        for r in result:
-            if isinstance(scope, (er.Instance, er.Class)) \
-                                and hasattr(r, 'get_descriptor_return'):
-                # handle descriptors
-                with common.ignored(KeyError):
-                    res_new += r.get_descriptor_return(scope)
-                    continue
-            res_new.append(r)
-        return res_new
-
-    if search_global:
-        scope_generator = get_names_of_scope(scope, position=position)
-    else:
-        if isinstance(scope, er.Instance):
-            scope_generator = scope.scope_generator()
-        else:
-            if isinstance(scope, (er.Class, pr.Module)):
-                # classes are only available directly via chaining?
-                # strange stuff...
-                names = scope.get_defined_names()
-            else:
-                names = get_defined_names_for_position(scope, position)
-            scope_generator = iter([(scope, names)])
-
-    if is_goto:
-        return filter_name(scope_generator)
-    return descriptor_check(remove_statements(filter_name(scope_generator)))
-
-
-def check_getattr(inst, name_str):
-    """Checks for both __getattr__ and __getattribute__ methods"""
-    result = []
-    # str is important to lose the NamePart!
-    module = builtin.Builtin.scope
-    name = pr.Call(module, str(name_str), pr.Call.STRING, (0, 0), inst)
-    with common.ignored(KeyError):
-        result = inst.execute_subscope_by_name('__getattr__', [name])
-    if not result:
-        # this is a little bit special. `__getattribute__` is executed
-        # before anything else. But: I know no use case, where this
-        # could be practical and the jedi would return wrong types. If
-        # you ever have something, let me know!
-        with common.ignored(KeyError):
-            result = inst.execute_subscope_by_name('__getattribute__', [name])
-    return result
-
-
-def get_iterator_types(inputs):
-    """Returns the types of any iterator (arrays, yields, __iter__, etc)."""
-    iterators = []
-    # Take the first statement (for has always only
-    # one, remember `in`). And follow it.
-    for it in inputs:
-        if isinstance(it, (er.Generator, er.Array, dynamic.ArrayInstance)):
-            iterators.append(it)
-        else:
-            if not hasattr(it, 'execute_subscope_by_name'):
-                debug.warning('iterator/for loop input wrong', it)
-                continue
-            try:
-                iterators += it.execute_subscope_by_name('__iter__')
-            except KeyError:
-                debug.warning('iterators: No __iter__ method found.')
-
-    result = []
-    for gen in iterators:
-        if isinstance(gen, er.Array):
-            # Array is a little bit special, since this is an internal
-            # array, but there's also the list builtin, which is
-            # another thing.
-            result += gen.get_index_types()
-        elif isinstance(gen, er.Instance):
-            # __iter__ returned an instance.
-            name = '__next__' if is_py3k else 'next'
-            try:
-                result += gen.execute_subscope_by_name(name)
-            except KeyError:
-                debug.warning('Instance has no __next__ function', gen)
-        else:
-            # is a generator
-            result += gen.iter_content()
-    return result
-
-
-def assign_tuples(tup, results, seek_name):
-    """
-    This is a normal assignment checker. In python functions and other things
-    can return tuples:
-    >>> a, b = 1, ""
-    >>> a, (b, c) = 1, ("", 1.0)
-
-    Here, if `seek_name` is "a", the number type will be returned.
-    The first part (before `=`) is the param tuples, the second one result.
-
-    :type tup: pr.Array
-    """
-    def eval_results(index):
-        types = []
-        for r in results:
-            try:
-                func = r.get_exact_index_types
-            except AttributeError:
-                debug.warning("invalid tuple lookup %s of result %s in %s"
-                                    % (tup, results, seek_name))
-            else:
-                with common.ignored(IndexError):
-                    types += func(index)
-        return types
-
-    result = []
-    for i, stmt in enumerate(tup):
-        # Used in assignments. There is just one call and no other things,
-        # therefore we can just assume, that the first part is important.
-        command = stmt.get_commands()[0]
-
-        if tup.type == pr.Array.NOARRAY:
-
-                # unnessecary braces -> just remove.
-            r = results
-        else:
-            r = eval_results(i)
-
-        # are there still tuples or is it just a Call.
-        if isinstance(command, pr.Array):
-            # These are "sub"-tuples.
-            result += assign_tuples(command, r, seek_name)
-        else:
-            if command.name.names[-1] == seek_name:
-                result += r
-    return result
-
-
-@recursion.RecursionDecorator
-@cache.memoize_default(default=())
-def follow_statement(stmt, seek_name=None):
-    """
-    The starting point of the completion. A statement always owns a call list,
-    which are the calls, that a statement does.
-    In case multiple names are defined in the statement, `seek_name` returns
-    the result for this name.
-
-    :param stmt: A `pr.Statement`.
-    :param seek_name: A string.
-    """
-    debug.dbg('follow_stmt %s (%s)' % (stmt, seek_name))
-    commands = stmt.get_commands()
-    debug.dbg('calls: %s' % commands)
-
-    result = follow_call_list(commands)
-
-    # Assignment checking is only important if the statement defines multiple
-    # variables.
-    if len(stmt.get_set_vars()) > 1 and seek_name and stmt.assignment_details:
-        new_result = []
-        for ass_commands, op in stmt.assignment_details:
-            new_result += assign_tuples(ass_commands[0], result, seek_name)
-        result = new_result
-    return set(result)
-
-
-@common.rethrow_uncaught
-def follow_call_list(call_list, follow_array=False):
-    """
-    `call_list` can be either `pr.Array` or `list of list`.
-    It is used to evaluate a two dimensional object, that has calls, arrays and
-    operators in it.
-    """
-    def evaluate_list_comprehension(lc, parent=None):
-        input = lc.input
-        nested_lc = lc.input.token_list[0]
-        if isinstance(nested_lc, pr.ListComprehension):
-            # is nested LC
-            input = nested_lc.stmt
-        module = input.get_parent_until()
-        # create a for loop, which does the same as list comprehensions
-        loop = pr.ForFlow(module, [input], lc.stmt.start_pos, lc.middle, True)
-
-        loop.parent = parent or lc.get_parent_until(pr.IsScope)
-
-        if isinstance(nested_lc, pr.ListComprehension):
-            loop = evaluate_list_comprehension(nested_lc, loop)
-        return loop
-
-    result = []
-    calls_iterator = iter(call_list)
-    for call in calls_iterator:
-        if pr.Array.is_type(call, pr.Array.NOARRAY):
-            r = list(itertools.chain.from_iterable(follow_statement(s)
-                                                   for s in call))
-            call_path = call.generate_call_path()
-            next(call_path, None)  # the first one has been used already
-            result += follow_paths(call_path, r, call.parent,
-                                  position=call.start_pos)
-        elif isinstance(call, pr.ListComprehension):
-            loop = evaluate_list_comprehension(call)
-            # Caveat: parents are being changed, but this doesn't matter,
-            # because nothing else uses it.
-            call.stmt.parent = loop
-            result += follow_statement(call.stmt)
-        else:
-            if isinstance(call, pr.Lambda):
-                result.append(er.Function(call))
-            # With things like params, these can also be functions...
-            elif isinstance(call, (er.Function, er.Class, er.Instance,
-                                            dynamic.ArrayInstance)):
-                result.append(call)
-            # The string tokens are just operations (+, -, etc.)
-            elif not isinstance(call, (str, unicode)):
-                if str(call.name) == 'if':
-                    # Ternary operators.
-                    while True:
-                        try:
-                            call = next(calls_iterator)
-                        except StopIteration:
-                            break
-                        with common.ignored(AttributeError):
-                            if str(call.name) == 'else':
-                                break
-                    continue
-                result += follow_call(call)
-            elif call == '*':
-                if [r for r in result if isinstance(r, er.Array)
-                                or isinstance(r, er.Instance)
-                                    and str(r.name) == 'str']:
-                    # if it is an iterable, ignore * operations
-                    next(calls_iterator)
-    return set(result)
-
-
-def follow_call(call):
-    """Follow a call is following a function, variable, string, etc."""
-    path = call.generate_call_path()
-
-    # find the statement of the Scope
-    s = call
-    while not s.parent.isinstance(pr.IsScope):
-        s = s.parent
-    return follow_call_path(path, s.parent, s.start_pos)
-
-
-def follow_call_path(path, scope, position):
-    """Follows a path generated by `pr.Call.generate_call_path()`"""
-    current = next(path)
-
-    if isinstance(current, pr.Array):
-        result = [er.Array(current)]
-    else:
-        if isinstance(current, pr.NamePart):
-            # This is the first global lookup.
-            scopes = find_name(scope, current, position=position,
-                                            search_global=True)
-        else:
-            if current.type in (pr.Call.STRING, pr.Call.NUMBER):
-                t = type(current.name).__name__
-                scopes = find_name(builtin.Builtin.scope, t)
-            else:
-                debug.warning('unknown type:', current.type, current)
-                scopes = []
-            # Make instances of those number/string objects.
-            scopes = [er.Instance(s, (current.name,)) for s in scopes]
-        result = imports.strip_imports(scopes)
-
-    return follow_paths(path, result, scope, position=position)
-
-
-def follow_paths(path, results, call_scope, position=None):
-    """
-    In each result, `path` must be followed. Copies the path iterator.
-    """
-    results_new = []
-    if results:
-        if len(results) > 1:
-            iter_paths = itertools.tee(path, len(results))
-        else:
-            iter_paths = [path]
-
-        for i, r in enumerate(results):
-            fp = follow_path(iter_paths[i], r, call_scope, position=position)
-            if fp is not None:
-                results_new += fp
-            else:
-                # This means stop iteration.
-                return results
-    return results_new
-
-
-def follow_path(path, scope, call_scope, position=None):
-    """
-    Uses a generator and tries to complete the path, e.g.::
-
-        foo.bar.baz
-
-    `follow_path` is only responsible for completing `.bar.baz`, the rest is
-    done in the `follow_call` function.
-    """
-    # current is either an Array or a Scope.
-    try:
-        current = next(path)
-    except StopIteration:
-        return None
-    debug.dbg('follow %s in scope %s' % (current, scope))
-
-    result = []
-    if isinstance(current, pr.Array):
-        # This must be an execution, either () or [].
-        if current.type == pr.Array.LIST:
-            if hasattr(scope, 'get_index_types'):
-                result = scope.get_index_types(current)
-        elif current.type not in [pr.Array.DICT]:
-            # Scope must be a class or func - make an instance or execution.
-            debug.dbg('exe', scope)
-            result = er.Execution(scope, current).get_return_types()
-        else:
-            # Curly braces are not allowed, because they make no sense.
-            debug.warning('strange function call with {}', current, scope)
-    else:
-        # The function must not be decorated with something else.
-        if scope.isinstance(er.Function):
-            scope = scope.get_magic_method_scope()
-        else:
-            # This is the typical lookup while chaining things.
-            if filter_private_variable(scope, call_scope, current):
-                return []
-        result = imports.strip_imports(find_name(scope, current,
-                                                    position=position))
-    return follow_paths(path, set(result), call_scope, position=position)
-
-
-def filter_private_variable(scope, call_scope, var_name):
-    """private variables begin with a double underline `__`"""
-    if isinstance(var_name, (str, unicode)) \
-            and var_name.startswith('__') and isinstance(scope, er.Instance):
-        s = call_scope.get_parent_until((pr.Class, er.Instance))
-        if s != scope and s != scope.base.base:
-            return True
-    return False
-
-
-def goto(stmt, call_path=None):
-    if call_path is None:
-        commands = stmt.get_commands()
-        assert len(commands) == 1
-        call = commands[0]
-        call_path = list(call.generate_call_path())
-
-    scope = stmt.get_parent_until(pr.IsScope)
-    pos = stmt.start_pos
-    call_path, search = call_path[:-1], call_path[-1]
-    pos = pos[0], pos[1] + 1
-
-    if call_path:
-        scopes = follow_call_path(iter(call_path), scope, pos)
-        search_global = False
-        pos = None
-    else:
-        scopes = [scope]
-        search_global = True
-    follow_res = []
-    for s in scopes:
-        follow_res += find_name(s, search, pos,
-                                    search_global=search_global, is_goto=True)
-    return follow_res, search

jedi/evaluate/__init__.py

@@ -0,0 +1,379 @@
+"""
+Evaluation of Python code in |jedi| is based on three assumptions:
+
+* The code uses as least side effects as possible. Jedi understands certain
+  list/tuple/set modifications, but there's no guarantee that Jedi detects
+  everything (list.append in different modules for example).
+* No magic is being used:
+
+  - metaclasses
+  - ``setattr()`` / ``__import__()``
+  - writing to ``globals()``, ``locals()``, ``object.__dict__``
+* The programmer is not a total dick, e.g. like `this
+  <https://github.com/davidhalter/jedi/issues/24>`_ :-)
+
+The actual algorithm is based on a principle called lazy evaluation. If you
+don't know about it, google it.  That said, the typical entry point for static
+analysis is calling ``eval_statement``. There's separate logic for
+autocompletion in the API, the evaluator is all about evaluating an expression.
+
+Now you need to understand what follows after ``eval_statement``. Let's
+make an example::
+
+    import datetime
+    datetime.date.toda# <-- cursor here
+
+First of all, this module doesn't care about completion. It really just cares
+about ``datetime.date``. At the end of the procedure ``eval_statement`` will
+return the ``date`` class.
+
+To *visualize* this (simplified):
+
+- ``Evaluator.eval_statement`` doesn't do much, because there's no assignment.
+- ``Evaluator.eval_element`` cares for resolving the dotted path
+- ``Evaluator.find_types`` searches for global definitions of datetime, which
+  it finds in the definition of an import, by scanning the syntax tree.
+- Using the import logic, the datetime module is found.
+- Now ``find_types`` is called again by ``eval_element`` to find ``date``
+  inside the datetime module.
+
+Now what would happen if we wanted ``datetime.date.foo.bar``? Two more
+calls to ``find_types``. However the second call would be ignored, because the
+first one would return nothing (there's no foo attribute in ``date``).
+
+What if the import would contain another ``ExprStmt`` like this::
+
+    from foo import bar
+    Date = bar.baz
+
+Well... You get it. Just another ``eval_statement`` recursion. It's really
+easy. Python can obviously get way more complicated then this. To understand
+tuple assignments, list comprehensions and everything else, a lot more code had
+to be written.
+
+Jedi has been tested very well, so you can just start modifying code. It's best
+to write your own test first for your "new" feature. Don't be scared of
+breaking stuff. As long as the tests pass, you're most likely to be fine.
+
+I need to mention now that lazy evaluation is really good because it
+only *evaluates* what needs to be *evaluated*. All the statements and modules
+that are not used are just being ignored.
+"""
+
+import copy
+from itertools import chain
+
+from jedi.parser import tree
+from jedi import debug
+from jedi.evaluate import representation as er
+from jedi.evaluate import imports
+from jedi.evaluate import recursion
+from jedi.evaluate import iterable
+from jedi.evaluate.cache import memoize_default
+from jedi.evaluate import stdlib
+from jedi.evaluate import finder
+from jedi.evaluate import compiled
+from jedi.evaluate import precedence
+from jedi.evaluate import param
+from jedi.evaluate import helpers
+
+
+class Evaluator(object):
+    def __init__(self, grammar):
+        self.grammar = grammar
+        self.memoize_cache = {}  # for memoize decorators
+        # To memorize modules -> equals `sys.modules`.
+        self.modules = {}  # like `sys.modules`.
+        self.compiled_cache = {}  # see `compiled.create()`
+        self.recursion_detector = recursion.RecursionDetector()
+        self.execution_recursion_detector = recursion.ExecutionRecursionDetector()
+        self.analysis = []
+
+    def wrap(self, element):
+        if isinstance(element, tree.Class):
+            return er.Class(self, element)
+        elif isinstance(element, tree.Function):
+            if isinstance(element, tree.Lambda):
+                return er.LambdaWrapper(self, element)
+            else:
+                return er.Function(self, element)
+        elif isinstance(element, (tree.Module)) \
+                and not isinstance(element, er.ModuleWrapper):
+            return er.ModuleWrapper(self, element)
+        else:
+            return element
+
+    def find_types(self, scope, name_str, position=None, search_global=False,
+                   is_goto=False):
+        """
+        This is the search function. The most important part to debug.
+        `remove_statements` and `filter_statements` really are the core part of
+        this completion.
+
+        :param position: Position of the last statement -> tuple of line, column
+        :return: List of Names. Their parents are the types.
+        """
+        f = finder.NameFinder(self, scope, name_str, position)
+        scopes = f.scopes(search_global)
+        if is_goto:
+            return f.filter_name(scopes)
+        return f.find(scopes, search_global)
+
+    @memoize_default(default=[], evaluator_is_first_arg=True)
+    @recursion.recursion_decorator
+    @debug.increase_indent
+    def eval_statement(self, stmt, seek_name=None):
+        """
+        The starting point of the completion. A statement always owns a call
+        list, which are the calls, that a statement does. In case multiple
+        names are defined in the statement, `seek_name` returns the result for
+        this name.
+
+        :param stmt: A `tree.ExprStmt`.
+        """
+        debug.dbg('eval_statement %s (%s)', stmt, seek_name)
+        types = self.eval_element(stmt.get_rhs())
+
+        if seek_name:
+            types = finder.check_tuple_assignments(types, seek_name)
+
+        first_operation = stmt.first_operation()
+        if first_operation not in ('=', None) and not isinstance(stmt, er.InstanceElement):  # TODO don't check for this.
+            # `=` is always the last character in aug assignments -> -1
+            operator = copy.copy(first_operation)
+            operator.value = operator.value[:-1]
+            name = str(stmt.get_defined_names()[0])
+            parent = self.wrap(stmt.get_parent_scope())
+            left = self.find_types(parent, name, stmt.start_pos, search_global=True)
+            if isinstance(stmt.get_parent_until(tree.ForStmt), tree.ForStmt):
+                # Iterate through result and add the values, that's possible
+                # only in for loops without clutter, because they are
+                # predictable.
+                for r in types:
+                    left = precedence.calculate(self, left, operator, [r])
+                types = left
+            else:
+                types = precedence.calculate(self, left, operator, types)
+        debug.dbg('eval_statement result %s', types)
+        return types
+
+    @memoize_default(evaluator_is_first_arg=True)
+    def eval_element(self, element):
+        if isinstance(element, iterable.AlreadyEvaluated):
+            return list(element)
+        elif isinstance(element, iterable.MergedNodes):
+            return iterable.unite(self.eval_element(e) for e in element)
+
+        debug.dbg('eval_element %s@%s', element, element.start_pos)
+        if isinstance(element, (tree.Name, tree.Literal)) or tree.is_node(element, 'atom'):
+            return self._eval_atom(element)
+        elif isinstance(element, tree.Keyword):
+            # For False/True/None
+            if element.value in ('False', 'True', 'None'):
+                return [compiled.builtin.get_by_name(element.value)]
+            else:
+                return []
+        elif element.isinstance(tree.Lambda):
+            return [er.LambdaWrapper(self, element)]
+        elif element.isinstance(er.LambdaWrapper):
+            return [element]  # TODO this is no real evaluation.
+        elif element.type == 'expr_stmt':
+            return self.eval_statement(element)
+        elif element.type == 'power':
+            types = self._eval_atom(element.children[0])
+            for trailer in element.children[1:]:
+                if trailer == '**':  # has a power operation.
+                    raise NotImplementedError
+                types = self.eval_trailer(types, trailer)
+
+            return types
+        elif element.type in ('testlist_star_expr', 'testlist',):
+            # The implicit tuple in statements.
+            return [iterable.ImplicitTuple(self, element)]
+        elif element.type in ('not_test', 'factor'):
+            types = self.eval_element(element.children[-1])
+            for operator in element.children[:-1]:
+                types = list(precedence.factor_calculate(self, types, operator))
+            return types
+        elif element.type == 'test':
+            # `x if foo else y` case.
+            return (self.eval_element(element.children[0]) +
+                    self.eval_element(element.children[-1]))
+        elif element.type == 'operator':
+            # Must be an ellipsis, other operators are not evaluated.
+            return []  # Ignore for now.
+        elif element.type == 'dotted_name':
+            types = self._eval_atom(element.children[0])
+            for next_name in element.children[2::2]:
+                types = list(chain.from_iterable(self.find_types(typ, next_name)
+                                                 for typ in types))
+            return types
+        else:
+            return precedence.calculate_children(self, element.children)
+
+    def _eval_atom(self, atom):
+        """
+        Basically to process ``atom`` nodes. The parser sometimes doesn't
+        generate the node (because it has just one child). In that case an atom
+        might be a name or a literal as well.
+        """
+        if isinstance(atom, tree.Name):
+            # This is the first global lookup.
+            stmt = atom.get_definition()
+            scope = stmt.get_parent_until(tree.IsScope, include_current=True)
+            if isinstance(stmt, tree.CompFor):
+                stmt = stmt.get_parent_until((tree.ClassOrFunc, tree.ExprStmt))
+            if stmt.type != 'expr_stmt':
+                # We only need to adjust the start_pos for statements, because
+                # there the name cannot be used.
+                stmt = atom
+            return self.find_types(scope, atom, stmt.start_pos, search_global=True)
+        elif isinstance(atom, tree.Literal):
+            return [compiled.create(self, atom.eval())]
+        else:
+            c = atom.children
+            # Parentheses without commas are not tuples.
+            if c[0] == '(' and not len(c) == 2 \
+                    and not(tree.is_node(c[1], 'testlist_comp')
+                            and len(c[1].children) > 1):
+                return self.eval_element(c[1])
+            try:
+                comp_for = c[1].children[1]
+            except (IndexError, AttributeError):
+                pass
+            else:
+                if isinstance(comp_for, tree.CompFor):
+                    return [iterable.Comprehension.from_atom(self, atom)]
+            return [iterable.Array(self, atom)]
+
+    def eval_trailer(self, types, trailer):
+        trailer_op, node = trailer.children[:2]
+        if node == ')':  # `arglist` is optional.
+            node = ()
+        new_types = []
+        for typ in types:
+            debug.dbg('eval_trailer: %s in scope %s', trailer, typ)
+            if trailer_op == '.':
+                new_types += self.find_types(typ, node)
+            elif trailer_op == '(':
+                new_types += self.execute(typ, node, trailer)
+            elif trailer_op == '[':
+                try:
+                    get = typ.get_index_types
+                except AttributeError:
+                    debug.warning("TypeError: '%s' object is not subscriptable"
+                                  % typ)
+                else:
+                    new_types += get(self, node)
+        return new_types
+
+    def execute_evaluated(self, obj, *args):
+        """
+        Execute a function with already executed arguments.
+        """
+        args = [iterable.AlreadyEvaluated([arg]) for arg in args]
+        return self.execute(obj, args)
+
+    @debug.increase_indent
+    def execute(self, obj, arguments=(), trailer=None):
+        if not isinstance(arguments, param.Arguments):
+            arguments = param.Arguments(self, arguments, trailer)
+
+        if obj.isinstance(er.Function):
+            obj = obj.get_decorated_func()
+
+        debug.dbg('execute: %s %s', obj, arguments)
+        try:
+            # Some stdlib functions like super(), namedtuple(), etc. have been
+            # hard-coded in Jedi to support them.
+            return stdlib.execute(self, obj, arguments)
+        except stdlib.NotInStdLib:
+            pass
+
+        try:
+            func = obj.py__call__
+        except AttributeError:
+            debug.warning("no execution possible %s", obj)
+            return []
+        else:
+            types = func(self, arguments)
+            debug.dbg('execute result: %s in %s', types, obj)
+            return types
+
+    def goto_definition(self, name):
+        def_ = name.get_definition()
+        if def_.type == 'expr_stmt' and name in def_.get_defined_names():
+            return self.eval_statement(def_, name)
+        call = helpers.call_of_name(name)
+        return self.eval_element(call)
+
+    def goto(self, name):
+        def resolve_implicit_imports(names):
+            for name in names:
+                if isinstance(name.parent, helpers.FakeImport):
+                    # Those are implicit imports.
+                    s = imports.ImportWrapper(self, name)
+                    for n in s.follow(is_goto=True):
+                        yield n
+                else:
+                    yield name
+
+        stmt = name.get_definition()
+        par = name.parent
+        if par.type == 'argument' and par.children[1] == '=' and par.children[0] == name:
+            # Named param goto.
+            trailer = par.parent
+            if trailer.type == 'arglist':
+                trailer = trailer.parent
+            if trailer.type != 'classdef':
+                if trailer.type == 'decorator':
+                    types = self.eval_element(trailer.children[1])
+                else:
+                    i = trailer.parent.children.index(trailer)
+                    to_evaluate = trailer.parent.children[:i]
+                    types = self.eval_element(to_evaluate[0])
+                    for trailer in to_evaluate[1:]:
+                        types = self.eval_trailer(types, trailer)
+                param_names = []
+                for typ in types:
+                    try:
+                        params = typ.params
+                    except AttributeError:
+                        pass
+                    else:
+                        param_names += [param.name for param in params
+                                        if param.name.value == name.value]
+                return param_names
+        elif isinstance(par, tree.ExprStmt) and name in par.get_defined_names():
+            # Only take the parent, because if it's more complicated than just
+            # a name it's something you can "goto" again.
+            return [name]
+        elif isinstance(par, (tree.Param, tree.Function, tree.Class)) and par.name is name:
+            return [name]
+        elif isinstance(stmt, tree.Import):
+            modules = imports.ImportWrapper(self, name).follow(is_goto=True)
+            return list(resolve_implicit_imports(modules))
+        elif par.type == 'dotted_name':  # Is a decorator.
+            index = par.children.index(name)
+            if index > 0:
+                new_dotted = helpers.deep_ast_copy(par)
+                new_dotted.children[index - 1:] = []
+                types = self.eval_element(new_dotted)
+                return resolve_implicit_imports(iterable.unite(
+                    self.find_types(typ, name, is_goto=True) for typ in types
+                ))
+
+        scope = name.get_parent_scope()
+        if tree.is_node(name.parent, 'trailer'):
+            call = helpers.call_of_name(name, cut_own_trailer=True)
+            types = self.eval_element(call)
+            return resolve_implicit_imports(iterable.unite(
+                self.find_types(typ, name, is_goto=True) for typ in types
+            ))
+        else:
+            if stmt.type != 'expr_stmt':
+                # We only need to adjust the start_pos for statements, because
+                # there the name cannot be used.
+                stmt = name
+            return self.find_types(scope, name, stmt.start_pos,
+                                   search_global=True, is_goto=True)

jedi/evaluate/analysis.py

@@ -0,0 +1,302 @@
+"""
+Module for statical analysis.
+"""
+from jedi import debug
+from jedi.parser import tree
+from jedi.evaluate.compiled import CompiledObject
+
+
+CODES = {
+    'attribute-error': (1, AttributeError, 'Potential AttributeError.'),
+    'name-error': (2, NameError, 'Potential NameError.'),
+    'import-error': (3, ImportError, 'Potential ImportError.'),
+    'type-error-generator': (4, TypeError, "TypeError: 'generator' object is not subscriptable."),
+    'type-error-too-many-arguments': (5, TypeError, None),
+    'type-error-too-few-arguments': (6, TypeError, None),
+    'type-error-keyword-argument': (7, TypeError, None),
+    'type-error-multiple-values': (8, TypeError, None),
+    'type-error-star-star': (9, TypeError, None),
+    'type-error-star': (10, TypeError, None),
+    'type-error-operation': (11, TypeError, None),
+}
+
+
+class Error(object):
+    def __init__(self, name, module_path, start_pos, message=None):
+        self.path = module_path
+        self._start_pos = start_pos
+        self.name = name
+        if message is None:
+            message = CODES[self.name][2]
+        self.message = message
+
+    @property
+    def line(self):
+        return self._start_pos[0]
+
+    @property
+    def column(self):
+        return self._start_pos[1]
+
+    @property
+    def code(self):
+        # The class name start
+        first = self.__class__.__name__[0]
+        return first + str(CODES[self.name][0])
+
+    def __unicode__(self):
+        return '%s:%s:%s: %s %s' % (self.path, self.line, self.column,
+                                    self.code, self.message)
+
+    def __str__(self):
+        return self.__unicode__()
+
+    def __eq__(self, other):
+        return (self.path == other.path and self.name == other.name
+                and self._start_pos == other._start_pos)
+
+    def __ne__(self, other):
+        return not self.__eq__(other)
+
+    def __hash__(self):
+        return hash((self.path, self._start_pos, self.name))
+
+    def __repr__(self):
+        return '<%s %s: %s@%s,%s>' % (self.__class__.__name__,
+                                      self.name, self.path,
+                                      self._start_pos[0], self._start_pos[1])
+
+
+class Warning(Error):
+    pass
+
+
+def add(evaluator, name, jedi_obj, message=None, typ=Error, payload=None):
+    from jedi.evaluate.iterable import MergedNodes
+    while isinstance(jedi_obj, MergedNodes):
+        if len(jedi_obj) != 1:
+            # TODO is this kosher?
+            return
+        jedi_obj = list(jedi_obj)[0]
+
+    exception = CODES[name][1]
+    if _check_for_exception_catch(evaluator, jedi_obj, exception, payload):
+        return
+
+    module_path = jedi_obj.get_parent_until().path
+    instance = typ(name, module_path, jedi_obj.start_pos, message)
+    debug.warning(str(instance))
+    evaluator.analysis.append(instance)
+
+
+def _check_for_setattr(instance):
+    """
+    Check if there's any setattr method inside an instance. If so, return True.
+    """
+    module = instance.get_parent_until()
+    try:
+        stmts = module.used_names['setattr']
+    except KeyError:
+        return False
+
+    return any(instance.start_pos < stmt.start_pos < instance.end_pos
+               for stmt in stmts)
+
+
+def add_attribute_error(evaluator, scope, name):
+    message = ('AttributeError: %s has no attribute %s.' % (scope, name))
+    from jedi.evaluate.representation import Instance
+    # Check for __getattr__/__getattribute__ existance and issue a warning
+    # instead of an error, if that happens.
+    if isinstance(scope, Instance):
+        typ = Warning
+        try:
+            scope.get_subscope_by_name('__getattr__')
+        except KeyError:
+            try:
+                scope.get_subscope_by_name('__getattribute__')
+            except KeyError:
+                if not _check_for_setattr(scope):
+                    typ = Error
+    else:
+        typ = Error
+
+    payload = scope, name
+    add(evaluator, 'attribute-error', name, message, typ, payload)
+
+
+def _check_for_exception_catch(evaluator, jedi_obj, exception, payload=None):
+    """
+    Checks if a jedi object (e.g. `Statement`) sits inside a try/catch and
+    doesn't count as an error (if equal to `exception`).
+    Also checks `hasattr` for AttributeErrors and uses the `payload` to compare
+    it.
+    Returns True if the exception was catched.
+    """
+    def check_match(cls, exception):
+        try:
+            return isinstance(cls, CompiledObject) and issubclass(exception, cls.obj)
+        except TypeError:
+            return False
+
+    def check_try_for_except(obj, exception):
+        # Only nodes in try
+        iterator = iter(obj.children)
+        for branch_type in iterator:
+            colon = next(iterator)
+            suite = next(iterator)
+            if branch_type == 'try' \
+                    and not (branch_type.start_pos < jedi_obj.start_pos <= suite.end_pos):
+                return False
+
+        for node in obj.except_clauses():
+            if node is None:
+                return True  # An exception block that catches everything.
+            else:
+                except_classes = evaluator.eval_element(node)
+                for cls in except_classes:
+                    from jedi.evaluate import iterable
+                    if isinstance(cls, iterable.Array) and cls.type == 'tuple':
+                        # multiple exceptions
+                        for c in cls.values():
+                            if check_match(c, exception):
+                                return True
+                    else:
+                        if check_match(cls, exception):
+                            return True
+
+    def check_hasattr(node, suite):
+        try:
+            assert suite.start_pos <= jedi_obj.start_pos < suite.end_pos
+            assert node.type == 'power'
+            base = node.children[0]
+            assert base.type == 'name' and base.value == 'hasattr'
+            trailer = node.children[1]
+            assert trailer.type == 'trailer'
+            arglist = trailer.children[1]
+            assert arglist.type == 'arglist'
+            from jedi.evaluate.param import Arguments
+            args = list(Arguments(evaluator, arglist).unpack())
+            # Arguments should be very simple
+            assert len(args) == 2
+
+            # Check name
+            key, values = args[1]
+            assert len(values) == 1
+            names = evaluator.eval_element(values[0])
+            assert len(names) == 1 and isinstance(names[0], CompiledObject)
+            assert names[0].obj == str(payload[1])
+
+            # Check objects
+            key, values = args[0]
+            assert len(values) == 1
+            objects = evaluator.eval_element(values[0])
+            return payload[0] in objects
+        except AssertionError:
+            return False
+
+    obj = jedi_obj
+    while obj is not None and not obj.isinstance(tree.Function, tree.Class):
+        if obj.isinstance(tree.Flow):
+            # try/except catch check
+            if obj.isinstance(tree.TryStmt) and check_try_for_except(obj, exception):
+                return True
+            # hasattr check
+            if exception == AttributeError and obj.isinstance(tree.IfStmt, tree.WhileStmt):
+                if check_hasattr(obj.children[1], obj.children[3]):
+                    return True
+        obj = obj.parent
+
+    return False
+
+
+def get_module_statements(module):
+    """
+    Returns the statements used in a module. All these statements should be
+    evaluated to check for potential exceptions.
+    """
+    def check_children(node):
+        try:
+            children = node.children
+        except AttributeError:
+            return []
+        else:
+            nodes = []
+            for child in children:
+                nodes += check_children(child)
+                if child.type == 'trailer':
+                    c = child.children
+                    if c[0] == '(' and c[1] != ')':
+                        if c[1].type != 'arglist':
+                            if c[1].type == 'argument':
+                                nodes.append(c[1].children[-1])
+                            else:
+                                nodes.append(c[1])
+                        else:
+                            for argument in c[1].children:
+                                if argument.type == 'argument':
+                                    nodes.append(argument.children[-1])
+                                elif argument.type != 'operator':
+                                    nodes.append(argument)
+            return nodes
+
+    def add_nodes(nodes):
+        new = set()
+        for node in nodes:
+            if isinstance(node, tree.Flow):
+                children = node.children
+                if node.type == 'for_stmt':
+                    children = children[2:]  # Don't want to include the names.
+                # Pick the suite/simple_stmt.
+                new |= add_nodes(children)
+            elif node.type in ('simple_stmt', 'suite'):
+                new |= add_nodes(node.children)
+            elif node.type in ('return_stmt', 'yield_expr'):
+                try:
+                    new.add(node.children[1])
+                except IndexError:
+                    pass
+            elif node.type not in ('whitespace', 'operator', 'keyword',
+                                   'parameters', 'decorated', 'except_clause') \
+                    and not isinstance(node, (tree.ClassOrFunc, tree.Import)):
+                new.add(node)
+
+                try:
+                    children = node.children
+                except AttributeError:
+                    pass
+                else:
+                    for next_node in children:
+                        new.update(check_children(node))
+                        if next_node.type != 'keyword' and node.type != 'expr_stmt':
+                            new.add(node)
+        return new
+
+    nodes = set()
+    import_names = set()
+    decorated_funcs = []
+    for scope in module.walk():
+        for imp in set(scope.imports):
+            import_names |= set(imp.get_defined_names())
+            if imp.is_nested():
+                import_names |= set(path[-1] for path in imp.paths())
+
+        children = scope.children
+        if isinstance(scope, tree.ClassOrFunc):
+            children = children[2:]  # We don't want to include the class name.
+        nodes |= add_nodes(children)
+
+        for flow in scope.flows:
+            if flow.type == 'for_stmt':
+                nodes.add(flow.children[3])
+            elif flow.type == 'try_stmt':
+                nodes.update(e for e in flow.except_clauses() if e is not None)
+
+        try:
+            decorators = scope.get_decorators()
+        except AttributeError:
+            pass
+        else:
+            if decorators:
+                decorated_funcs.append(scope)
+    return nodes, import_names, decorated_funcs

jedi/evaluate/cache.py

@@ -0,0 +1,58 @@
+"""
+- the popular ``memoize_default`` works like a typical memoize and returns the
+  default otherwise.
+- ``CachedMetaClass`` uses ``memoize_default`` to do the same with classes.
+"""
+
+import inspect
+
+NO_DEFAULT = object()
+
+
+def memoize_default(default=NO_DEFAULT, evaluator_is_first_arg=False, second_arg_is_evaluator=False):
+    """ This is a typical memoization decorator, BUT there is one difference:
+    To prevent recursion it sets defaults.
+
+    Preventing recursion is in this case the much bigger use than speed. I
+    don't think, that there is a big speed difference, but there are many cases
+    where recursion could happen (think about a = b; b = a).
+    """
+    def func(function):
+        def wrapper(obj, *args, **kwargs):
+            if evaluator_is_first_arg:
+                cache = obj.memoize_cache
+            elif second_arg_is_evaluator:  # needed for meta classes
+                cache = args[0].memoize_cache
+            else:
+                cache = obj._evaluator.memoize_cache
+
+            try:
+                memo = cache[function]
+            except KeyError:
+                memo = {}
+                cache[function] = memo
+
+            key = (obj, args, frozenset(kwargs.items()))
+            if key in memo:
+                return memo[key]
+            else:
+                if default is not NO_DEFAULT:
+                    memo[key] = default
+                rv = function(obj, *args, **kwargs)
+                if inspect.isgenerator(rv):
+                    rv = list(rv)
+                memo[key] = rv
+                return rv
+        return wrapper
+    return func
+
+
+class CachedMetaClass(type):
+    """
+    This is basically almost the same than the decorator above, it just caches
+    class initializations. Either you do it this way or with decorators, but
+    with decorators you lose class access (isinstance, etc).
+    """
+    @memoize_default(None, second_arg_is_evaluator=True)
+    def __call__(self, *args, **kwargs):
+        return super(CachedMetaClass, self).__call__(*args, **kwargs)

jedi/evaluate/compiled/__init__.py

@@ -0,0 +1,531 @@
+"""
+Imitate the parser representation.
+"""
+import inspect
+import re
+import sys
+import os
+from functools import partial
+
+from jedi._compatibility import builtins as _builtins, unicode
+from jedi import debug
+from jedi.cache import underscore_memoization, memoize_method
+from jedi.evaluate.sys_path import get_sys_path
+from jedi.parser.tree import Param, Base, Operator, zero_position_modifier
+from jedi.evaluate.helpers import FakeName
+from . import fake
+
+
+_sep = os.path.sep
+if os.path.altsep is not None:
+    _sep += os.path.altsep
+_path_re = re.compile('(?:\.[^{0}]+|[{0}]__init__\.py)$'.format(re.escape(_sep)))
+del _sep
+
+
+class CheckAttribute(object):
+    """Raises an AttributeError if the attribute X isn't available."""
+    def __init__(self, func):
+        self.func = func
+        # Remove the py in front of e.g. py__call__.
+        self.check_name = func.__name__[2:]
+
+    def __get__(self, instance, owner):
+        # This might raise an AttributeError. That's wanted.
+        getattr(instance.obj, self.check_name)
+        return partial(self.func, instance)
+
+
+class CompiledObject(Base):
+    # comply with the parser
+    start_pos = 0, 0
+    path = None  # modules have this attribute - set it to None.
+    used_names = {}  # To be consistent with modules.
+
+    def __init__(self, obj, parent=None):
+        self.obj = obj
+        self.parent = parent
+
+    @property
+    def py__call__(self):
+        def actual(evaluator, params):
+            if inspect.isclass(self.obj):
+                from jedi.evaluate.representation import Instance
+                return [Instance(evaluator, self, params)]
+            else:
+                return list(self._execute_function(evaluator, params))
+
+        # Might raise an AttributeError, which is intentional.
+        self.obj.__call__
+        return actual
+
+    @CheckAttribute
+    def py__class__(self, evaluator):
+        return CompiledObject(self.obj.__class__, parent=self.parent)
+
+    @CheckAttribute
+    def py__mro__(self, evaluator):
+        return tuple(create(evaluator, cls, self.parent) for cls in self.obj.__mro__)
+
+    @CheckAttribute
+    def py__bases__(self, evaluator):
+        return tuple(create(evaluator, cls) for cls in self.obj.__bases__)
+
+    def py__bool__(self):
+        return bool(self.obj)
+
+    def py__file__(self):
+        return self.obj.__file__
+
+    def is_class(self):
+        return inspect.isclass(self.obj)
+
+    @property
+    def doc(self):
+        return inspect.getdoc(self.obj) or ''
+
+    @property
+    def params(self):
+        params_str, ret = self._parse_function_doc()
+        tokens = params_str.split(',')
+        if inspect.ismethoddescriptor(self._cls().obj):
+            tokens.insert(0, 'self')
+        params = []
+        for p in tokens:
+            parts = [FakeName(part) for part in p.strip().split('=')]
+            if len(parts) > 1:
+                parts.insert(1, Operator(zero_position_modifier, '=', (0, 0)))
+            params.append(Param(parts, self))
+        return params
+
+    def __repr__(self):
+        return '<%s: %s>' % (type(self).__name__, repr(self.obj))
+
+    @underscore_memoization
+    def _parse_function_doc(self):
+        if self.doc is None:
+            return '', ''
+
+        return _parse_function_doc(self.doc)
+
+    def api_type(self):
+        if fake.is_class_instance(self.obj):
+            return 'instance'
+
+        cls = self._cls().obj
+        if inspect.isclass(cls):
+            return 'class'
+        elif inspect.ismodule(cls):
+            return 'module'
+        elif inspect.isbuiltin(cls) or inspect.ismethod(cls) \
+                or inspect.ismethoddescriptor(cls):
+            return 'function'
+
+    @property
+    def type(self):
+        """Imitate the tree.Node.type values."""
+        cls = self._cls().obj
+        if inspect.isclass(cls):
+            return 'classdef'
+        elif inspect.ismodule(cls):
+            return 'file_input'
+        elif inspect.isbuiltin(cls) or inspect.ismethod(cls) \
+                or inspect.ismethoddescriptor(cls):
+            return 'funcdef'
+
+    @underscore_memoization
+    def _cls(self):
+        # Ensures that a CompiledObject is returned that is not an instance (like list)
+        if fake.is_class_instance(self.obj):
+            try:
+                c = self.obj.__class__
+            except AttributeError:
+                # happens with numpy.core.umath._UFUNC_API (you get it
+                # automatically by doing `import numpy`.
+                c = type(None)
+            return CompiledObject(c, self.parent)
+        return self
+
+    @property
+    def names_dict(self):
+        # For compatibility with `representation.Class`.
+        return self.names_dicts(False)[0]
+
+    def names_dicts(self, search_global, is_instance=False):
+        return self._names_dict_ensure_one_dict(is_instance)
+
+    @memoize_method
+    def _names_dict_ensure_one_dict(self, is_instance):
+        """
+        search_global shouldn't change the fact that there's one dict, this way
+        there's only one `object`.
+        """
+        return [LazyNamesDict(self._cls(), is_instance)]
+
+    def get_subscope_by_name(self, name):
+        if name in dir(self._cls().obj):
+            return CompiledName(self._cls(), name).parent
+        else:
+            raise KeyError("CompiledObject doesn't have an attribute '%s'." % name)
+
+    def get_index_types(self, evaluator, index_array=()):
+        # If the object doesn't have `__getitem__`, just raise the
+        # AttributeError.
+        if not hasattr(self.obj, '__getitem__'):
+            debug.warning('Tried to call __getitem__ on non-iterable.')
+            return []
+        if type(self.obj) not in (str, list, tuple, unicode, bytes, bytearray, dict):
+            # Get rid of side effects, we won't call custom `__getitem__`s.
+            return []
+
+        result = []
+        from jedi.evaluate.iterable import create_indexes_or_slices
+        for typ in create_indexes_or_slices(evaluator, index_array):
+            index = None
+            try:
+                index = typ.obj
+                new = self.obj[index]
+            except (KeyError, IndexError, TypeError, AttributeError):
+                # Just try, we don't care if it fails, except for slices.
+                if isinstance(index, slice):
+                    result.append(self)
+            else:
+                result.append(CompiledObject(new))
+        if not result:
+            try:
+                for obj in self.obj:
+                    result.append(CompiledObject(obj))
+            except TypeError:
+                pass  # self.obj maynot have an __iter__ method.
+        return result
+
+    @property
+    def name(self):
+        # might not exist sometimes (raises AttributeError)
+        return FakeName(self._cls().obj.__name__, self)
+
+    def _execute_function(self, evaluator, params):
+        if self.type != 'funcdef':
+            return
+
+        for name in self._parse_function_doc()[1].split():
+            try:
+                bltn_obj = _create_from_name(builtin, builtin, name)
+            except AttributeError:
+                continue
+            else:
+                if isinstance(bltn_obj, CompiledObject) and bltn_obj.obj is None:
+                    # We want everything except None.
+                    continue
+                for result in evaluator.execute(bltn_obj, params):
+                    yield result
+
+    @property
+    @underscore_memoization
+    def subscopes(self):
+        """
+        Returns only the faked scopes - the other ones are not important for
+        internal analysis.
+        """
+        module = self.get_parent_until()
+        faked_subscopes = []
+        for name in dir(self._cls().obj):
+            f = fake.get_faked(module.obj, self.obj, name)
+            if f:
+                f.parent = self
+                faked_subscopes.append(f)
+        return faked_subscopes
+
+    def is_scope(self):
+        return True
+
+    def get_self_attributes(self):
+        return []  # Instance compatibility
+
+    def get_imports(self):
+        return []  # Builtins don't have imports
+
+
+class LazyNamesDict(object):
+    """
+    A names_dict instance for compiled objects, resembles the parser.tree.
+    """
+    def __init__(self, compiled_obj, is_instance):
+        self._compiled_obj = compiled_obj
+        self._is_instance = is_instance
+
+    def __iter__(self):
+        return (v[0].value for v in self.values())
+
+    @memoize_method
+    def __getitem__(self, name):
+        try:
+            getattr(self._compiled_obj.obj, name)
+        except AttributeError:
+            raise KeyError('%s in %s not found.' % (name, self._compiled_obj))
+        return [CompiledName(self._compiled_obj, name)]
+
+    def values(self):
+        obj = self._compiled_obj.obj
+
+        values = []
+        for name in dir(obj):
+            try:
+                values.append(self[name])
+            except KeyError:
+                # The dir function can be wrong.
+                pass
+
+        # dir doesn't include the type names.
+        if not inspect.ismodule(obj) and obj != type and not self._is_instance:
+            values += _type_names_dict.values()
+        return values
+
+
+class CompiledName(FakeName):
+    def __init__(self, obj, name):
+        super(CompiledName, self).__init__(name)
+        self._obj = obj
+        self.name = name
+
+    def __repr__(self):
+        try:
+            name = self._obj.name  # __name__ is not defined all the time
+        except AttributeError:
+            name = None
+        return '<%s: (%s).%s>' % (type(self).__name__, name, self.name)
+
+    def is_definition(self):
+        return True
+
+    @property
+    @underscore_memoization
+    def parent(self):
+        module = self._obj.get_parent_until()
+        return _create_from_name(module, self._obj, self.name)
+
+    @parent.setter
+    def parent(self, value):
+        pass  # Just ignore this, FakeName tries to overwrite the parent attribute.
+
+
+def dotted_from_fs_path(fs_path, sys_path=None):
+    """
+    Changes `/usr/lib/python3.4/email/utils.py` to `email.utils`.  I.e.
+    compares the path with sys.path and then returns the dotted_path. If the
+    path is not in the sys.path, just returns None.
+    """
+    if sys_path is None:
+        sys_path = get_sys_path()
+
+    if os.path.basename(fs_path).startswith('__init__.'):
+        # We are calculating the path. __init__ files are not interesting.
+        fs_path = os.path.dirname(fs_path)
+
+    # prefer
+    #   - UNIX
+    #     /path/to/pythonX.Y/lib-dynload
+    #     /path/to/pythonX.Y/site-packages
+    #   - Windows
+    #     C:\path\to\DLLs
+    #     C:\path\to\Lib\site-packages
+    # over
+    #   - UNIX
+    #     /path/to/pythonX.Y
+    #   - Windows
+    #     C:\path\to\Lib
+    path = ''
+    for s in sys_path:
+        if (fs_path.startswith(s) and len(path) < len(s)):
+            path = s
+    return _path_re.sub('', fs_path[len(path):].lstrip(os.path.sep)).replace(os.path.sep, '.')
+
+
+def load_module(path=None, name=None):
+    if path is not None:
+        dotted_path = dotted_from_fs_path(path)
+    else:
+        dotted_path = name
+
+    sys_path = get_sys_path()
+    if dotted_path is None:
+        p, _, dotted_path = path.partition(os.path.sep)
+        sys_path.insert(0, p)
+
+    temp, sys.path = sys.path, sys_path
+    try:
+        __import__(dotted_path)
+    except RuntimeError:
+        if 'PySide' in dotted_path or 'PyQt' in dotted_path:
+            # RuntimeError: the PyQt4.QtCore and PyQt5.QtCore modules both wrap
+            # the QObject class.
+            # See https://github.com/davidhalter/jedi/pull/483
+            return None
+        raise
+    except ImportError:
+        # If a module is "corrupt" or not really a Python module or whatever.
+        debug.warning('Module %s not importable.', path)
+        return None
+    finally:
+        sys.path = temp
+
+    # Just access the cache after import, because of #59 as well as the very
+    # complicated import structure of Python.
+    module = sys.modules[dotted_path]
+
+    return CompiledObject(module)
+
+
+docstr_defaults = {
+    'floating point number': 'float',
+    'character': 'str',
+    'integer': 'int',
+    'dictionary': 'dict',
+    'string': 'str',
+}
+
+
+def _parse_function_doc(doc):
+    """
+    Takes a function and returns the params and return value as a tuple.
+    This is nothing more than a docstring parser.
+
+    TODO docstrings like utime(path, (atime, mtime)) and a(b [, b]) -> None
+    TODO docstrings like 'tuple of integers'
+    """
+    # parse round parentheses: def func(a, (b,c))
+    try:
+        count = 0
+        start = doc.index('(')
+        for i, s in enumerate(doc[start:]):
+            if s == '(':
+                count += 1
+            elif s == ')':
+                count -= 1
+            if count == 0:
+                end = start + i
+                break
+        param_str = doc[start + 1:end]
+    except (ValueError, UnboundLocalError):
+        # ValueError for doc.index
+        # UnboundLocalError for undefined end in last line
+        debug.dbg('no brackets found - no param')
+        end = 0
+        param_str = ''
+    else:
+        # remove square brackets, that show an optional param ( = None)
+        def change_options(m):
+            args = m.group(1).split(',')
+            for i, a in enumerate(args):
+                if a and '=' not in a:
+                    args[i] += '=None'
+            return ','.join(args)
+
+        while True:
+            param_str, changes = re.subn(r' ?\[([^\[\]]+)\]',
+                                         change_options, param_str)
+            if changes == 0:
+                break
+    param_str = param_str.replace('-', '_')  # see: isinstance.__doc__
+
+    # parse return value
+    r = re.search('-[>-]* ', doc[end:end + 7])
+    if r is None:
+        ret = ''
+    else:
+        index = end + r.end()
+        # get result type, which can contain newlines
+        pattern = re.compile(r'(,\n|[^\n-])+')
+        ret_str = pattern.match(doc, index).group(0).strip()
+        # New object -> object()
+        ret_str = re.sub(r'[nN]ew (.*)', r'\1()', ret_str)
+
+        ret = docstr_defaults.get(ret_str, ret_str)
+
+    return param_str, ret
+
+
+class Builtin(CompiledObject):
+    @memoize_method
+    def get_by_name(self, name):
+        return self.names_dict[name][0].parent
+
+
+def _a_generator(foo):
+    """Used to have an object to return for generators."""
+    yield 42
+    yield foo
+
+
+def _create_from_name(module, parent, name):
+    faked = fake.get_faked(module.obj, parent.obj, name)
+    # only functions are necessary.
+    if faked is not None:
+        faked.parent = parent
+        return faked
+
+    try:
+        obj = getattr(parent.obj, name)
+    except AttributeError:
+        # happens e.g. in properties of
+        # PyQt4.QtGui.QStyleOptionComboBox.currentText
+        # -> just set it to None
+        obj = None
+    return CompiledObject(obj, parent)
+
+
+builtin = Builtin(_builtins)
+magic_function_class = CompiledObject(type(load_module), parent=builtin)
+generator_obj = CompiledObject(_a_generator(1.0))
+_type_names_dict = builtin.get_by_name('type').names_dict
+none_obj = builtin.get_by_name('None')
+false_obj = builtin.get_by_name('False')
+true_obj = builtin.get_by_name('True')
+object_obj = builtin.get_by_name('object')
+
+
+def keyword_from_value(obj):
+    if obj is None:
+        return none_obj
+    elif obj is False:
+        return false_obj
+    elif obj is True:
+        return true_obj
+    else:
+        raise NotImplementedError
+
+
+def compiled_objects_cache(func):
+    def wrapper(evaluator, obj, parent=builtin, module=None):
+        # Do a very cheap form of caching here.
+        key = id(obj), id(parent), id(module)
+        try:
+            return evaluator.compiled_cache[key][0]
+        except KeyError:
+            result = func(evaluator, obj, parent, module)
+            # Need to cache all of them, otherwise the id could be overwritten.
+            evaluator.compiled_cache[key] = result, obj, parent, module
+            return result
+    return wrapper
+
+
+@compiled_objects_cache
+def create(evaluator, obj, parent=builtin, module=None):
+    """
+    A very weird interface class to this module. The more options provided the
+    more acurate loading compiled objects is.
+    """
+
+    if not inspect.ismodule(obj):
+        faked = fake.get_faked(module and module.obj, obj)
+        if faked is not None:
+            faked.parent = parent
+            return faked
+
+    try:
+        if parent == builtin and obj.__module__ in ('builtins', '__builtin__'):
+            return builtin.get_by_name(obj.__name__)
+    except AttributeError:
+        pass
+
+    return CompiledObject(obj, parent)

jedi/evaluate/compiled/fake.py

@@ -0,0 +1,123 @@
+"""
+Loads functions that are mixed in to the standard library. E.g. builtins are
+written in C (binaries), but my autocompletion only understands Python code. By
+mixing in Python code, the autocompletion should work much better for builtins.
+"""
+
+import os
+import inspect
+
+from jedi._compatibility import is_py3, builtins, unicode
+from jedi.parser import Parser, load_grammar
+from jedi.parser import tree as pt
+from jedi.evaluate.helpers import FakeName
+
+modules = {}
+
+
+def _load_faked_module(module):
+    module_name = module.__name__
+    if module_name == '__builtin__' and not is_py3:
+        module_name = 'builtins'
+
+    try:
+        return modules[module_name]
+    except KeyError:
+        path = os.path.dirname(os.path.abspath(__file__))
+        try:
+            with open(os.path.join(path, 'fake', module_name) + '.pym') as f:
+                source = f.read()
+        except IOError:
+            modules[module_name] = None
+            return
+        grammar = load_grammar('grammar3.4')
+        module = Parser(grammar, unicode(source), module_name).module
+        modules[module_name] = module
+
+        if module_name == 'builtins' and not is_py3:
+            # There are two implementations of `open` for either python 2/3.
+            # -> Rename the python2 version (`look at fake/builtins.pym`).
+            open_func = search_scope(module, 'open')
+            open_func.children[1] = FakeName('open_python3')
+            open_func = search_scope(module, 'open_python2')
+            open_func.children[1] = FakeName('open')
+        return module
+
+
+def search_scope(scope, obj_name):
+    for s in scope.subscopes:
+        if str(s.name) == obj_name:
+            return s
+
+
+def get_module(obj):
+    if inspect.ismodule(obj):
+        return obj
+    try:
+        obj = obj.__objclass__
+    except AttributeError:
+        pass
+
+    try:
+        imp_plz = obj.__module__
+    except AttributeError:
+        # Unfortunately in some cases like `int` there's no __module__
+        return builtins
+    else:
+        return __import__(imp_plz)
+
+
+def _faked(module, obj, name):
+    # Crazy underscore actions to try to escape all the internal madness.
+    if module is None:
+        module = get_module(obj)
+
+    faked_mod = _load_faked_module(module)
+    if faked_mod is None:
+        return
+
+    # Having the module as a `parser.representation.module`, we need to scan
+    # for methods.
+    if name is None:
+        if inspect.isbuiltin(obj):
+            return search_scope(faked_mod, obj.__name__)
+        elif not inspect.isclass(obj):
+            # object is a method or descriptor
+            cls = search_scope(faked_mod, obj.__objclass__.__name__)
+            if cls is None:
+                return
+            return search_scope(cls, obj.__name__)
+    else:
+        if obj == module:
+            return search_scope(faked_mod, name)
+        else:
+            cls = search_scope(faked_mod, obj.__name__)
+            if cls is None:
+                return
+            return search_scope(cls, name)
+
+
+def get_faked(module, obj, name=None):
+    obj = obj.__class__ if is_class_instance(obj) else obj
+    result = _faked(module, obj, name)
+    if result is None or isinstance(result, pt.Class):
+        # We're not interested in classes. What we want is functions.
+        return None
+    else:
+        # Set the docstr which was previously not set (faked modules don't
+        # contain it).
+        doc = '"""%s"""' % obj.__doc__  # TODO need escapes.
+        suite = result.children[-1]
+        string = pt.String(pt.zero_position_modifier, doc, (0, 0), '')
+        new_line = pt.Whitespace('\n', (0, 0), '')
+        docstr_node = pt.Node('simple_stmt', [string, new_line])
+        suite.children.insert(2, docstr_node)
+        return result
+
+
+def is_class_instance(obj):
+    """Like inspect.* methods."""
+    return not (inspect.isclass(obj) or inspect.ismodule(obj)
+                or inspect.isbuiltin(obj) or inspect.ismethod(obj)
+                or inspect.ismethoddescriptor(obj) or inspect.iscode(obj)
+                or inspect.isgenerator(obj))

jedi/evaluate/compiled/fake/_functools.pym

@@ -5,5 +5,5 @@ class partial():
         self.__keywords = keywords
 
     def __call__(self, *args, **kwargs):
-        # I know this doesn't work in Python, but Jedi can this ;-)
-        return self.__func(*self.__args, *args, **self.keywords, **kwargs)
+        # TODO should be **dict(self.__keywords, **kwargs)
+        return self.__func(*(self.__args + args), **self.__keywords)

jedi/evaluate/compiled/fake/_sre.pym

@@ -1,48 +1,48 @@
 def compile():
     class SRE_Match():
-        endpos = 1
-        lastgroup = 0
-        lastindex = 1
-        pos = 0
-        string = 'a'
-        regs = ((0, 1),)
+        endpos = int()
+        lastgroup = int()
+        lastindex = int()
+        pos = int()
+        string = str()
+        regs = ((int(), int()),)
 
         def __init__(self, pattern):
             self.re = pattern
 
         def start(self):
-            return 0
+            return int()
 
         def end(self):
-            return 1
+            return int()
 
         def span(self):
-            return 0, 1
+            return int(), int()
 
         def expand(self):
-            return ''
+            return str()
 
         def group(self, nr):
-            return ''
+            return str()
 
         def groupdict(self):
-            return {'a', 'a'}
+            return {str(): str()}
 
         def groups(self):
-            return ('a',)
+            return (str(),)
 
     class SRE_Pattern():
-        flags = 0
+        flags = int()
         groupindex = {}
-        groups = 0
-        pattern = 'a'
+        groups = int()
+        pattern = str()
 
         def findall(self, string, pos=None, endpos=None):
             """
             findall(string[, pos[, endpos]]) --> list.
             Return a list of all non-overlapping matches of pattern in string.
             """
-            return ['a']
+            return [str()]
 
         def finditer(self, string, pos=None, endpos=None):
             """
@@ -77,7 +77,7 @@ def compile():
             split(string[, maxsplit = 0])  --> list.
             Split string by the occurrences of pattern.
             """
-            return ['a']
+            return [str()]
 
         def sub(self, repl, string, count=0):
             """
@@ -85,7 +85,7 @@ def compile():
             Return the string obtained by replacing the leftmost non-overlapping
             occurrences of pattern in string by the replacement repl.
             """
-            return ''
+            return str()
 
         def subn(self, repl, string, count=0):
             """
@@ -94,6 +94,6 @@ def compile():
             the leftmost non-overlapping occurrences of pattern with the
             replacement repl.
             """
-            return ('', 1)
+            return (str(), int())
 
     return SRE_Pattern()

jedi/evaluate/compiled/fake/_weakref.pym

@@ -1,7 +1,7 @@
 def proxy(object, callback=None):
     return object
 
-class ref():
+class weakref():
     def __init__(self, object, callback=None):
         self.__object = object
     def __call__(self):

jedi/evaluate/compiled/fake/builtins.pym

@@ -7,11 +7,14 @@ possible for the auto completion.
 
 
 def next(iterator, default=None):
-    if hasattr("next"):
-        return iterator.next()
+    if random.choice([0, 1]):
+        if hasattr("next"):
+            return iterator.next()
+        else:
+            return iterator.__next__()
     else:
-        return iterator.__next__()
-    return default
+        if default is not None:
+            return default
 
 
 def iter(collection, sentinel=None):
@@ -26,6 +29,13 @@ def range(start, stop=None, step=1):
     return [0]
 
 
+class file():
+    def __iter__(self):
+        yield ''
+    def next(self):
+        return ''
+
+
 class xrange():
     # Attention: this function doesn't exist in Py3k (there it is range).
     def __iter__(self):
@@ -38,6 +48,15 @@ class xrange():
         return 1
 
 
+def open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True):
+    import io
+    return io.TextIOWrapper(file, mode, buffering, encoding, errors, newline, closefd)
+
+
+def open_python2(name, mode=None, buffering=None):
+    return file(name, mode, buffering)
+
+
 #--------------------------------------------------------
 # descriptors
 #--------------------------------------------------------
@@ -137,7 +156,7 @@ class set():
             yield i
 
     def pop(self):
-        return self.__iterable.pop()
+        return list(self.__iterable)[-1]
 
     def copy(self):
         return self
@@ -183,6 +202,10 @@ class dict():
         except KeyError:
             return d
 
+    def setdefault(self, k, d):
+        # TODO maybe also return the content
+        return d
+
 
 class reversed():
     def __init__(self, sequence):
@@ -196,7 +219,11 @@ class reversed():
         return next(self.__iter__())
 
     def next(self):
-        return self.__next__()
+        return next(self.__iter__())
+
+
+def sorted(iterable, cmp=None, key=None, reverse=False):
+    return iterable
 
 
 #--------------------------------------------------------
@@ -211,8 +238,7 @@ class str():
     def __init__(self, obj):
         pass
 
-class object():
+
+class type():
     def mro():
-        """ mro() -> list
-        return a type's method resolution order """
         return [object]

jedi/evaluate/compiled/fake/io.pym

@@ -0,0 +1,6 @@
+class TextIOWrapper():
+    def __next__(self):
+        return str()
+
+    def __iter__(self):
+        yield str()

jedi/evaluate/docstrings.py

@@ -0,0 +1,195 @@
+"""
+Docstrings are another source of information for functions and classes.
+:mod:`jedi.evaluate.dynamic` tries to find all executions of functions, while
+the docstring parsing is much easier. There are two different types of
+docstrings that |jedi| understands:
+
+- `Sphinx <http://sphinx-doc.org/markup/desc.html#info-field-lists>`_
+- `Epydoc <http://epydoc.sourceforge.net/manual-fields.html>`_
+
+For example, the sphinx annotation ``:type foo: str`` clearly states that the
+type of ``foo`` is ``str``.
+
+As an addition to parameter searching, this module also provides return
+annotations.
+"""
+
+from ast import literal_eval
+import re
+from itertools import chain
+from textwrap import dedent
+
+from jedi.evaluate.cache import memoize_default
+from jedi.parser import Parser, load_grammar
+from jedi.common import indent_block
+from jedi.evaluate.iterable import Array, FakeSequence, AlreadyEvaluated
+
+
+DOCSTRING_PARAM_PATTERNS = [
+    r'\s*:type\s+%s:\s*([^\n]+)',  # Sphinx
+    r'\s*:param\s+(\w+)\s+%s:[^\n]+',  # Sphinx param with type
+    r'\s*@type\s+%s:\s*([^\n]+)',  # Epydoc
+]
+
+DOCSTRING_RETURN_PATTERNS = [
+    re.compile(r'\s*:rtype:\s*([^\n]+)', re.M),  # Sphinx
+    re.compile(r'\s*@rtype:\s*([^\n]+)', re.M),  # Epydoc
+]
+
+REST_ROLE_PATTERN = re.compile(r':[^`]+:`([^`]+)`')
+
+
+try:
+    from numpydoc.docscrape import NumpyDocString
+except ImportError:
+    def _search_param_in_numpydocstr(docstr, param_str):
+        return []
+else:
+    def _search_param_in_numpydocstr(docstr, param_str):
+        """Search `docstr` (in numpydoc format) for type(-s) of `param_str`."""
+        params = NumpyDocString(docstr)._parsed_data['Parameters']
+        for p_name, p_type, p_descr in params:
+            if p_name == param_str:
+                m = re.match('([^,]+(,[^,]+)*?)(,[ ]*optional)?$', p_type)
+                if m:
+                    p_type = m.group(1)
+
+                if p_type.startswith('{'):
+                    types = set(type(x).__name__ for x in literal_eval(p_type))
+                    return list(types)
+                else:
+                    return [p_type]
+        return []
+
+
+def _search_param_in_docstr(docstr, param_str):
+    """
+    Search `docstr` for type(-s) of `param_str`.
+
+    >>> _search_param_in_docstr(':type param: int', 'param')
+    ['int']
+    >>> _search_param_in_docstr('@type param: int', 'param')
+    ['int']
+    >>> _search_param_in_docstr(
+    ...   ':type param: :class:`threading.Thread`', 'param')
+    ['threading.Thread']
+    >>> bool(_search_param_in_docstr('no document', 'param'))
+    False
+    >>> _search_param_in_docstr(':param int param: some description', 'param')
+    ['int']
+
+    """
+    # look at #40 to see definitions of those params
+    patterns = [re.compile(p % re.escape(param_str))
+                for p in DOCSTRING_PARAM_PATTERNS]
+    for pattern in patterns:
+        match = pattern.search(docstr)
+        if match:
+            return [_strip_rst_role(match.group(1))]
+
+    return (_search_param_in_numpydocstr(docstr, param_str) or
+            [])
+
+
+def _strip_rst_role(type_str):
+    """
+    Strip off the part looks like a ReST role in `type_str`.
+
+    >>> _strip_rst_role(':class:`ClassName`')  # strip off :class:
+    'ClassName'
+    >>> _strip_rst_role(':py:obj:`module.Object`')  # works with domain
+    'module.Object'
+    >>> _strip_rst_role('ClassName')  # do nothing when not ReST role
+    'ClassName'
+
+    See also:
+    http://sphinx-doc.org/domains.html#cross-referencing-python-objects
+
+    """
+    match = REST_ROLE_PATTERN.match(type_str)
+    if match:
+        return match.group(1)
+    else:
+        return type_str
+
+
+def _evaluate_for_statement_string(evaluator, string, module):
+    code = dedent("""
+    def pseudo_docstring_stuff():
+        # Create a pseudo function for docstring statements.
+    %s
+    """)
+    if string is None:
+        return []
+
+    for element in re.findall('((?:\w+\.)*\w+)\.', string):
+        # Try to import module part in dotted name.
+        # (e.g., 'threading' in 'threading.Thread').
+        string = 'import %s\n' % element + string
+
+    # Take the default grammar here, if we load the Python 2.7 grammar here, it
+    # will be impossible to use `...` (Ellipsis) as a token. Docstring types
+    # don't need to conform with the current grammar.
+    p = Parser(load_grammar(), code % indent_block(string))
+    try:
+        pseudo_cls = p.module.subscopes[0]
+        # First pick suite, then simple_stmt (-2 for DEDENT) and then the node,
+        # which is also not the last item, because there's a newline.
+        stmt = pseudo_cls.children[-1].children[-2].children[-2]
+    except (AttributeError, IndexError):
+        return []
+
+    # Use the module of the param.
+    # TODO this module is not the module of the param in case of a function
+    # call. In that case it's the module of the function call.
+    # stuffed with content from a function call.
+    pseudo_cls.parent = module
+    return list(_execute_types_in_stmt(evaluator, stmt))
+
+
+def _execute_types_in_stmt(evaluator, stmt):
+    """
+    Executing all types or general elements that we find in a statement. This
+    doesn't include tuple, list and dict literals, because the stuff they
+    contain is executed. (Used as type information).
+    """
+    definitions = evaluator.eval_element(stmt)
+    return chain.from_iterable(_execute_array_values(evaluator, d) for d in definitions)
+
+
+def _execute_array_values(evaluator, array):
+    """
+    Tuples indicate that there's not just one return value, but the listed
+    ones.  `(str, int)` means that it returns a tuple with both types.
+    """
+    if isinstance(array, Array):
+        values = []
+        for typ in array.values():
+            objects = _execute_array_values(evaluator, typ)
+            values.append(AlreadyEvaluated(objects))
+        return [FakeSequence(evaluator, values, array.type)]
+    else:
+        return evaluator.execute(array)
+
+
+@memoize_default(None, evaluator_is_first_arg=True)
+def follow_param(evaluator, param):
+    func = param.parent_function
+
+    return [p
+            for param_str in _search_param_in_docstr(func.raw_doc,
+                                                     str(param.name))
+            for p in _evaluate_for_statement_string(evaluator, param_str,
+                                                    param.get_parent_until())]
+
+
+@memoize_default(None, evaluator_is_first_arg=True)
+def find_return_types(evaluator, func):
+    def search_return_in_docstr(code):
+        for p in DOCSTRING_RETURN_PATTERNS:
+            match = p.search(code)
+            if match:
+                return _strip_rst_role(match.group(1))
+
+    type_str = search_return_in_docstr(func.raw_doc)
+    return _evaluate_for_statement_string(evaluator, type_str, func.get_parent_until())

jedi/evaluate/dynamic.py

@@ -0,0 +1,146 @@
+"""
+One of the really important features of |jedi| is to have an option to
+understand code like this::
+
+    def foo(bar):
+        bar. # completion here
+    foo(1)
+
+There's no doubt wheter bar is an ``int`` or not, but if there's also a call
+like ``foo('str')``, what would happen? Well, we'll just show both. Because
+that's what a human would expect.
+
+It works as follows:
+
+- |Jedi| sees a param
+- search for function calls named ``foo``
+- execute these calls and check the input. This work with a ``ParamListener``.
+"""
+from itertools import chain
+
+from jedi._compatibility import unicode
+from jedi.parser import tree
+from jedi import settings
+from jedi import debug
+from jedi.evaluate.cache import memoize_default
+from jedi.evaluate import imports
+
+
+class ParamListener(object):
+    """
+    This listener is used to get the params for a function.
+    """
+    def __init__(self):
+        self.param_possibilities = []
+
+    def execute(self, params):
+        self.param_possibilities += params
+
+
+@debug.increase_indent
+def search_params(evaluator, param):
+    """
+    A dynamic search for param values. If you try to complete a type:
+
+    >>> def func(foo):
+    ...     foo
+    >>> func(1)
+    >>> func("")
+
+    It is not known what the type ``foo`` without analysing the whole code. You
+    have to look for all calls to ``func`` to find out what ``foo`` possibly
+    is.
+    """
+    if not settings.dynamic_params:
+        return []
+
+    func = param.get_parent_until(tree.Function)
+    debug.dbg('Dynamic param search for %s in %s.', param, str(func.name))
+    # Compare the param names.
+    names = [n for n in search_function_call(evaluator, func)
+             if n.value == param.name.value]
+    # Evaluate the ExecutedParams to types.
+    result = list(chain.from_iterable(n.parent.eval(evaluator) for n in names))
+    debug.dbg('Dynamic param result %s', result)
+    return result
+
+
+@memoize_default([], evaluator_is_first_arg=True)
+def search_function_call(evaluator, func):
+    """
+    Returns a list of param names.
+    """
+    from jedi.evaluate import representation as er
+
+    def get_params_for_module(module):
+        """
+        Returns the values of a param, or an empty array.
+        """
+        @memoize_default([], evaluator_is_first_arg=True)
+        def get_posibilities(evaluator, module, func_name):
+            try:
+                names = module.used_names[func_name]
+            except KeyError:
+                return []
+
+            for name in names:
+                parent = name.parent
+                if tree.is_node(parent, 'trailer'):
+                    parent = parent.parent
+
+                trailer = None
+                if tree.is_node(parent, 'power'):
+                    for t in parent.children[1:]:
+                        if t == '**':
+                            break
+                        if t.start_pos > name.start_pos and t.children[0] == '(':
+                            trailer = t
+                            break
+                if trailer is not None:
+                    types = evaluator.goto_definition(name)
+
+                    # We have to remove decorators, because they are not the
+                    # "original" functions, this way we can easily compare.
+                    # At the same time we also have to remove InstanceElements.
+                    undec = []
+                    for escope in types:
+                        if escope.isinstance(er.Function, er.Instance) \
+                                and escope.decorates is not None:
+                            undec.append(escope.decorates)
+                        elif isinstance(escope, er.InstanceElement):
+                            undec.append(escope.var)
+                        else:
+                            undec.append(escope)
+
+                    if evaluator.wrap(compare) in undec:
+                        # Only if we have the correct function we execute
+                        # it, otherwise just ignore it.
+                        evaluator.eval_trailer(types, trailer)
+            return listener.param_possibilities
+        return get_posibilities(evaluator, module, func_name)
+
+    current_module = func.get_parent_until()
+    func_name = unicode(func.name)
+    compare = func
+    if func_name == '__init__':
+        cls = func.get_parent_scope()
+        if isinstance(cls, tree.Class):
+            func_name = unicode(cls.name)
+            compare = cls
+
+    # add the listener
+    listener = ParamListener()
+    func.listeners.add(listener)
+
+    try:
+        result = []
+        # This is like backtracking: Get the first possible result.
+        for mod in imports.get_modules_containing_name(evaluator, [current_module], func_name):
+            result = get_params_for_module(mod)
+            if result:
+                break
+    finally:
+        # cleanup: remove the listener; important: should not stick.
+        func.listeners.remove(listener)
+
+    return result

jedi/evaluate/finder.py

@@ -0,0 +1,547 @@
+"""
+Searching for names with given scope and name. This is very central in Jedi and
+Python. The name resolution is quite complicated with descripter,
+``__getattribute__``, ``__getattr__``, ``global``, etc.
+
+Flow checks
++++++++++++
+
+Flow checks are not really mature. There's only a check for ``isinstance``.  It
+would check whether a flow has the form of ``if isinstance(a, type_or_tuple)``.
+Unfortunately every other thing is being ignored (e.g. a == '' would be easy to
+check for -> a is a string). There's big potential in these checks.
+"""
+from itertools import chain
+
+from jedi._compatibility import unicode, u
+from jedi.parser import tree
+from jedi import debug
+from jedi import common
+from jedi import settings
+from jedi.evaluate import representation as er
+from jedi.evaluate import dynamic
+from jedi.evaluate import compiled
+from jedi.evaluate import docstrings
+from jedi.evaluate import iterable
+from jedi.evaluate import imports
+from jedi.evaluate import analysis
+from jedi.evaluate import flow_analysis
+from jedi.evaluate import param
+from jedi.evaluate import helpers
+from jedi.evaluate.cache import memoize_default
+
+
+def filter_after_position(names, position):
+    """
+    Removes all names after a certain position. If position is None, just
+    returns the names list.
+    """
+    if position is None:
+        return names
+
+    names_new = []
+    for n in names:
+        # Filter positions and also allow list comprehensions and lambdas.
+        if n.start_pos[0] is not None and n.start_pos < position \
+                or isinstance(n.get_definition(), (tree.CompFor, tree.Lambda)):
+            names_new.append(n)
+    return names_new
+
+
+def filter_definition_names(names, origin, position=None):
+    """
+    Filter names that are actual definitions in a scope. Names that are just
+    used will be ignored.
+    """
+    # Just calculate the scope from the first
+    stmt = names[0].get_definition()
+    scope = stmt.get_parent_scope()
+
+    if not (isinstance(scope, er.FunctionExecution)
+            and isinstance(scope.base, er.LambdaWrapper)):
+        names = filter_after_position(names, position)
+    names = [name for name in names if name.is_definition()]
+
+    # Private name mangling (compile.c) disallows access on names
+    # preceeded by two underscores `__` if used outside of the class. Names
+    # that also end with two underscores (e.g. __id__) are not affected.
+    for name in list(names):
+        if name.value.startswith('__') and not name.value.endswith('__'):
+            if filter_private_variable(scope, origin):
+                names.remove(name)
+    return names
+
+
+class NameFinder(object):
+    def __init__(self, evaluator, scope, name_str, position=None):
+        self._evaluator = evaluator
+        # Make sure that it's not just a syntax tree node.
+        self.scope = evaluator.wrap(scope)
+        self.name_str = name_str
+        self.position = position
+
+    @debug.increase_indent
+    def find(self, scopes, search_global=False):
+        # TODO rename scopes to names_dicts
+        names = self.filter_name(scopes)
+        types = self._names_to_types(names, search_global)
+
+        if not names and not types \
+                and not (isinstance(self.name_str, tree.Name)
+                         and isinstance(self.name_str.parent.parent, tree.Param)):
+            if not isinstance(self.name_str, (str, unicode)):  # TODO Remove?
+                if search_global:
+                    message = ("NameError: name '%s' is not defined."
+                               % self.name_str)
+                    analysis.add(self._evaluator, 'name-error', self.name_str,
+                                 message)
+                else:
+                    analysis.add_attribute_error(self._evaluator,
+                                                 self.scope, self.name_str)
+
+        debug.dbg('finder._names_to_types: %s -> %s', names, types)
+        return types
+
+    def scopes(self, search_global=False):
+        if search_global:
+            return global_names_dict_generator(self._evaluator, self.scope, self.position)
+        else:
+            return ((n, None) for n in self.scope.names_dicts(search_global))
+
+    def names_dict_lookup(self, names_dict, position):
+        def get_param(scope, el):
+            if isinstance(el.get_parent_until(tree.Param), tree.Param):
+                return scope.param_by_name(str(el))
+            return el
+
+        search_str = str(self.name_str)
+        try:
+            names = names_dict[search_str]
+            if not names:  # We want names, otherwise stop.
+                return []
+        except KeyError:
+            return []
+
+        names = filter_definition_names(names, self.name_str, position)
+
+        name_scope = None
+        # Only the names defined in the last position are valid definitions.
+        last_names = []
+        for name in reversed(sorted(names, key=lambda name: name.start_pos)):
+            stmt = name.get_definition()
+            name_scope = self._evaluator.wrap(stmt.get_parent_scope())
+
+            if isinstance(self.scope, er.Instance) and not isinstance(name_scope, er.Instance):
+                # Instances should not be checked for positioning, because we
+                # don't know in which order the functions are called.
+                last_names.append(name)
+                continue
+
+            if isinstance(name_scope, compiled.CompiledObject):
+                # Let's test this. TODO need comment. shouldn't this be
+                # filtered before?
+                last_names.append(name)
+                continue
+
+            if isinstance(name, compiled.CompiledName) \
+                    or isinstance(name, er.InstanceName) and isinstance(name._origin_name, compiled.CompiledName):
+                last_names.append(name)
+                continue
+
+            if isinstance(self.name_str, tree.Name):
+                origin_scope = self.name_str.get_parent_until(tree.Scope, reverse=True)
+            else:
+                origin_scope = None
+            if isinstance(stmt.parent, compiled.CompiledObject):
+                # TODO seriously? this is stupid.
+                continue
+            check = flow_analysis.break_check(self._evaluator, name_scope,
+                                              stmt, origin_scope)
+            if check is not flow_analysis.UNREACHABLE:
+                last_names.append(name)
+            if check is flow_analysis.REACHABLE:
+                break
+
+        if isinstance(name_scope, er.FunctionExecution):
+            # Replace params
+            return [get_param(name_scope, n) for n in last_names]
+        return last_names
+
+    def filter_name(self, names_dicts):
+        """
+        Searches names that are defined in a scope (the different
+        `names_dicts`), until a name fits.
+        """
+        names = []
+        for names_dict, position in names_dicts:
+            names = self.names_dict_lookup(names_dict, position)
+            if names:
+                break
+
+        debug.dbg('finder.filter_name "%s" in (%s): %s@%s', self.name_str,
+                  self.scope, u(names), self.position)
+        return list(self._clean_names(names))
+
+    def _clean_names(self, names):
+        """
+        ``NameFinder.filter_name`` should only output names with correct
+        wrapper parents. We don't want to see AST classes out in the
+        evaluation, so remove them already here!
+        """
+        for n in names:
+            definition = n.parent
+            if isinstance(definition, (tree.Function, tree.Class, tree.Module)):
+                yield self._evaluator.wrap(definition).name
+            else:
+                yield n
+
+    def _check_getattr(self, inst):
+        """Checks for both __getattr__ and __getattribute__ methods"""
+        result = []
+        # str is important, because it shouldn't be `Name`!
+        name = compiled.create(self._evaluator, str(self.name_str))
+        with common.ignored(KeyError):
+            result = inst.execute_subscope_by_name('__getattr__', name)
+        if not result:
+            # this is a little bit special. `__getattribute__` is executed
+            # before anything else. But: I know no use case, where this
+            # could be practical and the jedi would return wrong types. If
+            # you ever have something, let me know!
+            with common.ignored(KeyError):
+                result = inst.execute_subscope_by_name('__getattribute__', name)
+        return result
+
+    def _names_to_types(self, names, search_global):
+        types = []
+
+        # Add isinstance and other if/assert knowledge.
+        if isinstance(self.name_str, tree.Name):
+            # Ignore FunctionExecution parents for now.
+            flow_scope = self.name_str
+            until = flow_scope.get_parent_until(er.FunctionExecution)
+            while not isinstance(until, er.FunctionExecution):
+                flow_scope = flow_scope.get_parent_scope(include_flows=True)
+                if flow_scope is None:
+                    break
+                # TODO check if result is in scope -> no evaluation necessary
+                n = check_flow_information(self._evaluator, flow_scope,
+                                           self.name_str, self.position)
+                if n:
+                    return n
+
+        for name in names:
+            new_types = _name_to_types(self._evaluator, name, self.scope)
+            if isinstance(self.scope, (er.Class, er.Instance)) and not search_global:
+                types += self._resolve_descriptors(name, new_types)
+            else:
+                types += new_types
+        if not names and isinstance(self.scope, er.Instance):
+            # handling __getattr__ / __getattribute__
+            types = self._check_getattr(self.scope)
+
+        return types
+
+    def _resolve_descriptors(self, name, types):
+        # The name must not be in the dictionary, but part of the class
+        # definition. __get__ is only called if the descriptor is defined in
+        # the class dictionary.
+        name_scope = name.get_definition().get_parent_scope()
+        if not isinstance(name_scope, (er.Instance, tree.Class)):
+            return types
+
+        result = []
+        for r in types:
+            try:
+                desc_return = r.get_descriptor_returns
+            except AttributeError:
+                result.append(r)
+            else:
+                result += desc_return(self.scope)
+        return result
+
+
+@memoize_default([], evaluator_is_first_arg=True)
+def _name_to_types(evaluator, name, scope):
+    types = []
+    typ = name.get_definition()
+    if typ.isinstance(tree.ForStmt):
+        for_types = evaluator.eval_element(typ.children[3])
+        for_types = iterable.get_iterator_types(for_types)
+        types += check_tuple_assignments(for_types, name)
+    elif typ.isinstance(tree.CompFor):
+        for_types = evaluator.eval_element(typ.children[3])
+        for_types = iterable.get_iterator_types(for_types)
+        types += check_tuple_assignments(for_types, name)
+    elif isinstance(typ, tree.Param):
+        types += _eval_param(evaluator, typ, scope)
+    elif typ.isinstance(tree.ExprStmt):
+        types += _remove_statements(evaluator, typ, name)
+    elif typ.isinstance(tree.WithStmt):
+        types += evaluator.eval_element(typ.node_from_name(name))
+    elif isinstance(typ, tree.Import):
+        types += imports.ImportWrapper(evaluator, name).follow()
+    elif isinstance(typ, tree.GlobalStmt):
+        # TODO theoretically we shouldn't be using search_global here, it
+        # doesn't make sense, because it's a local search (for that name)!
+        # However, globals are not that important and resolving them doesn't
+        # guarantee correctness in any way, because we don't check for when
+        # something is executed.
+        types += evaluator.find_types(typ.get_parent_scope(), str(name),
+                                      search_global=True)
+    elif isinstance(typ, tree.TryStmt):
+        # TODO an exception can also be a tuple. Check for those.
+        # TODO check for types that are not classes and add it to
+        # the static analysis report.
+        exceptions = evaluator.eval_element(name.prev_sibling().prev_sibling())
+        types = list(chain.from_iterable(
+                     evaluator.execute(t) for t in exceptions))
+    else:
+        if typ.isinstance(er.Function):
+            typ = typ.get_decorated_func()
+        types.append(typ)
+    return types
+
+
+def _remove_statements(evaluator, stmt, name):
+    """
+    This is the part where statements are being stripped.
+
+    Due to lazy evaluation, statements like a = func; b = a; b() have to be
+    evaluated.
+    """
+    types = []
+    # Remove the statement docstr stuff for now, that has to be
+    # implemented with the evaluator class.
+    #if stmt.docstr:
+        #res_new.append(stmt)
+
+    check_instance = None
+    if isinstance(stmt, er.InstanceElement) and stmt.is_class_var:
+        check_instance = stmt.instance
+        stmt = stmt.var
+
+    types += evaluator.eval_statement(stmt, seek_name=name)
+
+    if check_instance is not None:
+        # class renames
+        types = [er.get_instance_el(evaluator, check_instance, a, True)
+                 if isinstance(a, (er.Function, tree.Function))
+                 else a for a in types]
+    return types
+
+
+def _eval_param(evaluator, param, scope):
+    res_new = []
+    func = param.get_parent_scope()
+
+    cls = func.parent.get_parent_until((tree.Class, tree.Function))
+
+    from jedi.evaluate.param import ExecutedParam, Arguments
+    if isinstance(cls, tree.Class) and param.position_nr == 0 \
+            and not isinstance(param, ExecutedParam):
+        # This is where we add self - if it has never been
+        # instantiated.
+        if isinstance(scope, er.InstanceElement):
+            res_new.append(scope.instance)
+        else:
+            inst = er.Instance(evaluator, evaluator.wrap(cls),
+                               Arguments(evaluator, ()), is_generated=True)
+            res_new.append(inst)
+        return res_new
+
+    # Instances are typically faked, if the instance is not called from
+    # outside. Here we check it for __init__ functions and return.
+    if isinstance(func, er.InstanceElement) \
+            and func.instance.is_generated and str(func.name) == '__init__':
+        param = func.var.params[param.position_nr]
+
+    # Add docstring knowledge.
+    doc_params = docstrings.follow_param(evaluator, param)
+    if doc_params:
+        return doc_params
+
+    if isinstance(param, ExecutedParam):
+        return res_new + param.eval(evaluator)
+    else:
+        # Param owns no information itself.
+        res_new += dynamic.search_params(evaluator, param)
+        if not res_new:
+            if param.stars:
+                t = 'tuple' if param.stars == 1 else 'dict'
+                typ = evaluator.find_types(compiled.builtin, t)[0]
+                res_new = evaluator.execute(typ)
+        if param.default:
+            res_new += evaluator.eval_element(param.default)
+        return res_new
+
+
+def check_flow_information(evaluator, flow, search_name, pos):
+    """ Try to find out the type of a variable just with the information that
+    is given by the flows: e.g. It is also responsible for assert checks.::
+
+        if isinstance(k, str):
+            k.  # <- completion here
+
+    ensures that `k` is a string.
+    """
+    if not settings.dynamic_flow_information:
+        return None
+
+    result = []
+    if flow.is_scope():
+        # Check for asserts.
+        try:
+            names = reversed(flow.names_dict[search_name.value])
+        except (KeyError, AttributeError):
+            names = []
+
+        for name in names:
+            ass = name.get_parent_until(tree.AssertStmt)
+            if isinstance(ass, tree.AssertStmt) and pos is not None and ass.start_pos < pos:
+                result = _check_isinstance_type(evaluator, ass.assertion(), search_name)
+                if result:
+                    break
+
+    if isinstance(flow, (tree.IfStmt, tree.WhileStmt)):
+        element = flow.children[1]
+        result = _check_isinstance_type(evaluator, element, search_name)
+    return result
+
+
+def _check_isinstance_type(evaluator, element, search_name):
+    try:
+        assert element.type == 'power'
+        # this might be removed if we analyze and, etc
+        assert len(element.children) == 2
+        first, trailer = element.children
+        assert isinstance(first, tree.Name) and first.value == 'isinstance'
+        assert trailer.type == 'trailer' and trailer.children[0] == '('
+        assert len(trailer.children) == 3
+
+        # arglist stuff
+        arglist = trailer.children[1]
+        args = param.Arguments(evaluator, arglist, trailer)
+        lst = list(args.unpack())
+        # Disallow keyword arguments
+        assert len(lst) == 2 and lst[0][0] is None and lst[1][0] is None
+        name = lst[0][1][0]  # first argument, values, first value
+        # Do a simple get_code comparison. They should just have the same code,
+        # and everything will be all right.
+        classes = lst[1][1][0]
+        call = helpers.call_of_name(search_name)
+        assert name.get_code() == call.get_code()
+    except AssertionError:
+        return []
+
+    result = []
+    for typ in evaluator.eval_element(classes):
+        for typ in (typ.values() if isinstance(typ, iterable.Array) else [typ]):
+            result += evaluator.execute(typ)
+    return result
+
+
+def global_names_dict_generator(evaluator, scope, position):
+    """
+    For global name lookups. Yields tuples of (names_dict, position). If the
+    position is None, the position does not matter anymore in that scope.
+
+    This function is used to include names from outer scopes. For example, when
+    the current scope is function:
+
+    >>> from jedi._compatibility import u, no_unicode_pprint
+    >>> from jedi.parser import Parser, load_grammar
+    >>> parser = Parser(load_grammar(), u('''
+    ... x = ['a', 'b', 'c']
+    ... def func():
+    ...     y = None
+    ... '''))
+    >>> scope = parser.module.subscopes[0]
+    >>> scope
+    <Function: func@3-5>
+
+    `global_names_dict_generator` is a generator.  First it yields names from
+    most inner scope.
+
+    >>> from jedi.evaluate import Evaluator
+    >>> evaluator = Evaluator(load_grammar())
+    >>> scope = evaluator.wrap(scope)
+    >>> pairs = list(global_names_dict_generator(evaluator, scope, (4, 0)))
+    >>> no_unicode_pprint(pairs[0])
+    ({'func': [], 'y': [<Name: y@4,4>]}, (4, 0))
+
+    Then it yields the names from one level "lower". In this example, this
+    is the most outer scope. As you can see, the position in the tuple is now
+    None, because typically the whole module is loaded before the function is
+    called.
+
+    >>> no_unicode_pprint(pairs[1])
+    ({'func': [<Name: func@3,4>], 'x': [<Name: x@2,0>]}, None)
+
+    After that we have a few underscore names that are part of the module.
+
+    >>> sorted(pairs[2][0].keys())
+    ['__doc__', '__file__', '__name__', '__package__']
+    >>> pairs[3]  # global names -> there are none in our example.
+    ({}, None)
+    >>> pairs[4]  # package modules -> Also none.
+    ({}, None)
+
+    Finally, it yields names from builtin, if `include_builtin` is
+    true (default).
+
+    >>> pairs[5][0].values()                              #doctest: +ELLIPSIS
+    [[<CompiledName: ...>], ...]
+    """
+    in_func = False
+    while scope is not None:
+        if not (scope.type == 'classdef' and in_func):
+            # Names in methods cannot be resolved within the class.
+
+            for names_dict in scope.names_dicts(True):
+                yield names_dict, position
+            if scope.type == 'funcdef':
+                # The position should be reset if the current scope is a function.
+                in_func = True
+                position = None
+        scope = evaluator.wrap(scope.get_parent_scope())
+
+    # Add builtins to the global scope.
+    for names_dict in compiled.builtin.names_dicts(True):
+        yield names_dict, None
+
+
+def check_tuple_assignments(types, name):
+    """
+    Checks if tuples are assigned.
+    """
+    for index in name.assignment_indexes():
+        new_types = []
+        for r in types:
+            try:
+                func = r.get_exact_index_types
+            except AttributeError:
+                debug.warning("Invalid tuple lookup #%s of result %s in %s",
+                              index, types, name)
+            else:
+                try:
+                    new_types += func(index)
+                except IndexError:
+                    pass
+        types = new_types
+    return types
+
+
+def filter_private_variable(scope, origin_node):
+    """Check if a variable is defined inside the same class or outside."""
+    instance = scope.get_parent_scope()
+    coming_from = origin_node
+    while coming_from is not None \
+            and not isinstance(coming_from, (tree.Class, compiled.CompiledObject)):
+        coming_from = coming_from.get_parent_scope()
+
+    # CompiledObjects don't have double underscore attributes, but Jedi abuses
+    # those for fakes (builtins.pym -> list).
+    if isinstance(instance, compiled.CompiledObject):
+        return instance != coming_from
+    else:
+        return isinstance(instance, er.Instance) and instance.base.base != coming_from

jedi/evaluate/flow_analysis.py

@@ -0,0 +1,84 @@
+from jedi.parser import tree
+
+
+class Status(object):
+    lookup_table = {}
+
+    def __init__(self, value, name):
+        self._value = value
+        self._name = name
+        Status.lookup_table[value] = self
+
+    def invert(self):
+        if self is REACHABLE:
+            return UNREACHABLE
+        elif self is UNREACHABLE:
+            return REACHABLE
+        else:
+            return UNSURE
+
+    def __and__(self, other):
+        if UNSURE in (self, other):
+            return UNSURE
+        else:
+            return REACHABLE if self._value and other._value else UNREACHABLE
+
+    def __repr__(self):
+        return '<%s: %s>' % (type(self).__name__, self._name)
+
+
+REACHABLE = Status(True, 'reachable')
+UNREACHABLE = Status(False, 'unreachable')
+UNSURE = Status(None, 'unsure')
+
+
+def break_check(evaluator, base_scope, stmt, origin_scope=None):
+    element_scope = evaluator.wrap(stmt.get_parent_scope(include_flows=True))
+    # Direct parents get resolved, we filter scopes that are separate branches.
+    # This makes sense for autocompletion and static analysis. For actual
+    # Python it doesn't matter, because we're talking about potentially
+    # unreachable code.
+    # e.g. `if 0:` would cause all name lookup within the flow make
+    # unaccessible. This is not a "problem" in Python, because the code is
+    # never called. In Jedi though, we still want to infer types.
+    while origin_scope is not None:
+        if element_scope == origin_scope:
+            return REACHABLE
+        origin_scope = origin_scope.parent
+    return _break_check(evaluator, stmt, base_scope, element_scope)
+
+
+def _break_check(evaluator, stmt, base_scope, element_scope):
+    element_scope = evaluator.wrap(element_scope)
+    base_scope = evaluator.wrap(base_scope)
+
+    reachable = REACHABLE
+    if isinstance(element_scope, tree.IfStmt):
+        if element_scope.node_after_else(stmt):
+            for check_node in element_scope.check_nodes():
+                reachable = _check_if(evaluator, check_node)
+                if reachable in (REACHABLE, UNSURE):
+                    break
+            reachable = reachable.invert()
+        else:
+            node = element_scope.node_in_which_check_node(stmt)
+            reachable = _check_if(evaluator, node)
+    elif isinstance(element_scope, (tree.TryStmt, tree.WhileStmt)):
+        return UNSURE
+
+    # Only reachable branches need to be examined further.
+    if reachable in (UNREACHABLE, UNSURE):
+        return reachable
+
+    if base_scope != element_scope and base_scope != element_scope.parent:
+        return reachable & _break_check(evaluator, stmt, base_scope, element_scope.parent)
+    return reachable
+
+
+def _check_if(evaluator, node):
+    types = evaluator.eval_element(node)
+    values = set(x.py__bool__() for x in types)
+    if len(values) == 1:
+        return Status.lookup_table[values.pop()]
+    else:
+        return UNSURE

jedi/evaluate/helpers.py

@@ -0,0 +1,179 @@
+import copy
+from itertools import chain
+
+from jedi.parser import tree
+
+
+def deep_ast_copy(obj, parent=None, new_elements=None):
+    """
+    Much, much faster than copy.deepcopy, but just for Parser elements (Doesn't
+    copy parents).
+    """
+
+    if new_elements is None:
+        new_elements = {}
+
+    def copy_node(obj):
+        # If it's already in the cache, just return it.
+        try:
+            return new_elements[obj]
+        except KeyError:
+            # Actually copy and set attributes.
+            new_obj = copy.copy(obj)
+            new_elements[obj] = new_obj
+
+        # Copy children
+        new_children = []
+        for child in obj.children:
+            typ = child.type
+            if typ in ('whitespace', 'operator', 'keyword', 'number', 'string'):
+                # At the moment we're not actually copying those primitive
+                # elements, because there's really no need to. The parents are
+                # obviously wrong, but that's not an issue.
+                new_child = child
+            elif typ == 'name':
+                new_elements[child] = new_child = copy.copy(child)
+                new_child.parent = new_obj
+            else:  # Is a BaseNode.
+                new_child = copy_node(child)
+                new_child.parent = new_obj
+            new_children.append(new_child)
+        new_obj.children = new_children
+
+        # Copy the names_dict (if there is one).
+        try:
+            names_dict = obj.names_dict
+        except AttributeError:
+            pass
+        else:
+            try:
+                new_obj.names_dict = new_names_dict = {}
+            except AttributeError:  # Impossible to set CompFor.names_dict
+                pass
+            else:
+                for string, names in names_dict.items():
+                    new_names_dict[string] = [new_elements[n] for n in names]
+        return new_obj
+
+    if obj.type == 'name':
+        # Special case of a Name object.
+        new_elements[obj] = new_obj = copy.copy(obj)
+        if parent is not None:
+            new_obj.parent = parent
+    elif isinstance(obj, tree.BaseNode):
+        new_obj = copy_node(obj)
+        if parent is not None:
+            for child in new_obj.children:
+                if isinstance(child, (tree.Name, tree.BaseNode)):
+                    child.parent = parent
+    else:  # String literals and so on.
+        new_obj = obj  # Good enough, don't need to copy anything.
+    return new_obj
+
+
+def call_of_name(name, cut_own_trailer=False):
+    """
+    Creates a "call" node that consist of all ``trailer`` and ``power``
+    objects.  E.g. if you call it with ``append``::
+
+        list([]).append(3) or None
+
+    You would get a node with the content ``list([]).append`` back.
+
+    This generates a copy of the original ast node.
+    """
+    par = name
+    if tree.is_node(par.parent, 'trailer'):
+        par = par.parent
+
+    power = par.parent
+    if tree.is_node(power, 'power') and power.children[0] != name \
+            and not (power.children[-2] == '**' and
+                     name.start_pos > power.children[-1].start_pos):
+        par = power
+        # Now the name must be part of a trailer
+        index = par.children.index(name.parent)
+        if index != len(par.children) - 1 or cut_own_trailer:
+            # Now we have to cut the other trailers away.
+            par = deep_ast_copy(par)
+            if not cut_own_trailer:
+                # Normally we would remove just the stuff after the index, but
+                # if the option is set remove the index as well. (for goto)
+                index = index + 1
+            par.children[index:] = []
+
+    return par
+
+
+def get_module_names(module, all_scopes):
+    """
+    Returns a dictionary with name parts as keys and their call paths as
+    values.
+    """
+    if all_scopes:
+        dct = module.used_names
+    else:
+        dct = module.names_dict
+    return chain.from_iterable(dct.values())
+
+
+class FakeImport(tree.ImportName):
+    def __init__(self, name, parent, level=0):
+        super(FakeImport, self).__init__([])
+        self.parent = parent
+        self._level = level
+        self.name = name
+
+    def get_defined_names(self):
+        return [self.name]
+
+    def aliases(self):
+        return {}
+
+    @property
+    def level(self):
+        return self._level
+
+    @property
+    def start_pos(self):
+        return 0, 0
+
+    def paths(self):
+        return [[self.name]]
+
+    def is_definition(self):
+        return True
+
+
+class FakeName(tree.Name):
+    def __init__(self, name_str, parent=None, start_pos=(0, 0), is_definition=None):
+        """
+        In case is_definition is defined (not None), that bool value will be
+        returned.
+        """
+        super(FakeName, self).__init__(tree.zero_position_modifier, name_str, start_pos)
+        self.parent = parent
+        self._is_definition = is_definition
+
+    def get_definition(self):
+        return self.parent
+
+    def is_definition(self):
+        if self._is_definition is None:
+            return super(FakeName, self).is_definition()
+        else:
+            return self._is_definition
+
+
+class LazyName(FakeName):
+    def __init__(self, name, parent_callback, is_definition=None):
+        super(LazyName, self).__init__(name, is_definition=is_definition)
+        self._parent_callback = parent_callback
+
+    @property
+    def parent(self):
+        return self._parent_callback()
+
+    @parent.setter
+    def parent(self, value):
+        pass  # Do nothing, super classes can try to set the parent.

jedi/evaluate/imports.py

@@ -0,0 +1,507 @@
+"""
+:mod:`jedi.evaluate.imports` is here to resolve import statements and return
+the modules/classes/functions/whatever, which they stand for. However there's
+not any actual importing done. This module is about finding modules in the
+filesystem. This can be quite tricky sometimes, because Python imports are not
+always that simple.
+
+This module uses imp for python up to 3.2 and importlib for python 3.3 on; the
+correct implementation is delegated to _compatibility.
+
+This module also supports import autocompletion, which means to complete
+statements like ``from datetim`` (curser at the end would return ``datetime``).
+"""
+import imp
+import os
+import pkgutil
+import sys
+from itertools import chain
+
+from jedi._compatibility import find_module, unicode
+from jedi import common
+from jedi import debug
+from jedi import cache
+from jedi.parser import fast
+from jedi.parser import tree
+from jedi.evaluate import sys_path
+from jedi.evaluate import helpers
+from jedi import settings
+from jedi.common import source_to_unicode
+from jedi.evaluate import compiled
+from jedi.evaluate import analysis
+from jedi.evaluate.cache import memoize_default, NO_DEFAULT
+
+
+def completion_names(evaluator, imp, pos):
+    name = imp.name_for_position(pos)
+    module = evaluator.wrap(imp.get_parent_until())
+    if name is None:
+        level = 0
+        for node in imp.children:
+            if node.end_pos <= pos:
+                if node in ('.', '...'):
+                    level += len(node.value)
+        import_path = []
+    else:
+        # Completion on an existing name.
+
+        # The import path needs to be reduced by one, because we're completing.
+        import_path = imp.path_for_name(name)[:-1]
+        level = imp.level
+
+    importer = Importer(evaluator, tuple(import_path), module, level)
+    if isinstance(imp, tree.ImportFrom):
+        c = imp.children
+        only_modules = c[c.index('import')].start_pos >= pos
+    else:
+        only_modules = True
+    return importer.completion_names(evaluator, only_modules)
+
+
+class ImportWrapper(tree.Base):
+    def __init__(self, evaluator, name):
+        self._evaluator = evaluator
+        self._name = name
+
+        self._import = name.get_parent_until(tree.Import)
+        self.import_path = self._import.path_for_name(name)
+
+    @memoize_default()
+    def follow(self, is_goto=False):
+        if self._evaluator.recursion_detector.push_stmt(self._import):
+            # check recursion
+            return []
+
+        try:
+            module = self._evaluator.wrap(self._import.get_parent_until())
+            import_path = self._import.path_for_name(self._name)
+            from_import_name = None
+            try:
+                from_names = self._import.get_from_names()
+            except AttributeError:
+                # Is an import_name
+                pass
+            else:
+                if len(from_names) + 1 == len(import_path):
+                    # We have to fetch the from_names part first and then check
+                    # if from_names exists in the modules.
+                    from_import_name = import_path[-1]
+                    import_path = from_names
+
+            importer = Importer(self._evaluator, tuple(import_path),
+                                module, self._import.level)
+
+            types = importer.follow()
+
+            #if self._import.is_nested() and not self.nested_resolve:
+            #    scopes = [NestedImportModule(module, self._import)]
+
+            if from_import_name is not None:
+                types = list(chain.from_iterable(
+                    self._evaluator.find_types(t, unicode(from_import_name),
+                                               is_goto=is_goto)
+                    for t in types))
+
+                if not types:
+                    path = import_path + [from_import_name]
+                    importer = Importer(self._evaluator, tuple(path),
+                                        module, self._import.level)
+                    types = importer.follow()
+                    # goto only accepts `Name`
+                    if is_goto:
+                        types = [s.name for s in types]
+            else:
+                # goto only accepts `Name`
+                if is_goto:
+                    types = [s.name for s in types]
+
+            debug.dbg('after import: %s', types)
+        finally:
+            self._evaluator.recursion_detector.pop_stmt()
+        return types
+
+
+class NestedImportModule(tree.Module):
+    """
+    TODO while there's no use case for nested import module right now, we might
+        be able to use them for static analysis checks later on.
+    """
+    def __init__(self, module, nested_import):
+        self._module = module
+        self._nested_import = nested_import
+
+    def _get_nested_import_name(self):
+        """
+        Generates an Import statement, that can be used to fake nested imports.
+        """
+        i = self._nested_import
+        # This is not an existing Import statement. Therefore, set position to
+        # 0 (0 is not a valid line number).
+        zero = (0, 0)
+        names = [unicode(name) for name in i.namespace_names[1:]]
+        name = helpers.FakeName(names, self._nested_import)
+        new = tree.Import(i._sub_module, zero, zero, name)
+        new.parent = self._module
+        debug.dbg('Generated a nested import: %s', new)
+        return helpers.FakeName(str(i.namespace_names[1]), new)
+
+    def __getattr__(self, name):
+        return getattr(self._module, name)
+
+    def __repr__(self):
+        return "<%s: %s of %s>" % (self.__class__.__name__, self._module,
+                                   self._nested_import)
+
+
+def _add_error(evaluator, name, message=None):
+    if hasattr(name, 'parent'):
+        # Should be a name, not a string!
+        analysis.add(evaluator, 'import-error', name, message)
+
+
+def get_init_path(directory_path):
+    """
+    The __init__ file can be searched in a directory. If found return it, else
+    None.
+    """
+    for suffix, _, _ in imp.get_suffixes():
+        path = os.path.join(directory_path, '__init__' + suffix)
+        if os.path.exists(path):
+            return path
+    return None
+
+
+class Importer(object):
+    def __init__(self, evaluator, import_path, module, level=0):
+        """
+        An implementation similar to ``__import__``. Use `follow`
+        to actually follow the imports.
+
+        *level* specifies whether to use absolute or relative imports. 0 (the
+        default) means only perform absolute imports. Positive values for level
+        indicate the number of parent directories to search relative to the
+        directory of the module calling ``__import__()`` (see PEP 328 for the
+        details).
+
+        :param import_path: List of namespaces (strings or Names).
+        """
+        debug.speed('import %s' % (import_path,))
+        self._evaluator = evaluator
+        self.level = level
+        self.module = module
+        try:
+            self.file_path = module.py__file__()
+        except AttributeError:
+            # Can be None for certain compiled modules like 'builtins'.
+            self.file_path = None
+
+        if level:
+            base = module.py__package__().split('.')
+            if base == ['']:
+                base = []
+            if level > len(base):
+                path = module.py__file__()
+                import_path = list(import_path)
+                for i in range(level):
+                    path = os.path.dirname(path)
+                dir_name = os.path.basename(path)
+                # This is not the proper way to do relative imports. However, since
+                # Jedi cannot be sure about the entry point, we just calculate an
+                # absolute path here.
+                if dir_name:
+                    import_path.insert(0, dir_name)
+                else:
+                    _add_error(self._evaluator, import_path[-1])
+                    import_path = []
+                    # TODO add import error.
+                    debug.warning('Attempted relative import beyond top-level package.')
+            else:
+                # Here we basically rewrite the level to 0.
+                import_path = tuple(base) + import_path
+        self.import_path = import_path
+
+    @property
+    def str_import_path(self):
+        """Returns the import path as pure strings instead of `Name`."""
+        return tuple(str(name) for name in self.import_path)
+
+    @memoize_default()
+    def sys_path_with_modifications(self):
+        in_path = []
+        sys_path_mod = list(sys_path.sys_path_with_modifications(self._evaluator, self.module))
+        if self.file_path is not None:
+            # If you edit e.g. gunicorn, there will be imports like this:
+            # `from gunicorn import something`. But gunicorn is not in the
+            # sys.path. Therefore look if gunicorn is a parent directory, #56.
+            if self.import_path:  # TODO is this check really needed?
+                for path in sys_path.traverse_parents(self.file_path):
+                    if os.path.basename(path) == self.str_import_path[0]:
+                        in_path.append(os.path.dirname(path))
+
+            # Since we know nothing about the call location of the sys.path,
+            # it's a possibility that the current directory is the origin of
+            # the Python execution.
+            sys_path_mod.insert(0, os.path.dirname(self.file_path))
+
+        return in_path + sys_path_mod
+
+    @memoize_default(NO_DEFAULT)
+    def follow(self):
+        if not self.import_path:
+            return []
+        return self._do_import(self.import_path, self.sys_path_with_modifications())
+
+    def _do_import(self, import_path, sys_path):
+        """
+        This method is very similar to importlib's `_gcd_import`.
+        """
+        import_parts = [str(i) for i in import_path]
+
+        # Handle "magic" Flask extension imports:
+        # ``flask.ext.foo`` is really ``flask_foo`` or ``flaskext.foo``.
+        if len(import_path) > 2 and import_parts[:2] == ['flask', 'ext']:
+            # New style.
+            ipath = ('flask_' + str(import_parts[2]),) + import_path[3:]
+            modules = self._do_import(ipath, sys_path)
+            if modules:
+                return modules
+            else:
+                # Old style
+                return self._do_import(('flaskext',) + import_path[2:], sys_path)
+
+        module_name = '.'.join(import_parts)
+        try:
+            return [self._evaluator.modules[module_name]]
+        except KeyError:
+            pass
+
+        if len(import_path) > 1:
+            # This is a recursive way of importing that works great with
+            # the module cache.
+            bases = self._do_import(import_path[:-1], sys_path)
+            if not bases:
+                return []
+            # We can take the first element, because only the os special
+            # case yields multiple modules, which is not important for
+            # further imports.
+            base = bases[0]
+
+            # This is a huge exception, we follow a nested import
+            # ``os.path``, because it's a very important one in Python
+            # that is being achieved by messing with ``sys.modules`` in
+            # ``os``.
+            if [str(i) for i in import_path] == ['os', 'path']:
+                return self._evaluator.find_types(base, 'path')
+
+            try:
+                # It's possible that by giving it always the sys path (and not
+                # the __path__ attribute of the parent, we get wrong results
+                # and nested namespace packages don't work.  But I'm not sure.
+                paths = base.py__path__(sys_path)
+            except AttributeError:
+                # The module is not a package.
+                _add_error(self._evaluator, import_path[-1])
+                return []
+            else:
+                debug.dbg('search_module %s in paths %s', module_name, paths)
+                for path in paths:
+                    # At the moment we are only using one path. So this is
+                    # not important to be correct.
+                    try:
+                        module_file, module_path, is_pkg = \
+                            find_module(import_parts[-1], [path])
+                        break
+                    except ImportError:
+                        module_path = None
+                if module_path is None:
+                    _add_error(self._evaluator, import_path[-1])
+                    return []
+        else:
+            try:
+                debug.dbg('search_module %s in %s', import_parts[-1], self.file_path)
+                # Override the sys.path. It works only good that way.
+                # Injecting the path directly into `find_module` did not work.
+                sys.path, temp = sys_path, sys.path
+                try:
+                    module_file, module_path, is_pkg = \
+                        find_module(import_parts[-1])
+                finally:
+                    sys.path = temp
+            except ImportError:
+                # The module is not a package.
+                _add_error(self._evaluator, import_path[-1])
+                return []
+
+        source = None
+        if is_pkg:
+            # In this case, we don't have a file yet. Search for the
+            # __init__ file.
+            module_path = get_init_path(module_path)
+        elif module_file:
+            source = module_file.read()
+            module_file.close()
+
+        if module_file is None and not module_path.endswith('.py'):
+            module = compiled.load_module(module_path)
+        else:
+            module = _load_module(self._evaluator, module_path, source, sys_path)
+
+        self._evaluator.modules[module_name] = module
+        return [module]
+
+    def _generate_name(self, name):
+        return helpers.FakeName(name, parent=self.module)
+
+    def _get_module_names(self, search_path=None):
+        """
+        Get the names of all modules in the search_path. This means file names
+        and not names defined in the files.
+        """
+
+        names = []
+        # add builtin module names
+        if search_path is None:
+            names += [self._generate_name(name) for name in sys.builtin_module_names]
+
+        if search_path is None:
+            search_path = self.sys_path_with_modifications()
+        for module_loader, name, is_pkg in pkgutil.iter_modules(search_path):
+            names.append(self._generate_name(name))
+        return names
+
+    def completion_names(self, evaluator, only_modules=False):
+        """
+        :param only_modules: Indicates wheter it's possible to import a
+            definition that is not defined in a module.
+        """
+        from jedi.evaluate import finder
+        names = []
+        if self.import_path:
+            # flask
+            if self.str_import_path == ('flask', 'ext'):
+                # List Flask extensions like ``flask_foo``
+                for mod in self._get_module_names():
+                    modname = str(mod)
+                    if modname.startswith('flask_'):
+                        extname = modname[len('flask_'):]
+                        names.append(self._generate_name(extname))
+                # Now the old style: ``flaskext.foo``
+                for dir in self.sys_path_with_modifications():
+                    flaskext = os.path.join(dir, 'flaskext')
+                    if os.path.isdir(flaskext):
+                        names += self._get_module_names([flaskext])
+
+            for scope in self.follow():
+                # Non-modules are not completable.
+                if not scope.type == 'file_input':  # not a module
+                    continue
+
+                # namespace packages
+                if isinstance(scope, tree.Module) and scope.path.endswith('__init__.py'):
+                    paths = scope.py__path__(self.sys_path_with_modifications())
+                    names += self._get_module_names(paths)
+
+                if only_modules:
+                    # In the case of an import like `from x.` we don't need to
+                    # add all the variables.
+                    if ('os',) == self.str_import_path and not self.level:
+                        # os.path is a hardcoded exception, because it's a
+                        # ``sys.modules`` modification.
+                        names.append(self._generate_name('path'))
+
+                    continue
+
+                for names_dict in scope.names_dicts(search_global=False):
+                    _names = list(chain.from_iterable(names_dict.values()))
+                    if not _names:
+                        continue
+                    _names = finder.filter_definition_names(_names, scope)
+                    names += _names
+        else:
+            # Empty import path=completion after import
+            if not self.level:
+                names += self._get_module_names()
+
+            if self.file_path is not None:
+                path = os.path.abspath(self.file_path)
+                for i in range(self.level - 1):
+                    path = os.path.dirname(path)
+                names += self._get_module_names([path])
+
+        return names
+
+
+def _load_module(evaluator, path=None, source=None, sys_path=None):
+    def load(source):
+        dotted_path = path and compiled.dotted_from_fs_path(path, sys_path)
+        if path is not None and path.endswith('.py') \
+                and not dotted_path in settings.auto_import_modules:
+            if source is None:
+                with open(path, 'rb') as f:
+                    source = f.read()
+        else:
+            return compiled.load_module(path)
+        p = path
+        p = fast.FastParser(evaluator.grammar, common.source_to_unicode(source), p)
+        cache.save_parser(path, p)
+        return p.module
+
+    cached = cache.load_parser(path)
+    module = load(source) if cached is None else cached.module
+    module = evaluator.wrap(module)
+    return module
+
+
+def add_module(evaluator, module_name, module):
+    if '.' not in module_name:
+        # We cannot add paths with dots, because that would collide with
+        # the sepatator dots for nested packages. Therefore we return
+        # `__main__` in ModuleWrapper.py__name__(), which is similar to
+        # Python behavior.
+        evaluator.modules[module_name] = module
+
+
+def get_modules_containing_name(evaluator, mods, name):
+    """
+    Search a name in the directories of modules.
+    """
+    def check_python_file(path):
+        try:
+            return cache.parser_cache[path].parser.module
+        except KeyError:
+            try:
+                return check_fs(path)
+            except IOError:
+                return None
+
+    def check_fs(path):
+        with open(path, 'rb') as f:
+            source = source_to_unicode(f.read())
+            if name in source:
+                module_name = os.path.basename(path)[:-3]  # Remove `.py`.
+                module = _load_module(evaluator, path, source)
+                add_module(evaluator, module_name, module)
+                return module
+
+    # skip non python modules
+    mods = set(m for m in mods if not isinstance(m, compiled.CompiledObject))
+    mod_paths = set()
+    for m in mods:
+        mod_paths.add(m.path)
+        yield m
+
+    if settings.dynamic_params_for_other_modules:
+        paths = set(settings.additional_dynamic_modules)
+        for p in mod_paths:
+            if p is not None:
+                d = os.path.dirname(p)
+                for entry in os.listdir(d):
+                    if entry not in mod_paths:
+                        if entry.endswith('.py'):
+                            paths.add(d + os.path.sep + entry)
+
+        for p in sorted(paths):
+            # make testing easier, sort it - same results on every interpreter
+            c = check_python_file(p)
+            if c is not None and c not in mods and not isinstance(c, compiled.CompiledObject):
+                yield c

jedi/evaluate/iterable.py

@@ -0,0 +1,631 @@
+"""
+Contains all classes and functions to deal with lists, dicts, generators and
+iterators in general.
+
+Array modifications
+*******************
+
+If the content of an array (``set``/``list``) is requested somewhere, the
+current module will be checked for appearances of ``arr.append``,
+``arr.insert``, etc.  If the ``arr`` name points to an actual array, the
+content will be added
+
+This can be really cpu intensive, as you can imagine. Because |jedi| has to
+follow **every** ``append`` and check wheter it's the right array. However this
+works pretty good, because in *slow* cases, the recursion detector and other
+settings will stop this process.
+
+It is important to note that:
+
+1. Array modfications work only in the current module.
+2. Jedi only checks Array additions; ``list.pop``, etc are ignored.
+"""
+from itertools import chain
+
+from jedi import common
+from jedi import debug
+from jedi import settings
+from jedi._compatibility import use_metaclass, is_py3, unicode
+from jedi.parser import tree
+from jedi.evaluate import compiled
+from jedi.evaluate import helpers
+from jedi.evaluate.cache import CachedMetaClass, memoize_default
+from jedi.evaluate import analysis
+
+
+def unite(iterable):
+    """Turns a two dimensional array into a one dimensional."""
+    return list(chain.from_iterable(iterable))
+
+
+class IterableWrapper(tree.Base):
+    def is_class(self):
+        return False
+
+
+class GeneratorMixin(object):
+    @memoize_default()
+    def names_dicts(self, search_global=False):  # is always False
+        dct = {}
+        executes_generator = '__next__', 'send', 'next'
+        for names in compiled.generator_obj.names_dict.values():
+            for name in names:
+                if name.value in executes_generator:
+                    parent = GeneratorMethod(self, name.parent)
+                    dct[name.value] = [helpers.FakeName(name.name, parent, is_definition=True)]
+                else:
+                    dct[name.value] = [name]
+        yield dct
+
+    def get_index_types(self, evaluator, index_array):
+        #debug.warning('Tried to get array access on a generator: %s', self)
+        analysis.add(self._evaluator, 'type-error-generator', index_array)
+        return []
+
+    def get_exact_index_types(self, index):
+        """
+        Exact lookups are used for tuple lookups, which are perfectly fine if
+        used with generators.
+        """
+        return [self.iter_content()[index]]
+
+    def py__bool__(self):
+        return True
+
+
+class Generator(use_metaclass(CachedMetaClass, IterableWrapper, GeneratorMixin)):
+    """Handling of `yield` functions."""
+    def __init__(self, evaluator, func, var_args):
+        super(Generator, self).__init__()
+        self._evaluator = evaluator
+        self.func = func
+        self.var_args = var_args
+
+    def iter_content(self):
+        """ returns the content of __iter__ """
+        # Directly execute it, because with a normal call to py__call__ a
+        # Generator will be returned.
+        from jedi.evaluate.representation import FunctionExecution
+        f = FunctionExecution(self._evaluator, self.func, self.var_args)
+        return f.get_return_types(check_yields=True)
+
+    def __getattr__(self, name):
+        if name not in ['start_pos', 'end_pos', 'parent', 'get_imports',
+                        'doc', 'docstr', 'get_parent_until',
+                        'get_code', 'subscopes']:
+            raise AttributeError("Accessing %s of %s is not allowed."
+                                 % (self, name))
+        return getattr(self.func, name)
+
+    def __repr__(self):
+        return "<%s of %s>" % (type(self).__name__, self.func)
+
+
+class GeneratorMethod(IterableWrapper):
+    """``__next__`` and ``send`` methods."""
+    def __init__(self, generator, builtin_func):
+        self._builtin_func = builtin_func
+        self._generator = generator
+
+    def py__call__(self, evaluator, params):
+        # TODO add TypeError if params are given.
+        return self._generator.iter_content()
+
+    def __getattr__(self, name):
+        return getattr(self._builtin_func, name)
+
+
+class Comprehension(IterableWrapper):
+    @staticmethod
+    def from_atom(evaluator, atom):
+        mapping = {
+            '(': GeneratorComprehension,
+            '[': ListComprehension
+        }
+        return mapping[atom.children[0]](evaluator, atom)
+
+    def __init__(self, evaluator, atom):
+        self._evaluator = evaluator
+        self._atom = atom
+
+    @memoize_default()
+    def eval_node(self):
+        """
+        The first part `x + 1` of the list comprehension:
+
+            [x + 1 for x in foo]
+        """
+        comprehension = self._atom.children[1]
+        # For nested comprehensions we need to search the last one.
+        last = comprehension.children[-1]
+        last_comp = comprehension.children[1]
+        while True:
+            if isinstance(last, tree.CompFor):
+                last_comp = last
+            elif not tree.is_node(last, 'comp_if'):
+                break
+            last = last.children[-1]
+
+        return helpers.deep_ast_copy(comprehension.children[0], parent=last_comp)
+
+    def get_exact_index_types(self, index):
+        return [self._evaluator.eval_element(self.eval_node())[index]]
+
+    def __repr__(self):
+        return "<e%s of %s>" % (type(self).__name__, self._atom)
+
+
+class ArrayMixin(object):
+    @memoize_default()
+    def names_dicts(self, search_global=False):  # Always False.
+        # `array.type` is a string with the type, e.g. 'list'.
+        scope = self._evaluator.find_types(compiled.builtin, self.type)[0]
+        # builtins only have one class -> [0]
+        scope = self._evaluator.execute(scope, (AlreadyEvaluated((self,)),))[0]
+        return scope.names_dicts(search_global)
+
+    def py__bool__(self):
+        return None  # We don't know the length, because of appends.
+
+
+class ListComprehension(Comprehension, ArrayMixin):
+    type = 'list'
+
+    def get_index_types(self, evaluator, index):
+        return self.iter_content()
+
+    def iter_content(self):
+        return self._evaluator.eval_element(self.eval_node())
+
+    @property
+    def name(self):
+        return FakeSequence(self._evaluator, [], 'list').name
+
+
+class GeneratorComprehension(Comprehension, GeneratorMixin):
+    def iter_content(self):
+        return self._evaluator.eval_element(self.eval_node())
+
+
+class Array(IterableWrapper, ArrayMixin):
+    mapping = {'(': 'tuple',
+               '[': 'list',
+               '{': 'dict'}
+
+    def __init__(self, evaluator, atom):
+        self._evaluator = evaluator
+        self.atom = atom
+        self.type = Array.mapping[atom.children[0]]
+        """The builtin name of the array (list, set, tuple or dict)."""
+
+        c = self.atom.children
+        array_node = c[1]
+        if self.type == 'dict' and array_node != '}' \
+                and (not hasattr(array_node, 'children')
+                     or ':' not in array_node.children):
+            self.type = 'set'
+
+    @property
+    def name(self):
+        return helpers.FakeName(self.type, parent=self)
+
+    @memoize_default()
+    def get_index_types(self, evaluator, index=()):
+        """
+        Get the types of a specific index or all, if not given.
+
+        :param index: A subscriptlist node (or subnode).
+        """
+        indexes = create_indexes_or_slices(evaluator, index)
+        lookup_done = False
+        types = []
+        for index in indexes:
+            if isinstance(index, Slice):
+                types += [self]
+                lookup_done = True
+            elif isinstance(index, compiled.CompiledObject) \
+                    and isinstance(index.obj, (int, str, unicode)):
+                with common.ignored(KeyError, IndexError, TypeError):
+                    types += self.get_exact_index_types(index.obj)
+                    lookup_done = True
+
+        return types if lookup_done else self.values()
+
+    @memoize_default()
+    def values(self):
+        result = unite(self._evaluator.eval_element(v) for v in self._values())
+        result += check_array_additions(self._evaluator, self)
+        return result
+
+    def get_exact_index_types(self, mixed_index):
+        """ Here the index is an int/str. Raises IndexError/KeyError """
+        if self.type == 'dict':
+            for key, values in self._items():
+                # Because we only want the key to be a string.
+                keys = self._evaluator.eval_element(key)
+
+                for k in keys:
+                    if isinstance(k, compiled.CompiledObject) \
+                            and mixed_index == k.obj:
+                        for value in values:
+                            return self._evaluator.eval_element(value)
+            raise KeyError('No key found in dictionary %s.' % self)
+
+        # Can raise an IndexError
+        return self._evaluator.eval_element(self._items()[mixed_index])
+
+    def iter_content(self):
+        return self.values()
+
+    @common.safe_property
+    def parent(self):
+        return compiled.builtin
+
+    def get_parent_until(self):
+        return compiled.builtin
+
+    def __getattr__(self, name):
+        if name not in ['start_pos', 'get_only_subelement', 'parent',
+                        'get_parent_until', 'items']:
+            raise AttributeError('Strange access on %s: %s.' % (self, name))
+        return getattr(self.atom, name)
+
+    def _values(self):
+        """Returns a list of a list of node."""
+        if self.type == 'dict':
+            return list(chain.from_iterable(v for k, v in self._items()))
+        else:
+            return self._items()
+
+    def _items(self):
+        c = self.atom.children
+        array_node = c[1]
+        if array_node in (']', '}', ')'):
+            return []  # Direct closing bracket, doesn't contain items.
+
+        if tree.is_node(array_node, 'testlist_comp'):
+            return array_node.children[::2]
+        elif tree.is_node(array_node, 'dictorsetmaker'):
+            kv = []
+            iterator = iter(array_node.children)
+            for key in iterator:
+                op = next(iterator, None)
+                if op is None or op == ',':
+                    kv.append(key)  # A set.
+                elif op == ':':  # A dict.
+                    kv.append((key, [next(iterator)]))
+                    next(iterator, None)  # Possible comma.
+                else:
+                    raise NotImplementedError('dict/set comprehensions')
+            return kv
+        else:
+            return [array_node]
+
+    def __iter__(self):
+        return iter(self._items())
+
+    def __repr__(self):
+        return "<%s of %s>" % (type(self).__name__, self.atom)
+
+
+class _FakeArray(Array):
+    def __init__(self, evaluator, container, type):
+        self.type = type
+        self._evaluator = evaluator
+        self.atom = container
+
+
+class ImplicitTuple(_FakeArray):
+    def __init__(self, evaluator, testlist):
+        super(ImplicitTuple, self).__init__(evaluator, testlist, 'tuple')
+        self._testlist = testlist
+
+    def _items(self):
+        return self._testlist.children[::2]
+
+
+class FakeSequence(_FakeArray):
+    def __init__(self, evaluator, sequence_values, type):
+        super(FakeSequence, self).__init__(evaluator, sequence_values, type)
+        self._sequence_values = sequence_values
+
+    def _items(self):
+        return self._sequence_values
+
+    def get_exact_index_types(self, index):
+        value = self._sequence_values[index]
+        return self._evaluator.eval_element(value)
+
+
+class AlreadyEvaluated(frozenset):
+    """A simple container to add already evaluated objects to an array."""
+    def get_code(self):
+        # For debugging purposes.
+        return str(self)
+
+
+class MergedNodes(frozenset):
+    pass
+
+
+class FakeDict(_FakeArray):
+    def __init__(self, evaluator, dct):
+        super(FakeDict, self).__init__(evaluator, dct, 'dict')
+        self._dct = dct
+
+    def get_exact_index_types(self, index):
+        return list(chain.from_iterable(self._evaluator.eval_element(v)
+                                        for v in self._dct[index]))
+
+    def _items(self):
+        return self._dct.items()
+
+
+class MergedArray(_FakeArray):
+    def __init__(self, evaluator, arrays):
+        super(MergedArray, self).__init__(evaluator, arrays, arrays[-1].type)
+        self._arrays = arrays
+
+    def get_exact_index_types(self, mixed_index):
+        raise IndexError
+
+    def values(self):
+        return list(chain(*(a.values() for a in self._arrays)))
+
+    def __iter__(self):
+        for array in self._arrays:
+            for a in array:
+                yield a
+
+    def __len__(self):
+        return sum(len(a) for a in self._arrays)
+
+
+def get_iterator_types(inputs):
+    """Returns the types of any iterator (arrays, yields, __iter__, etc)."""
+    iterators = []
+    # Take the first statement (for has always only
+    # one, remember `in`). And follow it.
+    for it in inputs:
+        if isinstance(it, (Generator, Array, ArrayInstance, Comprehension)):
+            iterators.append(it)
+        else:
+            if not hasattr(it, 'execute_subscope_by_name'):
+                debug.warning('iterator/for loop input wrong: %s', it)
+                continue
+            try:
+                iterators += it.execute_subscope_by_name('__iter__')
+            except KeyError:
+                debug.warning('iterators: No __iter__ method found.')
+
+    result = []
+    from jedi.evaluate.representation import Instance
+    for it in iterators:
+        if isinstance(it, Array):
+            # Array is a little bit special, since this is an internal array,
+            # but there's also the list builtin, which is another thing.
+            result += it.values()
+        elif isinstance(it, Instance):
+            # __iter__ returned an instance.
+            name = '__next__' if is_py3 else 'next'
+            try:
+                result += it.execute_subscope_by_name(name)
+            except KeyError:
+                debug.warning('Instance has no __next__ function in %s.', it)
+        else:
+            # TODO this is not correct, __iter__ can return arbitrary input!
+            # Is a generator.
+            result += it.iter_content()
+    return result
+
+
+def check_array_additions(evaluator, array):
+    """ Just a mapper function for the internal _check_array_additions """
+    if array.type not in ('list', 'set'):
+        # TODO also check for dict updates
+        return []
+
+    is_list = array.type == 'list'
+    try:
+        current_module = array.atom.get_parent_until()
+    except AttributeError:
+        # If there's no get_parent_until, it's a FakeSequence or another Fake
+        # type. Those fake types are used inside Jedi's engine. No values may
+        # be added to those after their creation.
+        return []
+    return _check_array_additions(evaluator, array, current_module, is_list)
+
+
+@memoize_default([], evaluator_is_first_arg=True)
+def _check_array_additions(evaluator, compare_array, module, is_list):
+    """
+    Checks if a `Array` has "add" (append, insert, extend) statements:
+
+    >>> a = [""]
+    >>> a.append(1)
+    """
+    if not settings.dynamic_array_additions or isinstance(module, compiled.CompiledObject):
+        return []
+
+    def check_additions(arglist, add_name):
+        params = list(param.Arguments(evaluator, arglist).unpack())
+        result = []
+        if add_name in ['insert']:
+            params = params[1:]
+        if add_name in ['append', 'add', 'insert']:
+            for key, nodes in params:
+                result += unite(evaluator.eval_element(node) for node in nodes)
+        elif add_name in ['extend', 'update']:
+            for key, nodes in params:
+                iterators = unite(evaluator.eval_element(node) for node in nodes)
+                result += get_iterator_types(iterators)
+        return result
+
+    from jedi.evaluate import representation as er, param
+
+    def get_execution_parent(element):
+        """ Used to get an Instance/FunctionExecution parent """
+        if isinstance(element, Array):
+            node = element.atom
+        else:
+            # Is an Instance with an
+            # Arguments([AlreadyEvaluated([ArrayInstance])]) inside
+            # Yeah... I know... It's complicated ;-)
+            node = list(element.var_args.argument_node[0])[0].var_args.trailer
+        if isinstance(node, er.InstanceElement):
+            return node
+        return node.get_parent_until(er.FunctionExecution)
+
+    temp_param_add, settings.dynamic_params_for_other_modules = \
+        settings.dynamic_params_for_other_modules, False
+
+    search_names = ['append', 'extend', 'insert'] if is_list else ['add', 'update']
+    comp_arr_parent = get_execution_parent(compare_array)
+
+    added_types = []
+    for add_name in search_names:
+        try:
+            possible_names = module.used_names[add_name]
+        except KeyError:
+            continue
+        else:
+            for name in possible_names:
+                # Check if the original scope is an execution. If it is, one
+                # can search for the same statement, that is in the module
+                # dict. Executions are somewhat special in jedi, since they
+                # literally copy the contents of a function.
+                if isinstance(comp_arr_parent, er.FunctionExecution):
+                    if comp_arr_parent.start_pos < name.start_pos < comp_arr_parent.end_pos:
+                        name = comp_arr_parent.name_for_position(name.start_pos)
+                    else:
+                        # Don't check definitions that are not defined in the
+                        # same function. This is not "proper" anyway. It also
+                        # improves Jedi's speed for array lookups, since we
+                        # don't have to check the whole source tree anymore.
+                        continue
+                trailer = name.parent
+                power = trailer.parent
+                trailer_pos = power.children.index(trailer)
+                try:
+                    execution_trailer = power.children[trailer_pos + 1]
+                except IndexError:
+                    continue
+                else:
+                    if execution_trailer.type != 'trailer' \
+                            or execution_trailer.children[0] != '(' \
+                            or execution_trailer.children[1] == ')':
+                        continue
+                power = helpers.call_of_name(name, cut_own_trailer=True)
+                # InstanceElements are special, because they don't get copied,
+                # but have this wrapper around them.
+                if isinstance(comp_arr_parent, er.InstanceElement):
+                    power = er.get_instance_el(evaluator, comp_arr_parent.instance, power)
+
+                if evaluator.recursion_detector.push_stmt(power):
+                    # Check for recursion. Possible by using 'extend' in
+                    # combination with function calls.
+                    continue
+                if compare_array in evaluator.eval_element(power):
+                    # The arrays match. Now add the results
+                    added_types += check_additions(execution_trailer.children[1], add_name)
+
+                evaluator.recursion_detector.pop_stmt()
+    # reset settings
+    settings.dynamic_params_for_other_modules = temp_param_add
+    return added_types
+
+
+def check_array_instances(evaluator, instance):
+    """Used for set() and list() instances."""
+    if not settings.dynamic_array_additions:
+        return instance.var_args
+
+    ai = ArrayInstance(evaluator, instance)
+    from jedi.evaluate import param
+    return param.Arguments(evaluator, [AlreadyEvaluated([ai])])
+
+
+class ArrayInstance(IterableWrapper):
+    """
+    Used for the usage of set() and list().
+    This is definitely a hack, but a good one :-)
+    It makes it possible to use set/list conversions.
+
+    In contrast to Array, ListComprehension and all other iterable types, this
+    is something that is only used inside `evaluate/compiled/fake/builtins.py`
+    and therefore doesn't need `names_dicts`, `py__bool__` and so on, because
+    we don't use these operations in `builtins.py`.
+    """
+    def __init__(self, evaluator, instance):
+        self._evaluator = evaluator
+        self.instance = instance
+        self.var_args = instance.var_args
+
+    def iter_content(self):
+        """
+        The index is here just ignored, because of all the appends, etc.
+        lists/sets are too complicated too handle that.
+        """
+        items = []
+        for key, nodes in self.var_args.unpack():
+            for node in nodes:
+                for typ in self._evaluator.eval_element(node):
+                    items += get_iterator_types([typ])
+
+        module = self.var_args.get_parent_until()
+        is_list = str(self.instance.name) == 'list'
+        items += _check_array_additions(self._evaluator, self.instance, module, is_list)
+        return items
+
+
+class Slice(object):
+    def __init__(self, evaluator, start, stop, step):
+        self._evaluator = evaluator
+        # all of them are either a Precedence or None.
+        self._start = start
+        self._stop = stop
+        self._step = step
+
+    @property
+    def obj(self):
+        """
+        Imitate CompiledObject.obj behavior and return a ``builtin.slice()``
+        object.
+        """
+        def get(element):
+            if element is None:
+                return None
+
+            result = self._evaluator.eval_element(element)
+            if len(result) != 1:
+                # We want slices to be clear defined with just one type.
+                # Otherwise we will return an empty slice object.
+                raise IndexError
+            try:
+                return result[0].obj
+            except AttributeError:
+                return None
+
+        try:
+            return slice(get(self._start), get(self._stop), get(self._step))
+        except IndexError:
+            return slice(None, None, None)
+
+
+def create_indexes_or_slices(evaluator, index):
+    if tree.is_node(index, 'subscript'):  # subscript is a slice operation.
+        start, stop, step = None, None, None
+        result = []
+        for el in index.children:
+            if el == ':':
+                if not result:
+                    result.append(None)
+            elif tree.is_node(el, 'sliceop'):
+                if len(el.children) == 2:
+                    result.append(el.children[1])
+            else:
+                result.append(el)
+        result += [None] * (3 - len(result))
+
+        return (Slice(evaluator, *result),)
+    return evaluator.eval_element(index)

jedi/evaluate/param.py

@@ -0,0 +1,403 @@
+from collections import defaultdict
+from itertools import chain
+
+from jedi._compatibility import unicode, zip_longest
+from jedi import debug
+from jedi import common
+from jedi.parser import tree
+from jedi.evaluate import iterable
+from jedi.evaluate import analysis
+from jedi.evaluate import precedence
+from jedi.evaluate.helpers import FakeName
+from jedi.cache import underscore_memoization
+
+
+class Arguments(tree.Base):
+    def __init__(self, evaluator, argument_node, trailer=None):
+        """
+        The argument_node is either a parser node or a list of evaluated
+        objects. Those evaluated objects may be lists of evaluated objects
+        themselves (one list for the first argument, one for the second, etc).
+
+        :param argument_node: May be an argument_node or a list of nodes.
+        """
+        self.argument_node = argument_node
+        self._evaluator = evaluator
+        self.trailer = trailer  # Can be None, e.g. in a class definition.
+
+    def _split(self):
+        if isinstance(self.argument_node, (tuple, list)):
+            for el in self.argument_node:
+                yield 0, el
+        else:
+            if not tree.is_node(self.argument_node, 'arglist'):
+                yield 0, self.argument_node
+                return
+
+            iterator = iter(self.argument_node.children)
+            for child in iterator:
+                if child == ',':
+                    continue
+                elif child in ('*', '**'):
+                    yield len(child.value), next(iterator)
+                else:
+                    yield 0, child
+
+    def get_parent_until(self, *args, **kwargs):
+        if self.trailer is None:
+            try:
+                element = self.argument_node[0]
+                from jedi.evaluate.iterable import AlreadyEvaluated
+                if isinstance(element, AlreadyEvaluated):
+                    element = self._evaluator.eval_element(element)[0]
+            except IndexError:
+                return None
+            else:
+                return element.get_parent_until(*args, **kwargs)
+        else:
+            return self.trailer.get_parent_until(*args, **kwargs)
+
+    def as_tuple(self):
+        for stars, argument in self._split():
+            if tree.is_node(argument, 'argument'):
+                argument, default = argument.children[::2]
+            else:
+                default = None
+            yield argument, default, stars
+
+    def unpack(self, func=None):
+        named_args = []
+        for stars, el in self._split():
+            if stars == 1:
+                arrays = self._evaluator.eval_element(el)
+                iterators = [_iterate_star_args(self._evaluator, a, el, func)
+                             for a in arrays]
+                iterators = list(iterators)
+                for values in list(zip_longest(*iterators)):
+                    yield None, [v for v in values if v is not None]
+            elif stars == 2:
+                arrays = self._evaluator.eval_element(el)
+                dicts = [_star_star_dict(self._evaluator, a, el, func)
+                         for a in arrays]
+                for dct in dicts:
+                    for key, values in dct.items():
+                        yield key, values
+            else:
+                if tree.is_node(el, 'argument'):
+                    c = el.children
+                    if len(c) == 3:  # Keyword argument.
+                        named_args.append((c[0].value, (c[2],)))
+                    else:  # Generator comprehension.
+                        # Include the brackets with the parent.
+                        comp = iterable.GeneratorComprehension(
+                            self._evaluator, self.argument_node.parent)
+                        yield None, (iterable.AlreadyEvaluated([comp]),)
+                elif isinstance(el, (list, tuple)):
+                    yield None, el
+                else:
+                    yield None, (el,)
+
+        # Reordering var_args is necessary, because star args sometimes appear
+        # after named argument, but in the actual order it's prepended.
+        for key_arg in named_args:
+            yield key_arg
+
+    def _reorder_var_args(var_args):
+        named_index = None
+        new_args = []
+        for i, stmt in enumerate(var_args):
+            if isinstance(stmt, tree.ExprStmt):
+                if named_index is None and stmt.assignment_details:
+                    named_index = i
+
+                if named_index is not None:
+                    expression_list = stmt.expression_list()
+                    if expression_list and expression_list[0] == '*':
+                        new_args.insert(named_index, stmt)
+                        named_index += 1
+                        continue
+
+            new_args.append(stmt)
+        return new_args
+
+    def eval_argument_clinic(self, arguments):
+        """Uses a list with argument clinic information (see PEP 436)."""
+        iterator = self.unpack()
+        for i, (name, optional, allow_kwargs) in enumerate(arguments):
+            key, va_values = next(iterator, (None, []))
+            if key is not None:
+                raise NotImplementedError
+            if not va_values and not optional:
+                debug.warning('TypeError: %s expected at least %s arguments, got %s',
+                              name, len(arguments), i)
+                raise ValueError
+            values = list(chain.from_iterable(self._evaluator.eval_element(el)
+                                              for el in va_values))
+            if not values and not optional:
+                # For the stdlib we always want values. If we don't get them,
+                # that's ok, maybe something is too hard to resolve, however,
+                # we will not proceed with the evaluation of that function.
+                debug.warning('argument_clinic "%s" not resolvable.', name)
+                raise ValueError
+            yield values
+
+    def scope(self):
+        # Returns the scope in which the arguments are used.
+        return (self.trailer or self.argument_node).get_parent_until(tree.IsScope)
+
+    def eval_args(self):
+        # TODO this method doesn't work with named args and a lot of other
+        # things. Use unpack.
+        return [self._evaluator.eval_element(el) for stars, el in self._split()]
+
+    def __repr__(self):
+        return '<%s: %s>' % (type(self).__name__, self.argument_node)
+
+    def get_calling_var_args(self):
+        if tree.is_node(self.argument_node, 'arglist', 'argument') \
+                or self.argument_node == () and self.trailer is not None:
+            return _get_calling_var_args(self._evaluator, self)
+        else:
+            return None
+
+
+class ExecutedParam(tree.Param):
+    """Fake a param and give it values."""
+    def __init__(self, original_param, var_args, values):
+        self._original_param = original_param
+        self.var_args = var_args
+        self._values = values
+
+    def eval(self, evaluator):
+        types = []
+        for v in self._values:
+            types += evaluator.eval_element(v)
+        return types
+
+    @property
+    def position_nr(self):
+        # Need to use the original logic here, because it uses the parent.
+        return self._original_param.position_nr
+
+    @property
+    @underscore_memoization
+    def name(self):
+        return FakeName(str(self._original_param.name), self, self.start_pos)
+
+    def __getattr__(self, name):
+        return getattr(self._original_param, name)
+
+
+def _get_calling_var_args(evaluator, var_args):
+    old_var_args = None
+    while var_args != old_var_args:
+        old_var_args = var_args
+        for name, default, stars in reversed(list(var_args.as_tuple())):
+            if not stars or not isinstance(name, tree.Name):
+                continue
+
+            names = evaluator.goto(name)
+            if len(names) != 1:
+                break
+            param = names[0].get_definition()
+            if not isinstance(param, ExecutedParam):
+                if isinstance(param, tree.Param):
+                    # There is no calling var_args in this case - there's just
+                    # a param without any input.
+                    return None
+                break
+            # We never want var_args to be a tuple. This should be enough for
+            # now, we can change it later, if we need to.
+            if isinstance(param.var_args, Arguments):
+                var_args = param.var_args
+    return var_args.argument_node or var_args.trailer
+
+
+def get_params(evaluator, func, var_args):
+    param_names = []
+    param_dict = {}
+    for param in func.params:
+        param_dict[str(param.name)] = param
+    unpacked_va = list(var_args.unpack(func))
+    from jedi.evaluate.representation import InstanceElement
+    if isinstance(func, InstanceElement):
+        # Include self at this place.
+        unpacked_va.insert(0, (None, [iterable.AlreadyEvaluated([func.instance])]))
+    var_arg_iterator = common.PushBackIterator(iter(unpacked_va))
+
+    non_matching_keys = defaultdict(lambda: [])
+    keys_used = {}
+    keys_only = False
+    had_multiple_value_error = False
+    for param in func.params:
+        # The value and key can both be null. There, the defaults apply.
+        # args / kwargs will just be empty arrays / dicts, respectively.
+        # Wrong value count is just ignored. If you try to test cases that are
+        # not allowed in Python, Jedi will maybe not show any completions.
+        default = [] if param.default is None else [param.default]
+        key, va_values = next(var_arg_iterator, (None, default))
+        while key is not None:
+            keys_only = True
+            k = unicode(key)
+            try:
+                key_param = param_dict[unicode(key)]
+            except KeyError:
+                non_matching_keys[key] += va_values
+            else:
+                param_names.append(ExecutedParam(key_param, var_args, va_values).name)
+
+            if k in keys_used:
+                had_multiple_value_error = True
+                m = ("TypeError: %s() got multiple values for keyword argument '%s'."
+                     % (func.name, k))
+                calling_va = _get_calling_var_args(evaluator, var_args)
+                if calling_va is not None:
+                    analysis.add(evaluator, 'type-error-multiple-values',
+                                 calling_va, message=m)
+            else:
+                try:
+                    keys_used[k] = param_names[-1]
+                except IndexError:
+                    # TODO this is wrong stupid and whatever.
+                    pass
+            key, va_values = next(var_arg_iterator, (None, ()))
+
+        values = []
+        if param.stars == 1:
+            # *args param
+            lst_values = [iterable.MergedNodes(va_values)] if va_values else []
+            for key, va_values in var_arg_iterator:
+                # Iterate until a key argument is found.
+                if key:
+                    var_arg_iterator.push_back((key, va_values))
+                    break
+                if va_values:
+                    lst_values.append(iterable.MergedNodes(va_values))
+            seq = iterable.FakeSequence(evaluator, lst_values, 'tuple')
+            values = [iterable.AlreadyEvaluated([seq])]
+        elif param.stars == 2:
+            # **kwargs param
+            dct = iterable.FakeDict(evaluator, dict(non_matching_keys))
+            values = [iterable.AlreadyEvaluated([dct])]
+            non_matching_keys = {}
+        else:
+            # normal param
+            if va_values:
+                values = va_values
+            else:
+                # No value: Return an empty container
+                values = []
+                if not keys_only:
+                    calling_va = var_args.get_calling_var_args()
+                    if calling_va is not None:
+                        m = _error_argument_count(func, len(unpacked_va))
+                        analysis.add(evaluator, 'type-error-too-few-arguments',
+                                     calling_va, message=m)
+
+        # Now add to result if it's not one of the previously covered cases.
+        if (not keys_only or param.stars == 2):
+            param_names.append(ExecutedParam(param, var_args, values).name)
+            keys_used[unicode(param.name)] = param_names[-1]
+
+    if keys_only:
+        # All arguments should be handed over to the next function. It's not
+        # about the values inside, it's about the names. Jedi needs to now that
+        # there's nothing to find for certain names.
+        for k in set(param_dict) - set(keys_used):
+            param = param_dict[k]
+            values = [] if param.default is None else [param.default]
+            param_names.append(ExecutedParam(param, var_args, values).name)
+
+            if not (non_matching_keys or had_multiple_value_error
+                    or param.stars or param.default):
+                # add a warning only if there's not another one.
+                calling_va = _get_calling_var_args(evaluator, var_args)
+                if calling_va is not None:
+                    m = _error_argument_count(func, len(unpacked_va))
+                    analysis.add(evaluator, 'type-error-too-few-arguments',
+                                 calling_va, message=m)
+
+    for key, va_values in non_matching_keys.items():
+        m = "TypeError: %s() got an unexpected keyword argument '%s'." \
+            % (func.name, key)
+        for value in va_values:
+            analysis.add(evaluator, 'type-error-keyword-argument', value.parent, message=m)
+
+    remaining_params = list(var_arg_iterator)
+    if remaining_params:
+        m = _error_argument_count(func, len(unpacked_va))
+        # Just report an error for the first param that is not needed (like
+        # cPython).
+        first_key, first_values = remaining_params[0]
+        for v in first_values:
+            if first_key is not None:
+                # Is a keyword argument, return the whole thing instead of just
+                # the value node.
+                v = v.parent
+                try:
+                    non_kw_param = keys_used[first_key]
+                except KeyError:
+                    pass
+                else:
+                    origin_args = non_kw_param.parent.var_args.argument_node
+                    # TODO  calculate the var_args tree and check if it's in
+                    # the tree (if not continue).
+                    # print('\t\tnonkw', non_kw_param.parent.var_args.argument_node, )
+                    if origin_args not in [f.parent.parent for f in first_values]:
+                        continue
+            analysis.add(evaluator, 'type-error-too-many-arguments',
+                         v, message=m)
+    return param_names
+
+
+def _iterate_star_args(evaluator, array, input_node, func=None):
+    from jedi.evaluate.representation import Instance
+    if isinstance(array, iterable.Array):
+        for field_stmt in array:  # yield from plz!
+            yield field_stmt
+    elif isinstance(array, iterable.Generator):
+        for field_stmt in array.iter_content():
+            yield iterable.AlreadyEvaluated([field_stmt])
+    elif isinstance(array, Instance) and array.name.get_code() == 'tuple':
+        debug.warning('Ignored a tuple *args input %s' % array)
+    else:
+        if func is not None:
+            m = "TypeError: %s() argument after * must be a sequence, not %s" \
+                % (func.name.value, array)
+            analysis.add(evaluator, 'type-error-star', input_node, message=m)
+
+
+def _star_star_dict(evaluator, array, input_node, func):
+    dct = defaultdict(lambda: [])
+    from jedi.evaluate.representation import Instance
+    if isinstance(array, Instance) and array.name.get_code() == 'dict':
+        # For now ignore this case. In the future add proper iterators and just
+        # make one call without crazy isinstance checks.
+        return {}
+
+    if isinstance(array, iterable.FakeDict):
+        return array._dct
+    elif isinstance(array, iterable.Array) and array.type == 'dict':
+        # TODO bad call to non-public API
+        for key_node, values in array._items():
+            for key in evaluator.eval_element(key_node):
+                if precedence.is_string(key):
+                    dct[key.obj] += values
+
+    else:
+        if func is not None:
+            m = "TypeError: %s argument after ** must be a mapping, not %s" \
+                % (func.name.value, array)
+            analysis.add(evaluator, 'type-error-star-star', input_node, message=m)
+    return dict(dct)
+
+
+def _error_argument_count(func, actual_count):
+    default_arguments = sum(1 for p in func.params if p.default or p.stars)
+
+    if default_arguments == 0:
+        before = 'exactly '
+    else:
+        before = 'from %s to ' % (len(func.params) - default_arguments)
+    return ('TypeError: %s() takes %s%s arguments (%s given).'
+            % (func.name, before, len(func.params), actual_count))

jedi/evaluate/precedence.py

@@ -0,0 +1,174 @@
+"""
+Handles operator precedence.
+"""
+import operator
+
+from jedi._compatibility import unicode
+from jedi.parser import tree
+from jedi import debug
+from jedi.evaluate.compiled import (CompiledObject, create, builtin,
+                                    keyword_from_value, true_obj, false_obj)
+from jedi.evaluate import analysis
+
+# Maps Python syntax to the operator module.
+COMPARISON_OPERATORS = {
+    '==': operator.eq,
+    '!=': operator.ne,
+    'is': operator.is_,
+    'is not': operator.is_not,
+    '<': operator.lt,
+    '<=': operator.le,
+    '>': operator.gt,
+    '>=': operator.ge,
+}
+
+
+def _literals_to_types(evaluator, result):
+    # Changes literals ('a', 1, 1.0, etc) to its type instances (str(),
+    # int(), float(), etc).
+    for i, r in enumerate(result):
+        if is_literal(r):
+            # Literals are only valid as long as the operations are
+            # correct. Otherwise add a value-free instance.
+            cls = builtin.get_by_name(r.name.get_code())
+            result[i] = evaluator.execute(cls)[0]
+    return list(set(result))
+
+
+def calculate_children(evaluator, children):
+    """
+    Calculate a list of children with operators.
+    """
+    iterator = iter(children)
+    types = evaluator.eval_element(next(iterator))
+    for operator in iterator:
+        right = next(iterator)
+        if tree.is_node(operator, 'comp_op'):  # not in / is not
+            operator = ' '.join(str(c.value) for c in operator.children)
+
+        # handle lazy evaluation of and/or here.
+        if operator in ('and', 'or'):
+            left_bools = set([left.py__bool__() for left in types])
+            if left_bools == set([True]):
+                if operator == 'and':
+                    types = evaluator.eval_element(right)
+            elif left_bools == set([False]):
+                if operator != 'and':
+                    types = evaluator.eval_element(right)
+            # Otherwise continue, because of uncertainty.
+        else:
+            types = calculate(evaluator, types, operator,
+                              evaluator.eval_element(right))
+    debug.dbg('calculate_children types %s', types)
+    return types
+
+
+def calculate(evaluator, left_result, operator, right_result):
+    result = []
+    if not left_result or not right_result:
+        # illegal slices e.g. cause left/right_result to be None
+        result = (left_result or []) + (right_result or [])
+        result = _literals_to_types(evaluator, result)
+    else:
+        # I don't think there's a reasonable chance that a string
+        # operation is still correct, once we pass something like six
+        # objects.
+        if len(left_result) * len(right_result) > 6:
+            result = _literals_to_types(evaluator, left_result + right_result)
+        else:
+            for left in left_result:
+                for right in right_result:
+                    result += _element_calculate(evaluator, left, operator, right)
+    return result
+
+
+def factor_calculate(evaluator, types, operator):
+    """
+    Calculates `+`, `-`, `~` and `not` prefixes.
+    """
+    for typ in types:
+        if operator == '-':
+            if _is_number(typ):
+                yield create(evaluator, -typ.obj)
+        elif operator == 'not':
+            value = typ.py__bool__()
+            if value is None:  # Uncertainty.
+                return
+            yield keyword_from_value(not value)
+        else:
+            yield typ
+
+
+def _is_number(obj):
+    return isinstance(obj, CompiledObject) \
+        and isinstance(obj.obj, (int, float))
+
+
+def is_string(obj):
+    return isinstance(obj, CompiledObject) \
+        and isinstance(obj.obj, (str, unicode))
+
+
+def is_literal(obj):
+    return _is_number(obj) or is_string(obj)
+
+
+def _is_tuple(obj):
+    from jedi.evaluate import iterable
+    return isinstance(obj, iterable.Array) and obj.type == 'tuple'
+
+
+def _is_list(obj):
+    from jedi.evaluate import iterable
+    return isinstance(obj, iterable.Array) and obj.type == 'list'
+
+
+def _element_calculate(evaluator, left, operator, right):
+    from jedi.evaluate import iterable, representation as er
+    l_is_num = _is_number(left)
+    r_is_num = _is_number(right)
+    if operator == '*':
+        # for iterables, ignore * operations
+        if isinstance(left, iterable.Array) or is_string(left):
+            return [left]
+        elif isinstance(right, iterable.Array) or is_string(right):
+            return [right]
+    elif operator == '+':
+        if l_is_num and r_is_num or is_string(left) and is_string(right):
+            return [create(evaluator, left.obj + right.obj)]
+        elif _is_tuple(left) and _is_tuple(right) or _is_list(left) and _is_list(right):
+            return [iterable.MergedArray(evaluator, (left, right))]
+    elif operator == '-':
+        if l_is_num and r_is_num:
+            return [create(evaluator, left.obj - right.obj)]
+    elif operator == '%':
+        # With strings and numbers the left type typically remains. Except for
+        # `int() % float()`.
+        return [left]
+    elif operator in COMPARISON_OPERATORS:
+        operation = COMPARISON_OPERATORS[operator]
+        if isinstance(left, CompiledObject) and isinstance(right, CompiledObject):
+            # Possible, because the return is not an option. Just compare.
+            left = left.obj
+            right = right.obj
+
+        try:
+            return [keyword_from_value(operation(left, right))]
+        except TypeError:
+            # Could be True or False.
+            return [true_obj, false_obj]
+    elif operator == 'in':
+        return []
+
+    def check(obj):
+        """Checks if a Jedi object is either a float or an int."""
+        return isinstance(obj, er.Instance) and obj.name.get_code() in ('int', 'float')
+
+    # Static analysis, one is a number, the other one is not.
+    if operator in ('+', '-') and l_is_num != r_is_num \
+            and not (check(left) or check(right)):
+        message = "TypeError: unsupported operand type(s) for +: %s and %s"
+        analysis.add(evaluator, 'type-error-operation', operator,
+                     message % (left, right))
+
+    return [left, right]

jedi/evaluate/recursion.py

@@ -3,41 +3,44 @@ Recursions are the recipe of |jedi| to conquer Python code. However, someone
 must stop recursions going mad. Some settings are here to make |jedi| stop at
 the right time. You can read more about them :ref:`here <settings-recursion>`.
 
-Next to :mod:`cache` this module also makes |jedi| not thread-safe. Why?
-``ExecutionRecursionDecorator`` uses class variables to count the function
-calls.
+Next to :mod:`jedi.evaluate.cache` this module also makes |jedi| not
+thread-safe. Why?  ``execution_recursion_decorator`` uses class variables to
+count the function calls.
 """
-from jedi import parsing_representation as pr
 from jedi import debug
 from jedi import settings
-import evaluate_representation as er
-import builtin
+from jedi.evaluate import compiled
+from jedi.evaluate import iterable
 
 
-class RecursionDecorator(object):
+def recursion_decorator(func):
+    def run(evaluator, stmt, *args, **kwargs):
+        rec_detect = evaluator.recursion_detector
+        # print stmt, len(self.node_statements())
+        if rec_detect.push_stmt(stmt):
+            return []
+        else:
+            result = func(evaluator, stmt, *args, **kwargs)
+            rec_detect.pop_stmt()
+        return result
+    return run
+
+
+class RecursionDetector(object):
     """
     A decorator to detect recursions in statements. In a recursion a statement
     at the same place, in the same module may not be executed two times.
     """
-    def __init__(self, func):
-        self.func = func
-        self.reset()
-
-    def __call__(self, stmt, *args, **kwargs):
-        #print stmt, len(self.node_statements())
-        if self.push_stmt(stmt):
-            return []
-        else:
-            result = self.func(stmt, *args, **kwargs)
-            self.pop_stmt()
-        return result
+    def __init__(self):
+        self.top = None
+        self.current = None
 
     def push_stmt(self, stmt):
-        self.current = RecursionNode(stmt, self.current)
+        self.current = _RecursionNode(stmt, self.current)
         check = self._check_recursion()
-        if check:  # TODO remove False!!!!
-            debug.warning('catched stmt recursion: %s against %s @%s'
-                                % (stmt, check.stmt, stmt.start_pos))
+        if check:
+            debug.warning('catched stmt recursion: %s against %s @%s', stmt,
+                          check.stmt, stmt.start_pos)
             self.pop_stmt()
             return True
         return False
@@ -57,10 +60,6 @@ class RecursionDecorator(object):
             if not test:
                 return False
 
-    def reset(self):
-        self.top = None
-        self.current = None
-
     def node_statements(self):
         result = []
         n = self.current
@@ -70,7 +69,7 @@ class RecursionDecorator(object):
         return result
 
 
-class RecursionNode(object):
+class _RecursionNode(object):
     """ A node of the RecursionDecorator. """
     def __init__(self, stmt, parent):
         self.script = stmt.get_parent_until()
@@ -81,47 +80,56 @@ class RecursionNode(object):
         # Don't check param instances, they are not causing recursions
         # The same's true for the builtins, because the builtins are really
         # simple.
-        self.is_ignored = isinstance(stmt, pr.Param) \
-                                   or (self.script == builtin.Builtin.scope)
+        self.is_ignored = self.script == compiled.builtin
 
     def __eq__(self, other):
         if not other:
             return None
 
-        is_list_comp = lambda x: isinstance(x, pr.ForFlow) and x.is_list_comp
         return self.script == other.script \
-                    and self.position == other.position \
-                    and not is_list_comp(self.stmt.parent) \
-                    and not is_list_comp(other.parent) \
-                    and not self.is_ignored and not other.is_ignored
+            and self.position == other.position \
+            and not self.is_ignored and not other.is_ignored
 
 
-class ExecutionRecursionDecorator(object):
+def execution_recursion_decorator(func):
+    def run(execution, **kwargs):
+        detector = execution._evaluator.execution_recursion_detector
+        if detector.push_execution(execution):
+            result = []
+        else:
+            result = func(execution, **kwargs)
+        detector.pop_execution()
+        return result
+
+    return run
+
+
+class ExecutionRecursionDetector(object):
     """
     Catches recursions of executions.
     It is designed like a Singelton. Only one instance should exist.
     """
-    def __init__(self, func):
-        self.func = func
-        self.reset()
+    def __init__(self):
+        self.recursion_level = 0
+        self.parent_execution_funcs = []
+        self.execution_funcs = set()
+        self.execution_count = 0
 
-    def __call__(self, execution, evaluate_generator=False):
-        debug.dbg('Execution recursions: %s' % execution, self.recursion_level,
-                            self.execution_count, len(self.execution_funcs))
-        if self.check_recursion(execution, evaluate_generator):
+    def __call__(self, execution):
+        debug.dbg('Execution recursions: %s', execution, self.recursion_level,
+                  self.execution_count, len(self.execution_funcs))
+        if self.check_recursion(execution):
             result = []
         else:
-            result = self.func(execution, evaluate_generator)
-        self.cleanup()
+            result = self.func(execution)
+        self.pop_execution()
         return result
 
-    @classmethod
-    def cleanup(cls):
+    def pop_execution(cls):
         cls.parent_execution_funcs.pop()
         cls.recursion_level -= 1
 
-    @classmethod
-    def check_recursion(cls, execution, evaluate_generator):
+    def push_execution(cls, execution):
         in_par_execution_funcs = execution.base in cls.parent_execution_funcs
         in_execution_funcs = execution.base in cls.execution_funcs
         cls.recursion_level += 1
@@ -132,10 +140,10 @@ class ExecutionRecursionDecorator(object):
         if cls.execution_count > settings.max_executions:
             return True
 
-        if isinstance(execution.base, (er.Generator, er.Array)):
+        if isinstance(execution.base, (iterable.Array, iterable.Generator)):
             return False
         module = execution.get_parent_until()
-        if evaluate_generator or module == builtin.Builtin.scope:
+        if module == compiled.builtin:
             return False
 
         if in_par_execution_funcs:
@@ -147,10 +155,3 @@ class ExecutionRecursionDecorator(object):
         if cls.execution_count > settings.max_executions_without_builtins:
             return True
         return False
-
-    @classmethod
-    def reset(cls):
-        cls.recursion_level = 0
-        cls.parent_execution_funcs = []
-        cls.execution_funcs = set()
-        cls.execution_count = 0

jedi/evaluate/representation.py

@@ -0,0 +1,857 @@
+"""
+Like described in the :mod:`jedi.parser.tree` module,
+there's a need for an ast like module to represent the states of parsed
+modules.
+
+But now there are also structures in Python that need a little bit more than
+that. An ``Instance`` for example is only a ``Class`` before it is
+instantiated. This class represents these cases.
+
+So, why is there also a ``Class`` class here? Well, there are decorators and
+they change classes in Python 3.
+
+Representation modules also define "magic methods". Those methods look like
+``py__foo__`` and are typically mappable to the Python equivalents ``__call__``
+and others. Here's a list:
+
+====================================== ========================================
+**Method**                             **Description**
+-------------------------------------- ----------------------------------------
+py__call__(evaluator, params: Array)   On callable objects, returns types.
+py__bool__()                           Returns True/False/None; None means that
+                                       there's no certainty.
+py__bases__(evaluator)                 Returns a list of base classes.
+py__mro__(evaluator)                   Returns a list of classes (the mro).
+py__getattribute__(evaluator, name)    Returns a list of attribute values. The
+                                       name can be str or Name.
+====================================== ========================================
+
+__
+"""
+import os
+import pkgutil
+import imp
+import re
+from itertools import chain
+
+from jedi._compatibility import use_metaclass, unicode, Python3Method
+from jedi.parser import tree
+from jedi import debug
+from jedi import common
+from jedi.cache import underscore_memoization, cache_star_import
+from jedi.evaluate.cache import memoize_default, CachedMetaClass, NO_DEFAULT
+from jedi.evaluate import compiled
+from jedi.evaluate import recursion
+from jedi.evaluate import iterable
+from jedi.evaluate import docstrings
+from jedi.evaluate import helpers
+from jedi.evaluate import param
+from jedi.evaluate import flow_analysis
+from jedi.evaluate import imports
+
+
+class Executed(tree.Base):
+    """
+    An instance is also an executable - because __init__ is called
+    :param var_args: The param input array, consist of a parser node or a list.
+    """
+    def __init__(self, evaluator, base, var_args=()):
+        self._evaluator = evaluator
+        self.base = base
+        self.var_args = var_args
+
+    def is_scope(self):
+        return True
+
+    def get_parent_until(self, *args, **kwargs):
+        return tree.Base.get_parent_until(self, *args, **kwargs)
+
+    @common.safe_property
+    def parent(self):
+        return self.base.parent
+
+
+class Instance(use_metaclass(CachedMetaClass, Executed)):
+    """
+    This class is used to evaluate instances.
+    """
+    def __init__(self, evaluator, base, var_args, is_generated=False):
+        super(Instance, self).__init__(evaluator, base, var_args)
+        self.decorates = None
+        # Generated instances are classes that are just generated by self
+        # (No var_args) used.
+        self.is_generated = is_generated
+
+        if base.name.get_code() in ['list', 'set'] \
+                and compiled.builtin == base.get_parent_until():
+            # compare the module path with the builtin name.
+            self.var_args = iterable.check_array_instances(evaluator, self)
+        elif not is_generated:
+            # Need to execute the __init__ function, because the dynamic param
+            # searching needs it.
+            try:
+                method = self.get_subscope_by_name('__init__')
+            except KeyError:
+                pass
+            else:
+                evaluator.execute(method, self.var_args)
+
+    @property
+    def py__call__(self):
+        def actual(evaluator, params):
+            return evaluator.execute(method, params)
+
+        try:
+            method = self.get_subscope_by_name('__call__')
+        except KeyError:
+            # Means the Instance is not callable.
+            raise AttributeError
+
+        return actual
+
+    def py__class__(self, evaluator):
+        return self.base
+
+    def py__bool__(self):
+        # Signalize that we don't know about the bool type.
+        return None
+
+    @memoize_default()
+    def _get_method_execution(self, func):
+        func = get_instance_el(self._evaluator, self, func, True)
+        return FunctionExecution(self._evaluator, func, self.var_args)
+
+    def _get_func_self_name(self, func):
+        """
+        Returns the name of the first param in a class method (which is
+        normally self.
+        """
+        try:
+            return str(func.params[0].name)
+        except IndexError:
+            return None
+
+    def _self_names_dict(self, add_mro=True):
+        names = {}
+        # This loop adds the names of the self object, copies them and removes
+        # the self.
+        for sub in self.base.subscopes:
+            if isinstance(sub, tree.Class):
+                continue
+            # Get the self name, if there's one.
+            self_name = self._get_func_self_name(sub)
+            if self_name is None:
+                continue
+
+            if sub.name.value == '__init__' and not self.is_generated:
+                # ``__init__`` is special because the params need are injected
+                # this way. Therefore an execution is necessary.
+                if not sub.get_decorators():
+                    # __init__ decorators should generally just be ignored,
+                    # because to follow them and their self variables is too
+                    # complicated.
+                    sub = self._get_method_execution(sub)
+            for name_list in sub.names_dict.values():
+                for name in name_list:
+                    if name.value == self_name and name.prev_sibling() is None:
+                        trailer = name.next_sibling()
+                        if tree.is_node(trailer, 'trailer') \
+                                and len(trailer.children) == 2 \
+                                and trailer.children[0] == '.':
+                            name = trailer.children[1]  # After dot.
+                            if name.is_definition():
+                                arr = names.setdefault(name.value, [])
+                                arr.append(get_instance_el(self._evaluator, self, name))
+        return names
+
+    def get_subscope_by_name(self, name):
+        sub = self.base.get_subscope_by_name(name)
+        return get_instance_el(self._evaluator, self, sub, True)
+
+    def execute_subscope_by_name(self, name, *args):
+        method = self.get_subscope_by_name(name)
+        return self._evaluator.execute_evaluated(method, *args)
+
+    def get_descriptor_returns(self, obj):
+        """ Throws a KeyError if there's no method. """
+        # Arguments in __get__ descriptors are obj, class.
+        # `method` is the new parent of the array, don't know if that's good.
+        args = [obj, obj.base] if isinstance(obj, Instance) else [compiled.none_obj, obj]
+        try:
+            return self.execute_subscope_by_name('__get__', *args)
+        except KeyError:
+            return [self]
+
+    @memoize_default()
+    def names_dicts(self, search_global):
+        yield self._self_names_dict()
+
+        for s in self.base.py__mro__(self._evaluator)[1:]:
+            if not isinstance(s, compiled.CompiledObject):
+                # Compiled objects don't have `self.` names.
+                for inst in self._evaluator.execute(s):
+                    yield inst._self_names_dict(add_mro=False)
+
+        for names_dict in self.base.names_dicts(search_global=False, is_instance=True):
+            yield LazyInstanceDict(self._evaluator, self, names_dict)
+
+    def get_index_types(self, evaluator, index_array):
+        indexes = iterable.create_indexes_or_slices(self._evaluator, index_array)
+        if any([isinstance(i, iterable.Slice) for i in indexes]):
+            # Slice support in Jedi is very marginal, at the moment, so just
+            # ignore them in case of __getitem__.
+            # TODO support slices in a more general way.
+            indexes = []
+
+        try:
+            method = self.get_subscope_by_name('__getitem__')
+        except KeyError:
+            debug.warning('No __getitem__, cannot access the array.')
+            return []
+        else:
+            return self._evaluator.execute(method, [iterable.AlreadyEvaluated(indexes)])
+
+    @property
+    @underscore_memoization
+    def name(self):
+        name = self.base.name
+        return helpers.FakeName(unicode(name), self, name.start_pos)
+
+    def __getattr__(self, name):
+        if name not in ['start_pos', 'end_pos', 'get_imports', 'type',
+                        'doc', 'raw_doc']:
+            raise AttributeError("Instance %s: Don't touch this (%s)!"
+                                 % (self, name))
+        return getattr(self.base, name)
+
+    def __repr__(self):
+        dec = ''
+        if self.decorates is not None:
+            dec = " decorates " + repr(self.decorates)
+        return "<e%s of %s(%s)%s>" % (type(self).__name__, self.base,
+                                      self.var_args, dec)
+
+
+class LazyInstanceDict(object):
+    def __init__(self, evaluator, instance, dct):
+        self._evaluator = evaluator
+        self._instance = instance
+        self._dct = dct
+
+    def __getitem__(self, name):
+        return [get_instance_el(self._evaluator, self._instance, var, True)
+                for var in self._dct[name]]
+
+    def values(self):
+        return [self[key] for key in self._dct]
+
+
+class InstanceName(tree.Name):
+    def __init__(self, origin_name, parent):
+        super(InstanceName, self).__init__(tree.zero_position_modifier,
+                                           origin_name.value,
+                                           origin_name.start_pos)
+        self._origin_name = origin_name
+        self.parent = parent
+
+    def is_definition(self):
+        return self._origin_name.is_definition()
+
+
+def get_instance_el(evaluator, instance, var, is_class_var=False):
+    """
+    Returns an InstanceElement if it makes sense, otherwise leaves the object
+    untouched.
+
+    Basically having an InstanceElement is context information. That is needed
+    in quite a lot of cases, which includes Nodes like ``power``, that need to
+    know where a self name comes from for example.
+    """
+    if isinstance(var, tree.Name):
+        parent = get_instance_el(evaluator, instance, var.parent, is_class_var)
+        return InstanceName(var, parent)
+    elif var.type != 'funcdef' \
+            and isinstance(var, (Instance, compiled.CompiledObject, tree.Leaf,
+                           tree.Module, FunctionExecution)):
+        return var
+
+    var = evaluator.wrap(var)
+    return InstanceElement(evaluator, instance, var, is_class_var)
+
+
+class InstanceElement(use_metaclass(CachedMetaClass, tree.Base)):
+    """
+    InstanceElement is a wrapper for any object, that is used as an instance
+    variable (e.g. self.variable or class methods).
+    """
+    def __init__(self, evaluator, instance, var, is_class_var):
+        self._evaluator = evaluator
+        self.instance = instance
+        self.var = var
+        self.is_class_var = is_class_var
+
+    @common.safe_property
+    @memoize_default()
+    def parent(self):
+        par = self.var.parent
+        if isinstance(par, Class) and par == self.instance.base \
+                or isinstance(par, tree.Class) \
+                and par == self.instance.base.base:
+            par = self.instance
+        else:
+            par = get_instance_el(self._evaluator, self.instance, par,
+                                  self.is_class_var)
+        return par
+
+    def get_parent_until(self, *args, **kwargs):
+        return tree.BaseNode.get_parent_until(self, *args, **kwargs)
+
+    def get_definition(self):
+        return self.get_parent_until((tree.ExprStmt, tree.IsScope, tree.Import))
+
+    def get_decorated_func(self):
+        """ Needed because the InstanceElement should not be stripped """
+        func = self.var.get_decorated_func()
+        func = get_instance_el(self._evaluator, self.instance, func)
+        return func
+
+    def get_rhs(self):
+        return get_instance_el(self._evaluator, self.instance,
+                               self.var.get_rhs(), self.is_class_var)
+
+    def is_definition(self):
+        return self.var.is_definition()
+
+    @property
+    def children(self):
+        # Copy and modify the array.
+        return [get_instance_el(self._evaluator, self.instance, command, self.is_class_var)
+                for command in self.var.children]
+
+    @property
+    @memoize_default()
+    def name(self):
+        name = self.var.name
+        return helpers.FakeName(unicode(name), self, name.start_pos)
+
+    def __iter__(self):
+        for el in self.var.__iter__():
+            yield get_instance_el(self._evaluator, self.instance, el,
+                                  self.is_class_var)
+
+    def __getitem__(self, index):
+        return get_instance_el(self._evaluator, self.instance, self.var[index],
+                               self.is_class_var)
+
+    def __getattr__(self, name):
+        return getattr(self.var, name)
+
+    def isinstance(self, *cls):
+        return isinstance(self.var, cls)
+
+    def is_scope(self):
+        """
+        Since we inherit from Base, it would overwrite the action we want here.
+        """
+        return self.var.is_scope()
+
+    def py__call__(self, evaluator, params):
+        if isinstance(self.var, compiled.CompiledObject):
+            # This check is a bit strange, but CompiledObject itself is a bit
+            # more complicated than we would it actually like to be.
+            return self.var.py__call__(evaluator, params)
+        else:
+            return Function.py__call__(self, evaluator, params)
+
+    def __repr__(self):
+        return "<%s of %s>" % (type(self).__name__, self.var)
+
+
+class Wrapper(tree.Base):
+    def is_scope(self):
+        return True
+
+    def is_class(self):
+        return False
+
+    def py__bool__(self):
+        """
+        Since Wrapper is a super class for classes, functions and modules,
+        the return value will always be true.
+        """
+        return True
+
+    @property
+    @underscore_memoization
+    def name(self):
+        name = self.base.name
+        return helpers.FakeName(unicode(name), self, name.start_pos)
+
+
+class Class(use_metaclass(CachedMetaClass, Wrapper)):
+    """
+    This class is not only important to extend `tree.Class`, it is also a
+    important for descriptors (if the descriptor methods are evaluated or not).
+    """
+    def __init__(self, evaluator, base):
+        self._evaluator = evaluator
+        self.base = base
+
+    @memoize_default(default=())
+    def py__mro__(self, evaluator):
+        def add(cls):
+            if cls not in mro:
+                mro.append(cls)
+
+        mro = [self]
+        # TODO Do a proper mro resolution. Currently we are just listing
+        # classes. However, it's a complicated algorithm.
+        for cls in self.py__bases__(self._evaluator):
+            # TODO detect for TypeError: duplicate base class str,
+            # e.g.  `class X(str, str): pass`
+            try:
+                mro_method = cls.py__mro__
+            except AttributeError:
+                # TODO add a TypeError like:
+                """
+                >>> class Y(lambda: test): pass
+                Traceback (most recent call last):
+                  File "<stdin>", line 1, in <module>
+                TypeError: function() argument 1 must be code, not str
+                >>> class Y(1): pass
+                Traceback (most recent call last):
+                  File "<stdin>", line 1, in <module>
+                TypeError: int() takes at most 2 arguments (3 given)
+                """
+                pass
+            else:
+                add(cls)
+                for cls_new in mro_method(evaluator):
+                    add(cls_new)
+        return tuple(mro)
+
+    @memoize_default(default=())
+    def py__bases__(self, evaluator):
+        arglist = self.base.get_super_arglist()
+        if arglist:
+            args = param.Arguments(self._evaluator, arglist)
+            return list(chain.from_iterable(args.eval_args()))
+        else:
+            return [compiled.object_obj]
+
+    def py__call__(self, evaluator, params):
+        return [Instance(evaluator, self, params)]
+
+    def py__getattribute__(self, name):
+        return self._evaluator.find_types(self, name)
+
+    @property
+    def params(self):
+        return self.get_subscope_by_name('__init__').params
+
+    def names_dicts(self, search_global, is_instance=False):
+        if search_global:
+            yield self.names_dict
+        else:
+            for scope in self.py__mro__(self._evaluator):
+                if isinstance(scope, compiled.CompiledObject):
+                    yield scope.names_dicts(False, is_instance)[0]
+                else:
+                    yield scope.names_dict
+
+    def is_class(self):
+        return True
+
+    def get_subscope_by_name(self, name):
+        for s in self.py__mro__(self._evaluator):
+            for sub in reversed(s.subscopes):
+                if sub.name.value == name:
+                        return sub
+        raise KeyError("Couldn't find subscope.")
+
+    def __getattr__(self, name):
+        if name not in ['start_pos', 'end_pos', 'parent', 'raw_doc',
+                        'doc', 'get_imports', 'get_parent_until', 'get_code',
+                        'subscopes', 'names_dict', 'type']:
+            raise AttributeError("Don't touch this: %s of %s !" % (name, self))
+        return getattr(self.base, name)
+
+    def __repr__(self):
+        return "<e%s of %s>" % (type(self).__name__, self.base)
+
+
+class Function(use_metaclass(CachedMetaClass, Wrapper)):
+    """
+    Needed because of decorators. Decorators are evaluated here.
+    """
+    def __init__(self, evaluator, func, is_decorated=False):
+        """ This should not be called directly """
+        self._evaluator = evaluator
+        self.base = self.base_func = func
+        self.is_decorated = is_decorated
+        # A property that is set by the decorator resolution.
+        self.decorates = None
+
+    @memoize_default()
+    def get_decorated_func(self):
+        """
+        Returns the function, that should to be executed in the end.
+        This is also the places where the decorators are processed.
+        """
+        f = self.base_func
+        decorators = self.base_func.get_decorators()
+
+        if not decorators or self.is_decorated:
+            return self
+
+        # Only enter it, if has not already been processed.
+        if not self.is_decorated:
+            for dec in reversed(decorators):
+                debug.dbg('decorator: %s %s', dec, f)
+                dec_results = self._evaluator.eval_element(dec.children[1])
+                trailer = dec.children[2:-1]
+                if trailer:
+                    # Create a trailer and evaluate it.
+                    trailer = tree.Node('trailer', trailer)
+                    trailer.parent = dec
+                    dec_results = self._evaluator.eval_trailer(dec_results, trailer)
+
+                if not len(dec_results):
+                    debug.warning('decorator not found: %s on %s', dec, self.base_func)
+                    return self
+                decorator = dec_results.pop()
+                if dec_results:
+                    debug.warning('multiple decorators found %s %s',
+                                  self.base_func, dec_results)
+
+                # Create param array.
+                if isinstance(f, Function):
+                    old_func = f  # TODO this is just hacky. change.
+                else:
+                    old_func = Function(self._evaluator, f, is_decorated=True)
+
+                wrappers = self._evaluator.execute_evaluated(decorator, old_func)
+                if not len(wrappers):
+                    debug.warning('no wrappers found %s', self.base_func)
+                    return self
+                if len(wrappers) > 1:
+                    # TODO resolve issue with multiple wrappers -> multiple types
+                    debug.warning('multiple wrappers found %s %s',
+                                  self.base_func, wrappers)
+                f = wrappers[0]
+                if isinstance(f, (Instance, Function)):
+                    f.decorates = self
+
+                debug.dbg('decorator end %s', f)
+        return f
+
+    def names_dicts(self, search_global):
+        if search_global:
+            yield self.names_dict
+        else:
+            for names_dict in compiled.magic_function_class.names_dicts(False):
+                yield names_dict
+
+    @Python3Method
+    def py__call__(self, evaluator, params):
+        if self.base.is_generator():
+            return [iterable.Generator(evaluator, self, params)]
+        else:
+            return FunctionExecution(evaluator, self, params).get_return_types()
+
+    def __getattr__(self, name):
+        return getattr(self.base_func, name)
+
+    def __repr__(self):
+        dec = ''
+        if self.decorates is not None:
+            dec = " decorates " + repr(self.decorates)
+        return "<e%s of %s%s>" % (type(self).__name__, self.base_func, dec)
+
+
+class LambdaWrapper(Function):
+    def get_decorated_func(self):
+        return self
+
+
+class FunctionExecution(Executed):
+    """
+    This class is used to evaluate functions and their returns.
+
+    This is the most complicated class, because it contains the logic to
+    transfer parameters. It is even more complicated, because there may be
+    multiple calls to functions and recursion has to be avoided. But this is
+    responsibility of the decorators.
+    """
+    type = 'funcdef'
+
+    def __init__(self, evaluator, base, *args, **kwargs):
+        super(FunctionExecution, self).__init__(evaluator, base, *args, **kwargs)
+        self._copy_dict = {}
+        new_func = helpers.deep_ast_copy(base.base_func, self, self._copy_dict)
+        self.children = new_func.children
+        self.names_dict = new_func.names_dict
+
+    @memoize_default(default=())
+    @recursion.execution_recursion_decorator
+    def get_return_types(self, check_yields=False):
+        func = self.base
+
+        if func.isinstance(LambdaWrapper):
+            return self._evaluator.eval_element(self.children[-1])
+
+        if func.listeners:
+            # Feed the listeners, with the params.
+            for listener in func.listeners:
+                listener.execute(self._get_params())
+            # If we do have listeners, that means that there's not a regular
+            # execution ongoing. In this case Jedi is interested in the
+            # inserted params, not in the actual execution of the function.
+            return []
+
+        if check_yields:
+            types = []
+            returns = self.yields
+        else:
+            returns = self.returns
+            types = list(docstrings.find_return_types(self._evaluator, func))
+
+        for r in returns:
+            check = flow_analysis.break_check(self._evaluator, self, r)
+            if check is flow_analysis.UNREACHABLE:
+                debug.dbg('Return unreachable: %s', r)
+            else:
+                types += self._evaluator.eval_element(r.children[1])
+            if check is flow_analysis.REACHABLE:
+                debug.dbg('Return reachable: %s', r)
+                break
+        return types
+
+    def names_dicts(self, search_global):
+        yield self.names_dict
+
+    @memoize_default(default=NO_DEFAULT)
+    def _get_params(self):
+        """
+        This returns the params for an TODO and is injected as a
+        'hack' into the tree.Function class.
+        This needs to be here, because Instance can have __init__ functions,
+        which act the same way as normal functions.
+        """
+        return param.get_params(self._evaluator, self.base, self.var_args)
+
+    def param_by_name(self, name):
+        return [n for n in self._get_params() if str(n) == name][0]
+
+    def name_for_position(self, position):
+        return tree.Function.name_for_position(self, position)
+
+    def _copy_list(self, lst):
+        """
+        Copies a list attribute of a parser Function. Copying is very
+        expensive, because it is something like `copy.deepcopy`. However, these
+        copied objects can be used for the executions, as if they were in the
+        execution.
+        """
+        objects = []
+        for element in lst:
+            self._scope_copy(element.parent)
+            copied = helpers.deep_ast_copy(element, self._copy_dict)
+            objects.append(copied)
+        return objects
+
+    def __getattr__(self, name):
+        if name not in ['start_pos', 'end_pos', 'imports', 'name', 'type']:
+            raise AttributeError('Tried to access %s: %s. Why?' % (name, self))
+        return getattr(self.base, name)
+
+    def _scope_copy(self, scope):
+        raise NotImplementedError
+        """ Copies a scope (e.g. `if foo:`) in an execution """
+        if scope != self.base.base_func:
+            # Just make sure the parents been copied.
+            self._scope_copy(scope.parent)
+            helpers.deep_ast_copy(scope, self._copy_dict)
+
+    @common.safe_property
+    @memoize_default([])
+    def returns(self):
+        return tree.Scope._search_in_scope(self, tree.ReturnStmt)
+
+    @common.safe_property
+    @memoize_default([])
+    def yields(self):
+        return tree.Scope._search_in_scope(self, tree.YieldExpr)
+
+    @common.safe_property
+    @memoize_default([])
+    def statements(self):
+        return tree.Scope._search_in_scope(self, tree.ExprStmt)
+
+    @common.safe_property
+    @memoize_default([])
+    def subscopes(self):
+        return tree.Scope._search_in_scope(self, tree.Scope)
+
+    def __repr__(self):
+        return "<%s of %s>" % (type(self).__name__, self.base)
+
+
+class GlobalName(helpers.FakeName):
+    def __init__(self, name):
+        """
+        We need to mark global names somehow. Otherwise they are just normal
+        names that are not definitions.
+        """
+        super(GlobalName, self).__init__(name.value, name.parent,
+                                         name.start_pos, is_definition=True)
+
+
+class ModuleWrapper(use_metaclass(CachedMetaClass, tree.Module, Wrapper)):
+    def __init__(self, evaluator, module):
+        self._evaluator = evaluator
+        self.base = self._module = module
+
+    def names_dicts(self, search_global):
+        yield self.base.names_dict
+        yield self._module_attributes_dict()
+
+        for star_module in self.star_imports():
+            yield star_module.names_dict
+
+        yield dict((str(n), [GlobalName(n)]) for n in self.base.global_names)
+        yield self._sub_modules_dict()
+
+    # I'm not sure if the star import cache is really that effective anymore
+    # with all the other really fast import caches. Recheck. Also we would need
+    # to push the star imports into Evaluator.modules, if we reenable this.
+    #@cache_star_import
+    @memoize_default([])
+    def star_imports(self):
+        modules = []
+        for i in self.base.imports:
+            if i.is_star_import():
+                name = i.star_import_name()
+                new = imports.ImportWrapper(self._evaluator, name).follow()
+                for module in new:
+                    if isinstance(module, tree.Module):
+                        modules += module.star_imports()
+                modules += new
+        return modules
+
+    @memoize_default()
+    def _module_attributes_dict(self):
+        def parent_callback():
+            return self._evaluator.execute(compiled.create(self._evaluator, str))[0]
+
+        names = ['__file__', '__package__', '__doc__', '__name__']
+        # All the additional module attributes are strings.
+        return dict((n, [helpers.LazyName(n, parent_callback, is_definition=True)])
+                    for n in names)
+
+    @property
+    @memoize_default()
+    def name(self):
+        return helpers.FakeName(unicode(self.base.name), self, (1, 0))
+
+    def _get_init_directory(self):
+        for suffix, _, _ in imp.get_suffixes():
+            ending = '__init__' + suffix
+            if self.py__file__().endswith(ending):
+                # Remove the ending, including the separator.
+                return self.py__file__()[:-len(ending) - 1]
+        return None
+
+    def py__name__(self):
+        for name, module in self._evaluator.modules.items():
+            if module == self:
+                return name
+
+        return '__main__'
+
+    def py__file__(self):
+        """
+        In contrast to Python's __file__ can be None.
+        """
+        if self._module.path is None:
+            return None
+
+        return os.path.abspath(self._module.path)
+
+    def py__package__(self):
+        if self._get_init_directory() is None:
+            return re.sub(r'\.?[^\.]+$', '', self.py__name__())
+        else:
+            return self.py__name__()
+
+    @property
+    def py__path__(self):
+        """
+        Not seen here, since it's a property. The callback actually uses a
+        variable, so use it like::
+
+            foo.py__path__(sys_path)
+
+        In case of a package, this returns Python's __path__ attribute, which
+        is a list of paths (strings).
+        Raises an AttributeError if the module is not a package.
+        """
+        def return_value(search_path):
+            init_path = self.py__file__()
+            if os.path.basename(init_path) == '__init__.py':
+
+                with open(init_path, 'rb') as f:
+                    content = common.source_to_unicode(f.read())
+                    # these are strings that need to be used for namespace packages,
+                    # the first one is ``pkgutil``, the second ``pkg_resources``.
+                    options = ('declare_namespace(__name__)', 'extend_path(__path__')
+                    if options[0] in content or options[1] in content:
+                        # It is a namespace, now try to find the rest of the
+                        # modules on sys_path or whatever the search_path is.
+                        paths = set()
+                        for s in search_path:
+                            other = os.path.join(s, unicode(self.name))
+                            if os.path.isdir(other):
+                                paths.add(other)
+                        return list(paths)
+            # Default to this.
+            return [path]
+
+        path = self._get_init_directory()
+
+        if path is None:
+            raise AttributeError('Only packages have __path__ attributes.')
+        else:
+            return return_value
+
+    @memoize_default()
+    def _sub_modules_dict(self):
+        """
+        Lists modules in the directory of this module (if this module is a
+        package).
+        """
+        path = self._module.path
+        names = {}
+        if path is not None and path.endswith(os.path.sep + '__init__.py'):
+            mods = pkgutil.iter_modules([os.path.dirname(path)])
+            for module_loader, name, is_pkg in mods:
+                fake_n = helpers.FakeName(name)
+                # It's obviously a relative import to the current module.
+                imp = helpers.FakeImport(fake_n, self, level=1)
+                fake_n.parent = imp
+                names[name] = [fake_n]
+
+        # TODO add something like this in the future, its cleaner than the
+        #   import hacks.
+        # ``os.path`` is a hardcoded exception, because it's a
+        # ``sys.modules`` modification.
+        #if str(self.name) == 'os':
+        #    names.append(helpers.FakeName('path', parent=self))
+
+        return names
+
+    def __getattr__(self, name):
+        return getattr(self._module, name)
+
+    def __repr__(self):
+        return "<%s: %s>" % (type(self).__name__, self._module)

jedi/evaluate/stdlib.py

@@ -0,0 +1,256 @@
+"""
+Implementations of standard library functions, because it's not possible to
+understand them with Jedi.
+
+To add a new implementation, create a function and add it to the
+``_implemented`` dict at the bottom of this module.
+
+"""
+import collections
+import re
+
+from jedi._compatibility import unicode
+from jedi.evaluate import compiled
+from jedi.evaluate import representation as er
+from jedi.evaluate import iterable
+from jedi.parser import Parser
+from jedi.parser import tree
+from jedi import debug
+from jedi.evaluate import precedence
+from jedi.evaluate import param
+
+
+class NotInStdLib(LookupError):
+    pass
+
+
+def execute(evaluator, obj, params):
+    try:
+        obj_name = str(obj.name)
+    except AttributeError:
+        pass
+    else:
+        if obj.parent == compiled.builtin:
+            module_name = 'builtins'
+        elif isinstance(obj.parent, tree.Module):
+            module_name = str(obj.parent.name)
+        else:
+            module_name = ''
+
+        # for now we just support builtin functions.
+        try:
+            return _implemented[module_name][obj_name](evaluator, obj, params)
+        except KeyError:
+            pass
+    raise NotInStdLib()
+
+
+def _follow_param(evaluator, params, index):
+    try:
+        key, values = list(params.unpack())[index]
+    except IndexError:
+        return []
+    else:
+        return iterable.unite(evaluator.eval_element(v) for v in values)
+
+
+def argument_clinic(string, want_obj=False, want_scope=False):
+    """
+    Works like Argument Clinic (PEP 436), to validate function params.
+    """
+    clinic_args = []
+    allow_kwargs = False
+    optional = False
+    while string:
+        # Optional arguments have to begin with a bracket. And should always be
+        # at the end of the arguments. This is therefore not a proper argument
+        # clinic implementation. `range()` for exmple allows an optional start
+        # value at the beginning.
+        match = re.match('(?:(?:(\[),? ?|, ?|)(\w+)|, ?/)\]*', string)
+        string = string[len(match.group(0)):]
+        if not match.group(2):  # A slash -> allow named arguments
+            allow_kwargs = True
+            continue
+        optional = optional or bool(match.group(1))
+        word = match.group(2)
+        clinic_args.append((word, optional, allow_kwargs))
+
+    def f(func):
+        def wrapper(evaluator, obj, arguments):
+            try:
+                lst = list(arguments.eval_argument_clinic(clinic_args))
+            except ValueError:
+                return []
+            else:
+                kwargs = {}
+                if want_scope:
+                    kwargs['scope'] = arguments.scope()
+                if want_obj:
+                    kwargs['obj'] = obj
+                return func(evaluator, *lst, **kwargs)
+
+        return wrapper
+    return f
+
+
+@argument_clinic('object, name[, default], /')
+def builtins_getattr(evaluator, objects, names, defaults=None):
+    types = []
+    # follow the first param
+    for obj in objects:
+        if not isinstance(obj, (er.Instance, er.Class, tree.Module, compiled.CompiledObject)):
+            debug.warning('getattr called without instance')
+            continue
+
+        for name in names:
+            if precedence.is_string(name):
+                return evaluator.find_types(obj, name.obj)
+            else:
+                debug.warning('getattr called without str')
+                continue
+    return types
+
+
+@argument_clinic('object[, bases, dict], /')
+def builtins_type(evaluator, objects, bases, dicts):
+    if bases or dicts:
+        # metaclass... maybe someday...
+        return []
+    else:
+        return [o.base for o in objects if isinstance(o, er.Instance)]
+
+
+class SuperInstance(er.Instance):
+    """To be used like the object ``super`` returns."""
+    def __init__(self, evaluator, cls):
+        su = cls.py_mro()[1]
+        super().__init__(evaluator, su and su[0] or self)
+
+
+@argument_clinic('[type[, obj]], /', want_scope=True)
+def builtins_super(evaluator, types, objects, scope):
+    # TODO make this able to detect multiple inheritance super
+    accept = (tree.Function, er.FunctionExecution)
+    if scope.isinstance(*accept):
+        wanted = (tree.Class, er.Instance)
+        cls = scope.get_parent_until(accept + wanted,
+                                     include_current=False)
+        if isinstance(cls, wanted):
+            if isinstance(cls, tree.Class):
+                cls = er.Class(evaluator, cls)
+            elif isinstance(cls, er.Instance):
+                cls = cls.base
+            su = cls.py__bases__(evaluator)
+            if su:
+                return evaluator.execute(su[0])
+    return []
+
+
+@argument_clinic('sequence, /', want_obj=True)
+def builtins_reversed(evaluator, sequences, obj):
+    # Unpack the iterator values
+    objects = tuple(iterable.get_iterator_types(sequences))
+    rev = [iterable.AlreadyEvaluated([o]) for o in reversed(objects)]
+    # Repack iterator values and then run it the normal way. This is
+    # necessary, because `reversed` is a function and autocompletion
+    # would fail in certain cases like `reversed(x).__iter__` if we
+    # just returned the result directly.
+    rev = iterable.AlreadyEvaluated(
+        [iterable.FakeSequence(evaluator, rev, 'list')]
+    )
+    return [er.Instance(evaluator, obj, param.Arguments(evaluator, [rev]))]
+
+
+@argument_clinic('obj, type, /')
+def builtins_isinstance(evaluator, objects, types):
+    bool_results = set([])
+    for o in objects:
+        try:
+            mro_func = o.py__class__(evaluator).py__mro__
+        except AttributeError:
+            # This is temporary. Everything should have a class attribute in
+            # Python?! Maybe we'll leave it here, because some numpy objects or
+            # whatever might not.
+            return [compiled.true_obj, compiled.false_obj]
+
+        mro = mro_func(evaluator)
+
+        for cls_or_tup in types:
+            if cls_or_tup.is_class():
+                bool_results.add(cls_or_tup in mro)
+            else:
+                # Check for tuples.
+                classes = iterable.get_iterator_types([cls_or_tup])
+                bool_results.add(any(cls in mro for cls in classes))
+
+    return [compiled.keyword_from_value(x) for x in bool_results]
+
+
+def collections_namedtuple(evaluator, obj, params):
+    """
+    Implementation of the namedtuple function.
+
+    This has to be done by processing the namedtuple class template and
+    evaluating the result.
+
+    .. note:: |jedi| only supports namedtuples on Python >2.6.
+
+    """
+    # Namedtuples are not supported on Python 2.6
+    if not hasattr(collections, '_class_template'):
+        return []
+
+    # Process arguments
+    name = _follow_param(evaluator, params, 0)[0].obj
+    _fields = _follow_param(evaluator, params, 1)[0]
+    if isinstance(_fields, compiled.CompiledObject):
+        fields = _fields.obj.replace(',', ' ').split()
+    elif isinstance(_fields, iterable.Array):
+        try:
+            fields = [v.obj for v in _fields.values()]
+        except AttributeError:
+            return []
+    else:
+        return []
+
+    # Build source
+    source = collections._class_template.format(
+        typename=name,
+        field_names=fields,
+        num_fields=len(fields),
+        arg_list=', '.join(fields),
+        repr_fmt=', '.join(collections._repr_template.format(name=name) for name in fields),
+        field_defs='\n'.join(collections._field_template.format(index=index, name=name)
+                             for index, name in enumerate(fields))
+    )
+
+    # Parse source
+    generated_class = Parser(evaluator.grammar, unicode(source)).module.subscopes[0]
+    return [er.Class(evaluator, generated_class)]
+
+
+@argument_clinic('first, /')
+def _return_first_param(evaluator, firsts):
+    return firsts
+
+
+_implemented = {
+    'builtins': {
+        'getattr': builtins_getattr,
+        'type': builtins_type,
+        'super': builtins_super,
+        'reversed': builtins_reversed,
+        'isinstance': builtins_isinstance,
+    },
+    'copy': {
+        'copy': _return_first_param,
+        'deepcopy': _return_first_param,
+    },
+    'json': {
+        'load': lambda *args: [],
+        'loads': lambda *args: [],
+    },
+    'collections': {
+        'namedtuple': collections_namedtuple,
+    },
+}

jedi/evaluate/sys_path.py

@@ -0,0 +1,247 @@
+import glob
+import os
+import sys
+
+from jedi._compatibility import exec_function, unicode
+from jedi.parser import tree
+from jedi.parser import Parser
+from jedi.evaluate.cache import memoize_default
+from jedi import debug
+from jedi import common
+from jedi import cache
+
+
+def get_sys_path():
+    def check_virtual_env(sys_path):
+        """ Add virtualenv's site-packages to the `sys.path`."""
+        venv = os.getenv('VIRTUAL_ENV')
+        if not venv:
+            return
+        venv = os.path.abspath(venv)
+        p = _get_venv_sitepackages(venv)
+        if p not in sys_path:
+            sys_path.insert(0, p)
+
+        # Add all egg-links from the virtualenv.
+        for egg_link in glob.glob(os.path.join(p, '*.egg-link')):
+            with open(egg_link) as fd:
+                sys_path.insert(0, fd.readline().rstrip())
+
+    check_virtual_env(sys.path)
+    return [p for p in sys.path if p != ""]
+
+
+def _get_venv_sitepackages(venv):
+    if os.name == 'nt':
+        p = os.path.join(venv, 'lib', 'site-packages')
+    else:
+        p = os.path.join(venv, 'lib', 'python%d.%d' % sys.version_info[:2],
+                         'site-packages')
+    return p
+
+
+def _execute_code(module_path, code):
+    c = "import os; from os.path import *; result=%s"
+    variables = {'__file__': module_path}
+    try:
+        exec_function(c % code, variables)
+    except Exception:
+        debug.warning('sys.path manipulation detected, but failed to evaluate.')
+    else:
+        try:
+            res = variables['result']
+            if isinstance(res, str):
+                return [os.path.abspath(res)]
+        except KeyError:
+            pass
+    return []
+
+
+def _paths_from_assignment(evaluator, expr_stmt):
+    """
+    Extracts the assigned strings from an assignment that looks as follows::
+
+    >>> sys.path[0:0] = ['module/path', 'another/module/path']
+
+    This function is in general pretty tolerant (and therefore 'buggy').
+    However, it's not a big issue usually to add more paths to Jedi's sys_path,
+    because it will only affect Jedi in very random situations and by adding
+    more paths than necessary, it usually benefits the general user.
+    """
+    for assignee, operator in zip(expr_stmt.children[::2], expr_stmt.children[1::2]):
+        try:
+            assert operator in ['=', '+=']
+            assert tree.is_node(assignee, 'power') and len(assignee.children) > 1
+            c = assignee.children
+            assert c[0].type == 'name' and c[0].value == 'sys'
+            trailer = c[1]
+            assert trailer.children[0] == '.' and trailer.children[1].value == 'path'
+            # TODO Essentially we're not checking details on sys.path
+            # manipulation. Both assigment of the sys.path and changing/adding
+            # parts of the sys.path are the same: They get added to the current
+            # sys.path.
+            """
+            execution = c[2]
+            assert execution.children[0] == '['
+            subscript = execution.children[1]
+            assert subscript.type == 'subscript'
+            assert ':' in subscript.children
+            """
+        except AssertionError:
+            continue
+
+        from jedi.evaluate.iterable import get_iterator_types
+        from jedi.evaluate.precedence import is_string
+        for val in get_iterator_types(evaluator.eval_statement(expr_stmt)):
+            if is_string(val):
+                yield val.obj
+
+
+def _paths_from_list_modifications(module_path, trailer1, trailer2):
+    """ extract the path from either "sys.path.append" or "sys.path.insert" """
+    # Guarantee that both are trailers, the first one a name and the second one
+    # a function execution with at least one param.
+    if not (tree.is_node(trailer1, 'trailer') and trailer1.children[0] == '.'
+            and tree.is_node(trailer2, 'trailer') and trailer2.children[0] == '('
+            and len(trailer2.children) == 3):
+        return []
+
+    name = trailer1.children[1].value
+    if name not in ['insert', 'append']:
+        return []
+
+    arg = trailer2.children[1]
+    if name == 'insert' and len(arg.children) in (3, 4):  # Possible trailing comma.
+        arg = arg.children[2]
+    return _execute_code(module_path, arg.get_code())
+
+
+def _check_module(evaluator, module):
+    def get_sys_path_powers(names):
+        for name in names:
+            power = name.parent.parent
+            if tree.is_node(power, 'power'):
+                c = power.children
+                if isinstance(c[0], tree.Name) and c[0].value == 'sys' \
+                        and tree.is_node(c[1], 'trailer'):
+                    n = c[1].children[1]
+                    if isinstance(n, tree.Name) and n.value == 'path':
+                        yield name, power
+
+    sys_path = list(get_sys_path())  # copy
+    try:
+        possible_names = module.used_names['path']
+    except KeyError:
+        pass
+    else:
+        for name, power in get_sys_path_powers(possible_names):
+            stmt = name.get_definition()
+            if len(power.children) >= 4:
+                sys_path.extend(_paths_from_list_modifications(module.path, *power.children[2:4]))
+            elif name.get_definition().type == 'expr_stmt':
+                sys_path.extend(_paths_from_assignment(evaluator, stmt))
+    return sys_path
+
+
+@memoize_default(evaluator_is_first_arg=True, default=[])
+def sys_path_with_modifications(evaluator, module):
+    if module.path is None:
+        # Support for modules without a path is bad, therefore return the
+        # normal path.
+        return list(get_sys_path())
+
+    curdir = os.path.abspath(os.curdir)
+    with common.ignored(OSError):
+        os.chdir(os.path.dirname(module.path))
+
+    buildout_script_paths = set()
+
+    result = _check_module(evaluator, module)
+    result += _detect_django_path(module.path)
+    for buildout_script in _get_buildout_scripts(module.path):
+        for path in _get_paths_from_buildout_script(evaluator, buildout_script):
+            buildout_script_paths.add(path)
+    # cleanup, back to old directory
+    os.chdir(curdir)
+    return list(result) + list(buildout_script_paths)
+
+
+def _get_paths_from_buildout_script(evaluator, buildout_script):
+    def load(buildout_script):
+        try:
+            with open(buildout_script, 'rb') as f:
+                source = common.source_to_unicode(f.read())
+        except IOError:
+            debug.dbg('Error trying to read buildout_script: %s', buildout_script)
+            return
+
+        p = Parser(evaluator.grammar, source, buildout_script)
+        cache.save_parser(buildout_script, p)
+        return p.module
+
+    cached = cache.load_parser(buildout_script)
+    module = cached and cached.module or load(buildout_script)
+    if not module:
+        return
+
+    for path in _check_module(evaluator, module):
+        yield path
+
+
+def traverse_parents(path):
+    while True:
+        new = os.path.dirname(path)
+        if new == path:
+            return
+        path = new
+        yield path
+
+
+def _get_parent_dir_with_file(path, filename):
+    for parent in traverse_parents(path):
+        if os.path.isfile(os.path.join(parent, filename)):
+            return parent
+    return None
+
+
+def _detect_django_path(module_path):
+    """ Detects the path of the very well known Django library (if used) """
+    result = []
+
+    for parent in traverse_parents(module_path):
+        with common.ignored(IOError):
+            with open(parent + os.path.sep + 'manage.py'):
+                debug.dbg('Found django path: %s', module_path)
+                result.append(parent)
+    return result
+
+
+def _get_buildout_scripts(module_path):
+    """
+    if there is a 'buildout.cfg' file in one of the parent directories of the
+    given module it will return a list of all files in the buildout bin
+    directory that look like python files.
+
+    :param module_path: absolute path to the module.
+    :type module_path: str
+    """
+    project_root = _get_parent_dir_with_file(module_path, 'buildout.cfg')
+    if not project_root:
+        return []
+    bin_path = os.path.join(project_root, 'bin')
+    if not os.path.exists(bin_path):
+        return []
+    extra_module_paths = []
+    for filename in os.listdir(bin_path):
+        try:
+            filepath = os.path.join(bin_path, filename)
+            with open(filepath, 'r') as f:
+                firstline = f.readline()
+                if firstline.startswith('#!') and 'python' in firstline:
+                    extra_module_paths.append(filepath)
+        except IOError as e:
+            # either permission error or race cond. because file got deleted
+            # ignore
+            debug.warning(unicode(e))
+            continue
+    return extra_module_paths

jedi/evaluate_representation.py

@@ -1,901 +0,0 @@
-"""
-Like described in the :mod:`parsing_representation` module, there's a need for
-an ast like module to represent the states of parsed modules.
-
-But now there are also structures in Python that need a little bit more than
-that. An ``Instance`` for example is only a ``Class`` before it is
-instantiated. This class represents these cases.
-
-So, why is there also a ``Class`` class here? Well, there are decorators and
-they change classes in Python 3.
-"""
-from __future__ import with_statement
-
-import copy
-import itertools
-
-from jedi._compatibility import use_metaclass, next, hasattr
-from jedi import parsing_representation as pr
-from jedi import cache
-from jedi import helpers
-from jedi import debug
-from jedi import common
-import recursion
-import docstrings
-import imports
-import evaluate
-import builtin
-import dynamic
-
-
-class DecoratorNotFound(LookupError):
-    """
-    Decorators are sometimes not found, if that happens, that error is raised.
-    """
-    pass
-
-
-class Executable(pr.IsScope):
-    """
-    An instance is also an executable - because __init__ is called
-    :param var_args: The param input array, consist of `pr.Array` or list.
-    """
-    def __init__(self, base, var_args=()):
-        self.base = base
-        self.var_args = var_args
-
-    def get_parent_until(self, *args, **kwargs):
-        return self.base.get_parent_until(*args, **kwargs)
-
-    @property
-    def parent(self):
-        return self.base.parent
-
-
-class Instance(use_metaclass(cache.CachedMetaClass, Executable)):
-    """ This class is used to evaluate instances. """
-    def __init__(self, base, var_args=()):
-        super(Instance, self).__init__(base, var_args)
-        if str(base.name) in ['list', 'set'] \
-                    and builtin.Builtin.scope == base.get_parent_until():
-            # compare the module path with the builtin name.
-            self.var_args = dynamic.check_array_instances(self)
-        else:
-            # need to execute the __init__ function, because the dynamic param
-            # searching needs it.
-            with common.ignored(KeyError):
-                self.execute_subscope_by_name('__init__', self.var_args)
-        # Generated instances are classes that are just generated by self
-        # (No var_args) used.
-        self.is_generated = False
-
-    @cache.memoize_default()
-    def get_init_execution(self, func):
-        func = InstanceElement(self, func, True)
-        return Execution(func, self.var_args)
-
-    def get_func_self_name(self, func):
-        """
-        Returns the name of the first param in a class method (which is
-        normally self
-        """
-        try:
-            return func.params[0].used_vars[0].names[0]
-        except IndexError:
-            return None
-
-    def get_self_properties(self):
-        def add_self_dot_name(name):
-            n = copy.copy(name)
-            n.names = n.names[1:]
-            names.append(InstanceElement(self, n))
-
-        names = []
-        # This loop adds the names of the self object, copies them and removes
-        # the self.
-        for sub in self.base.subscopes:
-            if isinstance(sub, pr.Class):
-                continue
-            # Get the self name, if there's one.
-            self_name = self.get_func_self_name(sub)
-            if self_name:
-                # Check the __init__ function.
-                if sub.name.get_code() == '__init__':
-                    sub = self.get_init_execution(sub)
-                for n in sub.get_set_vars():
-                    # Only names with the selfname are being added.
-                    # It is also important, that they have a len() of 2,
-                    # because otherwise, they are just something else
-                    if n.names[0] == self_name and len(n.names) == 2:
-                        add_self_dot_name(n)
-
-        for s in self.base.get_super_classes():
-            if s == self.base:
-                # I don't know how this could happen... But saw it once.
-                continue
-            names += Instance(s).get_self_properties()
-
-        return names
-
-    def get_subscope_by_name(self, name):
-        sub = self.base.get_subscope_by_name(name)
-        return InstanceElement(self, sub, True)
-
-    def execute_subscope_by_name(self, name, args=()):
-        method = self.get_subscope_by_name(name)
-        return Execution(method, args).get_return_types()
-
-    def get_descriptor_return(self, obj):
-        """ Throws a KeyError if there's no method. """
-        # Arguments in __get__ descriptors are obj, class.
-        # `method` is the new parent of the array, don't know if that's good.
-        args = [obj, obj.base] if isinstance(obj, Instance) else [None, obj]
-        return self.execute_subscope_by_name('__get__', args)
-
-    @cache.memoize_default([])
-    def get_defined_names(self):
-        """
-        Get the instance vars of a class. This includes the vars of all
-        classes
-        """
-        names = self.get_self_properties()
-
-        class_names = self.base.get_defined_names()
-        for var in class_names:
-            names.append(InstanceElement(self, var, True))
-        return names
-
-    def scope_generator(self):
-        """
-        An Instance has two scopes: The scope with self names and the class
-        scope. Instance variables have priority over the class scope.
-        """
-        yield self, self.get_self_properties()
-
-        names = []
-        class_names = self.base.get_defined_names()
-        for var in class_names:
-            names.append(InstanceElement(self, var, True))
-        yield self, names
-
-    def get_index_types(self, index=None):
-        args = [] if index is None else [index]
-        try:
-            return self.execute_subscope_by_name('__getitem__', args)
-        except KeyError:
-            debug.warning('No __getitem__, cannot access the array.')
-            return []
-
-    def __getattr__(self, name):
-        if name not in ['start_pos', 'end_pos', 'name', 'get_imports',
-                'doc', 'docstr', 'asserts']:
-            raise AttributeError("Instance %s: Don't touch this (%s)!"
-                                    % (self, name))
-        return getattr(self.base, name)
-
-    def __repr__(self):
-        return "<e%s of %s (var_args: %s)>" % \
-                (type(self).__name__, self.base, len(self.var_args or []))
-
-
-class InstanceElement(use_metaclass(cache.CachedMetaClass)):
-    """
-    InstanceElement is a wrapper for any object, that is used as an instance
-    variable (e.g. self.variable or class methods).
-    """
-    def __init__(self, instance, var, is_class_var=False):
-        if isinstance(var, pr.Function):
-            var = Function(var)
-        elif isinstance(var, pr.Class):
-            var = Class(var)
-        self.instance = instance
-        self.var = var
-        self.is_class_var = is_class_var
-
-    @property
-    @cache.memoize_default()
-    def parent(self):
-        par = self.var.parent
-        if isinstance(par, Class) and par == self.instance.base \
-                        or isinstance(par, pr.Class) \
-                            and par == self.instance.base.base:
-            par = self.instance
-        elif not isinstance(par, pr.Module):
-            par = InstanceElement(self.instance, par, self.is_class_var)
-        return par
-
-    def get_parent_until(self, *args, **kwargs):
-        return pr.Simple.get_parent_until(self, *args, **kwargs)
-
-    def get_decorated_func(self):
-        """ Needed because the InstanceElement should not be stripped """
-        func = self.var.get_decorated_func()
-        if func == self.var:
-            return self
-        return func
-
-    def get_commands(self):
-        # Copy and modify the array.
-        return [InstanceElement(self.instance, command, self.is_class_var)
-                for command in self.var.get_commands()]
-
-    def __getattr__(self, name):
-        return getattr(self.var, name)
-
-    def isinstance(self, *cls):
-        return isinstance(self.var, cls)
-
-    def __repr__(self):
-        return "<%s of %s>" % (type(self).__name__, self.var)
-
-
-class Class(use_metaclass(cache.CachedMetaClass, pr.IsScope)):
-    """
-    This class is not only important to extend `pr.Class`, it is also a
-    important for descriptors (if the descriptor methods are evaluated or not).
-    """
-    def __init__(self, base):
-        self.base = base
-
-    @cache.memoize_default(default=())
-    def get_super_classes(self):
-        supers = []
-        # TODO care for mro stuff (multiple super classes).
-        for s in self.base.supers:
-            # Super classes are statements.
-            for cls in evaluate.follow_statement(s):
-                if not isinstance(cls, Class):
-                    debug.warning('Received non class, as a super class')
-                    continue  # Just ignore other stuff (user input error).
-                supers.append(cls)
-        if not supers and self.base.parent != builtin.Builtin.scope:
-            # add `object` to classes
-            supers += evaluate.find_name(builtin.Builtin.scope, 'object')
-        return supers
-
-    @cache.memoize_default(default=())
-    def get_defined_names(self):
-        def in_iterable(name, iterable):
-            """ checks if the name is in the variable 'iterable'. """
-            for i in iterable:
-                # Only the last name is important, because these names have a
-                # maximal length of 2, with the first one being `self`.
-                if i.names[-1] == name.names[-1]:
-                    return True
-            return False
-
-        result = self.base.get_defined_names()
-        super_result = []
-        # TODO mro!
-        for cls in self.get_super_classes():
-            # Get the inherited names.
-            for i in cls.get_defined_names():
-                if not in_iterable(i, result):
-                    super_result.append(i)
-        result += super_result
-        return result
-
-    def get_subscope_by_name(self, name):
-        for sub in reversed(self.subscopes):
-            if sub.name.get_code() == name:
-                return sub
-        raise KeyError("Couldn't find subscope.")
-
-    @property
-    def name(self):
-        return self.base.name
-
-    def __getattr__(self, name):
-        if name not in ['start_pos', 'end_pos', 'parent', 'asserts', 'docstr',
-                'doc', 'get_imports', 'get_parent_until', 'get_code',
-                'subscopes']:
-            raise AttributeError("Don't touch this: %s of %s !" % (name, self))
-        return getattr(self.base, name)
-
-    def __repr__(self):
-        return "<e%s of %s>" % (type(self).__name__, self.base)
-
-
-class Function(use_metaclass(cache.CachedMetaClass, pr.IsScope)):
-    """
-    Needed because of decorators. Decorators are evaluated here.
-    """
-    def __init__(self, func, is_decorated=False):
-        """ This should not be called directly """
-        self.base_func = func
-        self.is_decorated = is_decorated
-
-    @property
-    @cache.memoize_default()
-    def _decorated_func(self):
-        """
-        Returns the function, that is to be executed in the end.
-        This is also the places where the decorators are processed.
-        """
-        f = self.base_func
-
-        # Only enter it, if has not already been processed.
-        if not self.is_decorated:
-            for dec in reversed(self.base_func.decorators):
-                debug.dbg('decorator:', dec, f)
-                dec_results = evaluate.follow_statement(dec)
-                if not len(dec_results):
-                    debug.warning('decorator func not found: %s in stmt %s' %
-                                                        (self.base_func, dec))
-                    return None
-                if len(dec_results) > 1:
-                    debug.warning('multiple decorators found', self.base_func,
-                                                            dec_results)
-                decorator = dec_results.pop()
-                # Create param array.
-                old_func = Function(f, is_decorated=True)
-
-                wrappers = Execution(decorator, (old_func,)).get_return_types()
-                if not len(wrappers):
-                    debug.warning('no wrappers found', self.base_func)
-                    return None
-                if len(wrappers) > 1:
-                    debug.warning('multiple wrappers found', self.base_func,
-                                                                wrappers)
-                # This is here, that the wrapper gets executed.
-                f = wrappers[0]
-
-                debug.dbg('decorator end', f)
-        if f != self.base_func and isinstance(f, pr.Function):
-            f = Function(f)
-        return f
-
-    def get_decorated_func(self):
-        if self._decorated_func is None:
-            raise DecoratorNotFound()
-        if self._decorated_func == self.base_func:
-            return self
-        return self._decorated_func
-
-    def get_magic_method_names(self):
-        return builtin.Builtin.magic_function_scope.get_defined_names()
-
-    def get_magic_method_scope(self):
-        return builtin.Builtin.magic_function_scope
-
-    def __getattr__(self, name):
-        return getattr(self.base_func, name)
-
-    def __repr__(self):
-        dec = ''
-        if self._decorated_func != self.base_func:
-            dec = " is " + repr(self._decorated_func)
-        return "<e%s of %s%s>" % (type(self).__name__, self.base_func, dec)
-
-
-class Execution(Executable):
-    """
-    This class is used to evaluate functions and their returns.
-
-    This is the most complicated class, because it contains the logic to
-    transfer parameters. It is even more complicated, because there may be
-    multiple calls to functions and recursion has to be avoided. But this is
-    responsibility of the decorators.
-    """
-    def follow_var_arg(self, index):
-        try:
-            stmt = self.var_args[index]
-        except IndexError:
-            return []
-        else:
-            if isinstance(stmt, pr.Statement):
-                return evaluate.follow_statement(stmt)
-            else:
-                return [stmt]  # just some arbitrary object
-
-    @cache.memoize_default(default=())
-    @recursion.ExecutionRecursionDecorator
-    def get_return_types(self, evaluate_generator=False):
-        """ Get the return types of a function. """
-        stmts = []
-        if self.base.parent == builtin.Builtin.scope \
-                and not isinstance(self.base, (Generator, Array)):
-            func_name = str(self.base.name)
-
-            # some implementations of builtins:
-            if func_name == 'getattr':
-                # follow the first param
-                try:
-                    objects = self.follow_var_arg(0)
-                    names = self.follow_var_arg(1)
-                except IndexError:
-                    debug.warning('getattr() called with to few args.')
-                    return []
-
-                for obj in objects:
-                    if not isinstance(obj, (Instance, Class, pr.Module)):
-                        debug.warning('getattr called without instance')
-                        continue
-
-                    for arr_name in names:
-                        if len(arr_name.var_args) != 1:
-                            debug.warning('jedi getattr is too simple')
-                        key = arr_name.var_args[0]
-                        stmts += evaluate.follow_path(iter([key]), obj,
-                                                        self.base)
-                return stmts
-            elif func_name == 'type':
-                # otherwise it would be a metaclass
-                if len(self.var_args) == 1:
-                    objects = self.follow_var_arg(0)
-                    return [o.base for o in objects if isinstance(o, Instance)]
-            elif func_name == 'super':
-                # TODO make this able to detect multiple inheritance supers
-                accept = (pr.Function,)
-                func = self.var_args.get_parent_until(accept)
-                if func.isinstance(*accept):
-                    cls = func.get_parent_until(accept + (pr.Class,),
-                                                    include_current=False)
-                    if isinstance(cls, pr.Class):
-                        cls = Class(cls)
-                        su = cls.get_super_classes()
-                        if su:
-                            return [Instance(su[0])]
-                return []
-
-        if self.base.isinstance(Class):
-            # There maybe executions of executions.
-            stmts = [Instance(self.base, self.var_args)]
-        elif isinstance(self.base, Generator):
-            return self.base.iter_content()
-        else:
-            # Don't do this with exceptions, as usual, because some deeper
-            # exceptions could be catched - and I wouldn't know what happened.
-            try:
-                self.base.returns
-            except (AttributeError, DecoratorNotFound):
-                if hasattr(self.base, 'execute_subscope_by_name'):
-                    try:
-                        stmts = self.base.execute_subscope_by_name('__call__',
-                                                                self.var_args)
-                    except KeyError:
-                        debug.warning("no __call__ func available", self.base)
-                else:
-                    debug.warning("no execution possible", self.base)
-            else:
-                stmts = self._get_function_returns(evaluate_generator)
-
-        debug.dbg('exec result: %s in %s' % (stmts, self))
-
-        return imports.strip_imports(stmts)
-
-    def _get_function_returns(self, evaluate_generator):
-        """ A normal Function execution """
-        # Feed the listeners, with the params.
-        for listener in self.base.listeners:
-            listener.execute(self.get_params())
-        func = self.base.get_decorated_func()
-        if func.is_generator and not evaluate_generator:
-            return [Generator(func, self.var_args)]
-        else:
-            stmts = docstrings.find_return_types(func)
-            for r in self.returns:
-                if r is not None:
-                    stmts += evaluate.follow_statement(r)
-            return stmts
-
-    @cache.memoize_default(default=())
-    def get_params(self):
-        """
-        This returns the params for an Execution/Instance and is injected as a
-        'hack' into the pr.Function class.
-        This needs to be here, because Instance can have __init__ functions,
-        which act the same way as normal functions.
-        """
-        def gen_param_name_copy(param, keys=(), values=(), array_type=None):
-            """
-            Create a param with the original scope (of varargs) as parent.
-            """
-            if isinstance(self.var_args, pr.Array):
-                parent = self.var_args.parent
-                start_pos = self.var_args.start_pos
-            else:
-                parent = self.base
-                start_pos = 0, 0
-
-            new_param = copy.copy(param)
-            new_param.is_generated = True
-            if parent is not None:
-                new_param.parent = parent
-
-            # create an Array (-> needed for *args/**kwargs tuples/dicts)
-            arr = pr.Array(self._sub_module, start_pos, array_type, parent)
-            arr.values = values
-            key_stmts = []
-            for key in keys:
-                stmt = pr.Statement(self._sub_module, [], [], [],
-                                    start_pos, None)
-                stmt._commands = [key]
-                key_stmts.append(stmt)
-            arr.keys = key_stmts
-            arr.type = array_type
-
-            new_param._commands = [arr]
-
-            name = copy.copy(param.get_name())
-            name.parent = new_param
-            return name
-
-        result = []
-        start_offset = 0
-        if isinstance(self.base, InstanceElement):
-            # Care for self -> just exclude it and add the instance
-            start_offset = 1
-            self_name = copy.copy(self.base.params[0].get_name())
-            self_name.parent = self.base.instance
-            result.append(self_name)
-
-        param_dict = {}
-        for param in self.base.params:
-            param_dict[str(param.get_name())] = param
-        # There may be calls, which don't fit all the params, this just ignores
-        # it.
-        var_arg_iterator = self.get_var_args_iterator()
-
-        non_matching_keys = []
-        keys_used = set()
-        keys_only = False
-        for param in self.base.params[start_offset:]:
-            # The value and key can both be null. There, the defaults apply.
-            # args / kwargs will just be empty arrays / dicts, respectively.
-            # Wrong value count is just ignored. If you try to test cases that
-            # are not allowed in Python, Jedi will maybe not show any
-            # completions.
-            key, value = next(var_arg_iterator, (None, None))
-            while key:
-                keys_only = True
-                try:
-                    key_param = param_dict[str(key)]
-                except KeyError:
-                    non_matching_keys.append((key, value))
-                else:
-                    keys_used.add(str(key))
-                    result.append(gen_param_name_copy(key_param,
-                                                        values=[value]))
-                key, value = next(var_arg_iterator, (None, None))
-
-            commands = param.get_commands()
-            keys = []
-            values = []
-            array_type = None
-            ignore_creation = False
-            if commands[0] == '*':
-                # *args param
-                array_type = pr.Array.TUPLE
-                if value:
-                    values.append(value)
-                for key, value in var_arg_iterator:
-                    # Iterate until a key argument is found.
-                    if key:
-                        var_arg_iterator.push_back((key, value))
-                        break
-                    values.append(value)
-            elif commands[0] == '**':
-                # **kwargs param
-                array_type = pr.Array.DICT
-                if non_matching_keys:
-                    keys, values = zip(*non_matching_keys)
-            elif not keys_only:
-                # normal param
-                if value is not None:
-                    values = [value]
-                else:
-                    if param.assignment_details:
-                        # No value: return the default values.
-                        ignore_creation = True
-                        result.append(param.get_name())
-                        param.is_generated = True
-                    else:
-                        # If there is no assignment detail, that means there is
-                        # no assignment, just the result. Therefore nothing has
-                        # to be returned.
-                        values = []
-
-            # Just ignore all the params that are without a key, after one
-            # keyword argument was set.
-            if not ignore_creation and (not keys_only or commands[0] == '**'):
-                keys_used.add(str(key))
-                result.append(gen_param_name_copy(param, keys=keys,
-                                        values=values, array_type=array_type))
-
-        if keys_only:
-            # sometimes param arguments are not completely written (which would
-            # create an Exception, but we have to handle that).
-            for k in set(param_dict) - keys_used:
-                result.append(gen_param_name_copy(param_dict[k]))
-        return result
-
-    def get_var_args_iterator(self):
-        """
-        Yields a key/value pair, the key is None, if its not a named arg.
-        """
-        def iterate():
-            # `var_args` is typically an Array, and not a list.
-            for stmt in self.var_args:
-                if not isinstance(stmt, pr.Statement):
-                    if stmt is None:
-                        yield None, None
-                        continue
-                    old = stmt
-                    # generate a statement if it's not already one.
-                    module = builtin.Builtin.scope
-                    stmt = pr.Statement(module, [], [], [], (0, 0), None)
-                    stmt._commands = [old]
-
-                # *args
-                if stmt.get_commands()[0] == '*':
-                    arrays = evaluate.follow_call_list(stmt.get_commands()[1:])
-                    # *args must be some sort of an array, otherwise -> ignore
-                    for array in arrays:
-                        for field_stmt in array:  # yield from plz!
-                            yield None, field_stmt
-                # **kwargs
-                elif stmt.get_commands()[0] == '**':
-                    arrays = evaluate.follow_call_list(stmt.get_commands()[1:])
-                    for array in arrays:
-                        for key_stmt, value_stmt in array.items():
-                            # first index, is the key if syntactically correct
-                            call = key_stmt.get_commands()[0]
-                            if isinstance(call, pr.Name):
-                                yield call, value_stmt
-                            elif type(call) == pr.Call:
-                                yield call.name, value_stmt
-                # Normal arguments (including key arguments).
-                else:
-                    if stmt.assignment_details:
-                        key_arr, op = stmt.assignment_details[0]
-                        # named parameter
-                        if key_arr and isinstance(key_arr[0], pr.Call):
-                            yield key_arr[0].name, stmt
-                    else:
-                        yield None, stmt
-
-        return iter(common.PushBackIterator(iterate()))
-
-    def get_set_vars(self):
-        return self.get_defined_names()
-
-    def get_defined_names(self):
-        """
-        Call the default method with the own instance (self implements all
-        the necessary functions). Add also the params.
-        """
-        return self.get_params() + pr.Scope.get_set_vars(self)
-
-    @common.rethrow_uncaught
-    def copy_properties(self, prop):
-        """
-        Literally copies a property of a Function. Copying is very expensive,
-        because it is something like `copy.deepcopy`. However, these copied
-        objects can be used for the executions, as if they were in the
-        execution.
-        """
-        # Copy all these lists into this local function.
-        attr = getattr(self.base, prop)
-        objects = []
-        for element in attr:
-            if element is None:
-                copied = element
-            else:
-                copied = helpers.fast_parent_copy(element)
-                copied.parent = self._scope_copy(copied.parent)
-                if isinstance(copied, pr.Function):
-                    copied = Function(copied)
-            objects.append(copied)
-        return objects
-
-    def __getattr__(self, name):
-        if name not in ['start_pos', 'end_pos', 'imports', '_sub_module']:
-            raise AttributeError('Tried to access %s: %s. Why?' % (name, self))
-        return getattr(self.base, name)
-
-    @cache.memoize_default()
-    @common.rethrow_uncaught
-    def _scope_copy(self, scope):
-        """ Copies a scope (e.g. if) in an execution """
-        # TODO method uses different scopes than the subscopes property.
-
-        # just check the start_pos, sometimes it's difficult with closures
-        # to compare the scopes directly.
-        if scope.start_pos == self.start_pos:
-            return self
-        else:
-            copied = helpers.fast_parent_copy(scope)
-            copied.parent = self._scope_copy(copied.parent)
-            return copied
-
-    @property
-    @cache.memoize_default()
-    def returns(self):
-        return self.copy_properties('returns')
-
-    @property
-    @cache.memoize_default()
-    def asserts(self):
-        return self.copy_properties('asserts')
-
-    @property
-    @cache.memoize_default()
-    def statements(self):
-        return self.copy_properties('statements')
-
-    @property
-    @cache.memoize_default()
-    def subscopes(self):
-        return self.copy_properties('subscopes')
-
-    def get_statement_for_position(self, pos):
-        return pr.Scope.get_statement_for_position(self, pos)
-
-    def __repr__(self):
-        return "<%s of %s>" % \
-                (type(self).__name__, self.base)
-
-
-class Generator(use_metaclass(cache.CachedMetaClass, pr.Base)):
-    """ Cares for `yield` statements. """
-    def __init__(self, func, var_args):
-        super(Generator, self).__init__()
-        self.func = func
-        self.var_args = var_args
-
-    def get_defined_names(self):
-        """
-        Returns a list of names that define a generator, which can return the
-        content of a generator.
-        """
-        names = []
-        none_pos = (0, 0)
-        executes_generator = ('__next__', 'send')
-        for n in ('close', 'throw') + executes_generator:
-            name = pr.Name(builtin.Builtin.scope, [(n, none_pos)],
-                                none_pos, none_pos)
-            if n in executes_generator:
-                name.parent = self
-            names.append(name)
-        debug.dbg('generator names', names)
-        return names
-
-    def iter_content(self):
-        """ returns the content of __iter__ """
-        return Execution(self.func, self.var_args).get_return_types(True)
-
-    def get_index_types(self, index=None):
-        debug.warning('Tried to get array access on a generator', self)
-        return []
-
-    def __getattr__(self, name):
-        if name not in ['start_pos', 'end_pos', 'parent', 'get_imports',
-                'asserts', 'doc', 'docstr', 'get_parent_until', 'get_code',
-                'subscopes']:
-            raise AttributeError("Accessing %s of %s is not allowed."
-                                    % (self, name))
-        return getattr(self.func, name)
-
-    def __repr__(self):
-        return "<%s of %s>" % (type(self).__name__, self.func)
-
-
-class Array(use_metaclass(cache.CachedMetaClass, pr.Base)):
-    """
-    Used as a mirror to pr.Array, if needed. It defines some getter
-    methods which are important in this module.
-    """
-    def __init__(self, array):
-        self._array = array
-
-    def get_index_types(self, index_arr=None):
-        """ Get the types of a specific index or all, if not given """
-        if index_arr is not None:
-            if index_arr and [x for x in index_arr if ':' in x.get_commands()]:
-                # array slicing
-                return [self]
-
-            index_possibilities = self._follow_values(index_arr)
-            if len(index_possibilities) == 1:
-                # This is indexing only one element, with a fixed index number,
-                # otherwise it just ignores the index (e.g. [1+1]).
-                index = index_possibilities[0]
-                if isinstance(index, Instance) \
-                            and str(index.name) in ['int', 'str'] \
-                            and len(index.var_args) == 1:
-                    with common.ignored(KeyError, IndexError):
-                        return self.get_exact_index_types(index.var_args[0])
-
-        result = list(self._follow_values(self._array.values))
-        result += dynamic.check_array_additions(self)
-        return set(result)
-
-    def get_exact_index_types(self, mixed_index):
-        """ Here the index is an int/str. Raises IndexError/KeyError """
-        index = mixed_index
-        if self.type == pr.Array.DICT:
-            index = None
-            for i, key_statement in enumerate(self._array.keys):
-                # Because we only want the key to be a string.
-                key_commands = key_statement.get_commands()
-                if len(key_commands) != 1:  # cannot deal with complex strings
-                    continue
-                key = key_commands[0]
-                if isinstance(key, pr.Call) and key.type == pr.Call.STRING:
-                    str_key = key.name
-                elif isinstance(key, pr.Name):
-                    str_key = str(key)
-
-                if mixed_index == str_key:
-                    index = i
-                    break
-            if index is None:
-                raise KeyError('No key found in dictionary')
-
-        # Can raise an IndexError
-        values = [self._array.values[index]]
-        return self._follow_values(values)
-
-    def _follow_values(self, values):
-        """ helper function for the index getters """
-        return list(itertools.chain.from_iterable(evaluate.follow_statement(v)
-                                                  for v in values))
-
-    def get_defined_names(self):
-        """
-        This method generates all `ArrayMethod` for one pr.Array.
-        It returns e.g. for a list: append, pop, ...
-        """
-        # `array.type` is a string with the type, e.g. 'list'.
-        scope = evaluate.find_name(builtin.Builtin.scope, self._array.type)[0]
-        scope = Instance(scope)
-        names = scope.get_defined_names()
-        return [ArrayMethod(n) for n in names]
-
-    @property
-    def parent(self):
-        return builtin.Builtin.scope
-
-    def get_parent_until(self):
-        return builtin.Builtin.scope
-
-    def __getattr__(self, name):
-        if name not in ['type', 'start_pos', 'get_only_subelement', 'parent',
-                        'get_parent_until', 'items']:
-            raise AttributeError('Strange access on %s: %s.' % (self, name))
-        return getattr(self._array, name)
-
-    def __getitem__(self):
-        return self._array.__getitem__()
-
-    def __iter__(self):
-        return self._array.__iter__()
-
-    def __len__(self):
-        return self._array.__len__()
-
-    def __repr__(self):
-        return "<e%s of %s>" % (type(self).__name__, self._array)
-
-
-class ArrayMethod(object):
-    """
-    A name, e.g. `list.append`, it is used to access the original array
-    methods.
-    """
-    def __init__(self, name):
-        super(ArrayMethod, self).__init__()
-        self.name = name
-
-    def __getattr__(self, name):
-        # Set access privileges:
-        if name not in ['parent', 'names', 'start_pos', 'end_pos', 'get_code']:
-            raise AttributeError('Strange accesson %s: %s.' % (self, name))
-        return getattr(self.name, name)
-
-    def get_parent_until(self):
-        return builtin.Builtin.scope
-
-    def __repr__(self):
-        return "<%s of %s>" % (type(self).__name__, self.name)

jedi/fast_parser.py

@@ -1,424 +0,0 @@
-"""
-Basically a parser that is faster, because it tries to parse only parts and if
-anything changes, it only reparses the changed parts. But because it's not
-finished (and still not working as I want), I won't document it any further.
-"""
-import re
-
-from jedi._compatibility import use_metaclass
-from jedi import settings
-from jedi import parsing
-from jedi import parsing_representation as pr
-from jedi import cache
-from jedi import common
-
-
-SCOPE_CONTENTS = ['asserts', 'subscopes', 'imports', 'statements', 'returns']
-
-
-class Module(pr.Simple, pr.Module):
-    def __init__(self, parsers):
-        super(Module, self).__init__(self, (1, 0))
-        self.parsers = parsers
-        self.reset_caches()
-
-        self.start_pos = 1, 0
-        self.end_pos = None, None
-
-    def reset_caches(self):
-        """ This module does a whole lot of caching, because it uses different
-        parsers. """
-        self._used_names = None
-        for p in self.parsers:
-            p.user_scope = None
-            p.user_stmt = None
-
-    def __getattr__(self, name):
-        if name.startswith('__'):
-            raise AttributeError('Not available!')
-        else:
-            return getattr(self.parsers[0].module, name)
-
-    @property
-    def used_names(self):
-        if self._used_names is None:
-            dct = {}
-            for p in self.parsers:
-                for k, statement_set in p.module.used_names.items():
-                    if k in dct:
-                        dct[k] |= statement_set
-                    else:
-                        dct[k] = set(statement_set)
-
-            self._used_names = dct
-        return self._used_names
-
-    def __repr__(self):
-        return "<%s: %s@%s-%s>" % (type(self).__name__, self.name,
-                                    self.start_pos[0], self.end_pos[0])
-
-
-class CachedFastParser(type):
-    """ This is a metaclass for caching `FastParser`. """
-    def __call__(self, source, module_path=None, user_position=None):
-        if not settings.fast_parser:
-            return parsing.Parser(source, module_path, user_position)
-
-        pi = cache.parser_cache.get(module_path, None)
-        if pi is None or isinstance(pi.parser, parsing.Parser):
-            p = super(CachedFastParser, self).__call__(source, module_path,
-                                                            user_position)
-        else:
-            p = pi.parser  # pi is a `cache.ParserCacheItem`
-            p.update(source, user_position)
-        return p
-
-
-class ParserNode(object):
-    def __init__(self, parser, code, parent=None):
-        self.parent = parent
-        self.code = code
-        self.hash = hash(code)
-
-        self.children = []
-        # must be created before new things are added to it.
-        self.save_contents(parser)
-
-    def save_contents(self, parser):
-        self.parser = parser
-
-        try:
-            # with fast_parser we have either 1 subscope or only statements.
-            self.content_scope = parser.module.subscopes[0]
-        except IndexError:
-            self.content_scope = parser.module
-
-        scope = self.content_scope
-        self._contents = {}
-        for c in SCOPE_CONTENTS:
-            self._contents[c] = list(getattr(scope, c))
-        self._is_generator = scope.is_generator
-
-        self.old_children = self.children
-        self.children = []
-
-    def reset_contents(self):
-        scope = self.content_scope
-        for key, c in self._contents.items():
-            setattr(scope, key, list(c))
-        scope.is_generator = self._is_generator
-        self.parser.user_scope = self.parser.module
-
-        if self.parent is None:
-            # Global vars of the first one can be deleted, in the global scope
-            # they make no sense.
-            self.parser.module.global_vars = []
-
-        for c in self.children:
-            c.reset_contents()
-
-    def parent_until_indent(self, indent=None):
-        if indent is None or self.indent >= indent and self.parent:
-            self.old_children = []
-            if self.parent is not None:
-                return self.parent.parent_until_indent(indent)
-        return self
-
-    @property
-    def indent(self):
-        if not self.parent:
-            return 0
-        module = self.parser.module
-        try:
-            el = module.subscopes[0]
-        except IndexError:
-            try:
-                el = module.statements[0]
-            except IndexError:
-                try:
-                    el = module.imports[0]
-                except IndexError:
-                    try:
-                        el = [r for r in module.returns if r is not None][0]
-                    except IndexError:
-                        return self.parent.indent + 1
-        return el.start_pos[1]
-
-    def _set_items(self, parser, set_parent=False):
-        # insert parser objects into current structure
-        scope = self.content_scope
-        for c in SCOPE_CONTENTS:
-            content = getattr(scope, c)
-            items = getattr(parser.module, c)
-            if set_parent:
-                for i in items:
-                    if i is None:
-                        continue  # happens with empty returns
-                    i.parent = scope.use_as_parent
-                    if isinstance(i, (pr.Function, pr.Class)):
-                        for d in i.decorators:
-                            d.parent = scope.use_as_parent
-            content += items
-
-        # global_vars
-        cur = self
-        while cur.parent is not None:
-            cur = cur.parent
-        cur.parser.module.global_vars += parser.module.global_vars
-
-        scope.is_generator |= parser.module.is_generator
-
-    def add_node(self, node, set_parent=False):
-        """Adding a node means adding a node that was already added earlier"""
-        self.children.append(node)
-        self._set_items(node.parser, set_parent=set_parent)
-        node.old_children = node.children
-        node.children = []
-        return node
-
-    def add_parser(self, parser, code):
-        return self.add_node(ParserNode(parser, code, self), True)
-
-
-class FastParser(use_metaclass(CachedFastParser)):
-    def __init__(self, code, module_path=None, user_position=None):
-        # set values like `pr.Module`.
-        self.module_path = module_path
-        self.user_position = user_position
-        self._user_scope = None
-
-        self.current_node = None
-        self.parsers = []
-        self.module = Module(self.parsers)
-        self.reset_caches()
-
-        self._parse(code)
-
-    @property
-    def user_scope(self):
-        if self._user_scope is None:
-            for p in self.parsers:
-                if p.user_scope:
-                    if isinstance(p.user_scope, pr.SubModule):
-                        continue
-                    self._user_scope = p.user_scope
-
-        if isinstance(self._user_scope, pr.SubModule) \
-                    or self._user_scope is None:
-            self._user_scope = self.module
-        return self._user_scope
-
-    @property
-    def user_stmt(self):
-        if self._user_stmt is None:
-            for p in self.parsers:
-                if p.user_stmt:
-                    self._user_stmt = p.user_stmt
-                    break
-        return self._user_stmt
-
-    def update(self, code, user_position=None):
-        self.user_position = user_position
-        self.reset_caches()
-
-        self._parse(code)
-
-    def _scan_user_scope(self, sub_module):
-        """ Scan with self.user_position. """
-        for scope in sub_module.statements + sub_module.subscopes:
-            if isinstance(scope, pr.Scope):
-                if scope.start_pos <= self.user_position <= scope.end_pos:
-                    return self._scan_user_scope(scope) or scope
-        return None
-
-    def _split_parts(self, code):
-        """
-        Split the code into different parts. This makes it possible to parse
-        each part seperately and therefore cache parts of the file and not
-        everything.
-        """
-        def add_part():
-            txt = '\n'.join(current_lines)
-            if txt:
-                if add_to_last and parts:
-                    parts[-1] += '\n' + txt
-                else:
-                    parts.append(txt)
-                current_lines[:] = []
-
-        r_keyword = '^[ \t]*(def|class|@|%s)' % '|'.join(common.FLOWS)
-
-        lines = code.splitlines()
-        current_lines = []
-        parts = []
-        is_decorator = False
-        current_indent = 0
-        old_indent = 0
-        new_indent = False
-        in_flow = False
-        add_to_last = False
-        # All things within flows are simply being ignored.
-        for i, l in enumerate(lines):
-            # check for dedents
-            m = re.match('^([\t ]*)(.?)', l)
-            indent = len(m.group(1))
-            if m.group(2) in ['', '#']:
-                current_lines.append(l)  # just ignore comments and blank lines
-                continue
-
-            if indent < current_indent:  # -> dedent
-                current_indent = indent
-                new_indent = False
-                if not in_flow or indent < old_indent:
-                    add_part()
-                    add_to_last = False
-                in_flow = False
-            elif new_indent:
-                current_indent = indent
-                new_indent = False
-
-            # Check lines for functions/classes and split the code there.
-            if not in_flow:
-                m = re.match(r_keyword, l)
-                if m:
-                    in_flow = m.group(1) in common.FLOWS
-                    if not is_decorator and not in_flow:
-                        add_part()
-                        add_to_last = False
-                    is_decorator = '@' == m.group(1)
-                    if not is_decorator:
-                        old_indent = current_indent
-                        current_indent += 1  # it must be higher
-                        new_indent = True
-                elif is_decorator:
-                    is_decorator = False
-                    add_to_last = True
-
-            current_lines.append(l)
-        add_part()
-
-        return parts
-
-    def _parse(self, code):
-        """ :type code: str """
-        def empty_parser():
-            new, temp = self._get_parser('', '', 0, [], False)
-            return new
-
-        parts = self._split_parts(code)
-        self.parsers[:] = []
-
-        line_offset = 0
-        start = 0
-        p = None
-        is_first = True
-
-        for code_part in parts:
-            lines = code_part.count('\n') + 1
-            if is_first or line_offset >= p.end_pos[0]:
-                indent = len(re.match(r'[ \t]*', code_part).group(0))
-                if is_first and self.current_node is not None:
-                    nodes = [self.current_node]
-                else:
-                    nodes = []
-                if self.current_node is not None:
-
-                    self.current_node = \
-                                self.current_node.parent_until_indent(indent)
-                    nodes += self.current_node.old_children
-
-                # check if code_part has already been parsed
-                #print '#'*45,line_offset, p and p.end_pos, '\n', code_part
-                p, node = self._get_parser(code_part, code[start:],
-                                           line_offset, nodes, not is_first)
-
-                if is_first and p.module.subscopes:
-                    # special case, we cannot use a function subscope as a
-                    # base scope, subscopes would save all the other contents
-                    new = empty_parser()
-                    if self.current_node is None:
-                        self.current_node = ParserNode(new, '')
-                    else:
-                        self.current_node.save_contents(new)
-                    self.parsers.append(new)
-                    is_first = False
-
-                if is_first:
-                    if self.current_node is None:
-                        self.current_node = ParserNode(p, code_part)
-                    else:
-                        self.current_node.save_contents(p)
-                else:
-                    if node is None:
-                        self.current_node = \
-                                    self.current_node.add_parser(p, code_part)
-                    else:
-                        self.current_node = self.current_node.add_node(node)
-
-                if self.current_node.parent and (isinstance(p.user_scope,
-                                pr.SubModule) or p.user_scope is None) \
-                        and self.user_position \
-                        and p.start_pos <= self.user_position < p.end_pos:
-                    p.user_scope = self.current_node.parent.content_scope
-
-                self.parsers.append(p)
-
-                is_first = False
-            else:
-                #print '#'*45, line_offset, p.end_pos, 'theheck\n', code_part 
-                pass
-
-            line_offset += lines
-            start += len(code_part) + 1  # +1 for newline
-
-        if self.parsers:
-            self.current_node = self.current_node.parent_until_indent()
-        else:
-            self.parsers.append(empty_parser())
-
-        self.module.end_pos = self.parsers[-1].end_pos
-
-        #print(self.parsers[0].module.get_code())
-        del code
-
-    def _get_parser(self, code, parser_code, line_offset, nodes, no_docstr):
-        h = hash(code)
-        hashes = [n.hash for n in nodes]
-        node = None
-        try:
-            index = hashes.index(h)
-            if nodes[index].code != code:
-                raise ValueError()
-        except ValueError:
-            p = parsing.Parser(parser_code, self.module_path,
-                               self.user_position, offset=(line_offset, 0),
-                               is_fast_parser=True, top_module=self.module,
-                               no_docstr=no_docstr)
-            p.module.parent = self.module
-        else:
-            if nodes[index] != self.current_node:
-                offset = int(nodes[0] == self.current_node)
-                self.current_node.old_children.pop(index - offset)
-            node = nodes.pop(index)
-            p = node.parser
-            m = p.module
-            m.line_offset += line_offset + 1 - m.start_pos[0]
-            if self.user_position is not None and \
-                    m.start_pos[0] <= self.user_position[0] <= m.end_pos[0]:
-                # It's important to take care of the whole user
-                # positioning stuff, if no reparsing is being done.
-                p.user_stmt = m.get_statement_for_position(
-                            self.user_position, include_imports=True)
-                if p.user_stmt:
-                    p.user_scope = p.user_stmt.parent
-                else:
-                    p.user_scope = self._scan_user_scope(m) or m
-
-        return p, node
-
-    def reset_caches(self):
-        self._user_scope = None
-        self._user_stmt = None
-        self.module.reset_caches()
-        if self.current_node is not None:
-            self.current_node.reset_contents()

jedi/helpers.py

@@ -1,127 +0,0 @@
-from __future__ import with_statement
-
-import copy
-
-from jedi import common
-from jedi import parsing_representation as pr
-
-
-def fast_parent_copy(obj):
-    """
-    Much, much faster than copy.deepcopy, but just for certain elements.
-    """
-    new_elements = {}
-
-    def recursion(obj):
-        new_obj = copy.copy(obj)
-        new_elements[obj] = new_obj
-
-        try:
-            items = list(new_obj.__dict__.items())
-        except AttributeError:
-            # __dict__ not available, because of __slots__
-            items = []
-
-        before = ()
-        for cls in new_obj.__class__.__mro__:
-            with common.ignored(AttributeError):
-                if before == cls.__slots__:
-                    continue
-                before = cls.__slots__
-                items += [(n, getattr(new_obj, n)) for n in before]
-
-        for key, value in items:
-            # replace parent (first try _parent and then parent)
-            if key in ['parent', '_parent'] and value is not None:
-                if key == 'parent' and '_parent' in items:
-                    # parent can be a property
-                    continue
-                with common.ignored(KeyError):
-                    setattr(new_obj, key, new_elements[value])
-            elif key in ['parent_function', 'use_as_parent', '_sub_module']:
-                continue
-            elif isinstance(value, list):
-                setattr(new_obj, key, list_rec(value))
-            elif isinstance(value, (pr.Simple, pr.Call)):
-                setattr(new_obj, key, recursion(value))
-        return new_obj
-
-    def list_rec(list_obj):
-        copied_list = list_obj[:]   # lists, tuples, strings, unicode
-        for i, el in enumerate(copied_list):
-            if isinstance(el, (pr.Simple, pr.Call)):
-                copied_list[i] = recursion(el)
-            elif isinstance(el, list):
-                copied_list[i] = list_rec(el)
-        return copied_list
-    return recursion(obj)
-
-
-def check_arr_index(arr, pos):
-    positions = arr.arr_el_pos
-    for index, comma_pos in enumerate(positions):
-        if pos < comma_pos:
-            return index
-    return len(positions)
-
-
-def array_for_pos(stmt, pos, array_types=None):
-    """Searches for the array and position of a tuple"""
-    def search_array(arr, pos):
-        if arr.type == 'dict':
-            for stmt in arr.values + arr.keys:
-                new_arr, index = array_for_pos(stmt, pos, array_types)
-                if new_arr is not None:
-                    return new_arr, index
-        else:
-            for i, stmt in enumerate(arr):
-                new_arr, index = array_for_pos(stmt, pos, array_types)
-                if new_arr is not None:
-                    return new_arr, index
-                if arr.start_pos < pos <= stmt.end_pos:
-                    if not array_types or arr.type in array_types:
-                        return arr, i
-        if len(arr) == 0 and arr.start_pos < pos < arr.end_pos:
-            if not array_types or arr.type in array_types:
-                return arr, 0
-        return None, 0
-
-    def search_call(call, pos):
-        arr, index = None, 0
-        if call.next is not None:
-            if isinstance(call.next, pr.Array):
-                arr, index = search_array(call.next, pos)
-            else:
-                arr, index = search_call(call.next, pos)
-        if not arr and call.execution is not None:
-            arr, index = search_array(call.execution, pos)
-        return arr, index
-
-    if stmt.start_pos >= pos >= stmt.end_pos:
-        return None, 0
-
-    for command in stmt.get_commands():
-        arr = None
-        if isinstance(command, pr.Array):
-            arr, index = search_array(command, pos)
-        elif isinstance(command, pr.Call):
-            arr, index = search_call(command, pos)
-        if arr is not None:
-            return arr, index
-    return None, 0
-
-
-def search_function_definition(stmt, pos):
-    """
-    Returns the function Call that matches the position before.
-    """
-    # some parts will of the statement will be removed
-    stmt = fast_parent_copy(stmt)
-    arr, index = array_for_pos(stmt, pos, [pr.Array.TUPLE, pr.Array.NOARRAY])
-    if arr is not None and isinstance(arr.parent, pr.Call):
-        call = arr.parent
-        while isinstance(call.parent, pr.Call):
-            call = call.parent
-        arr.parent.execution = None
-        return call, index, False
-    return None, 0, False

jedi/imports.py

@@ -1,339 +0,0 @@
-"""
-:mod:`imports` is here to resolve import statements and return the
-modules/classes/functions/whatever, which they stand for. However there's not
-any actual importing done. This module is about finding modules in the
-filesystem. This can be quite tricky sometimes, because Python imports are not
-always that simple.
-
-This module uses imp for python up to 3.2 and importlib for python 3.3 on; the
-correct implementation is delegated to _compatibility.
-
-This module also supports import autocompletion, which means to complete
-statements like ``from datetim`` (curser at the end would return ``datetime``).
-"""
-from __future__ import with_statement
-
-import os
-import pkgutil
-import sys
-import itertools
-
-from jedi._compatibility import find_module
-from jedi import modules
-from jedi import common
-from jedi import debug
-from jedi import parsing_representation as pr
-from jedi import cache
-import builtin
-import evaluate
-
-# for debugging purposes only
-imports_processed = 0
-
-
-class ModuleNotFound(Exception):
-    pass
-
-
-class ImportPath(pr.Base):
-    """
-    An ImportPath is the path of a `pr.Import` object.
-    """
-    class _GlobalNamespace(object):
-        def __init__(self):
-            self.start_pos = 0, 0
-            self.line_offset = 0
-
-        def get_defined_names(self):
-            return []
-
-        def get_imports(self):
-            return []
-
-        def get_parent_until(self):
-            return None
-
-    GlobalNamespace = _GlobalNamespace()
-
-    def __init__(self, import_stmt, is_like_search=False, kill_count=0,
-                                                    direct_resolve=False):
-        self.import_stmt = import_stmt
-        self.is_like_search = is_like_search
-        self.direct_resolve = direct_resolve
-        self.is_partial_import = bool(max(0, kill_count))
-        path = import_stmt.get_parent_until().path
-        self.file_path = os.path.dirname(path) if path is not None else None
-
-        # rest is import_path resolution
-        self.import_path = []
-        if import_stmt.from_ns:
-            self.import_path += import_stmt.from_ns.names
-        if import_stmt.namespace:
-            if self.is_nested_import() and not direct_resolve:
-                self.import_path.append(import_stmt.namespace.names[0])
-            else:
-                self.import_path += import_stmt.namespace.names
-
-        for i in range(kill_count + int(is_like_search)):
-            self.import_path.pop()
-
-    def __repr__(self):
-        return '<%s: %s>' % (type(self).__name__, self.import_stmt)
-
-    def is_nested_import(self):
-        """
-        This checks for the special case of nested imports, without aliases and
-        from statement::
-
-            import foo.bar
-        """
-        return not self.import_stmt.alias and not self.import_stmt.from_ns \
-                and len(self.import_stmt.namespace.names) > 1 \
-                and not self.direct_resolve
-
-    def get_nested_import(self, parent):
-        """
-        See documentation of `self.is_nested_import`.
-        Generates an Import statement, that can be used to fake nested imports.
-        """
-        i = self.import_stmt
-        # This is not an existing Import statement. Therefore, set position to
-        # 0 (0 is not a valid line number).
-        zero = (0, 0)
-        names = i.namespace.names[1:]
-        n = pr.Name(i._sub_module, names, zero, zero, self.import_stmt)
-        new = pr.Import(i._sub_module, zero, zero, n)
-        new.parent = parent
-        debug.dbg('Generated a nested import: %s' % new)
-        return new
-
-    def get_defined_names(self, on_import_stmt=False):
-        names = []
-        for scope in self.follow():
-            if scope is ImportPath.GlobalNamespace:
-                if self.import_stmt.relative_count == 0:
-                    names += self.get_module_names()
-
-                if self.file_path is not None:
-                    path = os.path.abspath(self.file_path)
-                    for i in range(self.import_stmt.relative_count - 1):
-                        path = os.path.dirname(path)
-                    names += self.get_module_names([path])
-
-                    if self.import_stmt.relative_count:
-                        rel_path = self.get_relative_path() + '/__init__.py'
-                        with common.ignored(IOError):
-                            m = modules.Module(rel_path)
-                            names += m.parser.module.get_defined_names()
-            else:
-                if on_import_stmt and isinstance(scope, pr.Module) \
-                                        and scope.path.endswith('__init__.py'):
-                    pkg_path = os.path.dirname(scope.path)
-                    names += self.get_module_names([pkg_path])
-                for s, scope_names in evaluate.get_names_of_scope(scope,
-                                                    include_builtin=False):
-                    for n in scope_names:
-                        if self.import_stmt.from_ns is None \
-                                            or self.is_partial_import:
-                                # from_ns must be defined to access module
-                                # values plus a partial import means that there
-                                # is something after the import, which
-                                # automatically implies that there must not be
-                                # any non-module scope.
-                                continue
-                        names.append(n)
-        return names
-
-    def get_module_names(self, search_path=None):
-        """
-        Get the names of all modules in the search_path. This means file names
-        and not names defined in the files.
-        """
-        if not search_path:
-            search_path = self.sys_path_with_modifications()
-        names = []
-        for module_loader, name, is_pkg in pkgutil.iter_modules(search_path):
-            inf_pos = (float('inf'), float('inf'))
-            names.append(pr.Name(self.GlobalNamespace, [(name, inf_pos)],
-                                        inf_pos, inf_pos, self.import_stmt))
-        return names
-
-    def sys_path_with_modifications(self):
-        # If you edit e.g. gunicorn, there will be imports like this:
-        # `from gunicorn import something`. But gunicorn is not in the
-        # sys.path. Therefore look if gunicorn is a parent directory, #56.
-        parts = self.file_path.split(os.path.sep)
-        in_path = []
-        if self.import_path:
-            for i, p in enumerate(parts):
-                if p == self.import_path[0]:
-                    new = os.path.sep.join(parts[:i])
-                    in_path.append(new)
-
-        module = self.import_stmt.get_parent_until()
-        return in_path + modules.sys_path_with_modifications(module)
-
-    def follow(self, is_goto=False):
-        """
-        Returns the imported modules.
-        """
-        if evaluate.follow_statement.push_stmt(self.import_stmt):
-            # check recursion
-            return []
-
-        if self.import_path:
-            try:
-                scope, rest = self._follow_file_system()
-            except ModuleNotFound:
-                debug.warning('Module not found: ' + str(self.import_stmt))
-                evaluate.follow_statement.pop_stmt()
-                return []
-
-            scopes = [scope]
-            scopes += remove_star_imports(scope)
-
-            # follow the rest of the import (not FS -> classes, functions)
-            if len(rest) > 1 or rest and self.is_like_search:
-                scopes = []
-            elif rest:
-                if is_goto:
-                    scopes = itertools.chain.from_iterable(
-                                evaluate.find_name(s, rest[0], is_goto=True)
-                                for s in scopes)
-                else:
-                    scopes = itertools.chain.from_iterable(
-                                        evaluate.follow_path(iter(rest), s, s)
-                                        for s in scopes)
-            scopes = list(scopes)
-
-            if self.is_nested_import():
-                scopes.append(self.get_nested_import(scope))
-        else:
-            scopes = [ImportPath.GlobalNamespace]
-        debug.dbg('after import', scopes)
-
-        evaluate.follow_statement.pop_stmt()
-        return scopes
-
-    def get_relative_path(self):
-        path = self.file_path
-        for i in range(self.import_stmt.relative_count - 1):
-            path = os.path.dirname(path)
-        return path
-
-    def _follow_file_system(self):
-        """
-        Find a module with a path (of the module, like usb.backend.libusb10).
-        """
-        def follow_str(ns_path, string):
-            debug.dbg('follow_module', ns_path, string)
-            path = None
-            if ns_path:
-                path = ns_path
-            elif self.import_stmt.relative_count:
-                path = self.get_relative_path()
-
-            global imports_processed
-            imports_processed += 1
-            importing = None
-            if path is not None:
-                importing = find_module(string, [path])
-            else:
-                debug.dbg('search_module', string, self.file_path)
-                # Override the sys.path. It works only good that way.
-                # Injecting the path directly into `find_module` did not work.
-                sys.path, temp = sys_path_mod, sys.path
-                try:
-                    importing = find_module(string)
-                except ImportError:
-                    sys.path = temp
-                    raise
-                sys.path = temp
-
-            return importing
-
-        if self.file_path:
-            sys_path_mod = list(self.sys_path_with_modifications())
-            sys_path_mod.insert(0, self.file_path)
-        else:
-            sys_path_mod = list(modules.get_sys_path())
-
-        def module_not_found():
-            raise ModuleNotFound('The module you searched has not been found')
-
-        current_namespace = (None, None, None)
-        # now execute those paths
-        rest = []
-        for i, s in enumerate(self.import_path):
-            try:
-                current_namespace = follow_str(current_namespace[1], s)
-            except ImportError:
-                if self.import_stmt.relative_count \
-                                and len(self.import_path) == 1:
-                    # follow `from . import some_variable`
-                    rel_path = self.get_relative_path()
-                    with common.ignored(ImportError):
-                        current_namespace = follow_str(rel_path, '__init__')
-                if current_namespace[1]:
-                    rest = self.import_path[i:]
-                else:
-                    module_not_found()
-
-        if current_namespace == (None, None, False):
-            module_not_found()
-
-        sys_path_mod.pop(0)  # TODO why is this here?
-        path = current_namespace[1]
-        is_package_directory = current_namespace[2]
-
-        f = None
-        if is_package_directory or current_namespace[0]:
-            # is a directory module
-            if is_package_directory:
-                path += '/__init__.py'
-                with open(path) as f:
-                    source = f.read()
-            else:
-                source = current_namespace[0].read()
-                current_namespace[0].close()
-            if path.endswith('.py'):
-                f = modules.Module(path, source)
-            else:
-                f = builtin.BuiltinModule(path=path)
-        else:
-            f = builtin.BuiltinModule(name=path)
-
-        return f.parser.module, rest
-
-
-def strip_imports(scopes):
-    """
-    Here we strip the imports - they don't get resolved necessarily.
-    Really used anymore? Merge with remove_star_imports?
-    """
-    result = []
-    for s in scopes:
-        if isinstance(s, pr.Import):
-            result += ImportPath(s).follow()
-        else:
-            result.append(s)
-    return result
-
-
-@cache.cache_star_import
-def remove_star_imports(scope, ignored_modules=()):
-    """
-    Check a module for star imports:
-    >>> from module import *
-
-    and follow these modules.
-    """
-    modules = strip_imports(i for i in scope.get_imports() if i.star)
-    new = []
-    for m in modules:
-        if m not in ignored_modules:
-            new += remove_star_imports(m, modules)
-    modules += new
-
-    # Filter duplicate modules.
-    return set(modules)

jedi/modules.py

@@ -1,388 +0,0 @@
-"""
-Don't confuse these classes with :mod:`parsing_representation` modules, the
-modules here can access these representation with ``module.parser.module``.
-``Module`` exists mainly for caching purposes.
-
-Basically :mod:`modules` offers the classes:
-
-- ``CachedModule``, a base class for Cachedmodule.
-- ``Module`` the class for all normal Python modules (not builtins, they are at
-  home at :mod:`builtin`).
-- ``ModuleWithCursor``, holds the module information for :class:`api.Script`.
-
-Apart from those classes there's a ``sys.path`` fetching function, as well as
-`Virtual Env` and `Django` detection.
-"""
-from __future__ import with_statement
-
-import re
-import tokenizer as tokenize
-import sys
-import os
-from ast import literal_eval
-
-from jedi._compatibility import exec_function, unicode
-from jedi import cache
-from jedi import parsing_representation as pr
-from jedi import fast_parser
-from jedi import debug
-from jedi import common
-
-
-class CachedModule(object):
-    """
-    The base type for all modules, which is not to be confused with
-    `parsing_representation.Module`. Caching happens here.
-    """
-
-    def __init__(self, path=None, name=None):
-        self.path = path and os.path.abspath(path)
-        self.name = name
-        self._parser = None
-
-    @property
-    def parser(self):
-        """ get the parser lazy """
-        if self._parser is None:
-            self._parser = cache.load_module(self.path, self.name) \
-                                or self._load_module()
-        return self._parser
-
-    def _get_source(self):
-        raise NotImplementedError()
-
-    def _load_module(self):
-        source = self._get_source()
-        p = self.path or self.name
-        p = fast_parser.FastParser(source, p)
-        cache.save_module(self.path, self.name, p)
-        return p
-
-
-class Module(CachedModule):
-    """
-    Manages all files, that are parsed and caches them.
-
-    :param path: The module path of the file.
-    :param source: The source code of the file.
-    """
-    def __init__(self, path, source=None):
-        super(Module, self).__init__(path=path)
-        if source is None:
-            with open(path) as f:
-                source = f.read()
-        self.source = source_to_unicode(source)
-        self._line_cache = None
-
-    def _get_source(self):
-        """ Just one time """
-        s = self.source
-        del self.source  # memory efficiency
-        return s
-
-
-class ModuleWithCursor(Module):
-    """
-    Manages all files, that are parsed and caches them.
-    Important are the params source and path, one of them has to
-    be there.
-
-    :param source: The source code of the file.
-    :param path: The module path of the file or None.
-    :param position: The position, the user is currently in. Only important \
-    for the main file.
-    """
-    def __init__(self, path, source, position):
-        super(ModuleWithCursor, self).__init__(path, source)
-        self.position = position
-
-        # this two are only used, because there is no nonlocal in Python 2
-        self._line_temp = None
-        self._relevant_temp = None
-
-        self.source = source
-
-    @property
-    def parser(self):
-        """ get the parser lazy """
-        if not self._parser:
-            with common.ignored(KeyError):
-                parser = cache.parser_cache[self.path].parser
-                cache.invalidate_star_import_cache(parser.module)
-            # Call the parser already here, because it will be used anyways.
-            # Also, the position is here important (which will not be used by
-            # default), therefore fill the cache here.
-            self._parser = fast_parser.FastParser(self.source, self.path,
-                                                        self.position)
-            # don't pickle that module, because it's changing fast
-            cache.save_module(self.path, self.name, self._parser,
-                                                            pickling=False)
-        return self._parser
-
-    def get_path_until_cursor(self):
-        """ Get the path under the cursor. """
-        result = self._get_path_until_cursor()
-        self._start_cursor_pos = self._line_temp + 1, self._column_temp
-        return result
-
-    def _get_path_until_cursor(self, start_pos=None):
-        def fetch_line():
-            line = self.get_line(self._line_temp)
-            if self._is_first:
-                self._is_first = False
-                self._line_length = self._column_temp
-                line = line[:self._column_temp]
-            else:
-                self._line_length = len(line)
-                line = line + '\n'
-            # add lines with a backslash at the end
-            while 1:
-                self._line_temp -= 1
-                last_line = self.get_line(self._line_temp)
-                if last_line and last_line[-1] == '\\':
-                    line = last_line[:-1] + ' ' + line
-                else:
-                    break
-            return line[::-1]
-
-        self._is_first = True
-        if start_pos is None:
-            self._line_temp = self.position[0]
-            self._column_temp = self.position[1]
-        else:
-            self._line_temp, self._column_temp = start_pos
-
-        open_brackets = ['(', '[', '{']
-        close_brackets = [')', ']', '}']
-
-        gen = tokenize.generate_tokens(fetch_line)
-        string = ''
-        level = 0
-        force_point = False
-        last_type = None
-        try:
-            for token_type, tok, start, end, line in gen:
-                #print 'tok', token_type, tok, force_point
-                if last_type == token_type == tokenize.NAME:
-                    string += ' '
-
-                if level > 0:
-                    if tok in close_brackets:
-                        level += 1
-                    if tok in open_brackets:
-                        level -= 1
-                elif tok == '.':
-                    force_point = False
-                elif force_point:
-                    # it is reversed, therefore a number is getting recognized
-                    # as a floating point number
-                    if token_type == tokenize.NUMBER and tok[0] == '.':
-                        force_point = False
-                    else:
-                        break
-                elif tok in close_brackets:
-                    level += 1
-                elif token_type in [tokenize.NAME, tokenize.STRING]:
-                    force_point = True
-                elif token_type == tokenize.NUMBER:
-                    pass
-                else:
-                    break
-
-                self._column_temp = self._line_length - end[1]
-                string += tok
-                last_type = token_type
-        except tokenize.TokenError:
-            debug.warning("Tokenize couldn't finish", sys.exc_info)
-
-        # string can still contain spaces at the end
-        return string[::-1].strip()
-
-    def get_path_under_cursor(self):
-        """
-        Return the path under the cursor. If there is a rest of the path left,
-        it will be added to the stuff before it.
-        """
-        line = self.get_line(self.position[0])
-        after = re.search("[\w\d]*", line[self.position[1]:]).group(0)
-        return self.get_path_until_cursor() + after
-
-    def get_operator_under_cursor(self):
-        line = self.get_line(self.position[0])
-        after = re.match("[^\w\s]+", line[self.position[1]:])
-        before = re.match("[^\w\s]+", line[:self.position[1]][::-1])
-        return (before.group(0) if before is not None else '') \
-                + (after.group(0) if after is not None else '')
-
-    def get_context(self):
-        pos = self._start_cursor_pos
-        while pos > (1, 0):
-            # remove non important white space
-            line = self.get_line(pos[0])
-            while pos[1] > 0 and line[pos[1] - 1].isspace():
-                pos = pos[0], pos[1] - 1
-
-            try:
-                yield self._get_path_until_cursor(start_pos=pos)
-            except StopIteration:
-                yield ''
-            pos = self._line_temp, self._column_temp
-
-        while True:
-            yield ''
-
-    def get_line(self, line_nr):
-        if not self._line_cache:
-            self._line_cache = self.source.splitlines()
-            if not self.source:  # ''.splitlines() == []
-                self._line_cache = [self.source]
-
-        if line_nr == 0:
-            # This is a fix for the zeroth line. We need a newline there, for
-            # the backwards parser.
-            return ''
-        if line_nr < 0:
-            raise StopIteration()
-        try:
-            return self._line_cache[line_nr - 1]
-        except IndexError:
-            raise StopIteration()
-
-
-def get_sys_path():
-    def check_virtual_env(sys_path):
-        """ Add virtualenv's site-packages to the `sys.path`."""
-        venv = os.getenv('VIRTUAL_ENV')
-        if not venv:
-            return
-        venv = os.path.abspath(venv)
-        p = os.path.join(
-            venv, 'lib', 'python%d.%d' % sys.version_info[:2], 'site-packages')
-        sys_path.insert(0, p)
-
-    check_virtual_env(sys.path)
-    return [p for p in sys.path if p != ""]
-
-
-@cache.memoize_default([])
-def sys_path_with_modifications(module):
-    def execute_code(code):
-        c = "import os; from os.path import *; result=%s"
-        variables = {'__file__': module.path}
-        try:
-            exec_function(c % code, variables)
-        except Exception:
-            debug.warning('sys path detected, but failed to evaluate')
-            return None
-        try:
-            res = variables['result']
-            if isinstance(res, str):
-                return os.path.abspath(res)
-            else:
-                return None
-        except KeyError:
-            return None
-
-    def check_module(module):
-        try:
-            possible_stmts = module.used_names['path']
-        except KeyError:
-            return get_sys_path()
-
-        sys_path = list(get_sys_path())  # copy
-        for p in possible_stmts:
-            if not isinstance(p, pr.Statement):
-                continue
-            commands = p.get_commands()
-            if len(commands) != 1:  # sys.path command is just one thing.
-                continue
-            call = commands[0]
-            n = call.name
-            if not isinstance(n, pr.Name) or len(n.names) != 3:
-                continue
-            if n.names[:2] != ('sys', 'path'):
-                continue
-            array_cmd = n.names[2]
-            if call.execution is None:
-                continue
-            exe = call.execution
-            if not (array_cmd == 'insert' and len(exe) == 2
-                    or array_cmd == 'append' and len(exe) == 1):
-                continue
-
-            if array_cmd == 'insert':
-                exe_type, exe.type = exe.type, pr.Array.NOARRAY
-                exe_pop = exe.values.pop(0)
-                res = execute_code(exe.get_code())
-                if res is not None:
-                    sys_path.insert(0, res)
-                    debug.dbg('sys path inserted: %s' % res)
-                exe.type = exe_type
-                exe.values.insert(0, exe_pop)
-            elif array_cmd == 'append':
-                res = execute_code(exe.get_code())
-                if res is not None:
-                    sys_path.append(res)
-                    debug.dbg('sys path added: %s' % res)
-        return sys_path
-
-    if module.path is None:
-        return []  # support for modules without a path is intentionally bad.
-
-    curdir = os.path.abspath(os.curdir)
-    with common.ignored(OSError):
-        os.chdir(os.path.dirname(module.path))
-
-    result = check_module(module)
-    result += detect_django_path(module.path)
-
-    # cleanup, back to old directory
-    os.chdir(curdir)
-    return result
-
-
-def detect_django_path(module_path):
-    """ Detects the path of the very well known Django library (if used) """
-    result = []
-    while True:
-        new = os.path.dirname(module_path)
-        # If the module_path doesn't change anymore, we're finished -> /
-        if new == module_path:
-            break
-        else:
-            module_path = new
-
-        with common.ignored(IOError):
-            with open(module_path + os.path.sep + 'manage.py'):
-                debug.dbg('Found django path: %s' % module_path)
-                result.append(module_path)
-    return result
-
-
-def source_to_unicode(source, encoding=None):
-    def detect_encoding():
-        """ For the implementation of encoding definitions in Python, look at:
-        http://www.python.org/dev/peps/pep-0263/
-        http://docs.python.org/2/reference/lexical_analysis.html#encoding-\
-                                                                declarations
-        """
-        byte_mark = literal_eval(r"b'\xef\xbb\xbf'")
-        if source.startswith(byte_mark):
-            # UTF-8 byte-order mark
-            return 'utf-8'
-
-        first_two_lines = re.match(r'(?:[^\n]*\n){0,2}', str(source)).group(0)
-        possible_encoding = re.search(r"coding[=:]\s*([-\w.]+)",
-                                                            first_two_lines)
-        if possible_encoding:
-            return possible_encoding.group(1)
-        else:
-            # the default if nothing else has been set -> PEP 263
-            return encoding if encoding is not None else 'iso-8859-1'
-
-    if isinstance(source, unicode):
-        # only cast str/bytes
-        return source
-
-    # cast to unicode by default
-    return unicode(source, detect_encoding(), 'replace')

jedi/parser/__init__.py

@@ -0,0 +1,393 @@
+"""
+The ``Parser`` tries to convert the available Python code in an easy to read
+format, something like an abstract syntax tree. The classes who represent this
+tree, are sitting in the :mod:`jedi.parser.tree` module.
+
+The Python module ``tokenize`` is a very important part in the ``Parser``,
+because it splits the code into different words (tokens).  Sometimes it looks a
+bit messy. Sorry for that! You might ask now: "Why didn't you use the ``ast``
+module for this? Well, ``ast`` does a very good job understanding proper Python
+code, but fails to work as soon as there's a single line of broken code.
+
+There's one important optimization that needs to be known: Statements are not
+being parsed completely. ``Statement`` is just a representation of the tokens
+within the statement. This lowers memory usage and cpu time and reduces the
+complexity of the ``Parser`` (there's another parser sitting inside
+``Statement``, which produces ``Array`` and ``Call``).
+"""
+import os
+import re
+
+from jedi.parser import tree as pt
+from jedi.parser import tokenize
+from jedi.parser import token
+from jedi.parser.token import (DEDENT, INDENT, ENDMARKER, NEWLINE, NUMBER,
+                               STRING, OP, ERRORTOKEN)
+from jedi.parser.pgen2.pgen import generate_grammar
+from jedi.parser.pgen2.parse import PgenParser
+
+OPERATOR_KEYWORDS = 'and', 'for', 'if', 'else', 'in', 'is', 'lambda', 'not', 'or'
+# Not used yet. In the future I intend to add something like KeywordStatement
+STATEMENT_KEYWORDS = 'assert', 'del', 'global', 'nonlocal', 'raise', \
+    'return', 'yield', 'pass', 'continue', 'break'
+
+
+_loaded_grammars = {}
+
+
+def load_grammar(file='grammar3.4'):
+    # For now we only support two different Python syntax versions: The latest
+    # Python 3 and Python 2. This may change.
+    if file.startswith('grammar3'):
+        file = 'grammar3.4'
+    else:
+        file = 'grammar2.7'
+
+    global _loaded_grammars
+    path = os.path.join(os.path.dirname(__file__), file) + '.txt'
+    try:
+        return _loaded_grammars[path]
+    except KeyError:
+        return _loaded_grammars.setdefault(path, generate_grammar(path))
+
+
+class ErrorStatement(object):
+    def __init__(self, stack, next_token, position_modifier, next_start_pos):
+        self.stack = stack
+        self._position_modifier = position_modifier
+        self.next_token = next_token
+        self._next_start_pos = next_start_pos
+
+    @property
+    def next_start_pos(self):
+        s = self._next_start_pos
+        return s[0] + self._position_modifier.line, s[1]
+
+    @property
+    def first_pos(self):
+        first_type, nodes = self.stack[0]
+        return nodes[0].start_pos
+
+    @property
+    def first_type(self):
+        first_type, nodes = self.stack[0]
+        return first_type
+
+
+class ParserSyntaxError(object):
+    def __init__(self, message, position):
+        self.message = message
+        self.position = position
+
+
+class Parser(object):
+    """
+    This class is used to parse a Python file, it then divides them into a
+    class structure of different scopes.
+
+    :param grammar: The grammar object of pgen2. Loaded by load_grammar.
+    :param source: The codebase for the parser. Must be unicode.
+    :param module_path: The path of the module in the file system, may be None.
+    :type module_path: str
+    :param top_module: Use this module as a parent instead of `self.module`.
+    """
+    def __init__(self, grammar, source, module_path=None, tokenizer=None):
+        self._ast_mapping = {
+            'expr_stmt': pt.ExprStmt,
+            'classdef': pt.Class,
+            'funcdef': pt.Function,
+            'file_input': pt.Module,
+            'import_name': pt.ImportName,
+            'import_from': pt.ImportFrom,
+            'break_stmt': pt.KeywordStatement,
+            'continue_stmt': pt.KeywordStatement,
+            'return_stmt': pt.ReturnStmt,
+            'raise_stmt': pt.KeywordStatement,
+            'yield_expr': pt.YieldExpr,
+            'del_stmt': pt.KeywordStatement,
+            'pass_stmt': pt.KeywordStatement,
+            'global_stmt': pt.GlobalStmt,
+            'nonlocal_stmt': pt.KeywordStatement,
+            'assert_stmt': pt.AssertStmt,
+            'if_stmt': pt.IfStmt,
+            'with_stmt': pt.WithStmt,
+            'for_stmt': pt.ForStmt,
+            'while_stmt': pt.WhileStmt,
+            'try_stmt': pt.TryStmt,
+            'comp_for': pt.CompFor,
+            'decorator': pt.Decorator,
+            'lambdef': pt.Lambda,
+            'old_lambdef': pt.Lambda,
+            'lambdef_nocond': pt.Lambda,
+        }
+
+        self.syntax_errors = []
+
+        self._global_names = []
+        self._omit_dedent_list = []
+        self._indent_counter = 0
+        self._last_failed_start_pos = (0, 0)
+
+        # TODO do print absolute import detection here.
+        #try:
+        #    del python_grammar_no_print_statement.keywords["print"]
+        #except KeyError:
+        #    pass  # Doesn't exist in the Python 3 grammar.
+
+        #if self.options["print_function"]:
+        #    python_grammar = pygram.python_grammar_no_print_statement
+        #else:
+        self._used_names = {}
+        self._scope_names_stack = [{}]
+        self._error_statement_stacks = []
+
+        added_newline = False
+        # The Python grammar needs a newline at the end of each statement.
+        if not source.endswith('\n'):
+            source += '\n'
+            added_newline = True
+
+        # For the fast parser.
+        self.position_modifier = pt.PositionModifier()
+        p = PgenParser(grammar, self.convert_node, self.convert_leaf,
+                       self.error_recovery)
+        tokenizer = tokenizer or tokenize.source_tokens(source)
+        self.module = p.parse(self._tokenize(tokenizer))
+        if self.module.type != 'file_input':
+            # If there's only one statement, we get back a non-module. That's
+            # not what we want, we want a module, so we add it here:
+            self.module = self.convert_node(grammar,
+                                            grammar.symbol2number['file_input'],
+                                            [self.module])
+
+        if added_newline:
+            self.remove_last_newline()
+        self.module.used_names = self._used_names
+        self.module.path = module_path
+        self.module.global_names = self._global_names
+        self.module.error_statement_stacks = self._error_statement_stacks
+
+    def convert_node(self, grammar, type, children):
+        """
+        Convert raw node information to a Node instance.
+
+        This is passed to the parser driver which calls it whenever a reduction of a
+        grammar rule produces a new complete node, so that the tree is build
+        strictly bottom-up.
+        """
+        symbol = grammar.number2symbol[type]
+        try:
+            new_node = self._ast_mapping[symbol](children)
+        except KeyError:
+            new_node = pt.Node(symbol, children)
+
+        # We need to check raw_node always, because the same node can be
+        # returned by convert multiple times.
+        if symbol == 'global_stmt':
+            self._global_names += new_node.get_global_names()
+        elif isinstance(new_node, pt.Lambda):
+            new_node.names_dict = self._scope_names_stack.pop()
+        elif isinstance(new_node, (pt.ClassOrFunc, pt.Module)) \
+                and symbol in ('funcdef', 'classdef', 'file_input'):
+            # scope_name_stack handling
+            scope_names = self._scope_names_stack.pop()
+            if isinstance(new_node, pt.ClassOrFunc):
+                n = new_node.name
+                scope_names[n.value].remove(n)
+                # Set the func name of the current node
+                arr = self._scope_names_stack[-1].setdefault(n.value, [])
+                arr.append(n)
+            new_node.names_dict = scope_names
+        elif isinstance(new_node, pt.CompFor):
+            # The name definitions of comprehenions shouldn't be part of the
+            # current scope. They are part of the comprehension scope.
+            for n in new_node.get_defined_names():
+                self._scope_names_stack[-1][n.value].remove(n)
+        return new_node
+
+    def convert_leaf(self, grammar, type, value, prefix, start_pos):
+        #print('leaf', value, pytree.type_repr(type))
+        if type == tokenize.NAME:
+            if value in grammar.keywords:
+                if value in ('def', 'class', 'lambda'):
+                    self._scope_names_stack.append({})
+
+                return pt.Keyword(self.position_modifier, value, start_pos, prefix)
+            else:
+                name = pt.Name(self.position_modifier, value, start_pos, prefix)
+                # Keep a listing of all used names
+                arr = self._used_names.setdefault(name.value, [])
+                arr.append(name)
+                arr = self._scope_names_stack[-1].setdefault(name.value, [])
+                arr.append(name)
+                return name
+        elif type == STRING:
+            return pt.String(self.position_modifier, value, start_pos, prefix)
+        elif type == NUMBER:
+            return pt.Number(self.position_modifier, value, start_pos, prefix)
+        elif type in (NEWLINE, ENDMARKER):
+            return pt.Whitespace(self.position_modifier, value, start_pos, prefix)
+        else:
+            return pt.Operator(self.position_modifier, value, start_pos, prefix)
+
+    def error_recovery(self, grammar, stack, typ, value, start_pos, prefix,
+                       add_token_callback):
+        """
+        This parser is written in a dynamic way, meaning that this parser
+        allows using different grammars (even non-Python). However, error
+        recovery is purely written for Python.
+        """
+        def current_suite(stack):
+            # For now just discard everything that is not a suite or
+            # file_input, if we detect an error.
+            for index, (dfa, state, (typ, nodes)) in reversed(list(enumerate(stack))):
+                # `suite` can sometimes be only simple_stmt, not stmt.
+                symbol = grammar.number2symbol[typ]
+                if symbol == 'file_input':
+                    break
+                elif symbol == 'suite' and len(nodes) > 1:
+                    # suites without an indent in them get discarded.
+                    break
+                elif symbol == 'simple_stmt' and len(nodes) > 1:
+                    # simple_stmt can just be turned into a Node, if there are
+                    # enough statements. Ignore the rest after that.
+                    break
+            return index, symbol, nodes
+
+        index, symbol, nodes = current_suite(stack)
+        if symbol == 'simple_stmt':
+            index -= 2
+            (_, _, (typ, suite_nodes)) = stack[index]
+            symbol = grammar.number2symbol[typ]
+            suite_nodes.append(pt.Node(symbol, list(nodes)))
+            # Remove
+            nodes[:] = []
+            nodes = suite_nodes
+            stack[index]
+
+        #print('err', token.tok_name[typ], repr(value), start_pos, len(stack), index)
+        self._stack_removal(grammar, stack, index + 1, value, start_pos)
+        if typ == INDENT:
+            # For every deleted INDENT we have to delete a DEDENT as well.
+            # Otherwise the parser will get into trouble and DEDENT too early.
+            self._omit_dedent_list.append(self._indent_counter)
+
+        if value in ('import', 'from', 'class', 'def', 'try', 'while', 'return'):
+            # Those can always be new statements.
+            add_token_callback(typ, value, prefix, start_pos)
+        elif typ == DEDENT and symbol == 'suite':
+            # Close the current suite, with DEDENT.
+            # Note that this may cause some suites to not contain any
+            # statements at all. This is contrary to valid Python syntax. We
+            # keep incomplete suites in Jedi to be able to complete param names
+            # or `with ... as foo` names. If we want to use this parser for
+            # syntax checks, we have to check in a separate turn if suites
+            # contain statements or not. However, a second check is necessary
+            # anyway (compile.c does that for Python), because Python's grammar
+            # doesn't stop you from defining `continue` in a module, etc.
+            add_token_callback(typ, value, prefix, start_pos)
+
+    def _stack_removal(self, grammar, stack, start_index, value, start_pos):
+        def clear_names(children):
+            for c in children:
+                try:
+                    clear_names(c.children)
+                except AttributeError:
+                    if isinstance(c, pt.Name):
+                        try:
+                            self._scope_names_stack[-1][c.value].remove(c)
+                            self._used_names[c.value].remove(c)
+                        except ValueError:
+                            pass  # This may happen with CompFor.
+
+        for dfa, state, node in stack[start_index:]:
+            clear_names(children=node[1])
+
+        failed_stack = []
+        found = False
+        for dfa, state, (typ, nodes) in stack[start_index:]:
+            if nodes:
+                found = True
+            if found:
+                symbol = grammar.number2symbol[typ]
+                failed_stack.append((symbol, nodes))
+            if nodes and nodes[0] in ('def', 'class', 'lambda'):
+                self._scope_names_stack.pop()
+        if failed_stack:
+            err = ErrorStatement(failed_stack, value, self.position_modifier, start_pos)
+            self._error_statement_stacks.append(err)
+
+        self._last_failed_start_pos = start_pos
+
+        stack[start_index:] = []
+
+    def _tokenize(self, tokenizer):
+        for typ, value, start_pos, prefix in tokenizer:
+            #print(tokenize.tok_name[typ], repr(value), start_pos, repr(prefix))
+            if typ == DEDENT:
+                # We need to count indents, because if we just omit any DEDENT,
+                # we might omit them in the wrong place.
+                o = self._omit_dedent_list
+                if o and o[-1] == self._indent_counter:
+                    o.pop()
+                    continue
+
+                self._indent_counter -= 1
+            elif typ == INDENT:
+                self._indent_counter += 1
+            elif typ == ERRORTOKEN:
+                self._add_syntax_error('Strange token', start_pos)
+                continue
+
+            if typ == OP:
+                typ = token.opmap[value]
+            yield typ, value, prefix, start_pos
+
+    def _add_syntax_error(self, message, position):
+        self.syntax_errors.append(ParserSyntaxError(message, position))
+
+    def __repr__(self):
+        return "<%s: %s>" % (type(self).__name__, self.module)
+
+    def remove_last_newline(self):
+        """
+        In all of this we need to work with _start_pos, because if we worked
+        with start_pos, we would need to check the position_modifier as well
+        (which is accounted for in the start_pos property).
+        """
+        endmarker = self.module.children[-1]
+        # The newline is either in the endmarker as a prefix or the previous
+        # leaf as a newline token.
+        if endmarker.prefix.endswith('\n'):
+            endmarker.prefix = endmarker.prefix[:-1]
+            last_line = re.sub('.*\n', '', endmarker.prefix)
+            endmarker._start_pos = endmarker._start_pos[0] - 1, len(last_line)
+        else:
+            try:
+                newline = endmarker.get_previous()
+            except IndexError:
+                return  # This means that the parser is empty.
+            while True:
+                if newline.value == '':
+                    # Must be a DEDENT, just continue.
+                    try:
+                        newline = newline.get_previous()
+                    except IndexError:
+                        # If there's a statement that fails to be parsed, there
+                        # will be no previous leaf. So just ignore it.
+                        break
+                elif newline.value != '\n':
+                    # This may happen if error correction strikes and removes
+                    # a whole statement including '\n'.
+                    break
+                else:
+                    newline.value = ''
+                    if self._last_failed_start_pos > newline._start_pos:
+                        # It may be the case that there was a syntax error in a
+                        # function. In that case error correction removes the
+                        # right newline. So we use the previously assigned
+                        # _last_failed_start_pos variable to account for that.
+                        endmarker._start_pos = self._last_failed_start_pos
+                    else:
+                        endmarker._start_pos = newline._start_pos
+                    break

jedi/parser/fast.py

@@ -0,0 +1,580 @@
+"""
+Basically a parser that is faster, because it tries to parse only parts and if
+anything changes, it only reparses the changed parts. But because it's not
+finished (and still not working as I want), I won't document it any further.
+"""
+import re
+from itertools import chain
+
+from jedi._compatibility import use_metaclass
+from jedi import settings
+from jedi.parser import Parser
+from jedi.parser import tree
+from jedi import cache
+from jedi import debug
+from jedi.parser.tokenize import (source_tokens, NEWLINE,
+                                  ENDMARKER, INDENT, DEDENT)
+
+FLOWS = 'if', 'else', 'elif', 'while', 'with', 'try', 'except', 'finally', 'for'
+
+
+class FastModule(tree.Module):
+    type = 'file_input'
+
+    def __init__(self, module_path):
+        super(FastModule, self).__init__([])
+        self.modules = []
+        self.reset_caches()
+        self.names_dict = {}
+        self.path = module_path
+
+    def reset_caches(self):
+        self.modules = []
+        try:
+            del self._used_names  # Remove the used names cache.
+        except AttributeError:
+            pass  # It was never used.
+
+    @property
+    @cache.underscore_memoization
+    def used_names(self):
+        return MergedNamesDict([m.used_names for m in self.modules])
+
+    @property
+    def global_names(self):
+        return [name for m in self.modules for name in m.global_names]
+
+    @property
+    def error_statement_stacks(self):
+        return [e for m in self.modules for e in m.error_statement_stacks]
+
+    def __repr__(self):
+        return "<fast.%s: %s@%s-%s>" % (type(self).__name__, self.name,
+                                        self.start_pos[0], self.end_pos[0])
+
+    # To avoid issues with with the `parser.Parser`, we need setters that do
+    # nothing, because if pickle comes along and sets those values.
+    @global_names.setter
+    def global_names(self, value):
+        pass
+
+    @error_statement_stacks.setter
+    def error_statement_stacks(self, value):
+        pass
+
+    @used_names.setter
+    def used_names(self, value):
+        pass
+
+
+class MergedNamesDict(object):
+    def __init__(self, dicts):
+        self.dicts = dicts
+
+    def __iter__(self):
+        return iter(set(key for dct in self.dicts for key in dct))
+
+    def __getitem__(self, value):
+        return list(chain.from_iterable(dct.get(value, []) for dct in self.dicts))
+
+    def items(self):
+        dct = {}
+        for d in self.dicts:
+            for key, values in d.items():
+                try:
+                    dct_values = dct[key]
+                    dct_values += values
+                except KeyError:
+                    dct[key] = list(values)
+        return dct.items()
+
+    def values(self):
+        lst = []
+        for dct in self.dicts:
+            lst += dct.values()
+        return lst
+
+
+class CachedFastParser(type):
+    """ This is a metaclass for caching `FastParser`. """
+    def __call__(self, grammar, source, module_path=None):
+        if not settings.fast_parser:
+            return Parser(grammar, source, module_path)
+
+        pi = cache.parser_cache.get(module_path, None)
+        if pi is None or isinstance(pi.parser, Parser):
+            p = super(CachedFastParser, self).__call__(grammar, source, module_path)
+        else:
+            p = pi.parser  # pi is a `cache.ParserCacheItem`
+            p.update(source)
+        return p
+
+
+class ParserNode(object):
+    def __init__(self, fast_module, parser, source):
+        self._fast_module = fast_module
+        self.parent = None
+        self._node_children = []
+
+        self.source = source
+        self.hash = hash(source)
+        self.parser = parser
+
+        try:
+            # With fast_parser we have either 1 subscope or only statements.
+            self._content_scope = parser.module.subscopes[0]
+        except IndexError:
+            self._content_scope = parser.module
+        else:
+            self._rewrite_last_newline()
+
+        # We need to be able to reset the original children of a parser.
+        self._old_children = list(self._content_scope.children)
+
+    def _rewrite_last_newline(self):
+        """
+        The ENDMARKER can contain a newline in the prefix. However this prefix
+        really belongs to the function - respectively to the next function or
+        parser node. If we don't rewrite that newline, we end up with a newline
+        in the wrong position, i.d. at the end of the file instead of in the
+        middle.
+        """
+        c = self._content_scope.children
+        if tree.is_node(c[-1], 'suite'):  # In a simple_stmt there's no DEDENT.
+            end_marker = self.parser.module.children[-1]
+            # Set the DEDENT prefix instead of the ENDMARKER.
+            c[-1].children[-1].prefix = end_marker.prefix
+            end_marker.prefix = ''
+
+    def __repr__(self):
+        module = self.parser.module
+        try:
+            return '<%s: %s-%s>' % (type(self).__name__, module.start_pos, module.end_pos)
+        except IndexError:
+            # There's no module yet.
+            return '<%s: empty>' % type(self).__name__
+
+    def reset_node(self):
+        """
+        Removes changes that were applied in this class.
+        """
+        self._node_children = []
+        scope = self._content_scope
+        scope.children = list(self._old_children)
+        try:
+            # This works if it's a MergedNamesDict.
+            # We are correcting it, because the MergedNamesDicts are artificial
+            # and can change after closing a node.
+            scope.names_dict = scope.names_dict.dicts[0]
+        except AttributeError:
+            pass
+
+    def close(self):
+        """
+        Closes the current parser node. This means that after this no further
+        nodes should be added anymore.
+        """
+        # We only need to replace the dict if multiple dictionaries are used:
+        if self._node_children:
+            dcts = [n.parser.module.names_dict for n in self._node_children]
+            # Need to insert the own node as well.
+            dcts.insert(0, self._content_scope.names_dict)
+            self._content_scope.names_dict = MergedNamesDict(dcts)
+
+    def parent_until_indent(self, indent=None):
+        if (indent is None or self._indent >= indent) and self.parent is not None:
+            self.close()
+            return self.parent.parent_until_indent(indent)
+        return self
+
+    @property
+    def _indent(self):
+        if not self.parent:
+            return 0
+
+        return self.parser.module.children[0].start_pos[1]
+
+    def add_node(self, node, line_offset):
+        """Adding a node means adding a node that was already added earlier"""
+        # Changing the line offsets is very important, because if they don't
+        # fit, all the start_pos values will be wrong.
+        m = node.parser.module
+        node.parser.position_modifier.line = line_offset
+        self._fast_module.modules.append(m)
+        node.parent = self
+
+        self._node_children.append(node)
+
+        # Insert parser objects into current structure. We only need to set the
+        # parents and children in a good way.
+        scope = self._content_scope
+        for child in m.children:
+            child.parent = scope
+            scope.children.append(child)
+
+        return node
+
+    def all_sub_nodes(self):
+        """
+        Returns all nodes including nested ones.
+        """
+        for n in self._node_children:
+            yield n
+            for y in n.all_sub_nodes():
+                yield y
+
+    @cache.underscore_memoization  # Should only happen once!
+    def remove_last_newline(self):
+        self.parser.remove_last_newline()
+
+
+class FastParser(use_metaclass(CachedFastParser)):
+    _FLOWS_NEED_SPACE = 'if', 'elif', 'while', 'with', 'except', 'for'
+    _FLOWS_NEED_COLON = 'else', 'try', 'except', 'finally'
+    _keyword_re = re.compile('^[ \t]*(def |class |@|(?:%s)|(?:%s)\s*:)'
+                             % ('|'.join(_FLOWS_NEED_SPACE),
+                                '|'.join(_FLOWS_NEED_COLON)))
+
+    def __init__(self, grammar, source, module_path=None):
+        # set values like `tree.Module`.
+        self._grammar = grammar
+        self.module_path = module_path
+        self._reset_caches()
+        self.update(source)
+
+    def _reset_caches(self):
+        self.module = FastModule(self.module_path)
+        self.current_node = ParserNode(self.module, self, '')
+
+    def update(self, source):
+        # For testing purposes: It is important that the number of parsers used
+        # can be minimized. With these variables we can test against that.
+        self.number_parsers_used = 0
+        self.number_of_splits = 0
+        self.number_of_misses = 0
+        self.module.reset_caches()
+        try:
+            self._parse(source)
+        except:
+            # FastParser is cached, be careful with exceptions.
+            self._reset_caches()
+            raise
+
+    def _split_parts(self, source):
+        """
+        Split the source code into different parts. This makes it possible to
+        parse each part seperately and therefore cache parts of the file and
+        not everything.
+        """
+        def gen_part():
+            text = ''.join(current_lines)
+            del current_lines[:]
+            self.number_of_splits += 1
+            return text
+
+        def just_newlines(current_lines):
+            for line in current_lines:
+                line = line.lstrip('\t \n\r')
+                if line and line[0] != '#':
+                    return False
+            return True
+
+        # Split only new lines. Distinction between \r\n is the tokenizer's
+        # job.
+        # It seems like there's no problem with form feed characters here,
+        # because we're not counting lines.
+        self._lines = source.splitlines(True)
+        current_lines = []
+        is_decorator = False
+        # Use -1, because that indent is always smaller than any other.
+        indent_list = [-1, 0]
+        new_indent = False
+        parentheses_level = 0
+        flow_indent = None
+        previous_line = None
+        # All things within flows are simply being ignored.
+        for i, l in enumerate(self._lines):
+            # Handle backslash newline escaping.
+            if l.endswith('\\\n') or l.endswith('\\\r\n'):
+                if previous_line is not None:
+                    previous_line += l
+                else:
+                    previous_line = l
+                continue
+            if previous_line is not None:
+                l = previous_line + l
+                previous_line = None
+
+            # check for dedents
+            s = l.lstrip('\t \n\r')
+            indent = len(l) - len(s)
+            if not s or s[0] == '#':
+                current_lines.append(l)  # Just ignore comments and blank lines
+                continue
+
+            if new_indent:
+                if indent > indent_list[-2]:
+                    # Set the actual indent, not just the random old indent + 1.
+                    indent_list[-1] = indent
+                new_indent = False
+
+            while indent <= indent_list[-2]:  # -> dedent
+                indent_list.pop()
+                # This automatically resets the flow_indent if there was a
+                # dedent or a flow just on one line (with one simple_stmt).
+                new_indent = False
+                if flow_indent is None and current_lines and not parentheses_level:
+                    yield gen_part()
+                flow_indent = None
+
+            # Check lines for functions/classes and split the code there.
+            if flow_indent is None:
+                m = self._keyword_re.match(l)
+                if m:
+                    # Strip whitespace and colon from flows as a check.
+                    if m.group(1).strip(' \t\r\n:') in FLOWS:
+                        if not parentheses_level:
+                            flow_indent = indent
+                    else:
+                        if not is_decorator and not just_newlines(current_lines):
+                            yield gen_part()
+                    is_decorator = '@' == m.group(1)
+                    if not is_decorator:
+                        parentheses_level = 0
+                        # The new indent needs to be higher
+                        indent_list.append(indent + 1)
+                        new_indent = True
+                elif is_decorator:
+                    is_decorator = False
+
+            parentheses_level = \
+                max(0, (l.count('(') + l.count('[') + l.count('{')
+                        - l.count(')') - l.count(']') - l.count('}')))
+
+            current_lines.append(l)
+        if current_lines:
+            yield gen_part()
+
+    def _parse(self, source):
+        """ :type source: str """
+        added_newline = False
+        if not source or source[-1] != '\n':
+            # To be compatible with Pythons grammar, we need a newline at the
+            # end. The parser would handle it, but since the fast parser abuses
+            # the normal parser in various ways, we need to care for this
+            # ourselves.
+            source += '\n'
+            added_newline = True
+
+        next_line_offset = line_offset = 0
+        start = 0
+        nodes = list(self.current_node.all_sub_nodes())
+        # Now we can reset the node, because we have all the old nodes.
+        self.current_node.reset_node()
+        last_end_line = 1
+
+        for code_part in self._split_parts(source):
+            next_line_offset += code_part.count('\n')
+            # If the last code part parsed isn't equal to the current end_pos,
+            # we know that the parser went further (`def` start in a
+            # docstring). So just parse the next part.
+            if line_offset + 1 == last_end_line:
+                self.current_node = self._get_node(code_part, source[start:],
+                                                   line_offset, nodes)
+            else:
+                # Means that some lines where not fully parsed. Parse it now.
+                # This is a very rare case. Should only happens with very
+                # strange code bits.
+                self.number_of_misses += 1
+                while last_end_line < next_line_offset + 1:
+                    line_offset = last_end_line - 1
+                    # We could calculate the src in a more complicated way to
+                    # make caching here possible as well. However, this is
+                    # complicated and error-prone. Since this is not very often
+                    # called - just ignore it.
+                    src = ''.join(self._lines[line_offset:])
+                    self.current_node = self._get_node(code_part, src,
+                                                       line_offset, nodes)
+                    last_end_line = self.current_node.parser.module.end_pos[0]
+
+                debug.dbg('While parsing %s, line %s slowed down the fast parser.',
+                          self.module_path, line_offset + 1)
+
+            line_offset = next_line_offset
+            start += len(code_part)
+
+            last_end_line = self.current_node.parser.module.end_pos[0]
+
+        if added_newline:
+            self.current_node.remove_last_newline()
+
+        # Now that the for loop is finished, we still want to close all nodes.
+        self.current_node = self.current_node.parent_until_indent()
+        self.current_node.close()
+
+        debug.dbg('Parsed %s, with %s parsers in %s splits.'
+                  % (self.module_path, self.number_parsers_used,
+                     self.number_of_splits))
+
+    def _get_node(self, source, parser_code, line_offset, nodes):
+        """
+        Side effect: Alters the list of nodes.
+        """
+        indent = len(source) - len(source.lstrip('\t '))
+        self.current_node = self.current_node.parent_until_indent(indent)
+
+        h = hash(source)
+        for index, node in enumerate(nodes):
+            if node.hash == h and node.source == source:
+                node.reset_node()
+                nodes.remove(node)
+                break
+        else:
+            tokenizer = FastTokenizer(parser_code)
+            self.number_parsers_used += 1
+            p = Parser(self._grammar, parser_code, self.module_path, tokenizer=tokenizer)
+
+            end = line_offset + p.module.end_pos[0]
+            used_lines = self._lines[line_offset:end - 1]
+            code_part_actually_used = ''.join(used_lines)
+
+            node = ParserNode(self.module, p, code_part_actually_used)
+
+        self.current_node.add_node(node, line_offset)
+        return node
+
+
+class FastTokenizer(object):
+    """
+    Breaks when certain conditions are met, i.e. a new function or class opens.
+    """
+    def __init__(self, source):
+        self.source = source
+        self._gen = source_tokens(source)
+        self._closed = False
+
+        # fast parser options
+        self.current = self.previous = NEWLINE, '', (0, 0)
+        self._in_flow = False
+        self._is_decorator = False
+        self._first_stmt = True
+        self._parentheses_level = 0
+        self._indent_counter = 0
+        self._flow_indent_counter = 0
+        self._returned_endmarker = False
+        self._expect_indent = False
+
+    def __iter__(self):
+        return self
+
+    def next(self):
+        """ Python 2 Compatibility """
+        return self.__next__()
+
+    def __next__(self):
+        if self._closed:
+            return self._finish_dedents()
+
+        typ, value, start_pos, prefix = current = next(self._gen)
+        if typ == ENDMARKER:
+            self._closed = True
+            self._returned_endmarker = True
+            return current
+
+        self.previous = self.current
+        self.current = current
+
+        if typ == INDENT:
+            self._indent_counter += 1
+            if not self._expect_indent and not self._first_stmt and not self._in_flow:
+                # This does not mean that there is an actual flow, it means
+                # that the INDENT is syntactically wrong.
+                self._flow_indent_counter = self._indent_counter - 1
+                self._in_flow = True
+            self._expect_indent = False
+        elif typ == DEDENT:
+            self._indent_counter -= 1
+            if self._in_flow:
+                if self._indent_counter == self._flow_indent_counter:
+                    self._in_flow = False
+            else:
+                self._closed = True
+            return current
+
+        if value in ('def', 'class') and self._parentheses_level \
+                and re.search(r'\n[ \t]*\Z', prefix):
+            # Account for the fact that an open parentheses before a function
+            # will reset the parentheses counter, but new lines before will
+            # still be ignored. So check the prefix.
+
+            # TODO what about flow parentheses counter resets in the tokenizer?
+            self._parentheses_level = 0
+            return self._close()
+
+        # Parentheses ignore the indentation rules. The other three stand for
+        # new lines.
+        if self.previous[0] in (NEWLINE, INDENT, DEDENT) \
+                and not self._parentheses_level and typ not in (INDENT, DEDENT):
+            if not self._in_flow:
+                if value in FLOWS:
+                    self._flow_indent_counter = self._indent_counter
+                    self._first_stmt = False
+                elif value in ('def', 'class', '@'):
+                    # The values here are exactly the same check as in
+                    # _split_parts, but this time with tokenize and therefore
+                    # precise.
+                    if not self._first_stmt and not self._is_decorator:
+                        return self._close()
+
+                    self._is_decorator = '@' == value
+                    if not self._is_decorator:
+                        self._first_stmt = False
+                        self._expect_indent = True
+                elif self._expect_indent:
+                    return self._close()
+                else:
+                    self._first_stmt = False
+
+        if value in '([{' and value:
+            self._parentheses_level += 1
+        elif value in ')]}' and value:
+            # Ignore closing parentheses, because they are all
+            # irrelevant for the indentation.
+            self._parentheses_level = max(self._parentheses_level - 1, 0)
+        return current
+
+    def _close(self):
+        if self._first_stmt:
+            # Continue like nothing has happened, because we want to enter
+            # the first class/function.
+            if self.current[1] != '@':
+                self._first_stmt = False
+            return self.current
+        else:
+            self._closed = True
+            return self._finish_dedents()
+
+    def _finish_dedents(self):
+        if self._indent_counter:
+            self._indent_counter -= 1
+            return DEDENT, '', self.current[2], ''
+        elif not self._returned_endmarker:
+            self._returned_endmarker = True
+            return ENDMARKER, '', self.current[2], self._get_prefix()
+        else:
+            raise StopIteration
+
+    def _get_prefix(self):
+        """
+        We're using the current prefix for the endmarker to not loose any
+        information. However we care about "lost" lines. The prefix of the
+        current line (indent) will always be included in the current line.
+        """
+        cur = self.current
+        while cur[0] == DEDENT:
+            cur = next(self._gen)
+        prefix = cur[3]
+
+        # \Z for the end of the string. $ is bugged, because it has the
+        # same behavior with or without re.MULTILINE.
+        return re.sub(r'[^\n]+\Z', '', prefix)

jedi/parser/grammar2.7.txt

@@ -0,0 +1,152 @@
+# Grammar for 2to3. This grammar supports Python 2.x and 3.x.
+
+# Note:  Changing the grammar specified in this file will most likely
+#        require corresponding changes in the parser module
+#        (../Modules/parsermodule.c).  If you can't make the changes to
+#        that module yourself, please co-ordinate the required changes
+#        with someone who can; ask around on python-dev for help.  Fred
+#        Drake <fdrake@acm.org> will probably be listening there.
+
+# NOTE WELL: You should also follow all the steps listed in PEP 306,
+# "How to Change Python's Grammar"
+
+
+# Start symbols for the grammar:
+#	    file_input is a module or sequence of commands read from an input file;
+#	    single_input is a single interactive statement;
+#	    eval_input is the input for the eval() and input() functions.
+# NB: compound_stmt in single_input is followed by extra NEWLINE!
+file_input: (NEWLINE | stmt)* ENDMARKER
+single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE
+eval_input: testlist NEWLINE* ENDMARKER
+
+decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
+decorators: decorator+
+decorated: decorators (classdef | funcdef)
+funcdef: 'def' NAME parameters ['->' test] ':' suite
+parameters: '(' [typedargslist] ')'
+typedargslist: ((tfpdef ['=' test] ',')*
+                ('*' [tname] (',' tname ['=' test])* [',' '**' tname] | '**' tname)
+                | tfpdef ['=' test] (',' tfpdef ['=' test])* [','])
+tname: NAME [':' test]
+tfpdef: tname | '(' tfplist ')'
+tfplist: tfpdef (',' tfpdef)* [',']
+varargslist: ((vfpdef ['=' test] ',')*
+              ('*' [vname] (',' vname ['=' test])*  [',' '**' vname] | '**' vname)
+              | vfpdef ['=' test] (',' vfpdef ['=' test])* [','])
+vname: NAME
+vfpdef: vname | '(' vfplist ')'
+vfplist: vfpdef (',' vfpdef)* [',']
+
+stmt: simple_stmt | compound_stmt
+simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
+small_stmt: (expr_stmt | print_stmt  | del_stmt | pass_stmt | flow_stmt |
+             import_stmt | global_stmt | exec_stmt | assert_stmt)
+expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
+                     ('=' (yield_expr|testlist_star_expr))*)
+testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
+augassign: ('+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' |
+            '<<=' | '>>=' | '**=' | '//=')
+# For normal assignments, additional restrictions enforced by the interpreter
+print_stmt: 'print' ( [ test (',' test)* [','] ] |
+                      '>>' test [ (',' test)+ [','] ] )
+del_stmt: 'del' exprlist
+pass_stmt: 'pass'
+flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt
+break_stmt: 'break'
+continue_stmt: 'continue'
+return_stmt: 'return' [testlist]
+yield_stmt: yield_expr
+raise_stmt: 'raise' [test ['from' test | ',' test [',' test]]]
+import_stmt: import_name | import_from
+import_name: 'import' dotted_as_names
+# note below: the ('.' | '...') is necessary because '...' is tokenized as ELLIPSIS
+import_from: ('from' (('.' | '...')* dotted_name | ('.' | '...')+)
+              'import' ('*' | '(' import_as_names ')' | import_as_names))
+import_as_name: NAME ['as' NAME]
+dotted_as_name: dotted_name ['as' NAME]
+import_as_names: import_as_name (',' import_as_name)* [',']
+dotted_as_names: dotted_as_name (',' dotted_as_name)*
+dotted_name: NAME ('.' NAME)*
+global_stmt: ('global' | 'nonlocal') NAME (',' NAME)*
+exec_stmt: 'exec' expr ['in' test [',' test]]
+assert_stmt: 'assert' test [',' test]
+
+compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
+if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
+while_stmt: 'while' test ':' suite ['else' ':' suite]
+for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
+try_stmt: ('try' ':' suite
+           ((except_clause ':' suite)+
+	        ['else' ':' suite]
+	        ['finally' ':' suite] |
+	       'finally' ':' suite))
+with_stmt: 'with' with_item (',' with_item)*  ':' suite
+with_item: test ['as' expr]
+with_var: 'as' expr
+# NB compile.c makes sure that the default except clause is last
+except_clause: 'except' [test [(',' | 'as') test]]
+# Edit by David Halter: The stmt is now optional. This reflects how Jedi allows
+# classes and functions to be empty, which is beneficial for autocompletion.
+suite: simple_stmt | NEWLINE INDENT stmt* DEDENT
+
+# Backward compatibility cruft to support:
+# [ x for x in lambda: True, lambda: False if x() ]
+# even while also allowing:
+# lambda x: 5 if x else 2
+# (But not a mix of the two)
+testlist_safe: old_test [(',' old_test)+ [',']]
+old_test: or_test | old_lambdef
+old_lambdef: 'lambda' [varargslist] ':' old_test
+
+test: or_test ['if' or_test 'else' test] | lambdef
+or_test: and_test ('or' and_test)*
+and_test: not_test ('and' not_test)*
+not_test: 'not' not_test | comparison
+comparison: expr (comp_op expr)*
+comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not'
+star_expr: '*' expr
+expr: xor_expr ('|' xor_expr)*
+xor_expr: and_expr ('^' and_expr)*
+and_expr: shift_expr ('&' shift_expr)*
+shift_expr: arith_expr (('<<'|'>>') arith_expr)*
+arith_expr: term (('+'|'-') term)*
+term: factor (('*'|'/'|'%'|'//') factor)*
+factor: ('+'|'-'|'~') factor | power
+power: atom trailer* ['**' factor]
+atom: ('(' [yield_expr|testlist_comp] ')' |
+       '[' [testlist_comp] ']' |
+       '{' [dictorsetmaker] '}' |
+       '`' testlist1 '`' |
+       NAME | NUMBER | STRING+ | '.' '.' '.')
+# Modification by David Halter, remove `testlist_gexp` and `listmaker`
+testlist_comp: (test|star_expr) ( comp_for | (',' (test|star_expr))* [','] )
+lambdef: 'lambda' [varargslist] ':' test
+trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
+subscriptlist: subscript (',' subscript)* [',']
+subscript: test | [test] ':' [test] [sliceop]
+sliceop: ':' [test]
+exprlist: (expr|star_expr) (',' (expr|star_expr))* [',']
+testlist: test (',' test)* [',']
+# Modification by David Halter, dictsetmaker -> dictorsetmaker (so that it's
+# the same as in the 3.4 grammar).
+dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [','])) |
+                (test (comp_for | (',' test)* [','])) )
+
+classdef: 'class' NAME ['(' [arglist] ')'] ':' suite
+
+arglist: (argument ',')* (argument [',']
+                         |'*' test (',' argument)* [',' '**' test] 
+                         |'**' test)
+argument: test [comp_for] | test '=' test  # Really [keyword '='] test
+
+comp_iter: comp_for | comp_if
+comp_for: 'for' exprlist 'in' testlist_safe [comp_iter]
+comp_if: 'if' old_test [comp_iter]
+
+testlist1: test (',' test)*
+
+# not used in grammar, but may appear in "node" passed from Parser to Compiler
+encoding_decl: NAME
+
+yield_expr: 'yield' [testlist]

jedi/parser/grammar3.4.txt

@@ -0,0 +1,135 @@
+# Grammar for Python
+
+# Note:  Changing the grammar specified in this file will most likely
+#        require corresponding changes in the parser module
+#        (../Modules/parsermodule.c).  If you can't make the changes to
+#        that module yourself, please co-ordinate the required changes
+#        with someone who can; ask around on python-dev for help.  Fred
+#        Drake <fdrake@acm.org> will probably be listening there.
+
+# NOTE WELL: You should also follow all the steps listed in PEP 306,
+# "How to Change Python's Grammar"
+
+# Start symbols for the grammar:
+#       single_input is a single interactive statement;
+#       file_input is a module or sequence of commands read from an input file;
+#       eval_input is the input for the eval() functions.
+# NB: compound_stmt in single_input is followed by extra NEWLINE!
+file_input: (NEWLINE | stmt)* ENDMARKER
+single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE
+eval_input: testlist NEWLINE* ENDMARKER
+
+decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
+decorators: decorator+
+decorated: decorators (classdef | funcdef)
+funcdef: 'def' NAME parameters ['->' test] ':' suite
+parameters: '(' [typedargslist] ')'
+typedargslist: (tfpdef ['=' test] (',' tfpdef ['=' test])* [','
+       ['*' [tfpdef] (',' tfpdef ['=' test])* [',' '**' tfpdef] | '**' tfpdef]]
+     |  '*' [tfpdef] (',' tfpdef ['=' test])* [',' '**' tfpdef] | '**' tfpdef)
+tfpdef: NAME [':' test]
+varargslist: (vfpdef ['=' test] (',' vfpdef ['=' test])* [','
+       ['*' [vfpdef] (',' vfpdef ['=' test])* [',' '**' vfpdef] | '**' vfpdef]]
+     |  '*' [vfpdef] (',' vfpdef ['=' test])* [',' '**' vfpdef] | '**' vfpdef)
+vfpdef: NAME
+
+stmt: simple_stmt | compound_stmt
+simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
+small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
+             import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
+expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
+                     ('=' (yield_expr|testlist_star_expr))*)
+testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
+augassign: ('+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' |
+            '<<=' | '>>=' | '**=' | '//=')
+# For normal assignments, additional restrictions enforced by the interpreter
+del_stmt: 'del' exprlist
+pass_stmt: 'pass'
+flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt
+break_stmt: 'break'
+continue_stmt: 'continue'
+return_stmt: 'return' [testlist]
+yield_stmt: yield_expr
+raise_stmt: 'raise' [test ['from' test]]
+import_stmt: import_name | import_from
+import_name: 'import' dotted_as_names
+# note below: the ('.' | '...') is necessary because '...' is tokenized as ELLIPSIS
+import_from: ('from' (('.' | '...')* dotted_name | ('.' | '...')+)
+              'import' ('*' | '(' import_as_names ')' | import_as_names))
+import_as_name: NAME ['as' NAME]
+dotted_as_name: dotted_name ['as' NAME]
+import_as_names: import_as_name (',' import_as_name)* [',']
+dotted_as_names: dotted_as_name (',' dotted_as_name)*
+dotted_name: NAME ('.' NAME)*
+global_stmt: 'global' NAME (',' NAME)*
+nonlocal_stmt: 'nonlocal' NAME (',' NAME)*
+assert_stmt: 'assert' test [',' test]
+
+compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
+if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
+while_stmt: 'while' test ':' suite ['else' ':' suite]
+for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
+try_stmt: ('try' ':' suite
+           ((except_clause ':' suite)+
+            ['else' ':' suite]
+            ['finally' ':' suite] |
+           'finally' ':' suite))
+with_stmt: 'with' with_item (',' with_item)*  ':' suite
+with_item: test ['as' expr]
+# NB compile.c makes sure that the default except clause is last
+except_clause: 'except' [test ['as' NAME]]
+# Edit by David Halter: The stmt is now optional. This reflects how Jedi allows
+# classes and functions to be empty, which is beneficial for autocompletion.
+suite: simple_stmt | NEWLINE INDENT stmt* DEDENT
+
+test: or_test ['if' or_test 'else' test] | lambdef
+test_nocond: or_test | lambdef_nocond
+lambdef: 'lambda' [varargslist] ':' test
+lambdef_nocond: 'lambda' [varargslist] ':' test_nocond
+or_test: and_test ('or' and_test)*
+and_test: not_test ('and' not_test)*
+not_test: 'not' not_test | comparison
+comparison: expr (comp_op expr)*
+# <> isn't actually a valid comparison operator in Python. It's here for the
+# sake of a __future__ import described in PEP 401
+comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not'
+star_expr: '*' expr
+expr: xor_expr ('|' xor_expr)*
+xor_expr: and_expr ('^' and_expr)*
+and_expr: shift_expr ('&' shift_expr)*
+shift_expr: arith_expr (('<<'|'>>') arith_expr)*
+arith_expr: term (('+'|'-') term)*
+term: factor (('*'|'/'|'%'|'//') factor)*
+factor: ('+'|'-'|'~') factor | power
+power: atom trailer* ['**' factor]
+atom: ('(' [yield_expr|testlist_comp] ')' |
+       '[' [testlist_comp] ']' |
+       '{' [dictorsetmaker] '}' |
+       NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')
+testlist_comp: (test|star_expr) ( comp_for | (',' (test|star_expr))* [','] )
+trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' NAME
+subscriptlist: subscript (',' subscript)* [',']
+subscript: test | [test] ':' [test] [sliceop]
+sliceop: ':' [test]
+exprlist: (expr|star_expr) (',' (expr|star_expr))* [',']
+testlist: test (',' test)* [',']
+dictorsetmaker: ( (test ':' test (comp_for | (',' test ':' test)* [','])) |
+                  (test (comp_for | (',' test)* [','])) )
+
+classdef: 'class' NAME ['(' [arglist] ')'] ':' suite
+
+arglist: (argument ',')* (argument [',']
+                         |'*' test (',' argument)* [',' '**' test] 
+                         |'**' test)
+# The reason that keywords are test nodes instead of NAME is that using NAME
+# results in an ambiguity. ast.c makes sure it's a NAME.
+argument: test [comp_for] | test '=' test  # Really [keyword '='] test
+comp_iter: comp_for | comp_if
+comp_for: 'for' exprlist 'in' or_test [comp_iter]
+comp_if: 'if' test_nocond [comp_iter]
+
+# not used in grammar, but may appear in "node" passed from Parser to Compiler
+encoding_decl: NAME
+
+yield_expr: 'yield' [yield_arg]
+yield_arg: 'from' test | testlist

jedi/parser/pgen2/__init__.py

@@ -0,0 +1,8 @@
+# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved.
+# Licensed to PSF under a Contributor Agreement.
+
+# Modifications:
+# Copyright 2006 Google, Inc. All Rights Reserved.
+# Licensed to PSF under a Contributor Agreement.
+# Copyright 2014 David Halter. Integration into Jedi.
+# Modifications are dual-licensed: MIT and PSF.

jedi/parser/pgen2/grammar.py

@@ -0,0 +1,125 @@
+# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved.
+# Licensed to PSF under a Contributor Agreement.
+
+# Modifications:
+# Copyright 2014 David Halter. Integration into Jedi.
+# Modifications are dual-licensed: MIT and PSF.
+
+"""This module defines the data structures used to represent a grammar.
+
+These are a bit arcane because they are derived from the data
+structures used by Python's 'pgen' parser generator.
+
+There's also a table here mapping operators to their names in the
+token module; the Python tokenize module reports all operators as the
+fallback token code OP, but the parser needs the actual token code.
+
+"""
+
+# Python imports
+import pickle
+
+
+class Grammar(object):
+    """Pgen parsing tables conversion class.
+
+    Once initialized, this class supplies the grammar tables for the
+    parsing engine implemented by parse.py.  The parsing engine
+    accesses the instance variables directly.  The class here does not
+    provide initialization of the tables; several subclasses exist to
+    do this (see the conv and pgen modules).
+
+    The load() method reads the tables from a pickle file, which is
+    much faster than the other ways offered by subclasses.  The pickle
+    file is written by calling dump() (after loading the grammar
+    tables using a subclass).  The report() method prints a readable
+    representation of the tables to stdout, for debugging.
+
+    The instance variables are as follows:
+
+    symbol2number -- a dict mapping symbol names to numbers.  Symbol
+                     numbers are always 256 or higher, to distinguish
+                     them from token numbers, which are between 0 and
+                     255 (inclusive).
+
+    number2symbol -- a dict mapping numbers to symbol names;
+                     these two are each other's inverse.
+
+    states        -- a list of DFAs, where each DFA is a list of
+                     states, each state is a list of arcs, and each
+                     arc is a (i, j) pair where i is a label and j is
+                     a state number.  The DFA number is the index into
+                     this list.  (This name is slightly confusing.)
+                     Final states are represented by a special arc of
+                     the form (0, j) where j is its own state number.
+
+    dfas          -- a dict mapping symbol numbers to (DFA, first)
+                     pairs, where DFA is an item from the states list
+                     above, and first is a set of tokens that can
+                     begin this grammar rule (represented by a dict
+                     whose values are always 1).
+
+    labels        -- a list of (x, y) pairs where x is either a token
+                     number or a symbol number, and y is either None
+                     or a string; the strings are keywords.  The label
+                     number is the index in this list; label numbers
+                     are used to mark state transitions (arcs) in the
+                     DFAs.
+
+    start         -- the number of the grammar's start symbol.
+
+    keywords      -- a dict mapping keyword strings to arc labels.
+
+    tokens        -- a dict mapping token numbers to arc labels.
+
+    """
+
+    def __init__(self):
+        self.symbol2number = {}
+        self.number2symbol = {}
+        self.states = []
+        self.dfas = {}
+        self.labels = [(0, "EMPTY")]
+        self.keywords = {}
+        self.tokens = {}
+        self.symbol2label = {}
+        self.start = 256
+
+    def dump(self, filename):
+        """Dump the grammar tables to a pickle file."""
+        with open(filename, "wb") as f:
+            pickle.dump(self.__dict__, f, 2)
+
+    def load(self, filename):
+        """Load the grammar tables from a pickle file."""
+        with open(filename, "rb") as f:
+            d = pickle.load(f)
+        self.__dict__.update(d)
+
+    def copy(self):
+        """
+        Copy the grammar.
+        """
+        new = self.__class__()
+        for dict_attr in ("symbol2number", "number2symbol", "dfas", "keywords",
+                          "tokens", "symbol2label"):
+            setattr(new, dict_attr, getattr(self, dict_attr).copy())
+        new.labels = self.labels[:]
+        new.states = self.states[:]
+        new.start = self.start
+        return new
+
+    def report(self):
+        """Dump the grammar tables to standard output, for debugging."""
+        from pprint import pprint
+        print("s2n")
+        pprint(self.symbol2number)
+        print("n2s")
+        pprint(self.number2symbol)
+        print("states")
+        pprint(self.states)
+        print("dfas")
+        pprint(self.dfas)
+        print("labels")
+        pprint(self.labels)
+        print("start", self.start)

jedi/parser/pgen2/parse.py

@@ -0,0 +1,205 @@
+# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved.
+# Licensed to PSF under a Contributor Agreement.
+
+# Modifications:
+# Copyright 2014 David Halter. Integration into Jedi.
+# Modifications are dual-licensed: MIT and PSF.
+
+"""
+Parser engine for the grammar tables generated by pgen.
+
+The grammar table must be loaded first.
+
+See Parser/parser.c in the Python distribution for additional info on
+how this parsing engine works.
+"""
+
+# Local imports
+from jedi.parser import tokenize
+
+
+class ParseError(Exception):
+    """Exception to signal the parser is stuck."""
+
+    def __init__(self, msg, type, value, start_pos):
+        Exception.__init__(self, "%s: type=%r, value=%r, start_pos=%r" %
+                           (msg, tokenize.tok_name[type], value, start_pos))
+        self.msg = msg
+        self.type = type
+        self.value = value
+        self.start_pos = start_pos
+
+
+class PgenParser(object):
+    """Parser engine.
+
+    The proper usage sequence is:
+
+    p = Parser(grammar, [converter])  # create instance
+    p.setup([start])                  # prepare for parsing
+    <for each input token>:
+        if p.addtoken(...):           # parse a token; may raise ParseError
+            break
+    root = p.rootnode                 # root of abstract syntax tree
+
+    A Parser instance may be reused by calling setup() repeatedly.
+
+    A Parser instance contains state pertaining to the current token
+    sequence, and should not be used concurrently by different threads
+    to parse separate token sequences.
+
+    See driver.py for how to get input tokens by tokenizing a file or
+    string.
+
+    Parsing is complete when addtoken() returns True; the root of the
+    abstract syntax tree can then be retrieved from the rootnode
+    instance variable.  When a syntax error occurs, addtoken() raises
+    the ParseError exception.  There is no error recovery; the parser
+    cannot be used after a syntax error was reported (but it can be
+    reinitialized by calling setup()).
+
+    """
+
+    def __init__(self, grammar, convert_node, convert_leaf, error_recovery):
+        """Constructor.
+
+        The grammar argument is a grammar.Grammar instance; see the
+        grammar module for more information.
+
+        The parser is not ready yet for parsing; you must call the
+        setup() method to get it started.
+
+        The optional convert argument is a function mapping concrete
+        syntax tree nodes to abstract syntax tree nodes.  If not
+        given, no conversion is done and the syntax tree produced is
+        the concrete syntax tree.  If given, it must be a function of
+        two arguments, the first being the grammar (a grammar.Grammar
+        instance), and the second being the concrete syntax tree node
+        to be converted.  The syntax tree is converted from the bottom
+        up.
+
+        A concrete syntax tree node is a (type, nodes) tuple, where
+        type is the node type (a token or symbol number) and nodes
+        is a list of children for symbols, and None for tokens.
+
+        An abstract syntax tree node may be anything; this is entirely
+        up to the converter function.
+
+        """
+        self.grammar = grammar
+        self.convert_node = convert_node
+        self.convert_leaf = convert_leaf
+
+        # Prepare for parsing.
+        start = self.grammar.start
+        # Each stack entry is a tuple: (dfa, state, node).
+        # A node is a tuple: (type, children),
+        # where children is a list of nodes or None
+        newnode = (start, [])
+        stackentry = (self.grammar.dfas[start], 0, newnode)
+        self.stack = [stackentry]
+        self.rootnode = None
+        self.error_recovery = error_recovery
+
+    def parse(self, tokenizer):
+        for type, value, prefix, start_pos in tokenizer:
+            if self.addtoken(type, value, prefix, start_pos):
+                break
+        else:
+            # We never broke out -- EOF is too soon -- Unfinished statement.
+            self.error_recovery(self.grammar, self.stack, type, value,
+                                start_pos, prefix, self.addtoken)
+            # Add the ENDMARKER again.
+            if not self.addtoken(type, value, prefix, start_pos):
+                raise ParseError("incomplete input", type, value, start_pos)
+        return self.rootnode
+
+    def addtoken(self, type, value, prefix, start_pos):
+        """Add a token; return True if this is the end of the program."""
+        # Map from token to label
+        if type == tokenize.NAME:
+            # Check for reserved words (keywords)
+            try:
+                ilabel = self.grammar.keywords[value]
+            except KeyError:
+                ilabel = self.grammar.tokens[type]
+        else:
+            ilabel = self.grammar.tokens[type]
+
+        # Loop until the token is shifted; may raise exceptions
+        while True:
+            dfa, state, node = self.stack[-1]
+            states, first = dfa
+            arcs = states[state]
+            # Look for a state with this label
+            for i, newstate in arcs:
+                t, v = self.grammar.labels[i]
+                if ilabel == i:
+                    # Look it up in the list of labels
+                    assert t < 256
+                    # Shift a token; we're done with it
+                    self.shift(type, value, newstate, prefix, start_pos)
+                    # Pop while we are in an accept-only state
+                    state = newstate
+                    while states[state] == [(0, state)]:
+                        self.pop()
+                        if not self.stack:
+                            # Done parsing!
+                            return True
+                        dfa, state, node = self.stack[-1]
+                        states, first = dfa
+                    # Done with this token
+                    return False
+                elif t >= 256:
+                    # See if it's a symbol and if we're in its first set
+                    itsdfa = self.grammar.dfas[t]
+                    itsstates, itsfirst = itsdfa
+                    if ilabel in itsfirst:
+                        # Push a symbol
+                        self.push(t, itsdfa, newstate)
+                        break  # To continue the outer while loop
+            else:
+                if (0, state) in arcs:
+                    # An accepting state, pop it and try something else
+                    self.pop()
+                    if not self.stack:
+                        # Done parsing, but another token is input
+                        raise ParseError("too much input", type, value, start_pos)
+                else:
+                    self.error_recovery(self.grammar, self.stack, type,
+                                        value, start_pos, prefix, self.addtoken)
+                    break
+
+    def shift(self, type, value, newstate, prefix, start_pos):
+        """Shift a token.  (Internal)"""
+        dfa, state, node = self.stack[-1]
+        newnode = self.convert_leaf(self.grammar, type, value, prefix, start_pos)
+        node[-1].append(newnode)
+        self.stack[-1] = (dfa, newstate, node)
+
+    def push(self, type, newdfa, newstate):
+        """Push a nonterminal.  (Internal)"""
+        dfa, state, node = self.stack[-1]
+        newnode = (type, [])
+        self.stack[-1] = (dfa, newstate, node)
+        self.stack.append((newdfa, 0, newnode))
+
+    def pop(self):
+        """Pop a nonterminal.  (Internal)"""
+        popdfa, popstate, (type, children) = self.stack.pop()
+        # If there's exactly one child, return that child instead of creating a
+        # new node.  We still create expr_stmt and file_input though, because a
+        # lot of Jedi depends on its logic.
+        if len(children) == 1:
+            newnode = children[0]
+        else:
+            newnode = self.convert_node(self.grammar, type, children)
+
+        try:
+            # Equal to:
+            # dfa, state, node = self.stack[-1]
+            # symbol, children = node
+            self.stack[-1][2][1].append(newnode)
+        except IndexError:
+            # Stack is empty, set the rootnode.
+            self.rootnode = newnode

jedi/parser/pgen2/pgen.py

@@ -0,0 +1,394 @@
+# Copyright 2004-2005 Elemental Security, Inc. All Rights Reserved.
+# Licensed to PSF under a Contributor Agreement.
+
+# Modifications:
+# Copyright 2014 David Halter. Integration into Jedi.
+# Modifications are dual-licensed: MIT and PSF.
+
+# Pgen imports
+from . import grammar
+from jedi.parser import token
+from jedi.parser import tokenize
+
+
+class ParserGenerator(object):
+    def __init__(self, filename, stream=None):
+        close_stream = None
+        if stream is None:
+            stream = open(filename)
+            close_stream = stream.close
+        self.filename = filename
+        self.stream = stream
+        self.generator = tokenize.generate_tokens(stream.readline)
+        self.gettoken()  # Initialize lookahead
+        self.dfas, self.startsymbol = self.parse()
+        if close_stream is not None:
+            close_stream()
+        self.first = {}  # map from symbol name to set of tokens
+        self.addfirstsets()
+
+    def make_grammar(self):
+        c = grammar.Grammar()
+        names = list(self.dfas.keys())
+        names.sort()
+        names.remove(self.startsymbol)
+        names.insert(0, self.startsymbol)
+        for name in names:
+            i = 256 + len(c.symbol2number)
+            c.symbol2number[name] = i
+            c.number2symbol[i] = name
+        for name in names:
+            dfa = self.dfas[name]
+            states = []
+            for state in dfa:
+                arcs = []
+                for label, next in state.arcs.items():
+                    arcs.append((self.make_label(c, label), dfa.index(next)))
+                if state.isfinal:
+                    arcs.append((0, dfa.index(state)))
+                states.append(arcs)
+            c.states.append(states)
+            c.dfas[c.symbol2number[name]] = (states, self.make_first(c, name))
+        c.start = c.symbol2number[self.startsymbol]
+        return c
+
+    def make_first(self, c, name):
+        rawfirst = self.first[name]
+        first = {}
+        for label in rawfirst:
+            ilabel = self.make_label(c, label)
+            ##assert ilabel not in first # XXX failed on <> ... !=
+            first[ilabel] = 1
+        return first
+
+    def make_label(self, c, label):
+        # XXX Maybe this should be a method on a subclass of converter?
+        ilabel = len(c.labels)
+        if label[0].isalpha():
+            # Either a symbol name or a named token
+            if label in c.symbol2number:
+                # A symbol name (a non-terminal)
+                if label in c.symbol2label:
+                    return c.symbol2label[label]
+                else:
+                    c.labels.append((c.symbol2number[label], None))
+                    c.symbol2label[label] = ilabel
+                    return ilabel
+            else:
+                # A named token (NAME, NUMBER, STRING)
+                itoken = getattr(token, label, None)
+                assert isinstance(itoken, int), label
+                assert itoken in token.tok_name, label
+                if itoken in c.tokens:
+                    return c.tokens[itoken]
+                else:
+                    c.labels.append((itoken, None))
+                    c.tokens[itoken] = ilabel
+                    return ilabel
+        else:
+            # Either a keyword or an operator
+            assert label[0] in ('"', "'"), label
+            value = eval(label)
+            if value[0].isalpha():
+                # A keyword
+                if value in c.keywords:
+                    return c.keywords[value]
+                else:
+                    c.labels.append((token.NAME, value))
+                    c.keywords[value] = ilabel
+                    return ilabel
+            else:
+                # An operator (any non-numeric token)
+                itoken = token.opmap[value]  # Fails if unknown token
+                if itoken in c.tokens:
+                    return c.tokens[itoken]
+                else:
+                    c.labels.append((itoken, None))
+                    c.tokens[itoken] = ilabel
+                    return ilabel
+
+    def addfirstsets(self):
+        names = list(self.dfas.keys())
+        names.sort()
+        for name in names:
+            if name not in self.first:
+                self.calcfirst(name)
+            #print name, self.first[name].keys()
+
+    def calcfirst(self, name):
+        dfa = self.dfas[name]
+        self.first[name] = None  # dummy to detect left recursion
+        state = dfa[0]
+        totalset = {}
+        overlapcheck = {}
+        for label, next in state.arcs.items():
+            if label in self.dfas:
+                if label in self.first:
+                    fset = self.first[label]
+                    if fset is None:
+                        raise ValueError("recursion for rule %r" % name)
+                else:
+                    self.calcfirst(label)
+                    fset = self.first[label]
+                totalset.update(fset)
+                overlapcheck[label] = fset
+            else:
+                totalset[label] = 1
+                overlapcheck[label] = {label: 1}
+        inverse = {}
+        for label, itsfirst in overlapcheck.items():
+            for symbol in itsfirst:
+                if symbol in inverse:
+                    raise ValueError("rule %s is ambiguous; %s is in the"
+                                     " first sets of %s as well as %s" %
+                                     (name, symbol, label, inverse[symbol]))
+                inverse[symbol] = label
+        self.first[name] = totalset
+
+    def parse(self):
+        dfas = {}
+        startsymbol = None
+        # MSTART: (NEWLINE | RULE)* ENDMARKER
+        while self.type != token.ENDMARKER:
+            while self.type == token.NEWLINE:
+                self.gettoken()
+            # RULE: NAME ':' RHS NEWLINE
+            name = self.expect(token.NAME)
+            self.expect(token.OP, ":")
+            a, z = self.parse_rhs()
+            self.expect(token.NEWLINE)
+            #self.dump_nfa(name, a, z)
+            dfa = self.make_dfa(a, z)
+            #self.dump_dfa(name, dfa)
+            # oldlen = len(dfa)
+            self.simplify_dfa(dfa)
+            # newlen = len(dfa)
+            dfas[name] = dfa
+            #print name, oldlen, newlen
+            if startsymbol is None:
+                startsymbol = name
+        return dfas, startsymbol
+
+    def make_dfa(self, start, finish):
+        # To turn an NFA into a DFA, we define the states of the DFA
+        # to correspond to *sets* of states of the NFA.  Then do some
+        # state reduction.  Let's represent sets as dicts with 1 for
+        # values.
+        assert isinstance(start, NFAState)
+        assert isinstance(finish, NFAState)
+
+        def closure(state):
+            base = {}
+            addclosure(state, base)
+            return base
+
+        def addclosure(state, base):
+            assert isinstance(state, NFAState)
+            if state in base:
+                return
+            base[state] = 1
+            for label, next in state.arcs:
+                if label is None:
+                    addclosure(next, base)
+
+        states = [DFAState(closure(start), finish)]
+        for state in states:  # NB states grows while we're iterating
+            arcs = {}
+            for nfastate in state.nfaset:
+                for label, next in nfastate.arcs:
+                    if label is not None:
+                        addclosure(next, arcs.setdefault(label, {}))
+            for label, nfaset in arcs.items():
+                for st in states:
+                    if st.nfaset == nfaset:
+                        break
+                else:
+                    st = DFAState(nfaset, finish)
+                    states.append(st)
+                state.addarc(st, label)
+        return states  # List of DFAState instances; first one is start
+
+    def dump_nfa(self, name, start, finish):
+        print("Dump of NFA for", name)
+        todo = [start]
+        for i, state in enumerate(todo):
+            print("  State", i, state is finish and "(final)" or "")
+            for label, next in state.arcs:
+                if next in todo:
+                    j = todo.index(next)
+                else:
+                    j = len(todo)
+                    todo.append(next)
+                if label is None:
+                    print("    -> %d" % j)
+                else:
+                    print("    %s -> %d" % (label, j))
+
+    def dump_dfa(self, name, dfa):
+        print("Dump of DFA for", name)
+        for i, state in enumerate(dfa):
+            print("  State", i, state.isfinal and "(final)" or "")
+            for label, next in state.arcs.items():
+                print("    %s -> %d" % (label, dfa.index(next)))
+
+    def simplify_dfa(self, dfa):
+        # This is not theoretically optimal, but works well enough.
+        # Algorithm: repeatedly look for two states that have the same
+        # set of arcs (same labels pointing to the same nodes) and
+        # unify them, until things stop changing.
+
+        # dfa is a list of DFAState instances
+        changes = True
+        while changes:
+            changes = False
+            for i, state_i in enumerate(dfa):
+                for j in range(i + 1, len(dfa)):
+                    state_j = dfa[j]
+                    if state_i == state_j:
+                        #print "  unify", i, j
+                        del dfa[j]
+                        for state in dfa:
+                            state.unifystate(state_j, state_i)
+                        changes = True
+                        break
+
+    def parse_rhs(self):
+        # RHS: ALT ('|' ALT)*
+        a, z = self.parse_alt()
+        if self.value != "|":
+            return a, z
+        else:
+            aa = NFAState()
+            zz = NFAState()
+            aa.addarc(a)
+            z.addarc(zz)
+            while self.value == "|":
+                self.gettoken()
+                a, z = self.parse_alt()
+                aa.addarc(a)
+                z.addarc(zz)
+            return aa, zz
+
+    def parse_alt(self):
+        # ALT: ITEM+
+        a, b = self.parse_item()
+        while (self.value in ("(", "[") or
+               self.type in (token.NAME, token.STRING)):
+            c, d = self.parse_item()
+            b.addarc(c)
+            b = d
+        return a, b
+
+    def parse_item(self):
+        # ITEM: '[' RHS ']' | ATOM ['+' | '*']
+        if self.value == "[":
+            self.gettoken()
+            a, z = self.parse_rhs()
+            self.expect(token.OP, "]")
+            a.addarc(z)
+            return a, z
+        else:
+            a, z = self.parse_atom()
+            value = self.value
+            if value not in ("+", "*"):
+                return a, z
+            self.gettoken()
+            z.addarc(a)
+            if value == "+":
+                return a, z
+            else:
+                return a, a
+
+    def parse_atom(self):
+        # ATOM: '(' RHS ')' | NAME | STRING
+        if self.value == "(":
+            self.gettoken()
+            a, z = self.parse_rhs()
+            self.expect(token.OP, ")")
+            return a, z
+        elif self.type in (token.NAME, token.STRING):
+            a = NFAState()
+            z = NFAState()
+            a.addarc(z, self.value)
+            self.gettoken()
+            return a, z
+        else:
+            self.raise_error("expected (...) or NAME or STRING, got %s/%s",
+                             self.type, self.value)
+
+    def expect(self, type, value=None):
+        if self.type != type or (value is not None and self.value != value):
+            self.raise_error("expected %s/%s, got %s/%s",
+                             type, value, self.type, self.value)
+        value = self.value
+        self.gettoken()
+        return value
+
+    def gettoken(self):
+        tup = next(self.generator)
+        while tup[0] in (token.COMMENT, token.NL):
+            tup = next(self.generator)
+        self.type, self.value, self.begin, prefix = tup
+        #print tokenize.tok_name[self.type], repr(self.value)
+
+    def raise_error(self, msg, *args):
+        if args:
+            try:
+                msg = msg % args
+            except:
+                msg = " ".join([msg] + list(map(str, args)))
+        line = open(self.filename).readlines()[self.begin[0]]
+        raise SyntaxError(msg, (self.filename, self.begin[0],
+                                self.begin[1], line))
+
+
+class NFAState(object):
+    def __init__(self):
+        self.arcs = []  # list of (label, NFAState) pairs
+
+    def addarc(self, next, label=None):
+        assert label is None or isinstance(label, str)
+        assert isinstance(next, NFAState)
+        self.arcs.append((label, next))
+
+
+class DFAState(object):
+    def __init__(self, nfaset, final):
+        assert isinstance(nfaset, dict)
+        assert isinstance(next(iter(nfaset)), NFAState)
+        assert isinstance(final, NFAState)
+        self.nfaset = nfaset
+        self.isfinal = final in nfaset
+        self.arcs = {}  # map from label to DFAState
+
+    def addarc(self, next, label):
+        assert isinstance(label, str)
+        assert label not in self.arcs
+        assert isinstance(next, DFAState)
+        self.arcs[label] = next
+
+    def unifystate(self, old, new):
+        for label, next in self.arcs.items():
+            if next is old:
+                self.arcs[label] = new
+
+    def __eq__(self, other):
+        # Equality test -- ignore the nfaset instance variable
+        assert isinstance(other, DFAState)
+        if self.isfinal != other.isfinal:
+            return False
+        # Can't just return self.arcs == other.arcs, because that
+        # would invoke this method recursively, with cycles...
+        if len(self.arcs) != len(other.arcs):
+            return False
+        for label, next in self.arcs.items():
+            if next is not other.arcs.get(label):
+                return False
+        return True
+
+    __hash__ = None  # For Py3 compatibility.
+
+
+def generate_grammar(filename="Grammar.txt"):
+    p = ParserGenerator(filename)
+    return p.make_grammar()

jedi/parser/token.py

@@ -0,0 +1,84 @@
+from __future__ import absolute_import
+
+from jedi._compatibility import is_py3
+from token import *
+
+
+COMMENT = N_TOKENS
+tok_name[COMMENT] = 'COMMENT'
+N_TOKENS += 1
+
+NL = N_TOKENS
+tok_name[NL] = 'NL'
+N_TOKENS += 1
+
+if is_py3:
+    BACKQUOTE = N_TOKENS
+    tok_name[BACKQUOTE] = 'BACKQUOTE'
+    N_TOKENS += 1
+else:
+    RARROW = N_TOKENS
+    tok_name[RARROW] = 'RARROW'
+    N_TOKENS += 1
+    ELLIPSIS = N_TOKENS
+    tok_name[ELLIPSIS] = 'ELLIPSIS'
+    N_TOKENS += 1
+
+
+
+# Map from operator to number (since tokenize doesn't do this)
+
+opmap_raw = """\
+( LPAR
+) RPAR
+[ LSQB
+] RSQB
+: COLON
+, COMMA
+; SEMI
++ PLUS
+- MINUS
+* STAR
+/ SLASH
+| VBAR
+& AMPER
+< LESS
+> GREATER
+= EQUAL
+. DOT
+% PERCENT
+` BACKQUOTE
+{ LBRACE
+} RBRACE
+@ AT
+== EQEQUAL
+!= NOTEQUAL
+<> NOTEQUAL
+<= LESSEQUAL
+>= GREATEREQUAL
+~ TILDE
+^ CIRCUMFLEX
+<< LEFTSHIFT
+>> RIGHTSHIFT
+** DOUBLESTAR
++= PLUSEQUAL
+-= MINEQUAL
+*= STAREQUAL
+/= SLASHEQUAL
+%= PERCENTEQUAL
+&= AMPEREQUAL
+|= VBAREQUAL
+^= CIRCUMFLEXEQUAL
+<<= LEFTSHIFTEQUAL
+>>= RIGHTSHIFTEQUAL
+**= DOUBLESTAREQUAL
+// DOUBLESLASH
+//= DOUBLESLASHEQUAL
+-> RARROW
+... ELLIPSIS
+"""
+
+opmap = {}
+for line in opmap_raw.splitlines():
+    op, name = line.split()
+    opmap[op] = globals()[name]

jedi/parser/tokenize.py

@@ -0,0 +1,290 @@
+# -*- coding: utf-8 -*-
+"""
+This tokenizer has been copied from the ``tokenize.py`` standard library
+tokenizer. The reason was simple: The standard library tokenizer fails
+if the indentation is not right. The fast parser of jedi however requires
+"wrong" indentation.
+
+Basically this is a stripped down version of the standard library module, so
+you can read the documentation there. Additionally we included some speed and
+memory optimizations here.
+"""
+from __future__ import absolute_import
+
+import string
+import re
+from io import StringIO
+from jedi.parser.token import (tok_name, N_TOKENS, ENDMARKER, STRING, NUMBER,
+                               NAME, OP, ERRORTOKEN, NEWLINE, INDENT, DEDENT)
+from jedi._compatibility import is_py3
+
+
+cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
+
+
+if is_py3:
+    # Python 3 has str.isidentifier() to check if a char is a valid identifier
+    is_identifier = str.isidentifier
+else:
+    namechars = string.ascii_letters + '_'
+    is_identifier = lambda s: s in namechars
+
+
+COMMENT = N_TOKENS
+tok_name[COMMENT] = 'COMMENT'
+
+
+def group(*choices):
+    return '(' + '|'.join(choices) + ')'
+
+
+def maybe(*choices):
+    return group(*choices) + '?'
+
+
+# Note: we use unicode matching for names ("\w") but ascii matching for
+# number literals.
+whitespace = r'[ \f\t]*'
+comment = r'#[^\r\n]*'
+name = r'\w+'
+
+hex_number = r'0[xX][0-9a-fA-F]+'
+bin_number = r'0[bB][01]+'
+oct_number = r'0[oO][0-7]+'
+dec_number = r'(?:0+|[1-9][0-9]*)'
+int_number = group(hex_number, bin_number, oct_number, dec_number)
+exponent = r'[eE][-+]?[0-9]+'
+point_float = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+') + maybe(exponent)
+Expfloat = r'[0-9]+' + exponent
+float_number = group(point_float, Expfloat)
+imag_number = group(r'[0-9]+[jJ]', float_number + r'[jJ]')
+number = group(imag_number, float_number, int_number)
+
+# Tail end of ' string.
+single = r"[^'\\]*(?:\\.[^'\\]*)*'"
+# Tail end of " string.
+double = r'[^"\\]*(?:\\.[^"\\]*)*"'
+# Tail end of ''' string.
+single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''"
+# Tail end of """ string.
+double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""'
+triple = group("[uUbB]?[rR]?'''", '[uUbB]?[rR]?"""')
+# Single-line ' or " string.
+
+# Because of leftmost-then-longest match semantics, be sure to put the
+# longest operators first (e.g., if = came before ==, == would get
+# recognized as two instances of =).
+operator = group(r"\*\*=?", r">>=?", r"<<=?", r"!=",
+                 r"//=?", r"->",
+                 r"[+\-*/%&|^=<>]=?",
+                 r"~")
+
+bracket = '[][(){}]'
+special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
+funny = group(operator, bracket, special)
+
+# First (or only) line of ' or " string.
+cont_str = group(r"[bBuU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
+                 group("'", r'\\\r?\n'),
+                 r'[bBuU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
+                 group('"', r'\\\r?\n'))
+pseudo_extras = group(r'\\\r?\n', comment, triple)
+pseudo_token = group(whitespace) + \
+    group(pseudo_extras, number, funny, cont_str, name)
+
+
+def _compile(expr):
+    return re.compile(expr, re.UNICODE)
+
+
+pseudoprog, single3prog, double3prog = map(
+    _compile, (pseudo_token, single3, double3))
+
+endprogs = {"'": _compile(single), '"': _compile(double),
+            "'''": single3prog, '"""': double3prog,
+            "r'''": single3prog, 'r"""': double3prog,
+            "b'''": single3prog, 'b"""': double3prog,
+            "u'''": single3prog, 'u"""': double3prog,
+            "R'''": single3prog, 'R"""': double3prog,
+            "B'''": single3prog, 'B"""': double3prog,
+            "U'''": single3prog, 'U"""': double3prog,
+            "br'''": single3prog, 'br"""': double3prog,
+            "bR'''": single3prog, 'bR"""': double3prog,
+            "Br'''": single3prog, 'Br"""': double3prog,
+            "BR'''": single3prog, 'BR"""': double3prog,
+            "ur'''": single3prog, 'ur"""': double3prog,
+            "uR'''": single3prog, 'uR"""': double3prog,
+            "Ur'''": single3prog, 'Ur"""': double3prog,
+            "UR'''": single3prog, 'UR"""': double3prog,
+            'r': None, 'R': None, 'b': None, 'B': None}
+
+triple_quoted = {}
+for t in ("'''", '"""',
+          "r'''", 'r"""', "R'''", 'R"""',
+          "b'''", 'b"""', "B'''", 'B"""',
+          "u'''", 'u"""', "U'''", 'U"""',
+          "br'''", 'br"""', "Br'''", 'Br"""',
+          "bR'''", 'bR"""', "BR'''", 'BR"""',
+          "ur'''", 'ur"""', "Ur'''", 'Ur"""',
+          "uR'''", 'uR"""', "UR'''", 'UR"""'):
+    triple_quoted[t] = t
+single_quoted = {}
+for t in ("'", '"',
+          "r'", 'r"', "R'", 'R"',
+          "b'", 'b"', "B'", 'B"',
+          "u'", 'u"', "U'", 'U"',
+          "br'", 'br"', "Br'", 'Br"',
+          "bR'", 'bR"', "BR'", 'BR"',
+          "ur'", 'ur"', "Ur'", 'Ur"',
+          "uR'", 'uR"', "UR'", 'UR"'):
+    single_quoted[t] = t
+
+del _compile
+
+tabsize = 8
+
+ALWAYS_BREAK_TOKENS = (';', 'import', 'from', 'class', 'def', 'try', 'except',
+                       'finally', 'while', 'return')
+
+
+def source_tokens(source):
+    """Generate tokens from a the source code (string)."""
+    source = source + '\n'  # end with \n, because the parser needs it
+    readline = StringIO(source).readline
+    return generate_tokens(readline)
+
+
+def generate_tokens(readline):
+    """
+    A heavily modified Python standard library tokenizer.
+
+    Additionally to the default information, yields also the prefix of each
+    token. This idea comes from lib2to3. The prefix contains all information
+    that is irrelevant for the parser like newlines in parentheses or comments.
+    """
+    paren_level = 0  # count parentheses
+    indents = [0]
+    lnum = 0
+    numchars = '0123456789'
+    contstr = ''
+    contline = None
+    # We start with a newline. This makes indent at the first position
+    # possible. It's not valid Python, but still better than an INDENT in the
+    # second line (and not in the first). This makes quite a few things in
+    # Jedi's fast parser possible.
+    new_line = True
+    prefix = ''  # Should never be required, but here for safety
+    additional_prefix = ''
+    while True:            # loop over lines in stream
+        line = readline()  # readline returns empty when finished. See StringIO
+        if not line:
+            if contstr:
+                yield ERRORTOKEN, contstr, contstr_start, prefix
+            break
+
+        lnum += 1
+        pos, max = 0, len(line)
+
+        if contstr:                                         # continued string
+            endmatch = endprog.match(line)
+            if endmatch:
+                pos = endmatch.end(0)
+                yield STRING, contstr + line[:pos], contstr_start, prefix
+                contstr = ''
+                contline = None
+            else:
+                contstr = contstr + line
+                contline = contline + line
+                continue
+
+        while pos < max:
+            pseudomatch = pseudoprog.match(line, pos)
+            if not pseudomatch:                             # scan for tokens
+                txt = line[pos]
+                if line[pos] in '"\'':
+                    # If a literal starts but doesn't end the whole rest of the
+                    # line is an error token.
+                    txt = line[pos:]
+                yield ERRORTOKEN, txt, (lnum, pos), prefix
+                pos += 1
+                continue
+
+            prefix = additional_prefix + pseudomatch.group(1)
+            additional_prefix = ''
+            start, pos = pseudomatch.span(2)
+            spos = (lnum, start)
+            token, initial = line[start:pos], line[start]
+
+            if new_line and initial not in '\r\n#':
+                new_line = False
+                if paren_level == 0:
+                    if start > indents[-1]:
+                        yield INDENT, '', spos, ''
+                        indents.append(start)
+                    while start < indents[-1]:
+                        yield DEDENT, '', spos, ''
+                        indents.pop()
+
+            if (initial in numchars or                      # ordinary number
+                    (initial == '.' and token != '.' and token != '...')):
+                yield NUMBER, token, spos, prefix
+            elif initial in '\r\n':
+                if not new_line and paren_level == 0:
+                    yield NEWLINE, token, spos, prefix
+                else:
+                    additional_prefix = prefix + token
+                new_line = True
+            elif initial == '#':  # Comments
+                assert not token.endswith("\n")
+                additional_prefix = prefix + token
+            elif token in triple_quoted:
+                endprog = endprogs[token]
+                endmatch = endprog.match(line, pos)
+                if endmatch:                                # all on one line
+                    pos = endmatch.end(0)
+                    token = line[start:pos]
+                    yield STRING, token, spos, prefix
+                else:
+                    contstr_start = (lnum, start)           # multiple lines
+                    contstr = line[start:]
+                    contline = line
+                    break
+            elif initial in single_quoted or \
+                    token[:2] in single_quoted or \
+                    token[:3] in single_quoted:
+                if token[-1] == '\n':                       # continued string
+                    contstr_start = lnum, start
+                    endprog = (endprogs.get(initial) or endprogs.get(token[1])
+                               or endprogs.get(token[2]))
+                    contstr = line[start:]
+                    contline = line
+                    break
+                else:                                       # ordinary string
+                    yield STRING, token, spos, prefix
+            elif is_identifier(initial):                      # ordinary name
+                if token in ALWAYS_BREAK_TOKENS:
+                    paren_level = 0
+                    while True:
+                        indent = indents.pop()
+                        if indent > start:
+                            yield DEDENT, '', spos, ''
+                        else:
+                            indents.append(indent)
+                            break
+                yield NAME, token, spos, prefix
+            elif initial == '\\' and line[start:] in ('\\\n', '\\\r\n'):  # continued stmt
+                additional_prefix += prefix + line[start:]
+                break
+            else:
+                if token in '([{':
+                    paren_level += 1
+                elif token in ')]}':
+                    paren_level -= 1
+                yield OP, token, spos, prefix
+
+    end_pos = (lnum, max - 1)
+    # As the last position we just take the maximally possible position. We
+    # remove -1 for the last new line.
+    for indent in indents[1:]:
+        yield DEDENT, '', end_pos, ''
+    yield ENDMARKER, '', end_pos, prefix

jedi/parser/user_context.py

@@ -0,0 +1,339 @@
+import re
+import os
+import keyword
+
+from jedi import cache
+from jedi import common
+from jedi.parser import tokenize, Parser
+from jedi._compatibility import u
+from jedi.parser.fast import FastParser
+from jedi.parser import tree
+from jedi import debug
+from jedi.common import PushBackIterator
+
+
+REPLACE_STR = r"[bBuU]?[rR]?" + (r"(?:(')[^\n'\\]*(?:\\.[^\n'\\]*)*(?:'|$)" +
+                                 '|' +
+                                 r'(")[^\n"\\]*(?:\\.[^\n"\\]*)*(?:"|$))')
+REPLACE_STR = re.compile(REPLACE_STR)
+
+
+class UserContext(object):
+    """
+    :param source: The source code of the file.
+    :param position: The position, the user is currently in. Only important \
+    for the main file.
+    """
+    def __init__(self, source, position):
+        self.source = source
+        self.position = position
+        self._line_cache = None
+
+        self._relevant_temp = None
+
+    @cache.underscore_memoization
+    def get_path_until_cursor(self):
+        """ Get the path under the cursor. """
+        path, self._start_cursor_pos = self._calc_path_until_cursor(self.position)
+        return path
+
+    def _backwards_line_generator(self, start_pos):
+        self._line_temp, self._column_temp = start_pos
+        first_line = self.get_line(start_pos[0])[:self._column_temp]
+
+        self._line_length = self._column_temp
+        yield first_line[::-1] + '\n'
+
+        while True:
+            self._line_temp -= 1
+            line = self.get_line(self._line_temp)
+            self._line_length = len(line)
+            yield line[::-1] + '\n'
+
+    def _get_backwards_tokenizer(self, start_pos, line_gen=None):
+        if line_gen is None:
+            line_gen = self._backwards_line_generator(start_pos)
+        token_gen = tokenize.generate_tokens(lambda: next(line_gen))
+        for typ, tok_str, tok_start_pos, prefix in token_gen:
+            line = self.get_line(self._line_temp)
+            # Calculate the real start_pos of the token.
+            if tok_start_pos[0] == 1:
+                # We are in the first checked line
+                column = start_pos[1] - tok_start_pos[1]
+            else:
+                column = len(line) - tok_start_pos[1]
+            # Multi-line docstrings must be accounted for.
+            first_line = common.splitlines(tok_str)[0]
+            column -= len(first_line)
+            # Reverse the token again, so that it is in normal order again.
+            yield typ, tok_str[::-1], (self._line_temp, column), prefix[::-1]
+
+    def _calc_path_until_cursor(self, start_pos):
+        """
+        Something like a reverse tokenizer that tokenizes the reversed strings.
+        """
+        open_brackets = ['(', '[', '{']
+        close_brackets = [')', ']', '}']
+
+        start_cursor = start_pos
+        gen = PushBackIterator(self._get_backwards_tokenizer(start_pos))
+        string = u('')
+        level = 0
+        force_point = False
+        last_type = None
+        is_first = True
+        for tok_type, tok_str, tok_start_pos, prefix in gen:
+            if is_first:
+                if prefix:  # whitespace is not a path
+                    return u(''), start_cursor
+                is_first = False
+
+            if last_type == tok_type == tokenize.NAME:
+                string = ' ' + string
+
+            if level:
+                if tok_str in close_brackets:
+                    level += 1
+                elif tok_str in open_brackets:
+                    level -= 1
+            elif tok_str == '.':
+                force_point = False
+            elif force_point:
+                # Reversed tokenizing, therefore a number is recognized as a
+                # floating point number.
+                # The same is true for string prefixes -> represented as a
+                # combination of string and name.
+                if tok_type == tokenize.NUMBER and tok_str[-1] == '.' \
+                        or tok_type == tokenize.NAME and last_type == tokenize.STRING \
+                        and tok_str.lower() in ('b', 'u', 'r', 'br', 'ur'):
+                    force_point = False
+                else:
+                    break
+            elif tok_str in close_brackets:
+                level += 1
+            elif tok_type in [tokenize.NAME, tokenize.STRING]:
+                if keyword.iskeyword(tok_str) and string:
+                    # If there's already something in the string, a keyword
+                    # never adds any meaning to the current statement.
+                    break
+                force_point = True
+            elif tok_type == tokenize.NUMBER:
+                pass
+            else:
+                if tok_str == '-':
+                    next_tok = next(gen)
+                    if next_tok[1] == 'e':
+                        gen.push_back(next_tok)
+                    else:
+                        break
+                else:
+                    break
+
+            start_cursor = tok_start_pos
+            string = tok_str + prefix + string
+            last_type = tok_type
+
+        # Don't need whitespace around a statement.
+        return string.strip(), start_cursor
+
+    def get_path_under_cursor(self):
+        """
+        Return the path under the cursor. If there is a rest of the path left,
+        it will be added to the stuff before it.
+        """
+        return self.get_path_until_cursor() + self.get_path_after_cursor()
+
+    def get_path_after_cursor(self):
+        line = self.get_line(self.position[0])
+        return re.search("[\w\d]*", line[self.position[1]:]).group(0)
+
+    def get_operator_under_cursor(self):
+        line = self.get_line(self.position[0])
+        after = re.match("[^\w\s]+", line[self.position[1]:])
+        before = re.match("[^\w\s]+", line[:self.position[1]][::-1])
+        return (before.group(0) if before is not None else '') \
+            + (after.group(0) if after is not None else '')
+
+    def call_signature(self):
+        """
+        :return: Tuple of string of the call and the index of the cursor.
+        """
+        def get_line(pos):
+            def simplify_str(match):
+                """
+                To avoid having strings without end marks (error tokens) and
+                strings that just screw up all the call signatures, just
+                simplify everything.
+                """
+                mark = match.group(1) or match.group(2)
+                return mark + ' ' * (len(match.group(0)) - 2) + mark
+
+            line_gen = self._backwards_line_generator(pos)
+            for line in line_gen:
+                # We have to switch the already backwards lines twice, because
+                # we scan them from start.
+                line = line[::-1]
+                modified = re.sub(REPLACE_STR, simplify_str, line)
+                yield modified[::-1]
+
+        index = 0
+        level = 0
+        next_must_be_name = False
+        next_is_key = False
+        key_name = None
+        generator = self._get_backwards_tokenizer(self.position, get_line(self.position))
+        for tok_type, tok_str, start_pos, prefix in generator:
+            if tok_str in tokenize.ALWAYS_BREAK_TOKENS:
+                break
+            elif next_must_be_name:
+                if tok_type == tokenize.NUMBER:
+                    # If there's a number at the end of the string, it will be
+                    # tokenized as a number. So add it to the name.
+                    tok_type, t, _, _ = next(generator)
+                if tok_type == tokenize.NAME:
+                    end_pos = start_pos[0], start_pos[1] + len(tok_str)
+                    call, start_pos = self._calc_path_until_cursor(start_pos=end_pos)
+                    return call, index, key_name, start_pos
+                index = 0
+                next_must_be_name = False
+            elif next_is_key:
+                if tok_type == tokenize.NAME:
+                    key_name = tok_str
+                next_is_key = False
+
+            if tok_str == '(':
+                level += 1
+                if level == 1:
+                    next_must_be_name = True
+                    level = 0
+            elif tok_str == ')':
+                level -= 1
+            elif tok_str == ',':
+                index += 1
+            elif tok_str == '=':
+                next_is_key = True
+        return None, 0, None, (0, 0)
+
+    def get_context(self, yield_positions=False):
+        self.get_path_until_cursor()  # In case _start_cursor_pos is undefined.
+        pos = self._start_cursor_pos
+        while True:
+            # remove non important white space
+            line = self.get_line(pos[0])
+            while True:
+                if pos[1] == 0:
+                    line = self.get_line(pos[0] - 1)
+                    if line and line[-1] == '\\':
+                        pos = pos[0] - 1, len(line) - 1
+                        continue
+                    else:
+                        break
+
+                if line[pos[1] - 1].isspace():
+                    pos = pos[0], pos[1] - 1
+                else:
+                    break
+
+            try:
+                result, pos = self._calc_path_until_cursor(start_pos=pos)
+                if yield_positions:
+                    yield pos
+                else:
+                    yield result
+            except StopIteration:
+                if yield_positions:
+                    yield None
+                else:
+                    yield ''
+
+    def get_line(self, line_nr):
+        if not self._line_cache:
+            self._line_cache = common.splitlines(self.source)
+
+        if line_nr == 0:
+            # This is a fix for the zeroth line. We need a newline there, for
+            # the backwards parser.
+            return u('')
+        if line_nr < 0:
+            raise StopIteration()
+        try:
+            return self._line_cache[line_nr - 1]
+        except IndexError:
+            raise StopIteration()
+
+    def get_position_line(self):
+        return self.get_line(self.position[0])[:self.position[1]]
+
+
+class UserContextParser(object):
+    def __init__(self, grammar, source, path, position, user_context,
+                 parser_done_callback, use_fast_parser=True):
+        self._grammar = grammar
+        self._source = source
+        self._path = path and os.path.abspath(path)
+        self._position = position
+        self._user_context = user_context
+        self._use_fast_parser = use_fast_parser
+        self._parser_done_callback = parser_done_callback
+
+    @cache.underscore_memoization
+    def _parser(self):
+        cache.invalidate_star_import_cache(self._path)
+        if self._use_fast_parser:
+            parser = FastParser(self._grammar, self._source, self._path)
+            # Don't pickle that module, because the main module is changing quickly
+            cache.save_parser(self._path, parser, pickling=False)
+        else:
+            parser = Parser(self._grammar, self._source, self._path)
+        self._parser_done_callback(parser)
+        return parser
+
+    @cache.underscore_memoization
+    def user_stmt(self):
+        module = self.module()
+        debug.speed('parsed')
+        return module.get_statement_for_position(self._position)
+
+    @cache.underscore_memoization
+    def user_stmt_with_whitespace(self):
+        """
+        Returns the statement under the cursor even if the statement lies
+        before the cursor.
+        """
+        user_stmt = self.user_stmt()
+
+        if not user_stmt:
+            # for statements like `from x import ` (cursor not in statement)
+            # or `abs( ` where the cursor is out in the whitespace.
+            if self._user_context.get_path_under_cursor():
+                # We really should have a user_stmt, but the parser couldn't
+                # process it - probably a Syntax Error (or in a comment).
+                debug.warning('No statement under the cursor.')
+                return
+            pos = next(self._user_context.get_context(yield_positions=True))
+            user_stmt = self.module().get_statement_for_position(pos)
+        return user_stmt
+
+    @cache.underscore_memoization
+    def user_scope(self):
+        """
+        Returns the scope in which the user resides. This includes flows.
+        """
+        user_stmt = self.user_stmt()
+        if user_stmt is None:
+            def scan(scope):
+                for s in scope.children:
+                    if s.start_pos <= self._position <= s.end_pos:
+                        if isinstance(s, (tree.Scope, tree.Flow)):
+                                if isinstance(s, tree.Flow):
+                                    return s
+                                return scan(s) or s
+                        elif s.type in ('suite', 'decorated'):
+                            return scan(s)
+
+            return scan(self.module()) or self.module()
+        else:
+            return user_stmt.get_parent_scope(include_flows=True)
+
+    def module(self):
+        return self._parser().module

jedi/parsing.py

@@ -1,681 +0,0 @@
-"""
-The ``Parser`` tries to convert the available Python code in an easy to read
-format, something like an abstract syntax tree. The classes who represent this
-tree, are sitting in the :mod:`parsing_representation` module.
-
-The Python module ``tokenize`` is a very important part in the ``Parser``,
-because it splits the code into different words (tokens).  Sometimes it looks a
-bit messy. Sorry for that! You might ask now: "Why didn't you use the ``ast``
-module for this? Well, ``ast`` does a very good job understanding proper Python
-code, but fails to work as soon as there's a single line of broken code.
-
-There's one important optimization that needs to be known: Statements are not
-being parsed completely. ``Statement`` is just a representation of the tokens
-within the statement. This lowers memory usage and cpu time and reduces the
-complexity of the ``Parser`` (there's another parser sitting inside
-``Statement``, which produces ``Array`` and ``Call``).
-"""
-from __future__ import with_statement
-
-import tokenizer as tokenize
-import keyword
-
-from jedi._compatibility import next, StringIO
-from jedi import debug
-from jedi import common
-from jedi import parsing_representation as pr
-
-
-class ParserError(Exception):
-    pass
-
-
-class Parser(object):
-    """
-    This class is used to parse a Python file, it then divides them into a
-    class structure of different scopes.
-
-    :param source: The codebase for the parser.
-    :type source: str
-    :param module_path: The path of the module in the file system, may be None.
-    :type module_path: str
-    :param user_position: The line/column, the user is currently on.
-    :type user_position: tuple(int, int)
-    :param no_docstr: If True, a string at the beginning is not a docstr.
-    :param is_fast_parser: -> for fast_parser
-    :param top_module: Use this module as a parent instead of `self.module`.
-    """
-    def __init__(self, source, module_path=None, user_position=None,
-                        no_docstr=False, offset=(0, 0), is_fast_parser=None,
-                        top_module=None):
-        self.user_position = user_position
-        self.user_scope = None
-        self.user_stmt = None
-        self.no_docstr = no_docstr
-
-        self.start_pos = self.end_pos = 1 + offset[0], offset[1]
-        # initialize global Scope
-        self.module = pr.SubModule(module_path, self.start_pos, top_module)
-        self.scope = self.module
-        self.current = (None, None)
-
-        source = source + '\n'  # end with \n, because the parser needs it
-        buf = StringIO(source)
-        self._gen = common.NoErrorTokenizer(buf.readline, offset,
-                                            is_fast_parser)
-        self.top_module = top_module or self.module
-        try:
-            self._parse()
-        except (common.MultiLevelStopIteration, StopIteration):
-            # StopIteration needs to be added as well, because python 2 has a
-            # strange way of handling StopIterations.
-            # sometimes StopIteration isn't catched. Just ignore it.
-            pass
-
-        # clean up unused decorators
-        for d in self._decorators:
-            # set a parent for unused decorators, avoid NullPointerException
-            # because of `self.module.used_names`.
-            d.parent = self.module
-
-        if self.current[0] in (tokenize.NL, tokenize.NEWLINE):
-            # we added a newline before, so we need to "remove" it again.
-            self.end_pos = self._gen.previous[2]
-        if self.current[0] == tokenize.INDENT:
-            self.end_pos = self._gen.last_previous[2]
-
-        self.start_pos = self.module.start_pos
-        self.module.end_pos = self.end_pos
-        del self._gen
-
-    def __repr__(self):
-        return "<%s: %s>" % (type(self).__name__, self.module)
-
-    def _check_user_stmt(self, simple):
-        # this is not user checking, just update the used_names
-        for tok_name in self.module.temp_used_names:
-            try:
-                self.module.used_names[tok_name].add(simple)
-            except KeyError:
-                self.module.used_names[tok_name] = set([simple])
-        self.module.temp_used_names = []
-
-        if not self.user_position:
-            return
-        # the position is right
-        if simple.start_pos <= self.user_position <= simple.end_pos:
-            if self.user_stmt is not None:
-                # if there is already a user position (another import, because
-                # imports are splitted) the names are checked.
-                for n in simple.get_set_vars():
-                    if n.start_pos < self.user_position <= n.end_pos:
-                        self.user_stmt = simple
-            else:
-                self.user_stmt = simple
-
-    def _parse_dot_name(self, pre_used_token=None):
-        """
-        The dot name parser parses a name, variable or function and returns
-        their names.
-
-        :return: Tuple of Name, token_type, nexttoken.
-        :rtype: tuple(Name, int, str)
-        """
-        def append(el):
-            names.append(el)
-            self.module.temp_used_names.append(el[0])
-
-        names = []
-        if pre_used_token is None:
-            token_type, tok = self.next()
-            if token_type != tokenize.NAME and tok != '*':
-                return [], token_type, tok
-        else:
-            token_type, tok = pre_used_token
-
-        if token_type != tokenize.NAME and tok != '*':
-            # token maybe a name or star
-            return None, token_type, tok
-
-        append((tok, self.start_pos))
-        first_pos = self.start_pos
-        while True:
-            token_type, tok = self.next()
-            if tok != '.':
-                break
-            token_type, tok = self.next()
-            if token_type != tokenize.NAME:
-                break
-            append((tok, self.start_pos))
-
-        n = pr.Name(self.module, names, first_pos, self.end_pos) if names \
-                                                                else None
-        return n, token_type, tok
-
-    def _parse_import_list(self):
-        """
-        The parser for the imports. Unlike the class and function parse
-        function, this returns no Import class, but rather an import list,
-        which is then added later on.
-        The reason, why this is not done in the same class lies in the nature
-        of imports. There are two ways to write them:
-
-        - from ... import ...
-        - import ...
-
-        To distinguish, this has to be processed after the parser.
-
-        :return: List of imports.
-        :rtype: list
-        """
-        imports = []
-        brackets = False
-        continue_kw = [",", ";", "\n", ')'] \
-                        + list(set(keyword.kwlist) - set(['as']))
-        while True:
-            defunct = False
-            token_type, tok = self.next()
-            if brackets and tok == '\n':
-                self.next()
-            if tok == '(':  # python allows only one `(` in the statement.
-                brackets = True
-                self.next()
-            i, token_type, tok = self._parse_dot_name(self.current)
-            if not i:
-                defunct = True
-            name2 = None
-            if tok == 'as':
-                name2, token_type, tok = self._parse_dot_name()
-            imports.append((i, name2, defunct))
-            while tok not in continue_kw:
-                token_type, tok = self.next()
-            if not (tok == "," or brackets and tok == '\n'):
-                break
-        return imports
-
-    def _parse_parentheses(self):
-        """
-        Functions and Classes have params (which means for classes
-        super-classes). They are parsed here and returned as Statements.
-
-        :return: List of Statements
-        :rtype: list
-        """
-        names = []
-        tok = None
-        pos = 0
-        breaks = [',', ':']
-        while tok not in [')', ':']:
-            param, tok = self._parse_statement(added_breaks=breaks,
-                                                  stmt_class=pr.Param)
-            if param and tok == ':':
-                # parse annotations
-                annotation, tok = self._parse_statement(added_breaks=breaks)
-                if annotation:
-                    param.add_annotation(annotation)
-
-            # params without vars are usually syntax errors.
-            if param and (param.set_vars or param.used_vars):
-                param.position_nr = pos
-                names.append(param)
-                pos += 1
-
-        return names
-
-    def _parse_function(self):
-        """
-        The parser for a text functions. Process the tokens, which follow a
-        function definition.
-
-        :return: Return a Scope representation of the tokens.
-        :rtype: Function
-        """
-        first_pos = self.start_pos
-        token_type, fname = self.next()
-        if token_type != tokenize.NAME:
-            return None
-
-        fname = pr.Name(self.module, [(fname, self.start_pos)], self.start_pos,
-                                                                self.end_pos)
-
-        token_type, open = self.next()
-        if open != '(':
-            return None
-        params = self._parse_parentheses()
-
-        token_type, colon = self.next()
-        annotation = None
-        if colon in ['-', '->']:
-            # parse annotations
-            if colon == '-':
-                # The Python 2 tokenizer doesn't understand this
-                token_type, colon = self.next()
-                if colon != '>':
-                    return None
-            annotation, colon = self._parse_statement(added_breaks=[':'])
-
-        if colon != ':':
-            return None
-
-        # because of 2 line func param definitions
-        scope = pr.Function(self.module, fname, params, first_pos, annotation)
-        if self.user_scope and scope != self.user_scope \
-                        and self.user_position > first_pos:
-            self.user_scope = scope
-        return scope
-
-    def _parse_class(self):
-        """
-        The parser for a text class. Process the tokens, which follow a
-        class definition.
-
-        :return: Return a Scope representation of the tokens.
-        :rtype: Class
-        """
-        first_pos = self.start_pos
-        token_type, cname = self.next()
-        if token_type != tokenize.NAME:
-            debug.warning("class: syntax err, token is not a name@%s (%s: %s)"
-                % (self.start_pos[0], tokenize.tok_name[token_type], cname))
-            return None
-
-        cname = pr.Name(self.module, [(cname, self.start_pos)], self.start_pos,
-                                                                self.end_pos)
-
-        super = []
-        token_type, _next = self.next()
-        if _next == '(':
-            super = self._parse_parentheses()
-            token_type, _next = self.next()
-
-        if _next != ':':
-            debug.warning("class syntax: %s@%s" % (cname, self.start_pos[0]))
-            return None
-
-        # because of 2 line class initializations
-        scope = pr.Class(self.module, cname, super, first_pos)
-        if self.user_scope and scope != self.user_scope \
-                        and self.user_position > first_pos:
-            self.user_scope = scope
-        return scope
-
-    def _parse_statement(self, pre_used_token=None, added_breaks=None,
-                            stmt_class=pr.Statement):
-        """
-        Parses statements like::
-
-            a = test(b)
-            a += 3 - 2 or b
-
-        and so on. One line at a time.
-
-        :param pre_used_token: The pre parsed token.
-        :type pre_used_token: set
-        :return: Statement + last parsed token.
-        :rtype: (Statement, str)
-        """
-        set_vars = []
-        used_vars = []
-        level = 0  # The level of parentheses
-
-        if pre_used_token:
-            token_type, tok = pre_used_token
-        else:
-            token_type, tok = self.next()
-
-        while token_type == tokenize.COMMENT:
-            # remove newline and comment
-            self.next()
-            token_type, tok = self.next()
-
-        first_pos = self.start_pos
-        opening_brackets = ['{', '(', '[']
-        closing_brackets = ['}', ')', ']']
-
-        # the difference between "break" and "always break" is that the latter
-        # will even break in parentheses. This is true for typical flow
-        # commands like def and class and the imports, which will never be used
-        # in a statement.
-        breaks = set(['\n', ':', ')'])
-        always_break = [';', 'import', 'from', 'class', 'def', 'try', 'except',
-                        'finally', 'while', 'return', 'yield']
-        not_first_break = ['del', 'raise']
-        if added_breaks:
-            breaks |= set(added_breaks)
-
-        tok_list = []
-        while not (tok in always_break
-                or tok in not_first_break and not tok_list
-                or tok in breaks and level <= 0):
-            try:
-                #print 'parse_stmt', tok, tokenize.tok_name[token_type]
-                tok_list.append(self.current + (self.start_pos,))
-                if tok == 'as':
-                    token_type, tok = self.next()
-                    if token_type == tokenize.NAME:
-                        n, token_type, tok = self._parse_dot_name(self.current)
-                        if n:
-                            set_vars.append(n)
-                        tok_list.append(n)
-                    continue
-                elif tok in ['lambda', 'for', 'in']:
-                    # don't parse these keywords, parse later in stmt.
-                    if tok == 'lambda':
-                        breaks.discard(':')
-                elif token_type == tokenize.NAME:
-                    n, token_type, tok = self._parse_dot_name(self.current)
-                    # removed last entry, because we add Name
-                    tok_list.pop()
-                    if n:
-                        tok_list.append(n)
-                        used_vars.append(n)
-                    continue
-                elif tok.endswith('=') and tok not in ['>=', '<=', '==', '!=']:
-                    # there has been an assignement -> change vars
-                    if level == 0:
-                        set_vars += used_vars
-                        used_vars = []
-                elif tok in opening_brackets:
-                    level += 1
-                elif tok in closing_brackets:
-                    level -= 1
-
-                token_type, tok = self.next()
-            except (StopIteration, common.MultiLevelStopIteration):
-                # comes from tokenizer
-                break
-
-        if not tok_list:
-            return None, tok
-        #print 'new_stat', set_vars, used_vars
-        if self.freshscope and not self.no_docstr and len(tok_list) == 1 \
-                    and self.last_token[0] == tokenize.STRING:
-            self.scope.add_docstr(self.last_token[1])
-            return None, tok
-        else:
-            stmt = stmt_class(self.module, set_vars, used_vars, tok_list,
-                              first_pos, self.end_pos)
-
-            stmt.parent = self.top_module
-            self._check_user_stmt(stmt)
-
-        # Attribute docstring (PEP 224) support (sphinx uses it, e.g.)
-        with common.ignored(IndexError, AttributeError):
-            # If string literal is being parsed
-            first_tok = stmt.token_list[0]
-            if (not stmt.set_vars
-                    and not stmt.used_vars
-                    and len(stmt.token_list) == 1
-                    and first_tok[0] == tokenize.STRING):
-                # ... then set it as a docstring
-                self.scope.statements[-1].add_docstr(first_tok[1])
-
-        if tok in always_break + not_first_break:
-            self._gen.push_last_back()
-        return stmt, tok
-
-    def next(self):
-        return self.__next__()
-
-    def __iter__(self):
-        return self
-
-    def __next__(self):
-        """ Generate the next tokenize pattern. """
-        try:
-            typ, tok, start_pos, end_pos, self.parserline = next(self._gen)
-            # dedents shouldn't change positions
-            if typ != tokenize.DEDENT:
-                self.start_pos, self.end_pos = start_pos, end_pos
-        except (StopIteration, common.MultiLevelStopIteration):
-            # on finish, set end_pos correctly
-            s = self.scope
-            while s is not None:
-                if isinstance(s, pr.Module) \
-                                     and not isinstance(s, pr.SubModule):
-                    self.module.end_pos = self.end_pos
-                    break
-                s.end_pos = self.end_pos
-                s = s.parent
-            raise
-
-        if self.user_position and (self.start_pos[0] == self.user_position[0]
-                            or self.user_scope is None
-                            and self.start_pos[0] >= self.user_position[0]):
-            debug.dbg('user scope found [%s] = %s' %
-                    (self.parserline.replace('\n', ''), repr(self.scope)))
-            self.user_scope = self.scope
-        self.last_token = self.current
-        self.current = (typ, tok)
-        return self.current
-
-    def _parse(self):
-        """
-        The main part of the program. It analyzes the given code-text and
-        returns a tree-like scope. For a more detailed description, see the
-        class description.
-
-        :param text: The code which should be parsed.
-        :param type: str
-
-        :raises: IndentationError
-        """
-        extended_flow = ['else', 'elif', 'except', 'finally']
-        statement_toks = ['{', '[', '(', '`']
-
-        self._decorators = []
-        self.freshscope = True
-        self.iterator = iter(self)
-        # This iterator stuff is not intentional. It grew historically.
-        for token_type, tok in self.iterator:
-            self.module.temp_used_names = []
-            #debug.dbg('main: tok=[%s] type=[%s] indent=[%s]'\
-            #    % (tok, tokenize.tok_name[token_type], start_position[0]))
-
-            while token_type == tokenize.DEDENT and self.scope != self.module:
-                token_type, tok = self.next()
-                if self.start_pos[1] <= self.scope.start_pos[1]:
-                    self.scope.end_pos = self.start_pos
-                    self.scope = self.scope.parent
-                    if isinstance(self.scope, pr.Module) \
-                            and not isinstance(self.scope, pr.SubModule):
-                        self.scope = self.module
-
-            # check again for unindented stuff. this is true for syntax
-            # errors. only check for names, because thats relevant here. If
-            # some docstrings are not indented, I don't care.
-            while self.start_pos[1] <= self.scope.start_pos[1] \
-                    and (token_type == tokenize.NAME or tok in ['(', '['])\
-                    and self.scope != self.module:
-                self.scope.end_pos = self.start_pos
-                self.scope = self.scope.parent
-                if isinstance(self.scope, pr.Module) \
-                        and not isinstance(self.scope, pr.SubModule):
-                    self.scope = self.module
-
-            use_as_parent_scope = self.top_module if isinstance(self.scope,
-                                            pr.SubModule) else self.scope
-            first_pos = self.start_pos
-            if tok == 'def':
-                func = self._parse_function()
-                if func is None:
-                    debug.warning("function: syntax error@%s" %
-                                                        self.start_pos[0])
-                    continue
-                self.freshscope = True
-                self.scope = self.scope.add_scope(func, self._decorators)
-                self._decorators = []
-            elif tok == 'class':
-                cls = self._parse_class()
-                if cls is None:
-                    debug.warning("class: syntax error@%s" % self.start_pos[0])
-                    continue
-                self.freshscope = True
-                self.scope = self.scope.add_scope(cls, self._decorators)
-                self._decorators = []
-            # import stuff
-            elif tok == 'import':
-                imports = self._parse_import_list()
-                for count, (m, alias, defunct) in enumerate(imports):
-                    e = (alias or m or self).end_pos
-                    end_pos = self.end_pos if count + 1 == len(imports) else e
-                    i = pr.Import(self.module, first_pos, end_pos, m,
-                                                alias, defunct=defunct)
-                    self._check_user_stmt(i)
-                    self.scope.add_import(i)
-                if not imports:
-                    i = pr.Import(self.module, first_pos, self.end_pos, None,
-                                                                defunct=True)
-                    self._check_user_stmt(i)
-                self.freshscope = False
-            elif tok == 'from':
-                defunct = False
-                # take care for relative imports
-                relative_count = 0
-                while 1:
-                    token_type, tok = self.next()
-                    if tok != '.':
-                        break
-                    relative_count += 1
-                # the from import
-                mod, token_type, tok = self._parse_dot_name(self.current)
-                if str(mod) == 'import' and relative_count:
-                    self._gen.push_last_back()
-                    tok = 'import'
-                    mod = None
-                if not mod and not relative_count or tok != "import":
-                    debug.warning("from: syntax error@%s" % self.start_pos[0])
-                    defunct = True
-                    if tok != 'import':
-                        self._gen.push_last_back()
-                names = self._parse_import_list()
-                for count, (name, alias, defunct2) in enumerate(names):
-                    star = name is not None and name.names[0] == '*'
-                    if star:
-                        name = None
-                    e = (alias or name or self).end_pos
-                    end_pos = self.end_pos if count + 1 == len(names) else e
-                    i = pr.Import(self.module, first_pos, end_pos, name,
-                                        alias, mod, star, relative_count,
-                                        defunct=defunct or defunct2)
-                    self._check_user_stmt(i)
-                    self.scope.add_import(i)
-                self.freshscope = False
-            #loops
-            elif tok == 'for':
-                set_stmt, tok = self._parse_statement(added_breaks=['in'])
-                if tok == 'in':
-                    statement, tok = self._parse_statement()
-                    if tok == ':':
-                        s = [] if statement is None else [statement]
-                        f = pr.ForFlow(self.module, s, first_pos, set_stmt)
-                        self.scope = self.scope.add_statement(f)
-                    else:
-                        debug.warning('syntax err, for flow started @%s',
-                                                        self.start_pos[0])
-                        if statement is not None:
-                            statement.parent = use_as_parent_scope
-                        if set_stmt is not None:
-                            set_stmt.parent = use_as_parent_scope
-                else:
-                    debug.warning('syntax err, for flow incomplete @%s',
-                                                        self.start_pos[0])
-                    if set_stmt is not None:
-                        set_stmt.parent = use_as_parent_scope
-
-            elif tok in ['if', 'while', 'try', 'with'] + extended_flow:
-                added_breaks = []
-                command = tok
-                if command in ['except', 'with']:
-                    added_breaks.append(',')
-                # multiple inputs because of with
-                inputs = []
-                first = True
-                while first or command == 'with' \
-                                            and tok not in [':', '\n']:
-                    statement, tok = \
-                        self._parse_statement(added_breaks=added_breaks)
-                    if command == 'except' and tok in added_breaks:
-                        # the except statement defines a var
-                        # this is only true for python 2
-                        n, token_type, tok = self._parse_dot_name()
-                        if n:
-                            statement.set_vars.append(n)
-                    if statement:
-                        inputs.append(statement)
-                    first = False
-
-                if tok == ':':
-                    f = pr.Flow(self.module, command, inputs, first_pos)
-                    if command in extended_flow:
-                        # the last statement has to be another part of
-                        # the flow statement, because a dedent releases the
-                        # main scope, so just take the last statement.
-                        try:
-                            s = self.scope.statements[-1].set_next(f)
-                        except (AttributeError, IndexError):
-                            # If set_next doesn't exist, just add it.
-                            s = self.scope.add_statement(f)
-                    else:
-                        s = self.scope.add_statement(f)
-                    self.scope = s
-                else:
-                    for i in inputs:
-                        i.parent = use_as_parent_scope
-                    debug.warning('syntax err, flow started @%s',
-                                                        self.start_pos[0])
-            # returns
-            elif tok in ['return', 'yield']:
-                s = self.start_pos
-                self.freshscope = False
-                # add returns to the scope
-                func = self.scope.get_parent_until(pr.Function)
-                if tok == 'yield':
-                    func.is_generator = True
-
-                stmt, tok = self._parse_statement()
-                if stmt is not None:
-                    stmt.parent = use_as_parent_scope
-                try:
-                    func.returns.append(stmt)
-                    # start_pos is the one of the return statement
-                    stmt.start_pos = s
-                except AttributeError:
-                    debug.warning('return in non-function')
-            # globals
-            elif tok == 'global':
-                stmt, tok = self._parse_statement(self.current)
-                if stmt:
-                    self.scope.add_statement(stmt)
-                    for name in stmt.used_vars:
-                        # add the global to the top, because there it is
-                        # important.
-                        self.module.add_global(name)
-            # decorator
-            elif tok == '@':
-                stmt, tok = self._parse_statement()
-                self._decorators.append(stmt)
-            elif tok == 'pass':
-                continue
-            elif tok == 'assert':
-                stmt, tok = self._parse_statement()
-                stmt.parent = use_as_parent_scope
-                self.scope.asserts.append(stmt)
-            # default
-            elif token_type in [tokenize.NAME, tokenize.STRING,
-                                tokenize.NUMBER] \
-                    or tok in statement_toks:
-                # this is the main part - a name can be a function or a
-                # normal var, which can follow anything. but this is done
-                # by the statement parser.
-                stmt, tok = self._parse_statement(self.current)
-                if stmt:
-                    self.scope.add_statement(stmt)
-                self.freshscope = False
-            else:
-                if token_type not in [tokenize.COMMENT, tokenize.INDENT,
-                                      tokenize.NEWLINE, tokenize.NL]:
-                    debug.warning('token not classified', tok, token_type,
-                                                        self.start_pos[0])
-                continue
-            self.no_docstr = False

jedi/refactoring.py

@@ -12,14 +12,11 @@ following functions (sometimes bug-prone):
 - extract variable
 - inline variable
 """
-from __future__ import with_statement
-
 import difflib
 
 from jedi import common
-from jedi import modules
-from jedi import helpers
-from jedi import parsing_representation as pr
+from jedi.evaluate import helpers
+from jedi.parser import tree as pt
 
 
 class Refactoring(object):
@@ -64,8 +61,8 @@ def rename(script, new_name):
 
 def _rename(names, replace_str):
     """ For both rename and inline. """
-    order = sorted(names, key=lambda x: (x.module_path, x.start_pos),
-                            reverse=True)
+    order = sorted(names, key=lambda x: (x.module_path, x.line, x.column),
+                   reverse=True)
 
     def process(path, old_lines, new_lines):
         if new_lines is not None:  # goto next file, save last
@@ -86,13 +83,13 @@ def _rename(names, replace_str):
                 with open(current_path) as f:
                     source = f.read()
 
-            new_lines = modules.source_to_unicode(source).splitlines()
+            new_lines = common.splitlines(common.source_to_unicode(source))
             old_lines = new_lines[:]
 
-        nr, indent = name.start_pos
+        nr, indent = name.line, name.column
         line = new_lines[nr - 1]
         new_lines[nr - 1] = line[:indent] + replace_str + \
-                            line[indent + len(name.text):]
+            line[indent + len(name.name):]
     process(current_path, old_lines, new_lines)
     return dct
 
@@ -104,15 +101,15 @@ def extract(script, new_name):
     :type source: str
     :return: list of changed lines/changed files
     """
-    new_lines = modules.source_to_unicode(script.source).splitlines()
+    new_lines = common.splitlines(common.source_to_unicode(script.source))
     old_lines = new_lines[:]
 
-    user_stmt = script._parser.user_stmt
+    user_stmt = script._parser.user_stmt()
 
     # TODO care for multiline extracts
     dct = {}
     if user_stmt:
-        pos = script.pos
+        pos = script._pos
         line_index = pos[0] - 1
         arr, index = helpers.array_for_pos(user_stmt, pos)
         if arr is not None:
@@ -148,14 +145,14 @@ def extract(script, new_name):
             open_brackets = ['(', '[', '{']
             close_brackets = [')', ']', '}']
             if '\n' in text and not (text[0] in open_brackets and text[-1] ==
-                                close_brackets[open_brackets.index(text[0])]):
+                                     close_brackets[open_brackets.index(text[0])]):
                 text = '(%s)' % text
 
             # add new line before statement
             indent = user_stmt.start_pos[1]
             new = "%s%s = %s" % (' ' * indent, new_name, text)
             new_lines.insert(line_index, new)
-    dct[script.source_path] = script.source_path, old_lines, new_lines
+    dct[script.path] = script.path, old_lines, new_lines
     return Refactoring(dct)
 
 
@@ -163,40 +160,40 @@ def inline(script):
     """
     :type script: api.Script
     """
-    new_lines = modules.source_to_unicode(script.source).splitlines()
+    new_lines = common.splitlines(common.source_to_unicode(script.source))
 
     dct = {}
 
-    definitions = script.goto()
+    definitions = script.goto_assignments()
     with common.ignored(AssertionError):
         assert len(definitions) == 1
         stmt = definitions[0]._definition
         usages = script.usages()
         inlines = [r for r in usages
-                        if not stmt.start_pos <= r.start_pos <= stmt.end_pos]
-        inlines = sorted(inlines, key=lambda x: (x.module_path, x.start_pos),
-                                                reverse=True)
-        commands = stmt.get_commands()
+                   if not stmt.start_pos <= (r.line, r.column) <= stmt.end_pos]
+        inlines = sorted(inlines, key=lambda x: (x.module_path, x.line, x.column),
+                         reverse=True)
+        expression_list = stmt.expression_list()
         # don't allow multiline refactorings for now.
         assert stmt.start_pos[0] == stmt.end_pos[0]
         index = stmt.start_pos[0] - 1
 
         line = new_lines[index]
-        replace_str = line[commands[0].start_pos[1]:stmt.end_pos[1] + 1]
+        replace_str = line[expression_list[0].start_pos[1]:stmt.end_pos[1] + 1]
         replace_str = replace_str.strip()
         # tuples need parentheses
-        if commands and isinstance(commands[0], pr.Array):
-            arr = commands[0]
+        if expression_list and isinstance(expression_list[0], pr.Array):
+            arr = expression_list[0]
             if replace_str[0] not in ['(', '[', '{'] and len(arr) > 1:
                 replace_str = '(%s)' % replace_str
 
         # if it's the only assignment, remove the statement
-        if len(stmt.set_vars) == 1:
+        if len(stmt.get_defined_names()) == 1:
             line = line[:stmt.start_pos[1]] + line[stmt.end_pos[1]:]
 
         dct = _rename(inlines, replace_str)
         # remove the empty line
-        new_lines = dct[script.source_path][2]
+        new_lines = dct[script.path][2]
         if line.strip():
             new_lines[index] = line
         else:

jedi/settings.py

@@ -1,7 +1,6 @@
 """
-This module contains variables with global |jedi| setting. To change the
-behavior of |jedi|, change the variables defined in
-:mod:`jedi.settings`.
+This module contains variables with global |jedi| settings. To change the
+behavior of |jedi|, change the variables defined in :mod:`jedi.settings`.
 
 Plugins should expose an interface so that the user can adjust the
 configuration.
@@ -38,11 +37,11 @@ Parser
 Dynamic stuff
 ~~~~~~~~~~~~~
 
-.. autodata:: dynamic_arrays_instances
 .. autodata:: dynamic_array_additions
 .. autodata:: dynamic_params
 .. autodata:: dynamic_params_for_other_modules
 .. autodata:: additional_dynamic_modules
+.. autodata:: auto_import_modules
 
 
 .. _settings-recursion:
@@ -64,14 +63,14 @@ definitely worse in some cases. But a completion should also be fast.
 .. autodata:: max_function_recursion_level
 .. autodata:: max_executions_without_builtins
 .. autodata:: max_executions
-.. autodata:: scale_function_definition
+.. autodata:: scale_call_signatures
 
 
 Caching
 ~~~~~~~
 
 .. autodata:: star_import_cache_validity
-.. autodata:: function_definition_validity
+.. autodata:: call_signatures_validity
 
 
 """
@@ -127,8 +126,10 @@ cache_directory = os.path.expanduser(_cache_directory)
 """
 The path where all the caches can be found.
 
-On Linux, this defaults to ``~/.cache/jedi/``, on OS X to ``~/.jedi/`` and on
-Windows to ``%APPDATA%\\Jedi\\Jedi\\``.
+On Linux, this defaults to ``~/.cache/jedi/``, on OS X to
+``~/Library/Caches/Jedi/`` and on Windows to ``%APPDATA%\\Jedi\\Jedi\\``.
+On Linux, if environment variable ``$XDG_CACHE_HOME`` is set,
+``$XDG_CACHE_HOME/jedi`` is used instead of the default one.
 """
 
 # ----------------
@@ -146,14 +147,9 @@ function is being reparsed.
 # dynamic stuff
 # ----------------
 
-dynamic_arrays_instances = True
-"""
-Check for `append`, etc. on array instances like list()
-"""
-
 dynamic_array_additions = True
 """
-check for `append`, etc. on arrays: [], {}, ()
+check for `append`, etc. on arrays: [], {}, () as well as list/set calls.
 """
 
 dynamic_params = True
@@ -173,6 +169,20 @@ Additional modules in which |jedi| checks if statements are to be found. This
 is practical for IDEs, that want to administrate their modules themselves.
 """
 
+dynamic_flow_information = True
+"""
+Check for `isinstance` and other information to infer a type.
+"""
+
+auto_import_modules = [
+    'hashlib',  # setattr
+]
+"""
+Modules that are not analyzed but imported, although they contain Python code.
+This improves autocompletion for libraries that use ``setattr`` or
+``globals()`` modifications a lot.
+"""
+
 # ----------------
 # recursions
 # ----------------
@@ -204,9 +214,9 @@ max_executions = 250
 A maximum amount of time, the completion may use.
 """
 
-scale_function_definition = 0.1
+scale_call_signatures = 0.1
 """
-Because function_definition is normally used on every single key hit, it has
+Because call_signatures is normally used on every single key hit, it has
 to be faster than a normal completion. This is the factor that is used to
 scale `max_executions` and `max_until_execution_unique`:
 """
@@ -222,7 +232,7 @@ might be slow, therefore we do a star import caching, that lasts a certain
 time span (in seconds).
 """
 
-function_definition_validity = 3.0
+call_signatures_validity = 3.0
 """
 Finding function calls might be slow (0.1-0.5s). This is not acceptible for
 normal writing. Therefore cache it for a short time.

jedi/tokenizer.py

@@ -1,285 +0,0 @@
-"""
-This tokenizer has been copied from the ``tokenize.py`` standard library
-tokenizer. The reason was simple: The standanrd library  tokenizer fails
-if the indentation is not right. The fast parser of jedi however requires
-"wrong" indentation.
-
-Basically this is a stripped down version of the standard library module, so
-you can read the documentation there.
-"""
-
-import string
-import re
-from token import *
-from codecs import lookup, BOM_UTF8
-import collections
-cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
-
-namechars = string.ascii_letters + '_'
-
-
-COMMENT = N_TOKENS
-tok_name[COMMENT] = 'COMMENT'
-NL = N_TOKENS + 1
-tok_name[NL] = 'NL'
-ENCODING = N_TOKENS + 2
-tok_name[ENCODING] = 'ENCODING'
-N_TOKENS += 3
-
-
-class TokenInfo(collections.namedtuple('TokenInfo', 'type string start end line')):
-    def __repr__(self):
-        annotated_type = '%d (%s)' % (self.type, tok_name[self.type])
-        return ('TokenInfo(type=%s, string=%r, start=%r, end=%r, line=%r)' %
-                self._replace(type=annotated_type))
-
-
-def group(*choices):
-    return '(' + '|'.join(choices) + ')'
-
-
-def any(*choices):
-    return group(*choices) + '*'
-
-
-def maybe(*choices):
-    return group(*choices) + '?'
-
-
-# Note: we use unicode matching for names ("\w") but ascii matching for
-# number literals.
-Whitespace = r'[ \f\t]*'
-Comment = r'#[^\r\n]*'
-Ignore = Whitespace + any(r'\\\r?\n' + Whitespace) + maybe(Comment)
-Name = r'\w+'
-
-Hexnumber = r'0[xX][0-9a-fA-F]+'
-Binnumber = r'0[bB][01]+'
-Octnumber = r'0[oO][0-7]+'
-Decnumber = r'(?:0+|[1-9][0-9]*)'
-Intnumber = group(Hexnumber, Binnumber, Octnumber, Decnumber)
-Exponent = r'[eE][-+]?[0-9]+'
-Pointfloat = group(r'[0-9]+\.[0-9]*', r'\.[0-9]+') + maybe(Exponent)
-Expfloat = r'[0-9]+' + Exponent
-Floatnumber = group(Pointfloat, Expfloat)
-Imagnumber = group(r'[0-9]+[jJ]', Floatnumber + r'[jJ]')
-Number = group(Imagnumber, Floatnumber, Intnumber)
-
-# Tail end of ' string.
-Single = r"[^'\\]*(?:\\.[^'\\]*)*'"
-# Tail end of " string.
-Double = r'[^"\\]*(?:\\.[^"\\]*)*"'
-# Tail end of ''' string.
-Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''"
-# Tail end of """ string.
-Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""'
-Triple = group("[bB]?[rR]?'''", '[bB]?[rR]?"""')
-# Single-line ' or " string.
-String = group(r"[bB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
-               r'[bB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"')
-
-# Because of leftmost-then-longest match semantics, be sure to put the
-# longest operators first (e.g., if = came before ==, == would get
-# recognized as two instances of =).
-Operator = group(r"\*\*=?", r">>=?", r"<<=?", r"!=",
-                 r"//=?", r"->",
-                 r"[+\-*/%&|^=<>]=?",
-                 r"~")
-
-Bracket = '[][(){}]'
-Special = group(r'\r?\n', r'\.\.\.', r'[:;.,@]')
-Funny = group(Operator, Bracket, Special)
-
-PlainToken = group(Number, Funny, String, Name)
-Token = Ignore + PlainToken
-
-# First (or only) line of ' or " string.
-ContStr = group(r"[bB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
-                group("'", r'\\\r?\n'),
-                r'[bB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
-                group('"', r'\\\r?\n'))
-PseudoExtras = group(r'\\\r?\n', Comment, Triple)
-PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
-
-
-def _compile(expr):
-    return re.compile(expr, re.UNICODE)
-
-
-tokenprog, pseudoprog, single3prog, double3prog = map(
-    _compile, (Token, PseudoToken, Single3, Double3))
-endprogs = {"'": _compile(Single), '"': _compile(Double),
-            "'''": single3prog, '"""': double3prog,
-            "r'''": single3prog, 'r"""': double3prog,
-            "b'''": single3prog, 'b"""': double3prog,
-            "br'''": single3prog, 'br"""': double3prog,
-            "R'''": single3prog, 'R"""': double3prog,
-            "B'''": single3prog, 'B"""': double3prog,
-            "bR'''": single3prog, 'bR"""': double3prog,
-            "Br'''": single3prog, 'Br"""': double3prog,
-            "BR'''": single3prog, 'BR"""': double3prog,
-            'r': None, 'R': None, 'b': None, 'B': None}
-
-triple_quoted = {}
-for t in ("'''", '"""',
-          "r'''", 'r"""', "R'''", 'R"""',
-          "b'''", 'b"""', "B'''", 'B"""',
-          "br'''", 'br"""', "Br'''", 'Br"""',
-          "bR'''", 'bR"""', "BR'''", 'BR"""'):
-    triple_quoted[t] = t
-single_quoted = {}
-for t in ("'", '"',
-          "r'", 'r"', "R'", 'R"',
-          "b'", 'b"', "B'", 'B"',
-          "br'", 'br"', "Br'", 'Br"',
-          "bR'", 'bR"', "BR'", 'BR"'):
-    single_quoted[t] = t
-
-del _compile
-
-tabsize = 8
-
-
-class TokenError(Exception):
-    pass
-
-
-def generate_tokens(readline):
-    lnum = parenlev = continued = 0
-    numchars = '0123456789'
-    contstr, needcont = '', 0
-    contline = None
-    indents = [0]
-
-    while True:             # loop over lines in stream
-        try:
-            line = readline()
-        except StopIteration:
-            line = b''
-
-        lnum += 1
-        pos, max = 0, len(line)
-
-        if contstr:                            # continued string
-            if not line:
-                # multiline string has not been finished
-                break
-            endmatch = endprog.match(line)
-            if endmatch:
-                pos = end = endmatch.end(0)
-                yield TokenInfo(STRING, contstr + line[:end],
-                       strstart, (lnum, end), contline + line)
-                contstr, needcont = '', 0
-                contline = None
-            elif needcont and line[-2:] != '\\\n' and line[-3:] != '\\\r\n':
-                yield TokenInfo(ERRORTOKEN, contstr + line,
-                           strstart, (lnum, len(line)), contline)
-                contstr = ''
-                contline = None
-                continue
-            else:
-                contstr = contstr + line
-                contline = contline + line
-                continue
-
-        elif parenlev == 0 and not continued:  # new statement
-            if not line:
-                break
-            column = 0
-            while pos < max:                   # measure leading whitespace
-                if line[pos] == ' ':
-                    column += 1
-                elif line[pos] == '\t':
-                    column = (column // tabsize + 1) * tabsize
-                elif line[pos] == '\f':
-                    column = 0
-                else:
-                    break
-                pos += 1
-            if pos == max:
-                break
-
-            if line[pos] in '#\r\n':           # skip comments or blank lines
-                if line[pos] == '#':
-                    comment_token = line[pos:].rstrip('\r\n')
-                    nl_pos = pos + len(comment_token)
-                    yield TokenInfo(COMMENT, comment_token,
-                           (lnum, pos), (lnum, pos + len(comment_token)), line)
-                    yield TokenInfo(NL, line[nl_pos:],
-                           (lnum, nl_pos), (lnum, len(line)), line)
-                else:
-                    yield TokenInfo((NL, COMMENT)[line[pos] == '#'], line[pos:],
-                           (lnum, pos), (lnum, len(line)), line)
-                continue
-
-            if column > indents[-1]:           # count indents or dedents
-                indents.append(column)
-                yield TokenInfo(INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
-            while column < indents[-1]:
-                indents = indents[:-1]
-                yield TokenInfo(DEDENT, '', (lnum, pos), (lnum, pos), line)
-
-        else:                                  # continued statement
-            if not line:
-                # basically a statement has not been finished here.
-                break
-            continued = 0
-
-        while pos < max:
-            pseudomatch = pseudoprog.match(line, pos)
-            if pseudomatch:                                # scan for tokens
-                start, end = pseudomatch.span(1)
-                spos, epos, pos = (lnum, start), (lnum, end), end
-                token, initial = line[start:end], line[start]
-
-                if (initial in numchars or                  # ordinary number
-                        (initial == '.' and token != '.' and token != '...')):
-                    yield TokenInfo(NUMBER, token, spos, epos, line)
-                elif initial in '\r\n':
-                    yield TokenInfo(NL if parenlev > 0 else NEWLINE,
-                           token, spos, epos, line)
-                elif initial == '#':
-                    assert not token.endswith("\n")
-                    yield TokenInfo(COMMENT, token, spos, epos, line)
-                elif token in triple_quoted:
-                    endprog = endprogs[token]
-                    endmatch = endprog.match(line, pos)
-                    if endmatch:                           # all on one line
-                        pos = endmatch.end(0)
-                        token = line[start:pos]
-                        yield TokenInfo(STRING, token, spos, (lnum, pos), line)
-                    else:
-                        strstart = (lnum, start)           # multiple lines
-                        contstr = line[start:]
-                        contline = line
-                        break
-                elif initial in single_quoted or \
-                        token[:2] in single_quoted or \
-                        token[:3] in single_quoted:
-                    if token[-1] == '\n':                  # continued string
-                        strstart = (lnum, start)
-                        endprog = (endprogs[initial] or endprogs[token[1]] or
-                                   endprogs[token[2]])
-                        contstr, needcont = line[start:], 1
-                        contline = line
-                        break
-                    else:                                  # ordinary string
-                        yield TokenInfo(STRING, token, spos, epos, line)
-                elif initial in namechars:                 # ordinary name
-                    yield TokenInfo(NAME, token, spos, epos, line)
-                elif initial == '\\':                      # continued stmt
-                    continued = 1
-                else:
-                    if initial in '([{':
-                        parenlev += 1
-                    elif initial in ')]}':
-                        parenlev -= 1
-                    yield TokenInfo(OP, token, spos, epos, line)
-            else:
-                yield TokenInfo(ERRORTOKEN, line[pos],
-                           (lnum, pos), (lnum, pos + 1), line)
-                pos += 1
-
-    for indent in indents[1:]:                 # pop remaining indent levels
-        yield TokenInfo(DEDENT, '', (lnum, 0), (lnum, 0), '')
-    yield TokenInfo(ENDMARKER, '', (lnum, 0), (lnum, 0), '')

jedi/utils.py

@@ -0,0 +1,114 @@
+"""
+Utilities for end-users.
+"""
+
+from __future__ import absolute_import
+import __main__
+from collections import namedtuple
+import re
+import os
+import sys
+
+from jedi import Interpreter
+from jedi.api.helpers import completion_parts
+from jedi.parser.user_context import UserContext
+
+
+def setup_readline(namespace_module=__main__):
+    """
+    Install Jedi completer to :mod:`readline`.
+
+    This function setups :mod:`readline` to use Jedi in Python interactive
+    shell.  If you want to use a custom ``PYTHONSTARTUP`` file (typically
+    ``$HOME/.pythonrc.py``), you can add this piece of code::
+
+        try:
+            from jedi.utils import setup_readline
+            setup_readline()
+        except ImportError:
+            # Fallback to the stdlib readline completer if it is installed.
+            # Taken from http://docs.python.org/2/library/rlcompleter.html
+            print("Jedi is not installed, falling back to readline")
+            try:
+                import readline
+                import rlcompleter
+                readline.parse_and_bind("tab: complete")
+            except ImportError:
+                print("Readline is not installed either. No tab completion is enabled.")
+
+    This will fallback to the readline completer if Jedi is not installed.
+    The readline completer will only complete names in the global namespace,
+    so for example::
+
+        ran<TAB>
+
+    will complete to ``range``
+
+    with both Jedi and readline, but::
+
+        range(10).cou<TAB>
+
+    will show complete to ``range(10).count`` only with Jedi.
+
+    You'll also need to add ``export PYTHONSTARTUP=$HOME/.pythonrc.py`` to
+    your shell profile (usually ``.bash_profile`` or ``.profile`` if you use
+    bash).
+
+    """
+    class JediRL(object):
+        def complete(self, text, state):
+            """
+            This complete stuff is pretty weird, a generator would make
+            a lot more sense, but probably due to backwards compatibility
+            this is still the way how it works.
+
+            The only important part is stuff in the ``state == 0`` flow,
+            everything else has been copied from the ``rlcompleter`` std.
+            library module.
+            """
+            if state == 0:
+                sys.path.insert(0, os.getcwd())
+                # Calling python doesn't have a path, so add to sys.path.
+                try:
+                    interpreter = Interpreter(text, [namespace_module.__dict__])
+
+                    path = UserContext(text, (1, len(text))).get_path_until_cursor()
+                    path, dot, like = completion_parts(path)
+                    before = text[:len(text) - len(like)]
+                    completions = interpreter.completions()
+                finally:
+                    sys.path.pop(0)
+
+                self.matches = [before + c.name_with_symbols for c in completions]
+            try:
+                return self.matches[state]
+            except IndexError:
+                return None
+
+    try:
+        import readline
+    except ImportError:
+        print("Module readline not available.")
+    else:
+        readline.set_completer(JediRL().complete)
+        readline.parse_and_bind("tab: complete")
+        # jedi itself does the case matching
+        readline.parse_and_bind("set completion-ignore-case on")
+        # because it's easier to hit the tab just once
+        readline.parse_and_bind("set show-all-if-unmodified")
+        readline.parse_and_bind("set show-all-if-ambiguous on")
+        # don't repeat all the things written in the readline all the time
+        readline.parse_and_bind("set completion-prefix-display-length 2")
+        # No delimiters, Jedi handles that.
+        readline.set_completer_delims('')
+
+
+def version_info():
+    """
+    Returns a namedtuple of Jedi's version, similar to Python's
+    ``sys.version_info``.
+    """
+    Version = namedtuple('Version', 'major, minor, micro')
+    from jedi import __version__
+    tupl = re.findall('[a-z]+|\d+', __version__)
+    return Version(*[x if i == 3 else int(x) for i, x in enumerate(tupl)])

pytest.ini

@@ -2,7 +2,7 @@
 addopts = --doctest-modules
 
 # Ignore broken files in blackbox test directories
-norecursedirs = .* docs completion refactor
+norecursedirs = .* docs completion refactor absolute_import namespace_package scripts extensions speed static_analysis not_in_sys_path buildout_project egg-link init_extension_module
 
 # Activate `clean_jedi_cache` fixture for all tests.  This should be
 # fine as long as we are using `clean_jedi_cache` as a session scoped

scripts/memory_check.py

@@ -0,0 +1,58 @@
+#! /usr/bin/env python
+"""
+This is a convenience script to test the speed and memory usage of Jedi with
+large libraries.
+
+Each library is preloaded by jedi, recording the time and memory consumed by
+each operation.
+
+You can provide additional libraries via command line arguments.
+
+Note: This requires the psutil library, available on PyPI.
+"""
+import time
+import sys
+import os
+import psutil
+sys.path.insert(0, os.path.abspath(os.path.dirname(__file__) + '/..'))
+import jedi
+
+
+def used_memory():
+    """Return the total MB of System Memory in use."""
+    return psutil.virtual_memory().used / 2 ** 20
+
+
+def profile_preload(mod):
+    """Preload a module into Jedi, recording time and memory used."""
+    base = used_memory()
+    t0 = time.time()
+    jedi.preload_module(mod)
+    elapsed = time.time() - t0
+    used = used_memory() - base
+    return elapsed, used
+
+
+def main(mods):
+    """Preload the modules, and print the time and memory used."""
+    t0 = time.time()
+    baseline = used_memory()
+    print('Time (s) | Mem (MB) | Package')
+    print('------------------------------')
+    for mod in mods:
+        elapsed, used = profile_preload(mod)
+        if used > 0:
+            print('%8.2f | %8d | %s' % (elapsed, used, mod))
+    print('------------------------------')
+    elapsed = time.time() - t0
+    used = used_memory() - baseline
+    print('%8.2f | %8d | %s' % (elapsed, used, 'Total'))
+
+
+if __name__ == '__main__':
+    if sys.argv[1:]:
+        mods = sys.argv[1:]
+    else:
+        mods = ['re', 'numpy', 'scipy', 'scipy.sparse', 'scipy.stats',
+                'wx', 'decimal', 'PyQt4.QtGui', 'PySide.QtGui', 'Tkinter']
+    main(mods)

scripts/profile.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python
+"""
+Profile a piece of Python code with ``cProfile``. Tries a completion on a
+certain piece of code.
+
+Usage:
+  profile.py [<code>] [-n <number>] [-d] [-o] [-s <sort>]
+  profile.py -h | --help
+
+Options:
+  -h --help     Show this screen.
+  -n <number>   Number of passes before profiling [default: 1].
+  -d --debug    Enable Jedi internal debugging.
+  -o --omit     Omit profiler, just do a normal run.
+  -s <sort>     Sort the profile results, e.g. cum, name [default: time].
+"""
+
+import time
+import cProfile
+
+from docopt import docopt
+import jedi
+
+
+def run(code, index):
+    start = time.time()
+    result = jedi.Script(code).completions()
+    print('Used %ss for the %sth run.' % (time.time() - start, index + 1))
+    return result
+
+
+def main(args):
+    code = args['<code>']
+    n = int(args['-n'])
+    for i in range(n):
+        run(code, i)
+
+    jedi.set_debug_function(notices=args['--debug'])
+    if args['--omit']:
+        run(code, n)
+    else:
+        cProfile.runctx('run(code, n)', globals(), locals(), sort=args['-s'])
+
+
+if __name__ == '__main__':
+    args = docopt(__doc__)
+    if args['<code>'] is None:
+        args['<code>'] = 'import numpy; numpy.array([0])'
+    main(args)

scripts/wx_check.py

@@ -0,0 +1,61 @@
+#! /usr/bin/env python
+"""
+Depends: ``objgraph`` (third party Python library)
+
+``wx._core`` is a very nice module to test Jedi's speed and memory performance
+on big Python modules. Its size is ~16kLOC (one file). It also seems to look
+like a typical big Python modules. A mix between a lot of different Python
+things.
+
+You can view a markup version of it here:
+http://svn.wxwidgets.org/viewvc/wx/wxPython/trunk/src/gtk/_core.py?view=markup
+"""
+
+import resource
+import time
+import sys
+try:
+    import urllib.request as urllib2
+except ImportError:
+    import urllib2
+import gc
+from os.path import abspath, dirname
+
+import objgraph
+
+sys.path.insert(0, dirname(dirname(abspath(__file__))))
+import jedi
+
+
+def process_memory():
+    """
+    In kB according to
+    http://stackoverflow.com/questions/938733/total-memory-used-by-python-process
+    """
+    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+
+uri = 'http://svn.wxwidgets.org/viewvc/wx/wxPython/trunk/src/gtk/_core.py?revision=74740&content-type=text%2Fplain&view=co'
+
+wx_core = urllib2.urlopen(uri).read()
+
+
+def run():
+    start = time.time()
+    print('Process Memory before: %skB' % process_memory())
+    # After this the module should be cached.
+    # Need to invent a path so that it's really cached.
+    jedi.Script(wx_core, path='foobar.py').completions()
+
+    gc.collect()  # make sure that it's all fair and the gc did its job.
+    print('Process Memory after: %skB' % process_memory())
+
+    print(objgraph.most_common_types(limit=50))
+    print('\nIt took %s seconds to parse the file.' % (time.time() - start))
+
+
+print('First pass')
+run()
+print('\nSecond pass')
+run()
+print('\nThird pass')
+run()

setup.py

@@ -1,35 +1,40 @@
 #!/usr/bin/env python
 
 from __future__ import with_statement
-from setuptools import setup
+try:
+    from setuptools import setup
+except ImportError:
+    # Distribute is not actually required to install
+    from distutils.core import setup
 
 __AUTHOR__ = 'David Halter'
 __AUTHOR_EMAIL__ = 'davidhalter88@gmail.com'
 
 readme = open('README.rst').read() + '\n\n' + open('CHANGELOG.rst').read()
+packages = ['jedi', 'jedi.parser', 'jedi.parser.pgen2',
+            'jedi.evaluate', 'jedi.evaluate.compiled', 'jedi.api']
 
 import jedi
-VERSION = '.'.join(str(x) for x in jedi.__version__)
 
 setup(name='jedi',
-      version=VERSION,
+      version=jedi.__version__,
       description='An autocompletion tool for Python that can be used for text editors.',
       author=__AUTHOR__,
       author_email=__AUTHOR_EMAIL__,
       maintainer=__AUTHOR__,
       maintainer_email=__AUTHOR_EMAIL__,
       url='https://github.com/davidhalter/jedi',
-      license='LGPLv3',
+      license='MIT',
       keywords='python completion refactoring vim',
       long_description=readme,
-      packages=['jedi'],
-      package_data={'jedi': ['mixin/*.pym']},
+      packages=packages,
+      package_data={'jedi': ['evaluate/compiled/fake/*.pym', 'parser/grammar*.txt']},
       platforms=['any'],
       classifiers=[
           'Development Status :: 4 - Beta',
           'Environment :: Plugins',
           'Intended Audience :: Developers',
-          'License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)',
+          'License :: OSI Approved :: MIT License',
           'Operating System :: OS Independent',
           'Programming Language :: Python :: 2',
           'Programming Language :: Python :: 2.6',
@@ -37,8 +42,9 @@ setup(name='jedi',
           'Programming Language :: Python :: 3',
           'Programming Language :: Python :: 3.2',
           'Programming Language :: Python :: 3.3',
+          'Programming Language :: Python :: 3.4',
           'Topic :: Software Development :: Libraries :: Python Modules',
           'Topic :: Text Editors :: Integrated Development Environments (IDE)',
           'Topic :: Utilities',
       ],
-    )
+      )

sith.py

@@ -0,0 +1,216 @@
+#!/usr/bin/env python
+
+"""
+Sith attacks (and helps debugging) Jedi.
+
+Randomly search Python files and run Jedi on it.  Exception and used
+arguments are recorded to ``./record.json`` (specified by --record)::
+
+    ./sith.py random /path/to/sourcecode
+
+Redo recorded exception::
+
+    ./sith.py redo
+
+Show recorded exception::
+
+    ./sith.py show
+
+Run a specific operation
+
+    ./sith.py run <operation> </path/to/source/file.py> <line> <col>
+
+Where operation is one of completions, goto_assignments, goto_definitions,
+usages, or call_signatures.
+
+Note: Line numbers start at 1; columns start at 0 (this is consistent with
+many text editors, including Emacs).
+
+Usage:
+  sith.py [--pdb|--ipdb|--pudb] [-d] [-n=<nr>] [-f] [--record=<file>] random [-s] [<path>]
+  sith.py [--pdb|--ipdb|--pudb] [-d] [-f] [--record=<file>] redo
+  sith.py [--pdb|--ipdb|--pudb] [-d] [-f] run <operation> <path> <line> <column>
+  sith.py show [--record=<file>]
+  sith.py -h | --help
+
+Options:
+  -h --help             Show this screen.
+  --record=<file>       Exceptions are recorded in here [default: record.json].
+  -f, --fs-cache        By default, file system cache is off for reproducibility.
+  -n, --maxtries=<nr>   Maximum of random tries [default: 100]
+  -d, --debug           Jedi print debugging when an error is raised.
+  -s                    Shows the path/line numbers of every completion before it starts.
+  --pdb                 Launch pdb when error is raised.
+  --ipdb                Launch ipdb when error is raised.
+  --pudb                Launch pudb when error is raised.
+"""
+
+from __future__ import print_function, division, unicode_literals
+from docopt import docopt
+
+import json
+import os
+import random
+import sys
+import traceback
+
+import jedi
+
+
+class SourceFinder(object):
+    _files = None
+
+    @staticmethod
+    def fetch(file_path):
+        if not os.path.isdir(file_path):
+            yield file_path
+            return
+        for root, dirnames, filenames in os.walk(file_path):
+            for name in filenames:
+                if name.endswith('.py'):
+                    yield os.path.join(root, name)
+
+    @classmethod
+    def files(cls, file_path):
+        if cls._files is None:
+            cls._files = list(cls.fetch(file_path))
+        return cls._files
+
+
+class TestCase(object):
+    def __init__(self, operation, path, line, column, traceback=None):
+        if operation not in self.operations:
+            raise ValueError("%s is not a valid operation" % operation)
+
+        # Set other attributes
+        self.operation = operation
+        self.path = path
+        self.line = line
+        self.column = column
+        self.traceback = traceback
+
+    @classmethod
+    def from_cache(cls, record):
+        with open(record) as f:
+            args = json.load(f)
+        return cls(*args)
+
+    operations = [
+        'completions', 'goto_assignments', 'goto_definitions', 'usages',
+        'call_signatures']
+
+    @classmethod
+    def generate(cls, file_path):
+        operation = random.choice(cls.operations)
+
+        path = random.choice(SourceFinder.files(file_path))
+        with open(path) as f:
+            source = f.read()
+            lines = source.splitlines()
+
+        if not lines:
+            lines = ['']
+        line = random.randint(1, len(lines))
+        column = random.randint(0, len(lines[line - 1]))
+        return cls(operation, path, line, column)
+
+    def run(self, debugger, record=None, print_result=False):
+        try:
+            with open(self.path) as f:
+                self.script = jedi.Script(f.read(), self.line, self.column, self.path)
+            self.objects = getattr(self.script, self.operation)()
+            if print_result:
+                print("{path}: Line {line} column {column}".format(**self.__dict__))
+                self.show_location(self.line, self.column)
+                self.show_operation()
+        except jedi.NotFoundError:
+            pass
+        except Exception:
+            self.traceback = traceback.format_exc()
+            if record is not None:
+                call_args = (self.operation, self.path, self.line, self.column, self.traceback)
+                with open(record, 'w') as f:
+                    json.dump(call_args, f)
+            self.show_errors()
+            if debugger:
+                einfo = sys.exc_info()
+                pdb = __import__(debugger)
+                if debugger == 'pudb':
+                    pdb.post_mortem(einfo[2], einfo[0], einfo[1])
+                else:
+                    pdb.post_mortem(einfo[2])
+            exit(1)
+
+    def show_location(self, lineno, column, show=3):
+        # Three lines ought to be enough
+        lower = lineno - show if lineno - show > 0 else 0
+        prefix = '  |'
+        for i, line in enumerate(self.script.source.split('\n')[lower:lineno]):
+            print(prefix, lower + i + 1, line)
+        print(prefix, '   ', ' ' * (column + len(str(lineno))), '^')
+
+    def show_operation(self):
+        print("%s:\n" % self.operation.capitalize())
+        if self.operation == 'completions':
+            self.show_completions()
+        else:
+            self.show_definitions()
+
+    def show_completions(self):
+        for completion in self.objects:
+            print(completion.name)
+
+    def show_definitions(self):
+        for completion in self.objects:
+            print(completion.desc_with_module)
+            if completion.module_path is None:
+                continue
+            if os.path.abspath(completion.module_path) == os.path.abspath(self.path):
+                self.show_location(completion.line, completion.column)
+
+    def show_errors(self):
+        print(self.traceback)
+        print(("Error with running Script(...).{operation}() with\n"
+              "\tpath:   {path}\n"
+              "\tline:   {line}\n"
+              "\tcolumn: {column}").format(**self.__dict__))
+
+
+def main(arguments):
+    debugger = 'pdb' if arguments['--pdb'] else \
+               'ipdb' if arguments['--ipdb'] else \
+               'pudb' if arguments['--pudb'] else None
+    record = arguments['--record']
+
+    jedi.settings.use_filesystem_cache = arguments['--fs-cache']
+    if arguments['--debug']:
+        jedi.set_debug_function()
+
+    if arguments['redo'] or arguments['show']:
+        t = TestCase.from_cache(record)
+        if arguments['show']:
+            t.show_errors()
+        else:
+            t.run(debugger)
+    elif arguments['run']:
+            TestCase(
+                arguments['<operation>'], arguments['<path>'],
+                int(arguments['<line>']), int(arguments['<column>'])
+            ).run(debugger, print_result=True)
+    else:
+        for _ in range(int(arguments['--maxtries'])):
+            t = TestCase.generate(arguments['<path>'] or '.')
+            if arguments['-s']:
+                print('%s %s %s %s ' % (t.operation, t.path, t.line, t.column))
+                sys.stdout.flush()
+            else:
+                print('.', end='')
+            t.run(debugger, record)
+
+            sys.stdout.flush()
+        print()
+
+
+if __name__ == '__main__':
+    arguments = docopt(__doc__)
+    main(arguments)

test/completion/arrays.py

@@ -11,6 +11,8 @@
 [1,""][2]
 #? int() str()
 [1,""][20]
+#? int() str()
+[1,""][str(hello)]
 
 a = list()
 #? list()
@@ -28,6 +30,9 @@ b = [6,7]
 #? int()
 b[8-7]
 
+# -----------------
+# Slices
+# -----------------
 #? list()
 b[8:]
 
@@ -35,6 +40,15 @@ b[8:]
 b[int():]
 
 
+class _StrangeSlice():
+    def __getitem__(self, slice):
+        return slice
+
+# Should not result in an error, just because the slice itself is returned.
+#? []
+_StrangeSlice()[1:2]
+
+
 # -----------------
 # iterable multiplication
 # -----------------
@@ -42,14 +56,6 @@ a = ['']*2
 #? list()
 a
 
-a = 2*2
-#? int()
-a
-
-a = "a"*3
-#? str()
-a
-
 # -----------------
 # tuple assignments
 # -----------------
@@ -94,6 +100,28 @@ a4
 b4
 
 
+# -----------------
+# multiple assignments
+# -----------------
+a = b = 1
+#? int()
+a
+#? int()
+b
+
+(a, b) = (c, (e, f)) = ('2', (3, 4))
+#? str()
+a
+#? tuple()
+b
+#? str()
+c
+#? int()
+e
+#? int()
+f
+
+
 # -----------------
 # unnessecary braces
 # -----------------
@@ -128,6 +156,12 @@ def a(): return ''
 #? int()
 (tuple)().index()
 
+class C():
+    def __init__(self):
+        self.a = (str()).upper()
+
+#? str()
+C().a
 
 # -----------------
 # imbalanced sides
@@ -186,6 +220,18 @@ f()
 #? 9 ['str']
 {str: str}
 
+# iteration problem (detected with sith)
+d = dict({'a':''})
+def y(a):
+    return a
+#?
+y(**d)
+
+# problem with more complicated casts
+dic = {str(key): ''}
+#? str()
+dic['']
+
 # -----------------
 # with variable as index
 # -----------------
@@ -230,6 +276,17 @@ class GetItem():
 #? str()
 GetItem("")[1]
 
+class GetItemWithList():
+    def __getitem__(self, index):
+        return [1, 1.0, 's'][index]
+
+#? float()
+GetItemWithList()[1]
+
+for i in 0, 2:
+    #? int() str()
+    GetItemWithList()[i]
+
 # -----------------
 # conversions
 # -----------------
@@ -240,7 +297,7 @@ list(a)[1]
 
 #? int() str()
 list(a)[0]
-#? 
+#?
 set(a)[0]
 
 #? int() str()
@@ -280,3 +337,31 @@ tuple((1,))[0]
 # implementation detail for lists, should not be visible
 #? []
 list().__iterable
+
+# With a list comprehension.
+for i in set(a for a in [1]):
+    #? int()
+    i
+
+
+# -----------------
+# Recursions
+# -----------------
+
+def to_list(iterable):
+    return list(set(iterable))
+
+
+def recursion1(foo):
+    return to_list(to_list(foo)) + recursion1(foo)
+
+#? int()
+recursion1([1,2])[0]
+
+# -----------------
+# Merged Arrays
+# -----------------
+
+for x in [1] + ['']:
+    #? int() str()
+    x

test/completion/basic.py

@@ -23,18 +23,18 @@ a(0):.
 # if/else/elif
 # -----------------
 
-if 1:
+if (random.choice([0, 1])):
     1
-elif(3):
+elif(random.choice([0, 1])):
     a = 3
 else:
     a = ''
 #? int() str()
 a
 def func():
-    if 1:
+    if random.choice([0, 1]):
         1
-    elif(3):
+    elif(random.choice([0, 1])):
         a = 3
     else:
         a = ''
@@ -43,6 +43,18 @@ def func():
 #? int() str()
 func()
 
+# -----------------
+# keywords
+# -----------------
+
+#? list()
+assert []
+
+def focus_return():
+    #? list()
+    return []
+
+
 # -----------------
 # for loops
 # -----------------
@@ -93,6 +105,13 @@ for i in b:
 #? float() str()
 a[0]
 
+for i in [1,2,3]:
+    #? int()
+    i
+else:
+    i
+
+
 # -----------------
 # range()
 # -----------------
@@ -100,61 +119,6 @@ for i in range(10):
     #? int()
     i
 
-# -----------------
-# list comprehensions
-# -----------------
-
-# basics:
-
-a = ['' for a in [1]]
-#? str()
-a[0]
-
-a = [a for a in [1]]
-#? int()
-a[0]
-
-a = [a for a in 1,2]
-#? int()
-a[0]
-
-a = [a for a,b in [(1,'')]]
-#? int()
-a[0]
-
-arr = [1,'']
-a = [a for a in arr]
-#? int() str()
-a[0]
-
-a = [a if 1.0 else '' for a in [1] if [1.0]]
-#? int() str()
-a[0]
-
-# -----------------
-# nested list comprehensions
-# -----------------
-
-b = [a for arr in [[1]] for a in arr]
-#? int()
-b[0]
-
-b = [a for arr in [[1]] if '' for a in arr if '']
-#? int()
-b[0]
-
-b = [b for arr in [[[1.0]]] for a in arr for b in a]
-#? float()
-b[0]
-
-# jedi issue #26
-#? list()
-a = [[int(v) for v in line.strip().split() if v] for line in ["123", "123", "123"] if line]
-#? list()
-a[0]
-#? int()
-a[0][0]
-
 # -----------------
 # ternary operator
 # -----------------
@@ -179,6 +143,9 @@ ret()[0]
 with open('') as f:
     #? ['closed']
     f.closed
+    for line in f:
+        #? str()
+        line
 
 with open('') as f1, open('') as f2:
     #? ['closed']
@@ -209,6 +176,9 @@ def a():
     """
     pass
 
+#?
+# str literals in comment """ upper
+
 # -----------------
 # magic methods
 # -----------------
@@ -220,3 +190,80 @@ class B(): pass
 A.__init__
 #? ['__init__']
 B.__init__
+
+#? ['__init__']
+int().__init__
+
+# -----------------
+# comments
+# -----------------
+
+class A():
+    def __init__(self):
+        self.hello = {}  # comment shouldn't be a string
+#? dict()
+A().hello
+
+# -----------------
+# unicode
+# -----------------
+a = 'smörbröd'
+#? str()
+a
+xyz = 'smörbröd.py'
+if 1:
+    #? str()
+    xyz
+
+# -----------------
+# exceptions
+# -----------------
+try:
+    import math
+except ImportError as i_a:
+    #? ['i_a']
+    i_a
+    #? ImportError()
+    i_a
+try:
+    import math
+except ImportError, i_b:
+    # TODO check this only in Python2
+    ##? ['i_b']
+    i_b
+    ##? ImportError()
+    i_b
+
+
+class MyException(Exception):
+    def __init__(self, my_attr):
+        self.my_attr = my_attr
+
+try:
+    raise MyException(1)
+except MyException as e:
+    #? ['my_attr']
+    e.my_attr
+    #? 22 ['my_attr']
+    for x in e.my_attr:
+        pass
+
+
+# -----------------
+# continuations
+# -----------------
+
+foo = \
+1
+#? int()
+foo
+
+# -----------------
+# module attributes
+# -----------------
+
+# Don't move this to imports.py, because there's a star import.
+#? str()
+__file__
+#? ['__file__']
+__file__

test/completion/classes.py

@@ -119,6 +119,23 @@ strs.second
 #? ['var_class']
 TestClass.var_class.var_class.var_class.var_class
 
+# operations (+, *, etc) shouldn't be InstanceElements - #246
+class A():
+    def __init__(self):
+        self.addition = 1 + 2
+#? int()
+A().addition
+
+# should also work before `=`
+#? 8 int()
+A().addition = None
+#? 8 int()
+A(1).addition = None
+a = A()
+#? 8 int()
+a.addition = None
+
+
 # -----------------
 # inheritance
 # -----------------
@@ -162,6 +179,34 @@ SubClass.var
 #? ['class_sub', 'class_super']
 SubClass.class_
 
+# -----------------
+# inheritance of builtins
+# -----------------
+
+class Base(str):
+    pass
+
+#? ['upper']
+Base.upper
+#? ['upper']
+Base().upper
+
+# -----------------
+# dynamic inheritance
+# -----------------
+
+class Angry(object):
+    def shout(self):
+        return 'THIS IS MALARKEY!'
+
+def classgetter():
+    return Angry
+
+class Dude(classgetter()):
+    def react(self):
+        #? ['shout']
+        self.s
+
 # -----------------
 # __call__
 # -----------------
@@ -241,192 +286,6 @@ class A():
 #? list()
 A().b()
 
-# -----------------
-# descriptors
-# -----------------
-class RevealAccess(object):
-    """
-    A data descriptor that sets and returns values
-    normally and prints a message logging their access.
-    """
-    def __init__(self, initval=None, name='var'):
-        self.val = initval
-        self.name = name
-
-    def __get__(self, obj, objtype):
-        print('Retrieving', self.name)
-        return self.val
-
-    def __set__(self, obj, val):
-        print('Updating', self.name)
-        self.val = val
-
-    def just_a_method(self):
-        pass
-
-class C(object):
-    x = RevealAccess(10, 'var "x"')
-    #? RevealAccess()
-    x
-    #? ['just_a_method']
-    x.just_a_method
-    y = 5.0
-    def __init__(self):
-        #? int()
-        self.x
-
-        #? []
-        self.just_a_method
-        #? []
-        C.just_a_method
-
-m = C()
-#? int()
-m.x
-#? float()
-m.y
-#? int()
-C.x
-
-#? []
-m.just_a_method
-#? []
-C.just_a_method
-
-# -----------------
-# properties
-# -----------------
-class B():
-    @property
-    def r(self):
-        return 1
-    @r.setter
-    def r(self, value):
-        return ''
-    def t(self):
-        return ''
-    p = property(t)
-
-#? []
-B().r()
-#? int()
-B().r
-
-#? str()
-B().p
-#? []
-B().p()
-
-class PropClass():
-    def __init__(self, a):
-        self.a = a
-    @property
-    def ret(self):
-        return self.a
-
-    @ret.setter
-    def ret(self, value):
-        return 1.0
-
-    def ret2(self):
-        return self.a
-    ret2 = property(ret2)
-
-    @property
-    def nested(self):
-        """ causes recusions in properties, should work """
-        return self.ret
-
-    @property
-    def nested2(self):
-        """ causes recusions in properties, should not work """
-        return self.nested2
-
-    @property
-    def join1(self):
-        """ mutual recusion """
-        return self.join2
-
-    @property
-    def join2(self):
-        """ mutual recusion """
-        return self.join1
-
-#? str()
-PropClass("").ret
-#? []
-PropClass().ret.
-
-#? str()
-PropClass("").ret2
-#? 
-PropClass().ret2
-
-#? int()
-PropClass(1).nested
-#? []
-PropClass().nested.
-
-#? 
-PropClass(1).nested2
-#? []
-PropClass().nested2.
-
-#? 
-PropClass(1).join1
-# -----------------
-# staticmethod/classmethod
-# -----------------
-
-class E(object):
-    a = ''
-    def __init__(self, a):
-        self.a = a
-
-    def f(x):
-        return x
-    f = staticmethod(f)
-
-    @staticmethod
-    def g(x):
-        return x
-
-    def s(cls, x):
-        return x
-    s = classmethod(s)
-
-    @classmethod
-    def t(cls, x):
-        return x
-
-    @classmethod
-    def u(cls, x):
-        return cls.a
-
-e = E(1)
-#? int()
-e.f(1)
-#? int()
-E.f(1)
-#? int()
-e.g(1)
-#? int()
-E.g(1)
-
-#? int()
-e.s(1)
-#? int()
-E.s(1)
-#? int()
-e.t(1)
-#? int()
-E.t(1)
-
-#? str()
-e.u(1)
-#? str()
-E.u(1)
-
 # -----------------
 # recursions
 # -----------------
@@ -435,10 +294,10 @@ def Recursion():
         self.a = self.a
         self.b = self.b.recurse()
 
-#? 
+#?
 Recursion().a
 
-#? 
+#?
 Recursion().b
 
 # -----------------
@@ -480,14 +339,16 @@ getattr(str(), 'upper')
 getattr(str, 'upper')
 
 # some strange getattr calls
-#? 
+#?
 getattr(str, 1)
-#? 
+#?
 getattr()
-#? 
+#?
 getattr(str)
-#? 
+#?
 getattr(getattr, 1)
+#?
+getattr(str, [])
 
 
 class Base():
@@ -511,6 +372,14 @@ Wrapper(Base()).ret(3)
 #? int()
 Wrapper2(Base()).ret(3)
 
+class GetattrArray():
+    def __getattr__(self, name):
+        return [1]
+
+#? int()
+GetattrArray().something[0]
+
+
 # -----------------
 # private vars
 # -----------------
@@ -519,9 +388,11 @@ class PrivateVar():
         self.__var = 1
         #? int()
         self.__var
+        #? ['__var']
+        self.__var
 #? []
 PrivateVar().__var
-#? 
+#?
 PrivateVar().__var
 
 # -----------------
@@ -529,9 +400,11 @@ PrivateVar().__var
 # -----------------
 class Super(object):
     a = 3
+    def return_sup(self):
+        return 1
 
 class TestSuper(Super):
-    #? 
+    #?
     super()
     def test(self):
         #? Super()
@@ -542,9 +415,16 @@ class TestSuper(Super):
             #? Super()
             super()
         def a():
-            #? 
+            #?
             super()
 
+    def return_sup(self):
+        #? int()
+        return super().return_sup()
+
+#? int()
+TestSuper().return_sup()
+
 
 # -----------------
 # if flow at class level
@@ -564,3 +444,32 @@ class TestX(object):
         var = self.conditional_method()
         #? int()
         var
+
+# -----------------
+# mro method
+# -----------------
+
+class A(object):
+    a = 3
+
+#? ['mro']
+A.mro
+#? []
+A().mro
+
+
+# -----------------
+# mro resolution
+# -----------------
+
+class B(A()):
+    b = 3
+
+#?
+B.a
+#?
+B().a
+#? int()
+B.b
+#? int()
+B().b