pyastinterp

This is an AST interpreter (also known as tree-walking interpreter) for the Python programming language, written in Python itself. It is thus a meta-circular interpreter. While Python implementations of a Python VM (i.e. bytecode interpreters) are relatively well known (starting with PyPy), AST interpreters are much less common beyond those which implement a very limited subset of the syntax.

Pyastinterp strives to fulfil 2 goals:

1. Be as complete as possible (and see how much effort that takes and if there're any limits).
2. Yet be as minimal as possible, avoiding useless boilerplate and reimplementing what may be already available.

First goal is pretty obvious: ideally, we'd like Pyastinterp to be able to interpret (run) any application a native Python interpreter can.

Second goal is rooted in many reasons. To start with, Pyastinterp is a spiritual part of the Pycopy project, whose goal is the development of minimalist Python-compatible ecosystem. Besides, "bloated" meta-circular interpreters are well out there (the same PyPy) - there's lack on the opposite side of spectrum. Pyastinterp is also intended to be a "teaching" and "for-research" project. Surely, it's much easier to learn and reuse/repurpose a small, rather than large, project. These last requirements however also put bounds on "minimality": pyastinterp tries to have a clear focus (AST interpretation) and avoid unneeded boilerplate, but not be obfuscated, and actually tries to be clear and easily hackable (provide some reusable infrastructure), even at the expense of somewhat less optimality/performance.

To achieve the goals of minimality, pyastinterp does following:

1. Is concerned with syntax, not semantics (largely).
2. That means that it tries to interpret Python AST tree, but relies on the underlying Python runtime to implement the actual operations (semantics). In some cases pyastinterp actually has to deal with runtime semantics, but the point is that it should be done only when there's no other choice (i.e. rarely).
3. This in particular means that pyastinterp itself requires a (pretty) full Python implementation to run, it's not the case that it's written in "some subset of Python". (Contrast that with e.g. PyPy's RPython, which is a Python subset language, in which interpreters are written).
4. Another consequence is that there's no semantic separation between "metalevel" (level of the interpreter) and target application level. This is again unlike PyPy, where 2 are explicitly separated. Lack of separation allows to save on "marshalling" values between the 2 levels, but also make it possible to have "leaks" between levels, e.g. unexpected and unhandled exception in the interpreter to be delivered to the application, causing havoc and confusion. Pyastinterp's response to this concern is a pledge that there shouldn't be such unhandled interpreter-level exceptions. Of course, catching all such cases may take quite a lot of testing and debugging, and fixing them may affect code minimality and clarity. We'll see how it goes.
5. However, due to choices described above, implementation of many syntactic constructs is very clear, concise, and isomorphic between the target and meta levels. See e.g. how "if" statement is implemented: the implementation looks almost exactly how the usage of the "if" statement is.

Supported Python subset

Pyastinterp should implement almost all Python3.5 features, stopping at generators, which are not that easy to implement in a tree-walking interpreter. It also does not include import hook, so the imported modules are interpreted by the underlying Python VM, not by Pyastinterp.

Running

Quick way to run something interesting:

wget https://raw.githubusercontent.com/python/cpython/3.6/Lib/test/pystone.py
python3 -m astinterp pystone.my 10000

You can pass any other script on the command line and see how it goes.

Credits and licensing

Pyastinterp is (c) Paul Sokolovsky and is released under the MIT license.