How Mypy could simplify compiling Python

It’s the dream of every professional python programmer: Take an existing Python application, run it through a compiler, and generate high-speed, platform-native code that respects Python’s dynamic nature.

In theory, that’s possible to do right now―sort of. The problem is, every available path isfraught with limitations. You have to modify the source in non-standard ways (Cython), or use a replacement runtime that’s many times larger than the regular one and has its own limitations (PyPy), or work with tools that are still enormously unstable and experimental (Nuitka).

Now the development team forMypy, the optional static-typing syntax for Python that’s become a standard-issue part of the language , is mulling the possibility of using Mypy as a way to generate code that can be compiled to a binary.

In the vibrant discussion that has sprung up around the idea, one thing has been underscored: Tools for compiling Python need to make Python code faster without also making things more difficult for developers.

Theidea of using Mypy as a step toward a statically compiled version of Python has been kicking aroundever since the syntax arose―all the more so since Mypy was accepted as a Python standard.

Python creator Guido van Rossum haspreviously stated thatintroducing Mypy into Python wasn’t intended as a prelude to making Python statically typed.One of Python’s biggest boons has always been its dynamism; anything that makes Python less dynamic by default isn’t much of a win.

“The basic philosophy of the language is not going to change,” said van Rossum. Mypy is a way forcode linters and checkers to better guarantee the quality of Python code,while also preserving the dynamism.

That said, van Rossumisn’t opposed to the idea of implementations of Python that are statically compiled. To that end, Mypy’s syntax could be used as the basis for an intermediate representation or transpiled version of Python that’s fed to a compiler. Python as a language would remain dynamic; it would just has a new route to become static. (van Rossum has also given the thumbs-up to exploring ways Mypy could aid with static compilation.)

The discussion within the Mypy team is still in early stages, but a few key points have emerged in the debate. And if this were to be done, Mypy’s developers argue, what kind of code would be produced?

One method involves what has been described as a PyIR ―an intermediate representation for Python code, or IR, that takes inspiration from the WebAssembly project.Another possibility, which potentially involves far less reinventing from scratch, would be to leverage the Cython project.

Cython allows existing Python code to be compiled to C, often with dramatic speed-ups, by annotating Python code with markup to indicate static variable typing. But Cython’s markup doesn’t look anything like Mypy―and not much like C, either; it’s an idiosyncractic sub-dialect of Python. Still, it isn’t much of a stretch to imaginehow Mypy-annotated Python code could be converted to Cython (and thus to C).

As elegant as that solutionsounds, it’s also fraught with drawbacks. For one, some of the high-level type annotations in Mypy don’t map precisely to some close-to-the-machine Cython types. Developers would either have to add more type annotations or the compiler would have to make contextual guesses about typing.

The GitHub discussion also exposed the problem of being forced to choose between types that aid programmer productivity (Python’s forte) or typing that’s useful to the compiler (C’s specialty). Cython is mainly used to optimize individual modules and functions where speed is critical―so-called “hot spots”―rather than entire programs, so developers only need to provide type annotation to theparts of the app that need it. By contrast, the Mypy discussion takes the more holistic idea of feeding the compiler the entire Python source tree, and getting back a single, unified product.

If that’s the larger goal with a static compiler for Python, the real problem may not be with static compilation, but rather with how static compilation can be part of an end-to-end toolchain for the language. Toolchains in languages like C/C++―as well as derivatives like Rust, Go, and Nim―produce a binary without the need for an external runtime.Python has long been an interpreted language, and so has no native tradition of such things, apart from third-party packaging ( not compilation) tools like PyInstaller.

If Python gained such tools, it isn’t likely they would come from the core language development process, since the core language is first and foremost a dynamic, interpreted one. But there’s clearly interest in having more toolchains for Python that reliably produce platform-native executables―or at the very least lower-level code that can be compiled into same.

As commenter wtpayne said on the GitHub discussion, “I’m interested in building a *smooth* and *continuous* conveyor belt that takes ideas (algorithms) from inception to production.”