Better Package Management for Python Libraries

This post adds guidance for python libraries to the fantastic write-up, Better Package Management .

As a maintainer of a Python library, you can break down managing dependencies into 3 concerns.

How do I specify required and optional dependencies for consumers of my library? How do I specify dependencies required for contributors of my library? How do I make sure my builds are predictable and deterministic ?

The answer to all three of these concerns lies in the proper usage of pip-tools .

Specifying dependencies

For Python applications that install your library, you'll need to specify the library's depencies in setup.py -

from setuptools import setup setup( name='mypackage', install_requires=[ 'tornado>=4.2,<5', 'thriftrw>=1.1,<2', ], extras_require={ 'thisoptionalfeature': [ 'threadloop>=1,2', ], } )

Keep version constraints in this file as loose as possible, eg tornado>=4.2<5 instead of tornado==4.2.3 ; a good rule of thumb is to use Semantic Versioning , locking to the latest major release, aka mydep>=1,<2 .

For library maintainers

For contributors/maintainers who are actually working on the Python library - you'll want to provide additional dependencies to enable development. Extend the deps specified in setup.py in a requirements.in file -

# extend deps in setup.py -e . -e .[thisoptionalfeature] # debugging ipdb ipython # linting flake8 # releasing wheel zest.releaser

Now we can compile the requirements.in into a requirements.txt file like so -

(env) $ pip install pip-tools (env) $ pip-compile

The above gives us the following fully-pinned requirements.txt -

appnope==0.1.0 # via ipython backports-abc==0.4 # via tornado backports.ssl-match-hostname==3.5.0.1 # via tornado certifi==2016.2.28 # via tornado colorama==0.3.7 # via zest.releaser decorator==4.0.9 # via ipython, traitlets flake8==2.5.4 gnureadline==6.3.3 # via ipython ipdb==0.9.0 ipython-genutils==0.1.0 # via traitlets ipython==4.1.2 mccabe==0.4.0 # via flake8 path.py==8.1.2 # via pickleshare pep8==1.7.0 # via flake8 pexpect==4.0.1 # via ipython pickleshare==0.6 # via ipython pkginfo==1.2.1 # via twine ply==3.8 # via thriftrw ptyprocess==0.5.1 # via pexpect pyflakes==1.0.0 # via flake8 requests-toolbelt==0.6.0 # via twine requests==2.9.1 # via requests-toolbelt, twine simplegeneric==0.8.1 # via ipython singledispatch==3.4.0.3 # via tornado six==1.10.0 # via singledispatch, thriftrw, zest.releaser threadloop==1.0.2 thriftrw==1.2.4 tornado==4.3 # via threadloop traitlets==4.2.1 # via ipython twine==1.6.5 # via zest.releaser wheel==0.29.0 zest.releaser==6.6.4

This can be used with pip install -r requirements.txt , or with pip-sync as detailed in the following section.

Installing dependencies

Now that we have our compiled requirements.txt , we can make sure our virtualenv matches it using pip-sync -

(env) $ pip-sync

This makes sure our virtualenv contains exactly the dependencies listed in requirements.txt , and guarantees a deterministic build. Let's be nice and write a target in our Makefile as a peace-offering to future contributors -

.PHONY: install install: pip install pip-tools pip-sync python setup.py develop

Which should be ran from a virtualenv like so -

$ virtualenv env $ source env/bin/activate (env) $ make install

Cheers.