Using Private Packages in Python

With companies moving to microservices, it's becoming common to have a system scattered across multiple repositories. It's frequent to abstract common patterns and code into private repositories that are then included in each service.

But using packages from private repos with Python can be tricky. This guide will walk you through the necessary steps to correctly use private packages with pip or pipenv. We have also built a little app that will generate the right incantations for you to copy and paste.

Figuring out the URL

First, you need to choose which format you want to use: tarball (ie: .zip) or vcs (most likely git). I prefer just downloading the tarball because it doesn't require any additional vcs software. But if use pipenv, you'll have to use vcs (see below).

Authentication

In order to have access to the private repo or tarball, you'll need an authentication token. GitHub and other providers allow you to create a private access token for your user, but because we are going to hardcode such token in code, I strongly recommend to create a 'bot' user and use its access token.

Github: access token at https://github.com/settings/tokens. You can select just the repo scopes.
Githost: deploy token at https://<myorg>.githost.io/<org>/<project>/settings/repository. You can just select read_repository scope.
Gitlab: deploy token at https://gitlab.com/profile/<org>/<project>/settings/repository. You can just select read_repository scope.
Bitbucket: access token at https://bitbucket.org/account/user/<botusername>/app-passwords. You can just select repositories/read scope.

Versioning

You will also need to decide which version of your package you want to install. We'll call this the ref. You could target a branch, a commit, or a tag/version. I prefer to use version because it gives me a human-readable point of reference.

Building the url

Once you have your access token and your ref, the next step is figuring out a URL at which pip can download or clone the code.

Github

Tarball URL will be https://<access_token>@github.com/<myorg>/<myproject>/archive/<ref>.zip
VCS Tarball will be git+https://<access_token>@github.com/<myorg>/<myproject>.git@<ref>

Githost

Tarball URL will be https://<myorg>.githost.io/<myteam>/<myproject>/repository/archive.zip?private_token=<access_token>&ref=<ref>
VCS Tarball will be git+https://<token_username>:<deploy_token@<myorg>.githost.io/<myteam>/<myproject>.git@<ref>

Gitlab

Tarball URL will be https://gitlab.com/<myteam>/<myproject>/repository/archive.zip?private_token=<access_token>&ref=<ref>
VCS Tarball will be git+https://<token_username>:<deploy_token>@gitlab.com/<myorg>/<myproject>.git@<ref>

Bitbucket

Tarball URL will be https://<botusername>:<access_token>@bitbucket.org/<myorg>/<myproject>/get/<ref>.zip
VCS Tarball will be git+https://<botusername>:<access_token>@bitbucket.org/<myorg>/<myproject>.git@<ref>

Specifying the dependency

`pip` requirements file

Once you have the URL, you can just add it to you requirements files, or use with pip install:

$ pip install "<URL>"  # You'll need the quotes to escape the URL

`Pipfile`

To add the URL to you Pipfile, use this syntax:

[packages]
revsys-teams = {file = "<URL>"}

Private dependency of a private package

If you need to specify your private package as a dependency of another package, you'll have to add the URL in your the dependency's setup.py. Note that the exact syntax of the specification is a little tricky:

import os
from setuptools import setup, find_packages

setup(
    name='upstreamprivatepackage',
    install_requires=[
        # The 'version' is required. Doesn't have to be an actual version of the package.
        # but it must match what's in the `egg` fragment of its entry in `dependency_links`.
        "<myproject>==0.0.1",
    ],
    dependency_links=[
        # there must be a version identifier in the `egg=` fragment and it must match what's
        # in `install_requires`
        '<url>#egg=<myproject>-0.0.1',
    ],
)

When installing the upstream package, you'll also have to tell pip to actually parse the entries in dependency_links:

# `pip` has to be run with the `PIP_PROCESS_DEPENDENCY_LINKS` env var or with `--process-dependency-links` option.
# As of v9.0.1, `pip` will print a big red scary deprecation warning, but that's the only way.
PIP_PROCESS_DEPENDENCY_LINKS=1 pip install ...

# or
pip install ... --process-dependency-links

Check This Out – Using Private Packages in Python