Packaging applications to install on other machines with Python

In my last article in this series, I showed how to write a script in Python that returned a list of RPM-installed software installed on a machine. The output looks like this:

$ rpmqa_simple.py --limit 20
linux-firmware-20210818: 395,099,476
code-1.61.2: 303,882,220
brave-browser-1.31.87: 293,857,731
libreoffice-core-7.0.6.2: 287,370,064
thunderbird-91.1.0: 271,239,962
firefox-92.0: 266,349,777
glibc-all-langpacks-2.32: 227,552,812
mysql-workbench-community-8.0.23: 190,641,403
java-11-openjdk-headless-11.0.13.0.8: 179,469,639
iwl7260-firmware-25.30.13.0: 148,167,043
docker-ce-cli-20.10.10: 145,890,250
google-noto-sans-cjk-ttc-fonts-20190416: 136,611,853
containerd.io-1.4.11: 113,368,911
ansible-2.9.25: 101,106,247
docker-ce-20.10.10: 100,134,880
ibus-1.5.23: 90,840,441
llvm-libs-11.0.0: 87,593,600
gcc-10.3.1: 84,899,923
cldr-emoji-annotation-38: 80,832,870
kernel-core-5.14.12: 79,447,964

Now I want to package an application so that I can install it easily, including all the dependencies, on other machines. In this article, I'll show how to use the setuptools package to do that.

That's a lot to cover, so basic knowledge of Python is required. Even if you don't know much, the code is simple to follow, and the boilerplate code is small.

Set up your environment

My previous article explained how to install the code and use virtual environments, but here is a shortcut:

$ sudo dnf install -y python3-rpm
$ git clone git@github.com:josevnz/tutorials.git
$ cd rpm_query
$ python3 -m venv --system-site-packages ~/virtualenv/rpm_query
. ~/virtualenv/rpm_query/bin/activate
(rpm_query)$ 

Package and install the distribution

The next step is to package the application. However, I won't use RPM for that.

Why not use an RPM to package the Python application?

Well, there is no short answer to that.

RPM is great if you want to share your application with all the users of your system, especially because RPM can automatically install related dependencies.

For example, the RPM Python bindings (rpm-python3) are distributed this way, which makes sense as it is thinly tied to RPM.

In some cases, it is also a disadvantage:

• Users need root elevated access to install an RPM. If the code is malicious, it will take control of the server very easily. (That's why you always check the RPM signatures and download code from well-known sources, right?)
• Attempting to upgrade to an RPM that is incompatible with older dependent applications fails.
• RPM is not well suited to share "test" code created during continuous integration, at least in bare-metal deployments. With a container, that is probably a different story.
• If the Python code has dependencies, they will likely have to be packaged as RPMs.

Use virtual environments and pip + setuptools

Using virtual environments, pip, and setup tools solves these RPM limitations because:

• Virtual environments allow installing applications without elevated permissions.
• The application is self-contained in the virtual environment. Administrators can install different versions of the libraries without affecting the whole system.
• It is very easy to integrate a virtual environment with continuous integration and unit testing. After the tests pass, the environment can be recycled.
• setuptools solves the problem of packaging your application in a nice directory structure for making scripts and libraries available to users.
• setuptools also deals with keeping track of dependencies by using proper version checks to make the build process repeatable.
• setuptools works with pip, the Python package manager.
• Both virtual environments and setuptools have excellent support in IDEs like PyCharm and VS Code.

[ Decrease the complexity of getting into the cloud by downloading the Hybrid cloud strategy for dummies eBook. ]

Work with setuptools

Once you're ready to deploy the application, you can package it, copy its wheel file, and then install it in a new virtual environment. First, define a very important file, setup.py, which is used by setuptools.

The most important sections in the file below are:

• *_requires sections: Build and installation dependencies
• packages: The location of Python classes
• scripts: The scripts that the end user calls to interact with the libraries (if any)

"""
Project packaging and deployment
More details: https://setuptools.pypa.io/en/latest/userguide/quickstart.html
"""
import os
from setuptools import setup
from reporter import __version__

def __read__(file_name):
    return open(os.path.join(os.path.dirname(__file__), file_name)).read()

setup(
    name="rpm_query",
    version=__version__,
    author="Jose Vicente Nunez Zuleta",
    author_email="kodegeek.com@protonmail.com",
    description=__doc__,
    license="Apache",
    keywords="rpm query",
    url="https://github.com/josevnz/rpm_query",
    packages=[
        'reporter'
    ],
    long_description=__read__('README.md'),
    # https://pypi.org/pypi?%3Aaction=list_classifiers
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Topic :: Utilities",
        "Environment :: X11 Applications",
        "Intended Audience :: System Administrators"
        "License :: OSI Approved :: Apache Software License"
    ],
    setup_requires=[
        "setuptools==49.1.3",
        "wheel==0.37.0",
        "rich==9.5.1",
        "dearpygui==1.0.2"
    ],
    install_requires=[
        "rich==9.5.1",
    ],
    scripts=[
        "bin/rpmq_simple.py",
    ]
)

Things to note:

• I stored the version in the reporter/__init__.py package module to share with other parts of the application, not just setuptools. I also used a semantic version schema naming convention.
• The module's README is also stored as an external file. This makes editing the file much easier without worrying about size or breaking the Python syntax.
• Classifiers make it easier to see the application's intent.
• I can define packaging dependencies (setup_requires) and runtime dependencies (install_requires).
• I need the wheel package to create a precompiled distribution that is faster to install than other modes.

The pyproject.toml file is also very important. Here is the file:

[build-system]
requires = ["setuptools", "wheel"]

The pyproject.toml file specifies what is used for packaging the scripts and installing from source.

Quick check before uploading

Before you upload the wheel, you should ask Twine to check your settings for errors, like this:

(rpm_query) [josevnz@dmaf5 rpm_query]$ twine check dist/rpm_query-0.0.1-py3-none-any.whl 
Checking dist/rpm_query-0.0.1-py3-none-any.whl: FAILED
  `long_description` has syntax errors in markup and would not be rendered on PyPI.
    line 20: Warning: Bullet list ends without a blank line; unexpected unindent.
  warning: `long_description_content_type` missing. defaulting to `text/x-rst`.

The Markdown is incorrect in the file. One way to fix this issue is to install:

$ pip install readme_renderer[md]

Also, the long_description_content_type section is there:

    long_description_content_type="text/markdown",
    long_description=__read__('README.md'),

If you rerun it after making the changes above, you still see the warning:

rpm_query) [josevnz@dmaf5 rpm_query]$ twine check dist/rpm_query-0.0.1-py3-none-any.whl 
Checking dist/rpm_query-0.0.1-py3-none-any.whl: PASSED, with warnings
  warning: `long_description_content_type` missing. defaulting to `text/x-rst`.

No serious errors and one false alarm. You are ready to upload your wheel.

How to deploy while testing

You don't need to package and deploy the application in full mode. This is because setuptools has a very convenient develop mode that installs dependencies and allows editing the code while testing it:

(rpm_query)$ python setup.py develop

This mode creates special symbolic links that put the scripts (remember that scripts: section inside setup.py?) into your path.

By the way, once you are finished testing, you can remove development mode:

(rpm_query)$ python setup.py develop --uninstall

The official documentation recommends migrating from a setup.py configuration to setup.cfg. I decided to use setup.py because it is still the most popular format.

Create a precompiled distribution

Compiling is as simple as typing this:

(rpm_query)$ python setup.py bdist_wheel
running bdist_wheel
...  # Omitted output
(rpm_query)$ ls dist/
rpm_query-0.0.1-py3-none-any.whl

Then you can install it on the same machine or a new virtual machine:

(rpm_query)$ python setup.py install \
dist/rpm_query-1.0.0-py3-none-any.whl

Or with the new preferred way, using build. First make sure the module is installed:

(rpm_query) $ pip install build
Collecting build
  Downloading build-0.7.0-py3-none-any.whl (16 kB)
Collecting tomli>=1.0.0
  Downloading tomli-1.2.2-py3-none-any.whl (12 kB)
Requirement already satisfied: packaging>=19.0 in /usr/lib/python3.9/site-packages (from build) (20.4)
Collecting pep517>=0.9.1
  Downloading pep517-0.12.0-py2.py3-none-any.whl (19 kB)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/lib/python3.9/site-packages (from packaging>=19.0->build) (2.4.7)
Requirement already satisfied: six in /usr/lib/python3.9/site-packages (from packaging>=19.0->build) (1.15.0)
Installing collected packages: tomli, pep517, build
Successfully installed build-0.7.0 pep517-0.12.0 tomli-1.2.2

And then you can package your module like this (note that we do not tell build to use a virtual environment because we are already in one):

(rpm_query) $ python3 -m build --no-isolation
* Getting dependencies for wheel...
* Building wheel...
running bdist_wheel
running build
running build_scripts
# ... Omitted output 
whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'rpm_query-0.1.0.data/scripts/rpmq_dearpygui.py'
adding 'rpm_query-0.1.0.data/scripts/rpmq_rich.py'
adding 'rpm_query-0.1.0.data/scripts/rpmq_simple.py'
adding 'rpm_query-0.1.0.data/scripts/rpmq_tkinter.py'
adding 'rpm_query-0.1.0.dist-info/LICENSE.txt'
adding 'rpm_query-0.1.0.dist-info/METADATA'
adding 'rpm_query-0.1.0.dist-info/WHEEL'
adding 'rpm_query-0.1.0.dist-info/top_level.txt'
adding 'rpm_query-0.1.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built rpm_query-0.1.0-py3-none-any.whl

What if you want to share the .pip file with other users? I could copy the wheel file to the different machines and have the users install it, but there is a better way.

Set up a private PyPI server

Note: This setup is not production quality because:

• It is not secure because it uses passwords instead of tokens for authentication.
• There's no SSL encryption. HTTP sends clear-text passwords over the wire.
• There's no storage redundancy. Ideally, the pip storage should have some sort of redundancy and backups.

You can do a lot more than just installing from a wheel file. For example, you can set up a private server compatible with PyPI using a container running pypiserver.

[ Download the datasheet for information about Red Hat OpenShift Container Platform. ]

First, create a directory to store the packages:

$ mkdir -p -v $HOME/pypiserver
$ mkdir: created directory '/home/josevnz/pypiserver'

Then use htpasswd to set up a user and password to upload the packages:

htpasswd -c $HOME/.htpasswd josevnz
New password: 
Re-type new password: 
Adding password for user josevnz

After that, run the container in a detached mode:

$ docker run --detach --name privatepypiserver --publish 8080:8080 --volume ~/.htpasswd:/data/.htpasswd --volume $HOME/pypiserver:/data/packages pypiserver/pypiserver:latest -P .htpasswd --overwrite packages
f95f59a882b639db4509081de19a670fa8fdd93c63c3d4562c89e49e70bf6ee5
$ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED         STATUS         PORTS                                       NAMES
f95f59a882b6   pypiserver/pypiserver:latest   "/entrypoint.sh -P .…"   7 seconds ago   Up 6 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   privatepypiserver

Confirm it is running by pointing curl or lynx to the new privatepypiserver:

[josevnz@dmaf5 ~]$ lynx http://localhost:8080                                                                                                                                            Welcome to pypiserver!
                                                                         Welcome to pypiserver!

   This is a PyPI compatible package index serving 0 packages.

   To use this server with pip, run the following command:
        pip install --index-url http://localhost:8080/simple/ PACKAGE [PACKAGE2...]

   To use this server with easy_install, run the following command:
        easy_install --index-url http://localhost:8080/simple/ PACKAGE [PACKAGE2...]

   The complete list of all packages can be found here or via the simple index.

   This instance is running version 1.4.2 of the pypiserver software.

Next, upload the wheel to your private PyPI server.

Upload the application to a repository with Twine

The most common way to share Python code is to upload it to an artifact manager like Sonatype Nexus or pypiserver using a tool like Twine.

(rpm_query)$ pip install twine

The next step is to set up ~/.pypirc to allow passwordless uploads to the local PyPI server:

[distutils]
index-servers =
    pypi
    privatepypi

[pypi]
repository = https://upload.pypi.org/legacy/

[privatepypi]
repository = http://localhost:8080/
username = josevnz

It's best to not put the password = XXXX inside the file. Let Twine ask for it instead. Also, make the configuration accessible only to the owner:

$ chmod 600 ~/.pypirc

Finally, upload the wheel using the twine command:

(rpm_query) twine upload -r privatepypi dist/rpm_query-0.0.1-py3-none-any.whl 
Uploading distributions to http://localhost:8080/
Uploading rpm_query-0.0.1-py3-none-any.whl
100%|██████████████████████████████████

Confirm it was installed with lynx http://localhost:8080/packages/:

                                                                           Index of packages

   rpm_query-0.0.1-py3-none-any.whl

Commands: Use arrow keys to move, '?' for help, 'q' to quit, '<-' to go back.
  Arrow keys: Up and Down to move.  Right to follow a link; Left to go back.
 H)elp O)ptions P)rint G)o M)ain screen Q)uit /=search [delete]=history list

Install from the local privatepypi server

Wait, don't leave yet. It is time to install the package from your private PyPI server:

First, tell pip to look for packages on the private PyPI server:

$ mkdir --verbose --parents ~/.pip
$ cat<<PIPCONF>~/.pip/.pip.conf
[global]
extra-index-url = http://localhost:8080/simple/
trusted-host = http://localhost:8080/simple/
PIPCONF

To prove this works well, install it to a different virtual environment (or you can override any previous installation with pip install --force):

$ python3 -m venv ~/virtualenv/test2
$ . ~/virtualenv/test2/bin/activate
(test2)$ pip install --index-url http://localhost:8080/simple/ rpm_query
Looking in indexes: http://localhost:8080/simple/
Collecting rpm_query
  Downloading http://localhost:8080/packages/rpm_query-0.0.1-py3-none-any.whl (12 kB)
Collecting rich==9.5.1
  Using cached rich-9.5.1-py3-none-any.whl (180 kB)
Collecting pygments<3.0.0,>=2.6.0
  Downloading Pygments-2.10.0-py3-none-any.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 5.4 MB/s            
Collecting colorama<0.5.0,>=0.4.0
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting typing-extensions<4.0.0,>=3.7.4
  Downloading typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Collecting commonmark<0.10.0,>=0.9.0
  Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 5.8 MB/s             
Installing collected packages: typing-extensions, pygments, commonmark, colorama, rich, rpm-query
Successfully installed colorama-0.4.4 commonmark-0.9.1 pygments-2.10.0 rich-9.5.1 rpm-query-0.0.1 typing-extensions-3.10.0.2

What you've learned

I provided a lot of information here. Here's what I covered:

• Package the application with setuptools.
• Run a private PyPI server using a container.
• Upload the generated wheel package to the private repository using Twine.
• Install the wheel from the private repository instead of a file using pip.

In my next article in this series, I'll explore writing user interface applications in Python.