1// -*- mode:doc; -*- 2// vim: set syntax=asciidoc: 3 4=== Infrastructure for Python packages 5 6This infrastructure applies to Python packages that use the standard 7Python setuptools, pep517, flit or maturin mechanisms as their build 8system, generally recognizable by the usage of a +setup.py+ script or 9+pyproject.toml+ file. 10 11[[python-package-tutorial]] 12 13==== +python-package+ tutorial 14 15First, let's see how to write a +.mk+ file for a Python package, 16with an example : 17 18------------------------ 1901: ################################################################################ 2002: # 2103: # python-foo 2204: # 2305: ################################################################################ 2406: 2507: PYTHON_FOO_VERSION = 1.0 2608: PYTHON_FOO_SOURCE = python-foo-$(PYTHON_FOO_VERSION).tar.xz 2709: PYTHON_FOO_SITE = http://www.foosoftware.org/download 2810: PYTHON_FOO_LICENSE = BSD-3-Clause 2911: PYTHON_FOO_LICENSE_FILES = LICENSE 3012: PYTHON_FOO_ENV = SOME_VAR=1 3113: PYTHON_FOO_DEPENDENCIES = libmad 3214: PYTHON_FOO_SETUP_TYPE = setuptools 3315: 3416: $(eval $(python-package)) 35------------------------ 36 37On line 7, we declare the version of the package. 38 39On line 8 and 9, we declare the name of the tarball (xz-ed tarball 40recommended) and the location of the tarball on the Web. Buildroot 41will automatically download the tarball from this location. 42 43On line 10 and 11, we give licensing details about the package (its 44license on line 10, and the file containing the license text on line 4511). 46 47On line 12, we tell Buildroot to pass custom options to the Python 48+setup.py+ script when it is configuring the package. 49 50On line 13, we declare our dependencies, so that they are built 51before the build process of our package starts. 52 53On line 14, we declare the specific Python build system being used. In 54this case the +setuptools+ Python build system is used. The five 55supported ones are +flit+, +pep517+, +setuptools+, +setuptools-rust+ 56and +maturin+. 57 58Finally, on line 16, we invoke the +python-package+ macro that 59generates all the Makefile rules that actually allow the package to be 60built. 61 62[[python-package-reference]] 63 64==== +python-package+ reference 65 66As a policy, packages that merely provide Python modules should all be 67named +python-<something>+ in Buildroot. Other packages that use the 68Python build system, but are not Python modules, can freely choose 69their name (existing examples in Buildroot are +scons+ and 70+supervisor+). 71 72The main macro of the Python package infrastructure is 73+python-package+. It is similar to the +generic-package+ macro. It is 74also possible to create Python host packages with the 75+host-python-package+ macro. 76 77Just like the generic infrastructure, the Python infrastructure works 78by defining a number of variables before calling the +python-package+ 79or +host-python-package+ macros. 80 81All the package metadata information variables that exist in the 82xref:generic-package-reference[generic package infrastructure] also 83exist in the Python infrastructure: +PYTHON_FOO_VERSION+, 84+PYTHON_FOO_SOURCE+, +PYTHON_FOO_PATCH+, +PYTHON_FOO_SITE+, 85+PYTHON_FOO_SUBDIR+, +PYTHON_FOO_DEPENDENCIES+, +PYTHON_FOO_LICENSE+, 86+PYTHON_FOO_LICENSE_FILES+, +PYTHON_FOO_INSTALL_STAGING+, etc. 87 88Note that: 89 90 * It is not necessary to add +python+ or +host-python+ in the 91 +PYTHON_FOO_DEPENDENCIES+ variable of a package, since these basic 92 dependencies are automatically added as needed by the Python 93 package infrastructure. 94 95 * Similarly, it is not needed to add +host-python-setuptools+ to 96 +PYTHON_FOO_DEPENDENCIES+ for setuptools-based packages, since it's 97 automatically added by the Python infrastructure as needed. 98 99One variable specific to the Python infrastructure is mandatory: 100 101* +PYTHON_FOO_SETUP_TYPE+, to define which Python build system is used 102 by the package. The five supported values are +flit+, +pep517+ and 103 +setuptools+, +setuptools-rust+ and +maturin+. If you don't know 104 which one is used in your package, look at the +setup.py+ or 105 +pyproject.toml+ file in your package source code, and see whether 106 it imports things from the +flit+ module or the +setuptools+ 107 module. If the package is using a +pyproject.toml+ file without any 108 build-system requires and with a local in-tree backend-path one 109 should use +pep517+. 110 111A few additional variables, specific to the Python infrastructure, can 112optionally be defined, depending on the package's needs. Many of them 113are only useful in very specific cases, typical packages will 114therefore only use a few of them, or none. 115 116* +PYTHON_FOO_SUBDIR+ may contain the name of a subdirectory inside the 117 package that contains the main +setup.py+ or +pyproject.toml+ file. 118 This is useful, if for example, the main +setup.py+ or +pyproject.toml+ 119 file is not at the root of the tree extracted by the tarball. If 120 +HOST_PYTHON_FOO_SUBDIR+ is not specified, it defaults to 121 +PYTHON_FOO_SUBDIR+. 122 123* +PYTHON_FOO_ENV+, to specify additional environment variables to 124 pass to the Python +setup.py+ script (for setuptools packages) or 125 the +support/scripts/pyinstaller.py+ script (for flit/pep517 126 packages) for both the build and install steps. Note that the 127 infrastructure is automatically passing several standard variables, 128 defined in +PKG_PYTHON_SETUPTOOLS_ENV+ (for setuptools target 129 packages), +HOST_PKG_PYTHON_SETUPTOOLS_ENV+ (for setuptools host 130 packages), +PKG_PYTHON_PEP517_ENV+ (for flit/pep517 target packages) 131 and +HOST_PKG_PYTHON_PEP517_ENV+ (for flit/pep517 host packages). 132 133* +PYTHON_FOO_BUILD_OPTS+, to specify additional options to pass to 134 the Python +setup.py+ script during the build step, this generally 135 only makes sense to use for setuptools based packages as flit/pep517 136 based packages do not pass these options to a +setup.py+ script but 137 instead pass them to +support/scripts/pyinstaller.py+. 138 139* +PYTHON_FOO_INSTALL_TARGET_OPTS+, +PYTHON_FOO_INSTALL_STAGING_OPTS+, 140 +HOST_PYTHON_FOO_INSTALL_OPTS+ to specify additional options to pass 141 to the Python +setup.py+ script (for setuptools packages) or 142 +support/scripts/pyinstaller.py+ (for flit/pep517 packages) during 143 the target installation step, the staging installation step or the 144 host installation, respectively. 145 146With the Python infrastructure, all the steps required to build and 147install the packages are already defined, and they generally work well 148for most Python-based packages. However, when required, it is still 149possible to customize what is done in any particular step: 150 151* By adding a post-operation hook (after extract, patch, configure, 152 build or install). See xref:hooks[] for details. 153 154* By overriding one of the steps. For example, even if the Python 155 infrastructure is used, if the package +.mk+ file defines its own 156 +PYTHON_FOO_BUILD_CMDS+ variable, it will be used instead of the 157 default Python one. However, using this method should be restricted 158 to very specific cases. Do not use it in the general case. 159 160[[scanpypi]] 161 162==== Generating a +python-package+ from a PyPI repository 163 164If the Python package for which you would like to create a Buildroot 165package is available on PyPI, you may want to use the +scanpypi+ tool 166located in +utils/+ to automate the process. 167 168You can find the list of existing PyPI packages 169https://pypi.python.org[here]. 170 171+scanpypi+ requires Python's +setuptools+ package to be installed on 172your host. 173 174When at the root of your buildroot directory just do : 175 176----------------------- 177utils/scanpypi foo bar -o package 178----------------------- 179 180This will generate packages +python-foo+ and +python-bar+ in the package 181folder if they exist on https://pypi.python.org. 182 183Find the +external python modules+ menu and insert your package inside. 184Keep in mind that the items inside a menu should be in alphabetical order. 185 186Please keep in mind that you'll most likely have to manually check the 187package for any mistakes as there are things that cannot be guessed by 188the generator (e.g. dependencies on any of the python core modules 189such as BR2_PACKAGE_PYTHON_ZLIB). Also, please take note that the 190license and license files are guessed and must be checked. You also 191need to manually add the package to the +package/Config.in+ file. 192 193If your Buildroot package is not in the official Buildroot tree but in 194a br2-external tree, use the -o flag as follows: 195 196----------------------- 197utils/scanpypi foo bar -o other_package_dir 198----------------------- 199 200This will generate packages +python-foo+ and +python-bar+ in the 201+other_package_directory+ instead of +package+. 202 203Option +-h+ will list the available options: 204 205----------------------- 206utils/scanpypi -h 207----------------------- 208 209[[python-package-cffi-backend]] 210 211==== +python-package+ CFFI backend 212 213C Foreign Function Interface for Python (CFFI) provides a convenient 214and reliable way to call compiled C code from Python using interface 215declarations written in C. Python packages relying on this backend can 216be identified by the appearance of a +cffi+ dependency in the 217+install_requires+ field of their +setup.py+ file. 218 219Such a package should: 220 221 * add +python-cffi+ as a runtime dependency in order to install the 222compiled C library wrapper on the target. This is achieved by adding 223+select BR2_PACKAGE_PYTHON_CFFI+ to the package +Config.in+. 224 225------------------------ 226config BR2_PACKAGE_PYTHON_FOO 227 bool "python-foo" 228 select BR2_PACKAGE_PYTHON_CFFI # runtime 229------------------------ 230 231 * add +host-python-cffi+ as a build-time dependency in order to 232cross-compile the C wrapper. This is achieved by adding 233+host-python-cffi+ to the +PYTHON_FOO_DEPENDENCIES+ variable. 234 235------------------------ 236################################################################################ 237# 238# python-foo 239# 240################################################################################ 241 242... 243 244PYTHON_FOO_DEPENDENCIES = host-python-cffi 245 246$(eval $(python-package)) 247------------------------ 248