1// -*- mode:doc; -*- 2// vim: set syntax=asciidoc: 3 4=== Infrastructure for Python packages 5 6This infrastructure applies to Python packages that use the standard 7Python setuptools mechanism as their build system, generally 8recognizable by the usage of a +setup.py+ script. 9 10[[python-package-tutorial]] 11 12==== +python-package+ tutorial 13 14First, let's see how to write a +.mk+ file for a Python package, 15with an example : 16 17------------------------ 1801: ################################################################################ 1902: # 2003: # python-foo 2104: # 2205: ################################################################################ 2306: 2407: PYTHON_FOO_VERSION = 1.0 2508: PYTHON_FOO_SOURCE = python-foo-$(PYTHON_FOO_VERSION).tar.xz 2609: PYTHON_FOO_SITE = http://www.foosoftware.org/download 2710: PYTHON_FOO_LICENSE = BSD-3-Clause 2811: PYTHON_FOO_LICENSE_FILES = LICENSE 2912: PYTHON_FOO_ENV = SOME_VAR=1 3013: PYTHON_FOO_DEPENDENCIES = libmad 3114: PYTHON_FOO_SETUP_TYPE = distutils 3215: 3316: $(eval $(python-package)) 34------------------------ 35 36On line 7, we declare the version of the package. 37 38On line 8 and 9, we declare the name of the tarball (xz-ed tarball 39recommended) and the location of the tarball on the Web. Buildroot 40will automatically download the tarball from this location. 41 42On line 10 and 11, we give licensing details about the package (its 43license on line 10, and the file containing the license text on line 4411). 45 46On line 12, we tell Buildroot to pass custom options to the Python 47+setup.py+ script when it is configuring the package. 48 49On line 13, we declare our dependencies, so that they are built 50before the build process of our package starts. 51 52On line 14, we declare the specific Python build system being used. In 53this case the +distutils+ Python build system is used. The two 54supported ones are +distutils+ and +setuptools+. 55 56Finally, on line 16, we invoke the +python-package+ macro that 57generates all the Makefile rules that actually allow the package to be 58built. 59 60[[python-package-reference]] 61 62==== +python-package+ reference 63 64As a policy, packages that merely provide Python modules should all be 65named +python-<something>+ in Buildroot. Other packages that use the 66Python build system, but are not Python modules, can freely choose 67their name (existing examples in Buildroot are +scons+ and 68+supervisor+). 69 70Packages that are only compatible with one version of Python (as in: 71Python 2 or Python 3) should depend on that version explicitely in 72their +Config.in+ file (+BR2_PACKAGE_PYTHON+ for Python 2, 73+BR2_PACKAGE_PYTHON3+ for Python 3). Packages that are compatible 74with both versions should not explicitely depend on them in their 75+Config.in+ file, since that condition is already expressed for the 76whole "External python modules" menu. 77 78The main macro of the Python package infrastructure is 79+python-package+. It is similar to the +generic-package+ macro. It is 80also possible to create Python host packages with the 81+host-python-package+ macro. 82 83Just like the generic infrastructure, the Python infrastructure works 84by defining a number of variables before calling the +python-package+ 85or +host-python-package+ macros. 86 87All the package metadata information variables that exist in the 88xref:generic-package-reference[generic package infrastructure] also 89exist in the Python infrastructure: +PYTHON_FOO_VERSION+, 90+PYTHON_FOO_SOURCE+, +PYTHON_FOO_PATCH+, +PYTHON_FOO_SITE+, 91+PYTHON_FOO_SUBDIR+, +PYTHON_FOO_DEPENDENCIES+, +PYTHON_FOO_LICENSE+, 92+PYTHON_FOO_LICENSE_FILES+, +PYTHON_FOO_INSTALL_STAGING+, etc. 93 94Note that: 95 96 * It is not necessary to add +python+ or +host-python+ in the 97 +PYTHON_FOO_DEPENDENCIES+ variable of a package, since these basic 98 dependencies are automatically added as needed by the Python 99 package infrastructure. 100 101 * Similarly, it is not needed to add +host-setuptools+ to 102 +PYTHON_FOO_DEPENDENCIES+ for setuptools-based packages, since it's 103 automatically added by the Python infrastructure as needed. 104 105One variable specific to the Python infrastructure is mandatory: 106 107* +PYTHON_FOO_SETUP_TYPE+, to define which Python build system is used 108 by the package. The two supported values are +distutils+ and 109 +setuptools+. If you don't know which one is used in your package, 110 look at the +setup.py+ file in your package source code, and see 111 whether it imports things from the +distutils+ module or the 112 +setuptools+ module. 113 114A few additional variables, specific to the Python infrastructure, can 115optionally be defined, depending on the package's needs. Many of them 116are only useful in very specific cases, typical packages will 117therefore only use a few of them, or none. 118 119* +PYTHON_FOO_SUBDIR+ may contain the name of a subdirectory inside the 120 package that contains the main +setup.py+ file. This is useful, 121 if for example, the main +setup.py+ file is not at the root of 122 the tree extracted by the tarball. If +HOST_PYTHON_FOO_SUBDIR+ is not 123 specified, it defaults to +PYTHON_FOO_SUBDIR+. 124 125* +PYTHON_FOO_ENV+, to specify additional environment variables to 126 pass to the Python +setup.py+ script (for both the build and install 127 steps). Note that the infrastructure is automatically passing 128 several standard variables, defined in +PKG_PYTHON_DISTUTILS_ENV+ 129 (for distutils target packages), +HOST_PKG_PYTHON_DISTUTILS_ENV+ 130 (for distutils host packages), +PKG_PYTHON_SETUPTOOLS_ENV+ (for 131 setuptools target packages) and +HOST_PKG_PYTHON_SETUPTOOLS_ENV+ 132 (for setuptools host packages). 133 134* +PYTHON_FOO_BUILD_OPTS+, to specify additional options to pass to the 135 Python +setup.py+ script during the build step. For target distutils 136 packages, the +PKG_PYTHON_DISTUTILS_BUILD_OPTS+ options are already 137 passed automatically by the infrastructure. 138 139* +PYTHON_FOO_INSTALL_TARGET_OPTS+, +PYTHON_FOO_INSTALL_STAGING_OPTS+, 140 +HOST_PYTHON_FOO_INSTALL_OPTS+ to specify additional options to pass 141 to the Python +setup.py+ script during the target installation step, 142 the staging installation step or the host installation, 143 respectively. Note that the infrastructure is automatically passing 144 some options, defined in +PKG_PYTHON_DISTUTILS_INSTALL_TARGET_OPTS+ 145 or +PKG_PYTHON_DISTUTILS_INSTALL_STAGING_OPTS+ (for target distutils 146 packages), +HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS+ (for host 147 distutils packages), +PKG_PYTHON_SETUPTOOLS_INSTALL_TARGET_OPTS+ or 148 +PKG_PYTHON_SETUPTOOLS_INSTALL_STAGING_OPTS+ (for target setuptools 149 packages) and +HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS+ (for host 150 setuptools packages). 151 152* +HOST_PYTHON_FOO_NEEDS_HOST_PYTHON+, to define the host python 153 interpreter. The usage of this variable is limited to host 154 packages. The two supported value are +python2+ and +python3+. It 155 will ensure the right host python package is available and will 156 invoke it for the build. If some build steps are overloaded, the 157 right python interpreter must be explicitly called in the commands. 158 159With the Python infrastructure, all the steps required to build and 160install the packages are already defined, and they generally work well 161for most Python-based packages. However, when required, it is still 162possible to customize what is done in any particular step: 163 164* By adding a post-operation hook (after extract, patch, configure, 165 build or install). See xref:hooks[] for details. 166 167* By overriding one of the steps. For example, even if the Python 168 infrastructure is used, if the package +.mk+ file defines its own 169 +PYTHON_FOO_BUILD_CMDS+ variable, it will be used instead of the 170 default Python one. However, using this method should be restricted 171 to very specific cases. Do not use it in the general case. 172 173[[scanpypi]] 174 175==== Generating a +python-package+ from a PyPI repository 176 177If the Python package for which you would like to create a Buildroot 178package is available on PyPI, you may want to use the +scanpypi+ tool 179located in +utils/+ to automate the process. 180 181You can find the list of existing PyPI packages 182https://pypi.python.org[here]. 183 184+scanpypi+ requires Python's +setuptools+ package to be installed on 185your host. 186 187When at the root of your buildroot directory just do : 188 189----------------------- 190utils/scanpypi foo bar -o package 191----------------------- 192 193This will generate packages +python-foo+ and +python-bar+ in the package 194folder if they exist on https://pypi.python.org. 195 196Find the +external python modules+ menu and insert your package inside. 197Keep in mind that the items inside a menu should be in alphabetical order. 198 199Please keep in mind that you'll most likely have to manually check the 200package for any mistakes as there are things that cannot be guessed by 201the generator (e.g. dependencies on any of the python core modules 202such as BR2_PACKAGE_PYTHON_ZLIB). Also, please take note that the 203license and license files are guessed and must be checked. You also 204need to manually add the package to the +package/Config.in+ file. 205 206If your Buildroot package is not in the official Buildroot tree but in 207a br2-external tree, use the -o flag as follows: 208 209----------------------- 210utils/scanpypi foo bar -o other_package_dir 211----------------------- 212 213This will generate packages +python-foo+ and +python-bar+ in the 214+other_package_directory+ instead of +package+. 215 216Option +-h+ will list the available options: 217 218----------------------- 219utils/scanpypi -h 220----------------------- 221 222[[python-package-cffi-backend]] 223 224==== +python-package+ CFFI backend 225 226C Foreign Function Interface for Python (CFFI) provides a convenient 227and reliable way to call compiled C code from Python using interface 228declarations written in C. Python packages relying on this backend can 229be identified by the appearance of a +cffi+ dependency in the 230+install_requires+ field of their +setup.py+ file. 231 232Such a package should: 233 234 * add +python-cffi+ as a runtime dependency in order to install the 235compiled C library wrapper on the target. This is achieved by adding 236+select BR2_PACKAGE_PYTHON_CFFI+ to the package +Config.in+. 237 238------------------------ 239config BR2_PACKAGE_PYTHON_FOO 240 bool "python-foo" 241 select BR2_PACKAGE_PYTHON_CFFI # runtime 242------------------------ 243 244 * add +host-python-cffi+ as a build-time dependency in order to 245cross-compile the C wrapper. This is achieved by adding 246+host-python-cffi+ to the +PYTHON_FOO_DEPENDENCIES+ variable. 247 248------------------------ 249################################################################################ 250# 251# python-foo 252# 253################################################################################ 254 255... 256 257PYTHON_FOO_DEPENDENCIES = host-python-cffi 258 259$(eval $(python-package)) 260------------------------ 261