In terms of license, contributing guide, etc, all of that information is described in the top [README.md](README.md) file, and it applies here as well. This document attempts to cover only technical aspects of Python packages, and maybe some explanations about how things are (and why they are as they are).
This sub-tree came to exist after a number of contributions (Python packages) were made to this repo, and the [lang](lang) subtree grew to a point where a decision was made to move all Python packages under [lang/python](lang/python).
It contains the Python 3 interpreter and Python packages. Most of the Python packages are downloaded from [pypi.org](https://pypi.org/). Python packages from [pypi.org](https://pypi.org/) are typically preferred when adding new packages.
If more packages (than the ones packaged here) are needed, they can be downloaded via [pip](https://pip.pypa.io). Note that the versions of `pip`&`setuptools` [available in this repo] are the ones that are packaged inside the Python package (yes, Python comes packaged with `pip`&`setuptools`).
Python 2 [will not be maintained past 2020](https://www.python.org/dev/peps/pep-0373/). All Python 2 packages have been removed from the packages feed (this repo) and archived in the [abandoned packages feed](https://github.com/openwrt/packages-abandoned).
**One important consideration:**: if the local name is not `packages`, it's something else, like `openwrt-packages`. And in `feeds.conf[.default]` it's:
include $(TOPDIR)/feeds/openwrt-packages/lang/python/python3-package.mk
```
Each maintainer[s] of external packages feeds is responsible for the local name, and relative inclusion path back to this feed (which is named `packages` by default).
In case there is a need/requirement such that the local package feed is named something else than `packages`, one approach to make the package flexible to change is:
This should solve the corner-case where the `python3-package.mk` can be in some other feed, or if the packages feed will be named something else locally.
In order to build the Python interpreter, a host Python interpreter needs to be built, in order to process some of the build for the target Python build. The host Python interpreter is also needed so that Python bytecodes are generated, so the host interpreter needs to be the exact version as on the target. And finally, the host Python interpreter also provides pip, so that it may be used to install some Python packages that are required to build other Python packages.
That's why you'll also see a Python build & staging directories.
As you're probably thinking, this sounds [and is] somewhat too much complication [just for packaging], but the status of things is-as-it-is, and it's probably much worse than what's currently visible on the surface [with respect to packaging Python & packages].
The thought/discussion matrix derives a bit like this:
* shipping both Python source-code & bytecodes takes too much space on some devices ; Python source code & byte-code take about similar disk-size
* shipping only Python source code has a big performance penalty [on some lower end systems] ; something like 500 msecs (Python source-only) -> 70 msecs (Python byte-codes) time reduction for a simple "Hello World" script
By default, automatic Python byte-code generation is disabled when running a Python script, in order to prevent a disk from accidentally filling up. Since some disks reside in RAM, this also means not filling up the RAM. If someone wants to convert Python source to byte-code then he/she is free to compile it [directly on the device] manually via the Python interpreter & library.
This is a normal OpenWrt package, which will build the Python interpreter. This also provides `python3-pip`&`python3-setuptools`. Each Python package is actually split into multiple sub-packages [e.g. python3-email, python3-sqlite3, etc]. This can be viewed inside [lang/python/python3/files](lang/python/python3/files).
The reason for this splitting, is purely to offer a way for some people to package Python in as-minimal-as-possible-and-still-runable way, and also to be somewhat maintainable when packaging. A standard Python installation can take ~20-30 MBs of disk, which can be somewhat big for some people, so there is the `python3-base` package which brings that down to ~5 MBs. This seems to be good enough (and interesting) for a number of people.
*`python3-base`, which is just the minimal package to startup Python and run basic commands
*`python3` is a meta-package, which installs almost everything (python3-base [plus] Python library [minus] some unit-tests & some windows-y things)
*`python3-light` is `python3` [minus] packages that are in [lang/python/python3/files](lang/python/python3/files) ; the size of this package may be sensible (and interesting) to another group of people
All other Python packages (aside from the intepreter) typically use these files:
* **python3-host.mk** - this file contains paths and build rules for running the Python interpreter on the host-side; they also provide paths to host interprete, host Python lib-dir & so on
* **python3-package.mk**
* includes **python3-host.mk**
* contains all the default build rules for Python packages; these will be detailed below in the [Building a Python package](#Building a Python package) section
**Note** that Python packages don't need to use these files (i.e. `python3-package.mk`&`python3-host.mk`), but they do provide some ease-of-use & reduction of duplicate code. And they do contain some learned-lessons about packaging Python packages, so it's a good idea to use them.
If the package source code will be downloaded from [pypi.org](https://pypi.org/), including `pypi.mk` can help simplify the package Makefile.
To use `pypi.mk`, add this **before**`include $(INCLUDE_DIR)/package.mk`:
```
include ../pypi.mk
```
`pypi.mk` has several `PYPI_*` variables that must/can be set (see below); these should be set before `pypi.mk` is included, i.e. before the `include ../pypi.mk` line.
`pypi.mk` also provides default values for `PKG_SOURCE` and `PKG_SOURCE_URL`, so these variables may be omitted.
One variable is required:
*`PYPI_NAME`: Package name on pypi.org. This should match the PyPI name exactly.
For example (from the `python-yaml` package):
```
PYPI_NAME:=PyYAML
```
These variables are optional:
*`PYPI_SOURCE_NAME`: Package name component of the source tarball filename
Default: Same value as `PYPI_NAME`
*`PYPI_SOURCE_EXT`: File extension of the source tarball filename
Default: `tar.gz`
`pypi.mk` constructs the default `PKG_SOURCE` value from these variables (and `PKG_VERSION`):
Some considerations here (based on the example above):
*`VARIANT=python3` must be added
* typically the package is named `Package/python3-<something>` ; this convention makes things easier to follow, though it could work without naming things this way
*`TITLE` can be something a bit more verbose/neat ; typically the name is short as seen above
The `$(eval $(call Py3Package,python3-lxml))` part will instantiate all the default Python build rules so that the final Python package is packaged into an OpenWrt.
And `$(eval $(call BuildPackage,python3-lxml))` will bind all the rules generated with `$(eval $(call Py3Package,python3-lxml))` into the OpenWrt build system.
The default build process calls `setup.py install` inside the directory where the Python source package is extracted (`PKG_BUILD_DIR`). This "installs" the Python package to an intermediate location (`PKG_INSTALL_DIR`) where it is used by the default install process.
There are several Makefile variables that can be used to customize this process (all optional):
*`PYTHON3_PKG_SETUP_VARS`: Additional environment variables to set for the call to `setup.py`. Should be in the form of `VARIABLE1=value VARIABLE2=value ...`.
The default install process copies some/all of the files from `PKG_INSTALL_DIR`, placed there by the build process, to a location passed to the install rule as the first argument (`$(1)`). The OpenWrt build system will then take those files and create the actual .ipk package archives.
*`Py3Package/<package>/filespec` which are Python library files relative to `/usr/lib/pythonX.Y` ; by default this is `/usr/lib/python$(PYTHON3_VERSION)/site-packages` (`PYTHON3_PKG_DIR`) ; most Python packages generate files that get installed in this sub-folder
*`Py3Package/<package>/install` is similar to `Package/<package>/install` ; this allows binary (or other files) to be installed on the target
Both the 2 above rules generate a `Package/<package>/install` build rule, which gets picked up by the build system. Both can be used together (they are not mutually exclusive), and provide a good enough flexibility for specifying Python packages.
The initial character controls the action that will be taken:
*`+`: Install the given path. If the path is a directory, all files and subdirectories inside are installed.
* If file permissions is specified (optional), then the file or directory (and all files and subdirectories) are assigned the given permissions; if omitted, then the file or directory retains its original permissions.
*`-`: Remove the given path. Useful when most of a directory should be installed except for a few files or subdirectories.
* File permissions is not used / ignored in this case.
*`=`: Assign the given file permissions to the given path. File permissions is required in this case.
If the package installs a `example_package` directory inside `PYTHON3_PKG_DIR`, and there is an `examples` directory and `test_*.py` files that can be omitted to save space, this can be specified as:
These can be installed via pip and ideally they should only be installed like this, because it's a bit simpler than running them through the OpenWrt build system. Build variants on the host-side build are more complicated (and nearly impossible to do sanely) in the current OpenWrt build system.
Which is why [for example] if you need python cffi on the host build, it's easier to just add it via:
This works reasonably well in the current OpenWrt build system, as binaries get built for this package and get installed in the staging-dir `$(STAGING_DIR)/usr/lib/pythonX.Y/site-packages`.
