Commit graph

170 commits

Author SHA1 Message Date
Rosen Penev
fadf7190ea python3: fix compilation
Because of e52d0487e88c3c8c57e1310d1a02b18eae0d142e , these flags no
longer get passed. Add them back to fix compilation.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-11-29 13:58:13 +02:00
Alexandru Ardelean
cc332fddaa python3: bump to version 3.9.0
Refreshed patches.

Dropped 'patches-setuptools/004-site-patch.patch'
Does not apply anymore. Setuptools has removed site.py support:
   https://github.com/pypa/setuptools/issues/2165
If this is still needed, we may need to re-think it's implementation.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2020-10-19 11:01:12 +03:00
Jeffery To
001cd3c571
python3: Add config PYTHON3_HOST_PIP_CACHE_WORLD_READABLE
This adds a config option PYTHON3_HOST_PIP_CACHE_WORLD_READABLE; if
enabled, chmod will be run after pip install to make all
files/directories in the host pip cache world-readable.

Supersedes https://github.com/openwrt/packages/pull/13012.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-08-16 06:09:45 +08:00
Rosen Penev
b40c40151c
python3: fix host compilation with clang
Matched rpath parameter with Makefile.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-08-02 00:49:30 -07:00
Jeffery To
7d6fcf20e7
python3: Update to 3.8.5
This version includes fixes for:
* CVE-2020-15801 - Fixes python3x._pth being ignored on Windows
* CVE-2019-20907 - Avoid infinite loop when reading specially crafted
  TAR files using the tarfile module

This also:
* Remove patches that are included in the update
* Add a dependency in python3-distutils for python3-email[1]

[1]: https://github.com/python/cpython/blob/v3.8.5/Lib/distutils/dist.py#L10

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-07-27 04:33:28 +08:00
Jeffery To
1a3cef77d4
python3: Update to 3.8.4, refresh/rework patches, backport patches
This version includes fixes for:
* CVE-2020-14422: Hash collisions in IPv4Interface and IPv6Interface
* CVE-2020-15523: Python uses invalid DLL path after calling Py_SetPath
  on Windows

This version also includes support for OpenSSL 1.1.x builds that use
'no-deprecated' and '--api=1.1.0'[1], and so this removes the previous
OpenSSL-related patches.

This also backports fixes for security issues, including:
* CVE-2019-20907: Infinite loop in the tarfile module

This also updates the setuptools and pip packages to 47.1.0 and 20.1.1,
respectively.

[1]: https://github.com/python/cpython/pull/20566

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-07-20 17:30:14 +08:00
Jeffery To
0973d21787
python3: Use default _PYTHON_HOST_PLATFORM
This lets the Python build process set _PYTHON_HOST_PLATFORM instead of
forcing an explicit value.

Also:

* Save the target _PYTHON_HOST_PLATFORM value during Build/InstallDev
  for use when building target Python packages (in python3-package.mk).

* Use the (mostly) default PYTHON_FOR_BUILD value, instead patch
  configure to remove the platform triplet from the sysconfigdata file
  name.

* Remove the "CROSS_COMPILE=yes" make variable (there is no indication
  that this variable is necessary).

* Force host pip to build packages from source instead of downloading
  binary wheels.

  Previously, host pip can download universal (platform-independent)
  wheels but not platform-specific wheels, because of the custom
  _PYTHON_HOST_PLATFORM value. (Packages that do not have universal
  wheels would be compiled from source.)

  With a correct _PYTHON_HOST_PLATFORM, host pip can install
  platform-specific wheels as well. However, the pre-built shared object
  (.so) files in these wheels will have the host's platform triplet in
  their file names. When target Python packages are built (using the
  target's _PYTHON_HOST_PLATFORM), Python will not use these shared
  object files.

  By forcing host pip to build packages from source, the built shared
  object files will not have the platform triplet in their file names.
  (Host Python has been patched to remove the platform triplet from file
  names.) This allows these packages to be used when building target
  Python packages.

  (The net effect of this complete change is that platform-dependent
  packages will continue to be compiled from source, while
  platform-independent packages will now also be compiled from source.)

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-07-08 17:11:42 +08:00
Jeffery To
3c36696a56 python-packages: Add usr/bin symlinks without "3" suffix
Previously, binaries installed by Python packages will have a
non-suffixed Python 2 version and a suffixed Python 3 version, e.g. pip
and pip3. With the removal of Python 2, the non-suffixed names are no
longer taken.

This adds symlinks for the non-suffixed names linking to the suffixed
scripts (or in the case of pip, easy_install, and python-config, to the
fully-versioned scripts).

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-05-22 03:57:55 +08:00
Jeffery To
70a7f736c5 python3-setuptools: Fix error when installing from source
When a Python package is installed from source (i.e. using setup.py)
into a custom location (with --home), setuptools may want to create a
site.py file in the custom location. This file is created based on the
source code of site-patch.py, a file bundled with setuptools.

Because the normal OpenWrt setuptools package does not contain Python
source code, this file is missing and the installation will end with an
error.

This copies site-patch.py to site-patch.py.txt so that it will be
included in python3-setuptools, and patches setuptools to look for this
file.

See https://github.com/openwrt/packages/issues/12223

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-05-21 20:02:13 +08:00
Jeffery To
0254a4bea9 python3: Update to 3.8.3, refresh patches
Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-05-20 17:40:38 +08:00
Jeffery To
d2b0cc523d python3-openssl: Add dependency on ca-certs
The ssl module assumes OpenSSL can load the default trust anchors (root
CA certificates).

From https://github.com/openwrt/packages/issues/12209

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-05-20 04:30:45 +08:00
Jeffery To
be9a554a26 python3: Enable compile-time optimizations
This adds the --enable-optimizations configure option (for
profile-guided optimization) for both host and target Python, and the
--with-lto configure option (for link-time optimization) for target
Python (for non-MIPS platforms).

Currently, compiling Python with LTO leads to link errors on mips and
mipsel. (Compiling with LTO appears to succeed on mips64 but there is
only one mips64 target available for convenient testing.)

This also cleans up the host and target configure options:

* Sort options/variables
  - Alphabetically
  - Flags/options before child-process environment variables
  - Group options by type (enable/disable/with/without)
  - Static options/variables before conditional ones

* Remove the prefix/dir options, as they are the same as the defaults
  set by the build system

* Remove --with-threads, as it is no longer a valid option (threads are
  always enabled)

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-05-18 15:01:17 +08:00
Jeffery To
9f81ab895e python3: Fix host build tool names recorded in target files
Python will record the values of CC, CXX, AR, and READELF (and other
configure options) used during compilation. pip will use these programs
when asked to compile extension modules on the target device.

* If ccache is used during build, CC and CXX will be ccache_cc and
  ccache_cxx, respectively, which are not available on-device (#11912).

* If an external toolchain is used during build, the values of these
  variables will contain the external toolchain prefix, which may not be
  available on target.

* If the normal toolchain is used during build, AR and READELF will
  contain the toolchain prefix, but the names of ar and readelf
  on-device do not contain the prefix; they are named "ar" and
  "readelf".

This changes the values of these variables in Python's files to match
the names available on-device, and without any toolchain prefix.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-04-27 14:10:53 +08:00
Jeffery To
11bc05763d python3-pip: Fix error when installing packages that require compilation
This addresses one of the issues raised in #11912.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-04-24 21:09:37 +08:00
Jeffery To
89ae10ed71 python3: Change PYTHON_VERSION references to PYTHON3_VERSION
PYTHON_VERSION is a holdover from Python 2; all Python 3 variables are
prefixed with PYTHON3 (or some variation with "3").

This updates all uses of PYTHON_VERSION to PYTHON3_VERSION.

This also sets PYTHON3_PKG_BUILD:=0 before python3-package.mk is
included in the python3 Makefile.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-04-19 01:56:23 +08:00
Alexandru Ardelean
eeda0f939b python3: symlink 'python' to '/usr/bin/python3'
Time to make it more official.
Running 'python' should get you an interpreter running, and that one is now
Python3.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2020-04-08 14:40:26 +03:00
Jeffery To
932bfe4cf3 python3-pip: Symlink pip3 to pip3.8
Currently, python3-pip installs the same script as pip3 and pip3.8 to
usr/bin. This changes pip3 to be a symlink to pip3.8.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-04-04 06:21:39 +08:00
Jeffery To
7513448d20 python3: Update to 3.8.2, fix python3-xml dependency
xml.sax.saxutils has a dependency on urllib[1].

[1]: https://github.com/python/cpython/blob/v3.8.2/Lib/xml/sax/saxutils.py#L6

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2020-03-20 16:30:22 +08:00
Rosen Penev
d869dc8010
python3: Fix compilation without deprecated APIs
This is based on upstream work here:

https://github.com/python/cpython/pull/3934
and
https://github.com/python/cpython/pull/39430

First patch was kept as is (just rebased) while the second was combined
into one.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2020-01-14 14:43:58 -08:00
Josef Schlehofer
d2c997f39e
python3: Updated to version 3.8.1
Removed upstream patch: 002-fix-blake2-detection.patch

Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
2019-12-21 20:04:30 +01:00
Josef Schlehofer
ffeb852e61
python3: bump PKG_RELEASE
Fixes: 85eee2d7f8 (python3: fix build when
Blake2 not enabled in OpenSSL")

Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
2019-11-21 15:19:15 +01:00
Yorkie Liu
85eee2d7f8 python3: fix build when Blake2 not enabled in OpenSSL
Refs:

- 7c20888e71
- https://bugs.python.org/issue38684

Signed-off-by: Yorkie Liu <yorkiefixer@gmail.com>
2019-11-21 18:18:14 +08:00
Alexandru Ardelean
0c8b0b0bf7 python3: fix build on Ubuntu/Debian multi-arch
This issue was introduced after upgrading to 3.8.
Patch `006-remove-multi-arch-and-local-paths.patch` was dropped. It was
tested on x86, but when using the x86_64 compiler in OpenWrt, the issue is
present.

The issue seems to be about Ubuntu/Debian's multi-arch support in
Python/Python3 setup [which I forgot about]. The code runs it regardless of
whether it cross-compiles or not, and for OpenWrt, this causes issues, as
it introduces absolute include paths from the host system.

Fixes https://github.com/openwrt/packages/issues/10452

Fixes: 48277ec915 ("python3: bump to version 3.8")
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-11-04 14:10:36 +02:00
Alexandru Ardelean
48277ec915 python3: bump to version 3.8
This required a bit work to get working, compared to other versions. So,
some things have changed a bit more significantly.

Some highlights:
* there is no longer a pgen executable, seems this is now part of
  libpython; let's see what this means for us in the future
* blake2 hash (from OpenSSL) detection needs some fixing; will upstream
  added patch 002-fix-blake2-detection.patch
* removed all bpo patches; those should be fixed in upstream
* some needed to be manually re-applied as stuff changed:
  - 001-enable-zlib.patch  - file changed
  - 004-do-not-write-bytes-codes.patch  - file changed
  - 015-abort-on-failed-modules.patch - variable was renamed
    cross_compiling -> CROSS_COMPILING
* 017_lib2to3_fix_pyc_search.patch - the code changed, it does not seem to
     have the original problem with respect to file-extension, as there
     does not seem to be any special extension logic anymore there
* 006-remove-multi-arch-and-local-paths.patch - dropped patch; I can't
     remember the full-details of this issue; it was something with
     Debian/Ubuntu's multi-arch stuff; it was probably added maybe due to
     some overzealous (on my part) thingy caused by some weird reports,
     that I could never solve; let's have this patch dropped and see
* make package/python3/refresh to reduce fuzz for the rest

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-10-21 15:24:51 +03:00
Jeffery To
a3be5cfa92 python,python3: Fix CVE-2019-16056, CVE-2019-16935
These patches address issues:
CVE-2019-16056: email.utils.parseaddr mistakenly parse an email
CVE-2019-16935: A reflected XSS in python/Lib/DocXMLRPCServer.py (for
Python 2.7)

CVE-2019-16935 was fixed for python3 in #10109

Links to Python issues:
https://bugs.python.org/issue34155
https://bugs.python.org/issue38243

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-10-04 23:54:46 +08:00
Josef Schlehofer
80def9e653
python3: backport three security patches
Fixes: CVE-2019-16935

Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
2019-09-29 23:18:29 +02:00
Alexandru Ardelean
ed0e77f3c3 python,python3: split python[3]-pkg-resources from setuptools
This package is required by other packages to run some binaries via
`load_entry_point`.

So, this splits this package away from setuptools.
setuptools is pretty big, akd pkg-resources is also big, but not as big.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-09-20 13:42:40 +03:00
Jeffery To
229c23a440 python,python3: Fix host Python compilation for macOS
* Remove $$$$(pkg-config --static --libs libcrypto libssl) from
  HOST_LDFLAGS

  Having this leads to an "unknown type name 'u_int'" error on Mac.
  Removing it doesn't appear to affect Python's ability to find
  buildroot LibreSSL.

* Change -Wl,-rpath=... to -Wl,-rpath,... in HOST_LDFLAGS

  The equals sign version is not supported by the Mac linker (appears to
  be an GNU extension). The comma version is supported; -rpath and its
  argument will be separated by a space when passed to the linker.

* Add ac_cv_header_libintl_h=no to HOST_CONFIGURE_VARS for Mac

  Python on Mac doesn't expect to use libintl, but if gettext-full is
  compiled for host, it will try, leading to undefined symbol errors
  during compilation. This prevents configure from finding libintl.h.

Fixes #7171.
Fixes #9621.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-08-17 04:45:58 +08:00
Jeffery To
e14c4e5c0a python,python3: Fix ctypes.util.find_library()
Python's ctypes.util.find_library() function currently doesn't work for
musl libraries/systems[1].

This adds a patch to fix this function, based on a patch from Alpine
Linux[2].

Fixes #9448.

[1]: https://bugs.python.org/issue21622
[2]: https://git.alpinelinux.org/aports/tree/main/python2/musl-find_library.patch

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-07-22 04:42:54 +08:00
Josef Schlehofer
ce769db007
python3: Update to version 3.7.4
- Update setuptools to 40.8.0
- Update pip to 19.0.3
- Refreshed patches
- Removed 4 patches (2 of them was included in 3.7.3 and other two are
included in this release)

Makefile python3:
- Move PKG_MAINTAINER above PKG_LICENSE

Signed-off-by: Josef Schlehofer <pepe.schlehofer@gmail.com>
2019-07-10 10:11:50 +02:00
Eneas U de Queiroz
523c52f6f2 python3: use _python_sysroot for cross-compilation
This patch, taken from buildroot, avoids the use of host paths when
compiling third-party extensions.

Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
2019-06-26 18:50:07 -03:00
Jeffery To
64959a1dd0 python,python3: Move python-config to $(STAGING_DIR)/host
Build/InstallDev is passed a second argument, a path where host binaries
should be placed (ultimately $(STAGING_DIR)/host).

This change moves python[3]-config to that directory.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-06-10 20:33:06 +08:00
Rosen Penev
8f0f613af0
Merge pull request #9131 from jefferyto/python-bpo-35907
python,python3: Fix CVE-2019-9948 - local_file:// allowed in urllib
2019-06-05 01:23:46 -07:00
Jeffery To
53838903fe python,python3: Fix CVE-2019-9948 - local_file:// allowed in urllib
These patches address issue:
CVE-2019-9948: Unnecessary URL scheme exists to allow local_file://
reading file in urllib

Link to Python issue:
https://bugs.python.org/issue35907

Issue 35907 is still currently open, waiting for a decision for
Python 3.5; these patches for Python 2.7 and 3.7 have been merged.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-06-04 22:20:21 +08:00
Hannu Nyman
1e39f3189b
Merge pull request #9114 from jefferyto/python-host-setuptools-pip-installed-pkg-release
python,python3: Record PKG_RELEASE for host setuptools/pip
2019-06-03 17:02:44 +03:00
Rosen Penev
15a0606403
Merge pull request #9100 from jefferyto/isolate-host-python
python,python3: Better isolate host Python
2019-06-03 02:11:20 -07:00
Jeffery To
9331fbb1a0 python,python3: Fix CVE-2019-9740 and CVE-2019-9947
These patches address issues:
CVE-2019-9740: Python urllib CRLF injection vulnerability
CVE-2019-9947: Header Injection in urllib

Links to Python issues:
https://bugs.python.org/issue36276 (resolved duplicated of 30458)
https://bugs.python.org/issue35906 (resolved duplicated of 30458)
https://bugs.python.org/issue30458

Issue 30458 is still currently open, waiting for a decision for
Python 3.5; these patches for Python 2.7 and 3.7 have been merged.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-06-03 00:38:53 +08:00
Jeffery To
1f1eacc484 python,python3: Record PKG_RELEASE for host setuptools/pip
This changes the "patched" indicator files for host setuptools and pip
to include their PKG_RELEASE values. This also removes host setuptools
and/or pip before host install, if the installed copy does not match the
version (and PKG_RELEASE) of the copy to be installed.

This will allow added or removed patches to affect host setuptools /
pip, since these changes will cause PKG_RELEASE to be incremented.

This also fixes the host install error, when the install tries to patch
an already patched copy of setuptools. (This error occurs because the
existing indicator files do not have version numbers in their file
names, whereas host install expected version numbers to be present.)

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-06-01 03:52:23 +08:00
Hannu Nyman
e3dbc87bd9
Merge pull request #9103 from jefferyto/python-host-patch-upgraded-setuptools-pip
python,python3: Allow upgraded host setuptools/pip to be patched
2019-05-30 11:11:26 +03:00
Hannu Nyman
b7e689aefd
Merge pull request #9101 from jefferyto/python-setuptools-pip-install
python-setuptools,python-pip: Installation changes
2019-05-30 11:10:23 +03:00
Jeffery To
e074cbc285 python,python3: Allow upgraded host setuptools/pip to be patched
This adds the current setuptools/pip version numbers to the indicator
files' names, which should allow upgraded versions to be patched.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-05-30 01:48:29 +08:00
Jeffery To
b8e7a197db python3-setuptools,python3-pip: Use more predictable pycache cleanup
This uses two find commands to delete __pycache__ contents then the
__pycache__ directories, rather than a for loop.

The second command omits a -empty test, so that if the first command
doesn't remove all directory contents for some reason, the second
command will return an error (find will not delete a non-empty
directory).

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-05-29 22:42:02 +08:00
Jeffery To
be751236aa python-setuptools,python-pip: Change prefix to /usr
This changes the --prefix option, passed to host pip when "installing"
target setuptools and pip, to /usr, in case the prefix is recorded in
the packages.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-05-29 22:38:30 +08:00
Jeffery To
688825f37a python-setuptools,python-pip: Set pip cache dir, disable version check
This adds --cache-dir and --disable-pip-version-check options for host
pip, when "installing" target setuptools and pip.

This also changes the pip command to use $(HOST_PYTHON[3]_PIP) from
python[3]-host.mk.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-05-29 22:29:20 +08:00
Jeffery To
87a31e49f7 python,python3: Disable user site-packages for host Python
Normally, Python will include the user's site-packages directory
(~/.local/lib/python$(PYTHON_VERSION)/site-packages) in it's internal
search path for modules.

This disables this default inclusion for host Python.

This change is applied during Host/Configure instead of as a patch to
keep this setting unchanged for target Python.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-05-29 22:03:18 +08:00
Jeffery To
af5288827e python,python3: Use ensurepip=upgrade for host Python
This changes --with-ensurepip=install to upgrade, to upgrade host
versions of setuptools and pip to the Python-bundled versions.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-05-28 20:27:13 +08:00
Jeffery To
da35e6770d python-setuptools: Add reproducibility patches from Debian
This adds the ability to patch setuptools (and pip), and adds 3
reproducibility patches from Debian[1].
(003-PKG-INFO-output-reproducible.patch addresses the issue identified
in #9039.)

The patching is not perfect, in that the patches are applied to
setuptools and pip after they have been installed, since they are
installed from wheels which are already "precompiled".

Also, patching for the host install cannot be updated in place, for
example if a patch is added or removed.

[1]: https://sources.debian.org/patches/python-setuptools/40.8.0-1/

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-05-24 13:18:09 +08:00
Eneas U de Queiroz
5f447b9020
python-python3-pip: add LICENSE information
CVE id was left commented out as it is not handled by uscan.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
2019-05-21 14:46:13 -03:00
Eneas U de Queiroz
bbd4930ccd
python-python3-setuptools: add LICENSE information
CVE id was left commented out as it is not handled by uscan.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
2019-05-21 14:44:40 -03:00
Eneas U de Queiroz
00f90974b6
python3: refresh LICENSE information
Changed PKG_LICENSE to reflect spdx license tag, and PKG_LICENSE_FILES
to include all lincense-related files applicable to the parts of the
code we are actually using to build and/or distributing.  The
Windows-only files, and the python-bundled Tools we're not using have
been left out.

Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
2019-05-20 18:20:43 -03:00
Jeffery To
8de8ff4f25 python,python3: Include python-config for target Python in InstallDev
This installs python{2.7,3.7}-config in $(STAGING_DIR)/usr/bin as part
of Build/InstallDev, to be used by other packages to get build
configuration for target Python.

The treatment for Python 2 and 3 are a bit different:

* For Python 2, python-config is a Python script that is expected to be
  run with, and return data for, the installed Python interpreter. This
  installs a modified version of this script, to be run using host
  Python, and read/return data for target Python.

* Python 3 includes a shell script version of python-config (expected to
  be used in cross-compilation scenarios). This simply installs the
  script into the right place.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-03-31 19:33:12 +08:00
Jeffery To
c23bea8dc8 python,python3: Fix CVE-2019-9636 - urlsplit missing NFKC normalization
These patches address issue:
CVE-2019-9636: urlsplit does not handle NFKC normalization

Link to Python issue:
https://bugs.python.org/issue36216

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-03-31 05:18:14 +08:00
Jeffery To
e436e73f1c python,python3: Use only target package data for Build/InstallDev
This changes Build/InstallDev for both Python 2 and 3 to only copy files
from target Python, not from host Python, since InstallDev files are
used for target packages to link to other target packages.

In particular, usr/lib/python{2.7,3.7}/_sysconfigdata.py holds system
configuration data generated at build time, and is different for target
Python and host Python.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2019-03-27 03:44:31 +08:00
Alexandru Ardelean
b485a90aa3 python3: remove OpenSSL deprecated API patch
Fixes: https://github.com/openwrt/packages/issues/8399

These 2 patches cause some breakage for other packages.
For now, we drop them and wait for upstream to finalize a fix.
We can live with deprecated SSL APIs for a while. No need to hurry, since
this doesn't seem to help.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-03-19 14:09:03 +08:00
Hannu Nyman
f9dd076a46
Merge pull request #8371 from neheb/patch-1
python: Add PKG_CPE_ID for proper CVE tracking
2019-03-12 19:06:41 +02:00
Rosen Penev
3b935060e8 python3: Fix compilation with deprecated OpenSSL APIs
Also refreshed patches.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2019-03-11 15:08:09 -07:00
Rosen Penev
1d04e54651 python: Add PKG_CPE_ID for proper CVE tracking
Signed-off-by: Rosen Penev <rosenp@gmail.com>
2019-03-11 13:14:26 -07:00
Hannu Nyman
c722f6e5e1
Merge pull request #8308 from commodo/python3-pip-fix
python3-pip: fix install rule
2019-03-01 23:07:09 +02:00
Hannu Nyman
b710d098f2
Merge pull request #8307 from commodo/python3-split-urllib
python3: split python3-urllib from python3-light
2019-03-01 23:06:25 +02:00
Alexandru Ardelean
998ff3471a python,python3: add Jeffery To as co-maintainer
For a while now, Jeffery has helped quite a lot with Python, and is now
unofficial go-to guy [for problems] with Python packages.

This change adds him as co-maintainer [if he also agrees].

I'm not going away; I'll be still doing the same work for Python.
This change serves to recognize Jeffery in an official way, since he's
already taking on these things. And 2 co-maintainers is better in case one
kicks the bucket [by accident].

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-02-27 12:33:27 +02:00
Alexandru Ardelean
1b96dc0171 python3-pip: fix install rule
Fixes: https://github.com/openwrt/packages/issues/8301

This seems to have slipped for some time. No idea if it ever worked.
It could be that this worked at some point.

In any case, the shebang is properly updated now.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-02-27 12:18:19 +02:00
Alexandru Ardelean
82981d1d55 python3: split python3-urllib from python3-light
This is the result of this discussion:
  https://github.com/openwrt/packages/issues/8285

`urllib.request` requires the `email` module/lib, which was part of
python3-light.
This change moves the Lib/urllib folder from the python3-light into it's
own package, making it lighter. At least this way, users that want `urllib`
(on top of `python3-light`) will be forced to install it via opkg and this
will make sure `python3-email` gets installed as well.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-02-26 15:37:48 +02:00
Karel Kočí
7175b2b47d
python,python3: do not package binary files to base packages
Thanks to fix 200a5a2eec all base packages
now contain all binaries that are generated as part of python
installation. That causes collision between those packages with package
managers that consider this such as Turris updater-ng. This is also just
wrong. Those binaries were not included and should not be after
mentioned fix as well.

This just adds empty install definition. The idea is to override the
default one that is otherwise used.

Signed-off-by: Karel Kočí <karel.koci@nic.cz>
2019-02-21 09:33:49 +01:00
Alexandru Ardelean
c767598097 python,python3: fix CVE-2019-5010 - Denial of Service
This patch addresses issue:
  [ssl][CVE-2019-5010] TALOS-2018-0758 Denial of Service

Link to Python issue:
  https://bugs.python.org/issue35746

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-02-12 12:31:55 +02:00
Alexandru Ardelean
ed862da936 python,python3: move shebang handle in install script
This extends the Python[3] shebang fixup to all packages.
Only Python scripts in `/usr/bin` will be handled at the moment. Later it
may make sense to also cover executables in `/bin`, though typically Python
executables shouldn't be placed there.

Previously the shebang handling was only done for python[3]-pip &
python[3]-setuptools.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-02-12 12:31:55 +02:00
Alexandru Ardelean
1bf7679211 python,python3: make deletion tolerant for paths with spaces
Piping to xargs does not handle spaces in paths too well, because it splits
up the paths.
For deleting empty dirs, we also need to do several retries, otherwise
`find` will try to go through the directories after they're deleted.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-02-12 12:31:54 +02:00
Daniel F. Dickinson
682cdf3b9c python3: install pkgconfig to staging
Some programs that use the Python C API have difficulties finding
cross-compiled Python3 without the pkgconfig, so make sure we have
python3.pc and python-3.7.pc in pkgconfig staging dir.

CircleCI requires a package Makefile change to actually
do the CI check, so bump PKG_RELEASE.

Signed-off-by: Daniel F. Dickinson <cshored@thecshore.com>
2019-01-18 04:40:43 -05:00
Nj Hsiong
c98b12d9a9 python3: enable lib2to3 to also search and fix .pyc files.
python3's lib2to3 would fail in silence if python3 and its packages are installed as compiled .pyc files. Root cause is, in Lib/lib2to3/refactor.py, the function get_all_fix_names only searches '.py' fix names.

Signed-off-by: Nj Hsiong <nj.hsiong@gmail.com>
2019-01-09 22:06:52 +08:00
Alexandru Ardelean
a16dea9627 python,python3: use setuptools & pip versions from python{3}-version.mk
`setuptools` & `pip` whl files were selected via wildcards, because it was
easier in the beginning.
Also, initially there weren't any PYTHON{3}_{SETUTPTOOLS/PIP}_VERSION
variables. But now since these vars exist, it makes sense to use them,
because we can catch easier (at build) time if Python/Python3 bump these
versions.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2019-01-02 23:42:01 +02:00
Alexandru Ardelean
1e9a248a56 python3: bump to version 3.7.2
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-12-31 19:26:46 +02:00
Jeffery To
2f55f2a7b5 python/python3: fix .dist-info missing for setuptools and pip
Without .dist-info (similar to .egg-info), setuptools and pip are not
discoverable by pkg_resources.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2018-12-22 22:16:52 +08:00
Alexandru Ardelean
178ffb41f1 python3: bump to version 3.7.1
This change bumps Python3 version to 3.7.1.
Patch `002-fix-implicit-dh-free-declaration.patch` is now included in
upstream.
This also fixes CVE-2018-1061.
   https://www.cvedetails.com/cve/CVE-2018-1061/

Compile & run-tested on x86.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-11-02 09:58:57 +02:00
Jan Kardell
6ade5a1e3a Python3: Fix host build on OpenSUSE
The linker option -rpath is required to find libs in staging_dir. Now it
is included when building host modules. Without it the import test of
the _ctypes and _uuid modules would fail. The _ctypes module uses
libffi.so.6 from staging, but OpenSUSE LEAP 15 has libffi.so.7.
It will also fail on LEAP 42.x, Fedora28 and 29 and future or old
versions of Ubuntu.

Fix needed in master and 18.06 branches.

Signed-off-by: Jan Kardell <jan.kardell@telliq.com>
2018-10-08 11:47:08 +02:00
Alexandru Ardelean
768c675123 python3: fix build with SSL module
No idea how this creeped up. Probably OpenSSL been has updated recently.
Will send this patch upstream as well, but in the meantime we should fix
the Python3 build.

Build error seems to be:
```
<openwrt>/build_dir/target-i386_pentium4_musl/Python-3.7.0/Modules/_ssl.c:4000:5: error: implicit declaration of function 'DH_free'; did you mean 'lh_free'? [-Werror=implicit-function-declaration]
     DH_free(dh);
     ^~~~~~~
     lh_free
cc1: some warnings being treated as errors

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_tkinter              _uuid                 nis
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time                  zlib

Failed to build these modules:
_ssl

Makefile:618: recipe for target 'sharedmods' failed
```

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-09-17 12:53:57 +03:00
Karol Wrona
23b7d14b24 python3: fix libuuid missing dependency for python3-light
Signed-off-by: Karol Wrona <wrona.vy@gmail.com>
2018-07-13 12:57:44 +02:00
Martin Strobel
2080550f76 python3: add override for uuid.h header check
Signed-off-by: Martin Strobel <arctus@crza.de>
2018-07-11 06:35:36 +02:00
Martin Strobel
fe01c8fc05 python3: update to python 3.7.0
Fixes python3 host SSL build -> host-pip can use https:// urls again.
Updates Pip to 10.0.1
rebased patches

Signed-off-by: Martin Strobel <arctus@crza.de>
2018-07-11 06:34:54 +02:00
Alexandru Ardelean
b7d1d2149e python3: update to version 3.6.5
Add PKG_RELEASE for setuptools and pip.
Refresh all patches.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-05-03 14:46:35 +03:00
Hauke Mehrtens
f9016b5977 python/python3: remove zlib/host dependency
zlib is now a host tool and the zlib/host package was removed. this
dependency is not needed any more as there will always be a zlib host
library.

Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
2018-04-28 16:20:44 +02:00
Alexandru Ardelean
b6973e5deb python,python3: remove --with-system-ffi=no config arg
Report https://github.com/openwrt/packages/issues/5638
It was mentioned that this causes build failures on Mac OS X.

The default behavior [in the setup.py script] is to check whether
`--with-system-ffi` is present in the CONFIG_ARGS env var.

However that back-fires a bit when `--with-system-ffi=no`, because the
condition `not '--with-system-ffi' in sysconfig.get_config_var("CONFIG_ARGS")`
evaluates to true.

This is a small bug in the `setup.py` script, but it looks like the
easiest/cleanest way to address it on our end is to just remove it entirely
from the HOST_CONFIGURE_ARGS.
At least that's how it looks like when testing on a Linux machine.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-03-14 16:21:41 +02:00
Alexandru Ardelean
ea62d1217c python,python3: limit underlinking behavior to linux platforms
This change was introduced in commit 1c54e2b0fb to address build
issues on Ubuntu 12.04.

However it was reported to cause issues on Mac OS X.
Report: https://github.com/openwrt/packages/issues/5310

It was also reported that removing this on MacOS X fixes the issue.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-03-14 16:18:30 +02:00
Alexandru Ardelean
63dfa19812 python,python3: disable dependencies between python src packages
Related to:
 https://github.com/openwrt/packages/issues/5424

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-01-16 16:18:47 +02:00
Hannu Nyman
f460719bf4
Merge pull request #5360 from commodo/python_reproducible
python,python3: honour the SOURCE_DATE_EPOCH for the build-info
2018-01-14 17:32:57 +02:00
Alexandru Ardelean
ccdc6bc530 python,python3: export mk files outside of python package dirs
Since `lang/python` is it's own folder of Python packages
(for both Python 2 & 3), and these build rules are needed
in a lot of packages [especially Python packages],
putting them here makes sense architecturally,
to be shared.

This also helps get rid of the `include_mk` construct
which relies on OpenWrt core to provide, and seems
like a broken design idea that has persisted for a while.
Reason is: it requires that Python 2/3 be built to provide
these mk files for other Python packages,
which seems like a bad idea.

Long-term, there could be an issue where some other feeds
would require these mk files [e.g. telephony] for
some Python packages.
We'll see how we handle this a bit later.

For now we limit this to this feed.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-01-10 23:01:51 +02:00
Matthias Schiffer
55c0b5ae19 python,python3: do not install .mk snippets
The .mk snippets are not really usable at the moment, as they cannot be
considered for metadata collection (package DUMP) when included through
include_mk. Python packages do not use include_mk anymore for this reason,
so the install commands can be removed as well.

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2018-01-10 23:01:51 +02:00
Alexandru Ardelean
c54b312043 python,python3: honour the SOURCE_DATE_EPOCH for the build-info
The Modules/getbuildinfo.c allows the use of DATE and TIME
macros to be defined via CFLAGS.

These vars, control the build date & time when the
interpreter is opened, and can be read via the
`platform._sys_version()` function.

So, a conversion from SOURCE_DATE_EPOCH to DATE & TIME
is required at build-time.

This is especially needed for `platform._sys_version()`
to work.
The installation of pip seems to rely on this.

The logic has been adapted from:
 https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal#Makefile

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-01-08 15:32:47 +02:00
Jeffery To
c5cbd44cea python,python3: undo egg-info removal
This reverts commits 4333d1dcbf and
074d2863be, making Python packages
discoverable again by pkg_resources.

Fixes #5361.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
2018-01-08 17:26:54 +08:00
Alexandru Ardelean
36bb5706a2 python,python3: add zlib/host to HOST_BUILD_DEPENDS (ver 2)
This reverts commit 3c6d14021e.
( which is a revert of commit c764f77dc1 )

The initiall commit ( c764f77dc1 )
was reverted, becase zlib did not have a host-build.

Now it does:
 cbe71649bc

So, now it should be good to put this in.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2018-01-02 18:59:53 +02:00
Alexandru Ardelean
4ec5d3d62f python3: bump to version 3.6.4
Re-worked patch `003-do-not-run-distutils-tests.patch`
to reduce patch-size.

Removed `011-fix-ncursesw-definition-colisions.patch`
it is fixed upstream.

Refreshed with `make package/python3/refresh`

Resetting PKG_RELEASE to 1.
This variable was never used for pip3 & setuptools, since
VERSION is specified in the package definitions.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-12-31 17:58:34 +02:00
Alexandru Ardelean
1b6dd4781f python,python3: add support for SOURCE_DATE_EPOCH var
See:
https://github.com/openwrt/packages/issues/5278

This should make Python & Python3 packages reproducible
when building.
In my local tests, I got the same sha256 for a sample
.pyc file, so likely this is the solution that should address
this.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-12-20 00:11:47 +02:00
Hannu Nyman
3c6d14021e Revert "python,python3: add zlib/host to HOST_BUILD_DEPENDS"
This reverts commit c764f77dc1.

The commit caused warnings to be displayed at make defconfig etc.
  WARNING: Makefile 'package/feeds/packages/python/python/Makefile'
  has a host build dependency on 'zlib/host' but
  'package/libs/zlib/Makefile' does not implement a 'host' build type

Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
2017-12-16 17:02:45 +02:00
Alexandru Ardelean
c764f77dc1 python,python3: add zlib/host to HOST_BUILD_DEPENDS
This should fix the zlibmodule build on the host side.

Usually, if zlib is not found, Python/Python3 builds fine
without it, but there are some cases where the Python/Python3
interpreter on the host-side requires zlib to run.

At the moment, zlib does not have a host-build.
This should be available when this PR gets merged:
https://github.com/lede-project/source/pull/1329
[ or a similar one that contains host-build support for zlib ].

In the meantime, this change can go into Python/Python3.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-12-13 21:07:46 +02:00
Alexandru Ardelean
59f0506622 python,python3: define config option for bluetooth support
It was reported via
https://github.com/openwrt/packages/pull/5122#issuecomment-347395472
that if bluez-libs is selected as an installable package,
then the error below will show up:
```
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for python-light:
 *	bluez-libs *
 * opkg_install_cmd: Cannot install package python-light.
```

This looks like a limitation in the design of package deps,
and maybe a misuse of conditional deps (i.e. PACKAGE_bluez-libs:bluez-libs).

So, to fix this, an idea we're adding an extra symbol
that enfoces installation of bluez-libs if selected.

We also need to add a way to disable bluetooth build
if PYTHON(3)_BLUETOOTH_SUPPORT is de-selected.
Otherwise, bluetooth is installed and the socket
module is broken due to linker errors.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-12-10 19:49:23 +02:00
Alexandru Ardelean
c2b3496977 python,python3: build bluetooth support if bluez-libs selected
This should hopefully fix the Python3 build on buildbot.

For a while I assumed it may be a build-bot issue, but
then looking through the packages repo [and finding
the bluez package] it looks like, if you try
to build all packages, Python3 detects the bluetooth
headers installed by bluez.

It looks like Python's bluetooth support was somewhat
broken ; it was not detecting the <bluetooth/bluetooth.h>
header, so a backport from Python3 to Python fixed that.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-11-14 13:50:14 +02:00
Alexandru Ardelean
8750a59d3d python,python3: apply PKG_FIXUP:=autoreconf
Should hopefully provide more consistent results
across distros.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-11-06 18:06:02 +02:00
Alexandru Ardelean
652553e552 python3: bump to version 3.6.3
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-11-06 18:05:59 +02:00
Mirko Vogt
32b60fa32b python3: package CGI component
Signed-off-by: Mirko Vogt <mirko-openwrt@nanl.de>
2017-10-31 13:45:29 +01:00
Alexandru Ardelean
074d2863be python,python3: add option to keep egg-info dirs for python packages
That way some python packages can choose
to keep their egg-info dirs, if they want to, or they're needed.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-10-03 10:40:26 +03:00
Alexandru Ardelean
21a18bd97f python3: fix install mk filenames
Copy + paste error.

Seems I fixed it while testing but forgot to update PR with this.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-09-15 08:59:47 +03:00
Alexandru Ardelean
2eb3979d28 python,python3: install mk files in both host build and InstallDev part
Not sure how it can happen that the files are not
installed via the host build.

Maybe some SDK-like build.

Let's make sure they are installed via InstallDev rule too.

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
2017-09-14 11:44:01 +03:00