Description Updated. I'll work on getting the patch for Jammy approved
and then I will create the patch for Focal. I expect (and would love)
feedback on wording for the new print statements I made.
** Description changed:
- Compiling a Python extension using Python2 (Python 2.7.18) is making use
- of wrong compiler flags, hence dropping required optimizations when
- required. This is happening only when python2 is installed from Ubuntu's
- repositories. By default, Python's distutils module uses compiler and
- linker flags used to compile Python itself to be used to compile
- extensions.
+ [ Impact ]
- Steps to reproduce:
- 1) On Ubuntu 20.04, install python2 using apt package manager.
- 2) After successful installation, verify the CFLAGS variable from sysconfig
module. On my machine, the output is
+ When compiling Python extensions using Python2, CFLAGS optimization
+ flags are dropped.
- Python 2.7.18 (default, Jul 1 2022, 12:27:04)
- [GCC 9.4.0] on linux2
- Type "help", "copyright", "credits" or "license" for more information.
- >>> import sysconfig
- >>> sysconfig.get_config_var('CFLAGS')
- '-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
-Wdate-time -D_FORTIFY_SOURCE=2 -g
-fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=.
-fstack-protector-strong -Wformat -Werror=format-security '
+ This behavior has been caused by our update in this patch
+
http://archive.ubuntu.com/ubuntu/pool/universe/p/python2.7/python2.7_2.7.18-1~20.04.3.diff.gz
+ which differs from upstream.
- 3) Build a test extension module using python2 and verify the compilation
flags.
- python2 setup.py build_ext --inplace
+ The fix modifies the portion of code in Lib/distutils/sysconfig.py which
+ gets the cflags from the environments, and includes the dropped OPT flag
+ from get_config_vars().
- Output from below command is not matching with our expected above CFLAGS.
- aarch64-linux-gnu-gcc -pthread -fno-strict-aliasing -Wdate-time
-D_FORTIFY_SOURCE=2 -g
-fdebug-prefix-map=/build/python2.7-vvQ8AI/python2.7-2.7.18=.
-fstack-protector-strong -Wformat -Werror=format-security -fPIC
-I/usr/include/python2.7 -c testmodule.c -o
build/temp.linux-aarch64-2.7/testmodule.o
+ [ Test Plan ]
+
+ There will be 2 separate tests for this bug:
+ * Ensuring no-change rebuilds are not changed
+ * Ensuring local builds are not changed unless environment variable is set
+
+ Test 1) No-change rebuilds
+
+ To test that no-change rebuilds are not changed, the package python-
+ stdlib-extensions will be built against the new python2.7, and confirm
+ the compiler flags are not altered. This will be a manual test and
+ visual inspection of the build logs.
+
+ Test 2) Functional test
+
+ 1. Create test container
+ $ lxc launch ubuntu:jammy jammy-2002043
+ $ lxc shell ubuntu:jammy jammy-2002043
+
+ 2. Install required packages
+ # apt update -y && apt install -y python2 python-pip
+
+ 3. Create test files
+ # mkdir testprog
+ # cd testprog
+ # cat >setup.py <<EOL
+ from setuptools import setup, Extension
- On further investigation, it looks like Ubuntu's specific patch applied on
libpython2.7-stdlib package is altering the original upstream implementation of
distutils/sysconfig.py code.
+ setup(
+ name="test",
+ ext_modules=[Extension("test", sources=["testmodule.c"])],
+ zip_safe=False
+ )
+ EOL
+ # cat >testmodule.c <<EOL
+ #include <stdio.h>
- Package - https://packages.ubuntu.com/focal/libpython2.7-stdlib
- Patch -
http://archive.ubuntu.com/ubuntu/pool/universe/p/python2.7/python2.7_2.7.18-1~20.04.3.diff.gz
+ int main(void)
+ {
+ printf("This is test program");
+ return 0;
+ }
+ EOL
- Below is the code block which is causing the issue, where the presence of
configure_cflags is modifying cflags. This code is result of ubuntu's patch and
doesn't come directly from upstream python implementation.
- File - /usr/lib/python2.7/distutils/sysconfig.py
- Part of code block:
- elif configure_cflags:
- cflags = ' '.join(str(x) for x in (basecflags, configure_cflags,
extra_cflags) if x)
- ldshared = ldshared + ' ' + configure_cflags
+ 4. Compile a test program
+ # python2 setup.py build_ext --inplace
+ 5. Check CFLAGS
+ # python2 -c "import sysconfig; print(sysconfig.get_config_var('CFLAGS'))"
+ -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
-Wdate-time -D_FORTIFY_SOURCE=2 -g
-ffile-prefix-map=/build/python2.7-W40Ff2/python2.7-2.7.18=. -flto=auto
-ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong
-Wformat -Werror=format-security
- I don't see problem on Python3 though we have extra code added from patch
there as well. Patch used on python3, is not modifying the cflags completely
and instead appending new flags to cflags.
- On python3 (tested on Ubuntu 20.04)
- File - /usr/lib/python3.8/distutils/sysconfig.py
- Part of code block which doesn't alter cflags completely
- elif configure_cflags:
- cflags = cflags + ' ' + configure_cflags
- ldshared = ldshared + ' ' + configure_cflags
+ 6. Check the flags used to compile the test wheel
+ # strings build/lib.linux-x86_64-2.7/test.so | grep -- -O
+ GNU GIMPLE 11.4.0 -mtune=generic -march=x86-64 -g -g -g -g -O2 -fno-openmp
-fno-openacc -fcf-protection=full -fno-strict-aliasing -fwrapv
-fstack-protector-strong -ffat-lto-objects -fstack-protector-strong -fPIC
-fltrans
+ 7. Install fixed python
+ Once updated, this will simply be an apt update && apt upgrade
+ # add-apt-repository ppa:mitchdz/python2.7-optimization-flags -y
+ # apt install -y python2.7
+ # dpkg -s python2.7 | grep Version:
+ Version: 2.7.18-13ubuntu1.2~jammy9
- Request to update the python2 patch to behave similar to what is been done on
python3.
+ 8. Clean build
+ # rm -rf build/ test.so
+
+ 9. Enable opt-in environment variable
+ # export APPLY_LP2002043_UBUNTU_CFLAGS_WORKAROUND=""
+
+ 9. Rebuild with new python2.7 installed
+ # python2 setup.py build_ext --inplace
+
+ 10. Check build flags
+ # strings build/lib.linux-x86_64-2.7/test.so | grep -- -O
+ GNU GIMPLE 11.4.0 -mtune=generic -march=x86-64 -g -g -g -g -O2 -O2
-fno-openmp -fno-openacc -fcf-protection=full -fno-strict-aliasing -fwrapv
-fstack-protector-strong -ffat-lto-objects -fstack-protector-strong -fPIC
-fltrans
+ GNU C17 11.4.0 -mtune=generic -march=x86-64 -g -g -O2 -fno-strict-aliasing
-fwrapv -flto -flto -ffat-lto-objects -fstack-protector-strong -fPIC
-fasynchronous-unwind-tables -fstack-protector-strong -fstack-clash-protection
-fcf-protection
+
+ [ Where problems could occur ]
+
+ * Changing optimization flags can cause a myriad of unattended side effects,
but the change being opt-in means the users should be aware a change is being
made
+ * The change is opt-in, so an informative message is printed to the console
to spread awareness of the issue and how to use the workaround. A questionably
made CI/CD system might see the new console output and flag it as a failure, so
it is important to choose wording carefully.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to python2.7 in Ubuntu.
https://bugs.launchpad.net/bugs/2002043
Title:
Python extension modules get built using wrong compiler flags with
python2
Status in python2.7 package in Ubuntu:
Invalid
Status in python2.7 source package in Bionic:
Won't Fix
Status in python2.7 source package in Focal:
In Progress
Status in python2.7 source package in Jammy:
In Progress
Status in python2.7 source package in Kinetic:
Invalid
Status in python2.7 source package in Lunar:
Invalid
Status in python2.7 source package in Mantic:
Invalid
Bug description:
[ Impact ]
When compiling Python extensions using Python2, CFLAGS optimization
flags are dropped.
This behavior has been caused by our update in this patch
http://archive.ubuntu.com/ubuntu/pool/universe/p/python2.7/python2.7_2.7.18-1~20.04.3.diff.gz
which differs from upstream.
The fix modifies the portion of code in Lib/distutils/sysconfig.py
which gets the cflags from the environments, and includes the dropped
OPT flag from get_config_vars().
[ Test Plan ]
There will be 2 separate tests for this bug:
* Ensuring no-change rebuilds are not changed
* Ensuring local builds are not changed unless environment variable is set
Test 1) No-change rebuilds
To test that no-change rebuilds are not changed, the package python-
stdlib-extensions will be built against the new python2.7, and confirm
the compiler flags are not altered. This will be a manual test and
visual inspection of the build logs.
Test 2) Functional test
1. Create test container
$ lxc launch ubuntu:jammy jammy-2002043
$ lxc shell ubuntu:jammy jammy-2002043
2. Install required packages
# apt update -y && apt install -y python2 python-pip
3. Create test files
# mkdir testprog
# cd testprog
# cat >setup.py <<EOL
from setuptools import setup, Extension
setup(
name="test",
ext_modules=[Extension("test", sources=["testmodule.c"])],
zip_safe=False
)
EOL
# cat >testmodule.c <<EOL
#include <stdio.h>
int main(void)
{
printf("This is test program");
return 0;
}
EOL
4. Compile a test program
# python2 setup.py build_ext --inplace
5. Check CFLAGS
# python2 -c "import sysconfig; print(sysconfig.get_config_var('CFLAGS'))"
-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
-Wdate-time -D_FORTIFY_SOURCE=2 -g
-ffile-prefix-map=/build/python2.7-W40Ff2/python2.7-2.7.18=. -flto=auto
-ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong
-Wformat -Werror=format-security
6. Check the flags used to compile the test wheel
# strings build/lib.linux-x86_64-2.7/test.so | grep -- -O
GNU GIMPLE 11.4.0 -mtune=generic -march=x86-64 -g -g -g -g -O2 -fno-openmp
-fno-openacc -fcf-protection=full -fno-strict-aliasing -fwrapv
-fstack-protector-strong -ffat-lto-objects -fstack-protector-strong -fPIC
-fltrans
7. Install fixed python
Once updated, this will simply be an apt update && apt upgrade
# add-apt-repository ppa:mitchdz/python2.7-optimization-flags -y
# apt install -y python2.7
# dpkg -s python2.7 | grep Version:
Version: 2.7.18-13ubuntu1.2~jammy9
8. Clean build
# rm -rf build/ test.so
9. Enable opt-in environment variable
# export APPLY_LP2002043_UBUNTU_CFLAGS_WORKAROUND=""
9. Rebuild with new python2.7 installed
# python2 setup.py build_ext --inplace
10. Check build flags
# strings build/lib.linux-x86_64-2.7/test.so | grep -- -O
GNU GIMPLE 11.4.0 -mtune=generic -march=x86-64 -g -g -g -g -O2 -O2
-fno-openmp -fno-openacc -fcf-protection=full -fno-strict-aliasing -fwrapv
-fstack-protector-strong -ffat-lto-objects -fstack-protector-strong -fPIC
-fltrans
GNU C17 11.4.0 -mtune=generic -march=x86-64 -g -g -O2 -fno-strict-aliasing
-fwrapv -flto -flto -ffat-lto-objects -fstack-protector-strong -fPIC
-fasynchronous-unwind-tables -fstack-protector-strong -fstack-clash-protection
-fcf-protection
[ Where problems could occur ]
* Changing optimization flags can cause a myriad of unattended side effects,
but the change being opt-in means the users should be aware a change is being
made
* The change is opt-in, so an informative message is printed to the console
to spread awareness of the issue and how to use the workaround. A questionably
made CI/CD system might see the new console output and flag it as a failure, so
it is important to choose wording carefully.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/2002043/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
