unix: link libpython with -Bsymbolic-functions

This changes the runtime loader lookup semantics for libpython so
function references are resolved in the local library first.

* May improve startup time by eliding symbol lookups in the executable
  and other libraries.
* Prevents symbols provided by libpython from resolving to other loaded
  libraries.
* Enables additional compiler+linker optimizations by guaranteeing that
  libpython symbols resolve to the local library. (e.g. more aggressive
  inlining across translations units.)

I believe this change is safe since we're already disabling semantic
interposition and PGO+LTO+BOLT result in substantial inlining.

However, this change can break `LD_PRELOAD` where a separate DSO providing
libpython symbols is injected at the front of the loader search path.
In this scenario, the libpython symbol will be used instead of the variant
injected via `LD_PRELOAD`. I'm unsure if any popular software (that isn't
malware) is relying on intercepting libpython symbols via `LD_PRELOAD`. But
because of our aggressive build optimizations, various functions would have
been inlined and wouldn't have been `LD_PRELOAD` interceptable anyway since
the PLT was already elided. So this change effectively finishes the
"migration" of making `LD_PRELOAD` unreliable.

Author: indygreg

cpython-unix/build-cpython.sh

@@ -85,6 +85,15 @@ if [ -n "${CROSS_COMPILING}" ]; then
     fi
 fi
 
+# Inject -Bsymbolic* on shared library to force direct symbol resolution
+# and potentially unlock more compiler+linker optimizations.
+#
+# Patch is safe on all arches but runs into context conflict on macOS.
+# It isn't needed for macOS. So workaround by not applying there.
+if [[ "${PYBUILD_PLATFORM}" != macos* ]]; then
+    patch -p1 -i "${ROOT}/patch-configure-linker-symbolic.patch"
+fi
+
 # LIBTOOL_CRUFT is unused and breaks cross-compiling on macOS. Nuke it.
 # Submitted upstream at https://github.com/python/cpython/pull/101048.
 if [ -n "${PYTHON_MEETS_MAXIMUM_VERSION_3_11}" ]; then

cpython-unix/patch-configure-linker-symbolic.patch

@@ -0,0 +1,23 @@
+diff --git a/configure.ac b/configure.ac
+index a1f4a567095..f7aa69556e5 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -3489,8 +3489,16 @@ then
+ 		LDSHARED='$(CC) -shared'
+ 		LDCXXSHARED='$(CXX) -shared';;
+ 	Linux*|GNU*|QNX*|VxWorks*|Haiku*)
+-		LDSHARED='$(CC) -shared'
+-		LDCXXSHARED='$(CXX) -shared';;
++		# See https://maskray.me/blog/2021-05-16-elf-interposition-and-bsymbolic.
++		# This is related to the -fno-semantic-interposition optimization that
++		# --enable-optimizations engages automatically. It changes the linkage
++		# behavior so the runtime loader searches the current DSO first before
++		# falling back to the default search mechanism of the executable plus
++		# all its loaded DSOs. It has much the same effect as
++		# -fno-semantic-interposition but can also enable optimizations across
++		# translation units.
++		LDSHARED='$(CC) -shared -Wl,-Bsymbolic-functions'
++		LDCXXSHARED='$(CXX) -shared -Wl,-Bsymbolic-functions';;
+ 	FreeBSD*)
+ 		if [[ "`$CC -dM -E - </dev/null | grep __ELF__`" != "" ]]
+ 		then

docs/quirks.rst

@@ -194,6 +194,22 @@ compile extensions, too.
 If there is a build time normalization that you think should be performed to
 make distributions more portable, please file a GitHub issue.
 
+.. _quirk_ld_preload_doesnt_work:
+
+``LD_PRELOAD`` Does Not Work
+============================
+
+This project applies various build optimizations that undermine the
+ability for ``LD_PRELOAD`` to reliably intercept symbols provided by
+``libpython``:
+
+* Functions can be aggressively inlined - bypassing the PLT - and inlined
+  function calls cannot be intercepted by ``LD_PRELOAD``'d libraries.
+* Use of the linker option ``-Bsymbolic-functions`` on ``libpython`` forces
+  symbols to resolve to the local library (``libpython``) instead of going
+  through the regular loader lookup sequence, which would otherwise process
+  ``LD_PRELOAD``'d libraries before ``libpython``.
+
 .. _quirk_former:
 .. _quirk_missing_libcrypt:
 .. _quirk_linux_libx11: