cmake: modernize CMake infrastructure; adds modules, bindings; build Windows wheels #21

gsmecher · 2025-12-17T19:16:33Z

This PR builds on the previous one (#20), so while it's a big change, it's not as sprawling as it seems.

My primary motivation is getting Windows wheels built for pygetdata. This created a fair cascade of tasks because:

autotools only works on UNIX-like environments (including MSYS2/cygwin), but
on Windows, native Python .pyd modules are really only supported using MSVC
If we do that, the MacOS and Linux wheel builders should come along for the ride.

Practically speaking, native Windows support needs CMake instead of autotools, and switching the MacOS/Linux wheel builders at the same time makes sense. The autotools flow remains in place (and the github regression tests still pass against it.)

This PR contains a mix of "omnibus commits" (an incremental modernization of the CMake infrastructure does not make sense) and "drive-by commits" for individual bug fixes or improvements.

Note that we have definitely left Python 2.x behind (Numpy 2.x never supported it anyways).

gsmecher · 2025-12-17T19:18:54Z

@ketiltrout and @arahlin, sorry to drop this on you before the holidays. This is a follow-up to #20 (and includes both series of commits).

… iterating Without this poison, the following tests segfault: 1426 - test_trunc (SEGFAULT) 1427 - test_trunc_dir (SEGFAULT) 1429 - test_trunc_truncsub (SEGFAULT)

…re support

…arch() This has no impact on the current codebase, but in a future commit we'll be adding Windows support (which means backslashes, which need to be escaped.)

… paths

…separators)

…n for "skip me")

We now have two Python packaging/backend combinations: - setup.py: setuptools, invoked explicitly by the autotools build, and - pyproject.toml: scikit-build-core, invoked by the cmake build It's slightly awkward having these two coexist side-by-side, but the two build systems already do that.

…n warnings Since cmake on Windows unconditionally defines GD_C89_API, Windows otherwise gives us warnings like: \path\to\alter_crecip89.c(22,9): warning C4005: 'GD_C89_API': macro redefinition

This works within GCC but not within MSVC, and the c++ preprocessor documentation (see 16.3.4 in N4594) seems to disavow this behaviour.

Now that CMake generates gd_config.h (consistent with the automake flow), this macro was redundant and can be safely removed.

…3.14

…sion number

Squashes the warning: Warning: cibuildwheel: Invalid skip selector: 'cp36*'. cibuildwheel 3.x no longer supports Python < 3.8. Please use the 2.x series or update `skip`. This selector will have no effect. Warning: cibuildwheel: Invalid skip selector: 'cp37*'. cibuildwheel 3.x no longer supports Python < 3.8. Please use the 2.x series or update `skip`. This selector will have no effect.

bindings/cxx/test/big_test.cpp

bindings/python/setup.py.in

cmake/CMakeLists.txt

arahlin · 2025-12-18T00:35:58Z

bindings/python/pyproject.toml

 dynamic = ["version"]

-[tool.setuptools]
-script-files = ["../../util/dirfile2ascii", "../../util/checkdirfile"]


These utilities are actually quite useful to include in the wheels if possible.

OK, halfway there. The wheels now

produce a python module that's dynamically linked,

include the corresponding .so/.dll, and

include whichever of these two utilities can be compiled on the platform.

Unfortunately, two limitations

dirfile2ascii uses getopt_long, which is unavailable on Windows, and

pyproject.toml only allows Python scripts to be installed (via [project.scripts]), so for these binaries to be visible in PATH, we'd need to create Python wrappers for them.

I am losing all steam on making the windows build work, to be honest. I'd just turn the utils off on the windows build altogether. For the linux/osx builds, I think you can tell cmake to install binary files in the top level bin directory (something like install(<target> DESTINATION bin)) inside the wheel package, without any additional content in [project.scripts]. Does that work?

Perhaps ${SKBUILD_SCRIPTS_DIR} is what you want? https://scikit-build-core.readthedocs.io/en/latest/guide/cmakelists.html#install-directories

arahlin · 2025-12-18T00:56:04Z

cmake/gd_config.h.in

+
+/* Package information */
+#define PACKAGE_NAME "GetData"
+#define PACKAGE_VERSION "0.11.0"


Should be able to pick this up from the PROJECT_VERSION cmake variable I think?

Yup - see d626c9. This parses version strings from m4/version.m4 - which is pretty clunky, but functional.

arahlin · 2025-12-18T00:59:31Z

cmake/CMakeLists.txt

-cmake_minimum_required(VERSION 2.6.4)
+cmake_minimum_required(VERSION 3.15)

 project(getdata)


Should include a version here. Perhaps also some logic for whether this is run by scikit-build or manually.

Similarly, d626c9. For the scikit-build vs. manual thing - what do you have in mind?

scikit-build-core uses setuptools-scm to get the version number. I think you probably want to use that package name and version in the CMakeLists file if you're triggering it from the pyproject file, and otherwise fall back to the m4 parsing. Perhaps something like this:

if (SKBUILD_PROJECT_NAME) project(${SKBUILD_PROJECT_NAME} VERSION ${SKBUILD_PROJECT_VERSION}) else() <parse m4/version.m4 here...> project(getdata VERSION ${GETDATA_VERSION}) endif()

This way you can use a single PROJECT_VERSION cmake variable everywhere else, which is set by the above project() call.

…file2ascii)

…nd them

…try point

This is a little clunky but reduces the number of redundant places where the version numbering must be updated.

arahlin · 2025-12-20T00:47:55Z

.github/workflows/package.yml

-          CIBW_REPAIR_WHEEL_COMMAND_LINUX: >
-            LD_LIBRARY_PATH=$PWD/src/.libs auditwheel repair -w {dest_dir} {wheel}
-          CIBW_REPAIR_WHEEL_COMMAND_MACOS: >
-            DYLD_LIBRARY_PATH=$PWD/src/.libs delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}


Are you sure these repair wheel command tweaks are not needed for linux and osx? IIRC I've had to set the library paths by hand like this for the repair tools to find the shared libraries and set RPATH correctly.

arahlin · 2025-12-20T00:53:22Z

.github/workflows/package.yml

-            yum install -y bzip2-devel flac-devel xz-devel pcre-devel zziplib-devel &&
-            .github/workflows/bootstrap.sh --disable-python &&
-            make
+            yum install -y cmake bzip2-devel flac-devel xz-devel pcre-devel zziplib-devel


Are all of these plugin libraries being found correctly by cmake? IIRC, I had to give hints to autotools with the bootstrap command to make sure that all of the. compression plugins built properly.

arahlin · 2025-12-20T01:00:31Z

cmake/CMakeLists.txt

+else()
+  math(EXPR GETDATA_INT_VERSION "${PROJECT_VERSION_MAJOR} * 10000 + ${PROJECT_VERSION_MINOR} * 100 + ${PROJECT_VERSION_PATCH}")
+endif()
+set(DEFINE_GD_GETDATA_INT_VERSION "#define GD_GETDATA_INT_VERSION ${GETDATA_INT_VERSION}")


This variable doesn't seem to be used anywhere?

gsmecher mentioned this pull request Dec 17, 2025

bindings/python: use Numpy 2.x; update wheel builds #20

Merged

gsmecher force-pushed the python-cmake branch 4 times, most recently from 667f835 to ccccf8e Compare December 17, 2025 20:28

gsmecher added 18 commits December 17, 2025 13:23

.gitignore: ignore CMakeCache.txt and CMakeFiles

5e144e7

msvc: don't define FILENAME_MAX if it already exists

f9cff67

Windows: MSVC: Poison GD_TRUNC for now - it is unsafe to unlink while…

3baa45e

… iterating Without this poison, the following tests segfault: 1426 - test_trunc (SEGFAULT) 1427 - test_trunc_dir (SEGFAULT) 1429 - test_trunc_truncsub (SEGFAULT)

big_test.py (Python tests): Python 2.x -> 3.x exception update

fe8536d

big_test.py (Python tests): gracefully handle builds without regex/pc…

2355082

…re support

big_test.py (Python tests): CheckEOS: escape input before using re.se…

f99b653

…arch() This has no impact on the current codebase, but in a future commit we'll be adding Windows support (which means backslashes, which need to be escaped.)

big_test.py (Python tests): use path.join for "expected results" with…

5092790

… paths

bindings/python: callback.py tests: win32 portability (CRLF and path …

7053593

…separators)

cmake modernization and python bindings

7580825

CMake: correctly wrangle tests with exit code 77 (autotools conventio…

665d827

…n for "skip me")

Github workflows: add CMake regression tests and Windows wheels

513dba5

test/c89*: place GD_C89_API in an #ifndef guard to squash redefinitio…

7dc518c

…n warnings Since cmake on Windows unconditionally defines GD_C89_API, Windows otherwise gives us warnings like: \path\to\alter_crecip89.c(22,9): warning C4005: 'GD_C89_API': macro redefinition

test/version_5: fix #ifdef within macro expansion

855ec88

This works within GCC but not within MSVC, and the c++ preprocessor documentation (see 16.3.4 in N4594) seems to disavow this behaviour.

Remove HAVE_CONFIG_H macro entirely

b91b7d6

Now that CMake generates gd_config.h (consistent with the automake flow), this macro was redundant and can be safely removed.

.github/workflows/package.yml: upgrade cibuildwheel; add Python 3.13/…

d509ce0

…3.14

bindings/python/pyproject.toml: Teach scikit-build to extract the ver…

5d13185

…sion number

gsmecher force-pushed the python-cmake branch from ccccf8e to 1da0110 Compare December 17, 2025 21:24

arahlin reviewed Dec 18, 2025

View reviewed changes

bindings/cxx/test/big_test.cpp Show resolved Hide resolved

bindings/python/setup.py.in Show resolved Hide resolved

cmake/CMakeLists.txt Show resolved Hide resolved

arahlin reviewed Dec 18, 2025

View reviewed changes

arahlin assigned gsmecher Dec 18, 2025

gsmecher added 2 commits December 18, 2025 09:03

CMake/wheels: properly configure and install utils (checkdirfile, dir…

938b4fd

…file2ascii)

MSVC: work around #define snprintf _snprintf in stdio.h

153982c

gsmecher force-pushed the python-cmake branch 11 times, most recently from b8e92dc to 5f66b02 Compare December 18, 2025 22:41

gsmecher added 2 commits December 18, 2025 15:13

wheel build: use shared libraries; teach wheel repair steps how to fi…

bdf7dde

…nd them

CMake: Add BUILD_CXX compile option; disable it for pyproject.toml en…

f7099e1

…try point

gsmecher force-pushed the python-cmake branch from 5f66b02 to f7099e1 Compare December 18, 2025 23:14

gsmecher added 2 commits December 18, 2025 16:08

CMake: Correct .dll placement for Windows

9ce55ba

CMake: extract GETDATA_VERSION macros from m4/version.m4

d626c9c

This is a little clunky but reduces the number of redundant places where the version numbering must be updated.

arahlin requested changes Dec 20, 2025

View reviewed changes

cmake: modernize CMake infrastructure; adds modules, bindings; build Windows wheels #21

Are you sure you want to change the base?

cmake: modernize CMake infrastructure; adds modules, bindings; build Windows wheels #21

Uh oh!

Conversation

gsmecher commented Dec 17, 2025

Uh oh!

gsmecher commented Dec 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gsmecher Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gsmecher Dec 19, 2025 •

edited

Loading