Docker build failure of titiler-pgstac-api with latest numpy

[gbegkas]

Similar to issue 107 using TitilerPgstacApiLambda from the python package build fails with:

ERROR: Problem encountered: NumPy requires GCC >= 9.3

I am using eoAPI-template with the latest version of eoAPI-cdk package (8.2.2).

Am I doing something wrong? Thanks for any help.

[hrodmn]

Thanks for the bug report @gbegkas. I spent some time testing out solutions and I think the simplest solution is to upgrade the Lambda Python runtime to 3.12 or 3.13.

The first thing I tried was pinning the numpy dependency to 2.2.6 but I kept running into issues with building numepxr which requires numpy as a build dependency. Unfortunately I could not find a way to control which version of numpy gets used while building numexpr (using --no-build-isolation opens a different can of worms).

Upgrading the Lambda Python runtime version is a pretty easy change, I'll try to get a new release out today.

For posterity, here is the full Docker build error:

henry@quercus:~/workspace/devseed/eoapi-cdk/lib$ docker build . --build-arg PYTHON_VERSION=3.11 -f titiler-pgstac-api/runtime/Dockerfile --no-cache
[+] Building 36.7s (10/19)                                                            docker:default
 => [internal] load build definition from Dockerfile                                            0.0s
 => => transferring dockerfile: 1.25kB                                                          0.0s
 => WARN: InvalidDefaultArgInFrom: Default value for ARG public.ecr.aws/lambda/python:${PYTHON  0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value   0.0s
 => [internal] load metadata for public.ecr.aws/lambda/python:3.11                              0.0s
 => [internal] load .dockerignore                                                               0.0s
 => => transferring context: 2B                                                                 0.0s
 => [ 1/15] FROM public.ecr.aws/lambda/python:3.11                                              0.0s
 => [internal] load build context                                                               0.1s
 => => transferring context: 379B                                                               0.1s
 => CACHED [ 2/15] WORKDIR /tmp                                                                 0.0s
 => [ 3/15] RUN python -m pip install pip -U                                                    3.1s
 => [ 4/15] RUN yum install -y gcc-c++                                                         23.4s
 => [ 5/15] COPY titiler-pgstac-api/runtime/requirements.txt requirements.txt                   0.0s
 => ERROR [ 6/15] RUN python -m pip install -r requirements.txt "mangum>=0.14,<0.15" -t /asse  10.2s
------
 > [ 6/15] RUN python -m pip install -r requirements.txt "mangum>=0.14,<0.15" -t /asset:
0.417 Collecting mangum<0.15,>=0.14
0.509   Downloading mangum-0.14.1-py3-none-any.whl.metadata (3.1 kB)
0.573 Collecting titiler.pgstac<1.8,>=1.7 (from -r requirements.txt (line 1))
0.599   Downloading titiler_pgstac-1.7.2-py3-none-any.whl.metadata (8.0 kB)
0.661 Collecting psycopg[binary,pool] (from -r requirements.txt (line 2))
0.679   Downloading psycopg-3.2.9-py3-none-any.whl.metadata (4.5 kB)
0.718 Collecting typing-extensions (from mangum<0.15,>=0.14)
0.737   Downloading typing_extensions-4.14.0-py3-none-any.whl.metadata (3.0 kB)
0.803 Collecting titiler.core<0.22,>=0.21 (from titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
0.820   Downloading titiler_core-0.21.1-py3-none-any.whl.metadata (3.9 kB)
0.882 Collecting titiler.mosaic<0.22,>=0.21 (from titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
0.901   Downloading titiler_mosaic-0.21.1-py3-none-any.whl.metadata (2.9 kB)
0.959 Collecting geojson-pydantic~=1.0 (from titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
0.982   Downloading geojson_pydantic-1.2.0-py3-none-any.whl.metadata (4.2 kB)
1.113 Collecting pydantic<3.0,>=2.4 (from titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.132   Downloading pydantic-2.11.5-py3-none-any.whl.metadata (67 kB)
1.185 Collecting pydantic-settings~=2.0 (from titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.204   Downloading pydantic_settings-2.9.1-py3-none-any.whl.metadata (3.8 kB)
1.241 Collecting annotated-types>=0.6.0 (from pydantic<3.0,>=2.4->titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.258   Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
1.658 Collecting pydantic-core==2.33.2 (from pydantic<3.0,>=2.4->titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.679   Downloading pydantic_core-2.33.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)
1.712 Collecting typing-inspection>=0.4.0 (from pydantic<3.0,>=2.4->titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.730   Downloading typing_inspection-0.4.1-py3-none-any.whl.metadata (2.6 kB)
1.767 Collecting python-dotenv>=0.21.0 (from pydantic-settings~=2.0->titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.790   Downloading python_dotenv-1.1.0-py3-none-any.whl.metadata (24 kB)
1.841 Collecting fastapi>=0.108.0 (from titiler.core<0.22,>=0.21->titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.859   Downloading fastapi-0.115.12-py3-none-any.whl.metadata (27 kB)
1.895 Collecting jinja2<4.0.0,>=2.11.2 (from titiler.core<0.22,>=0.21->titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
1.914   Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
2.064 Collecting numpy (from titiler.core<0.22,>=0.21->titiler.pgstac<1.8,>=1.7->-r requirements.txt (line 1))
2.083   Downloading numpy-2.3.0.tar.gz (20.4 MB)
3.462      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.4/20.4 MB 14.9 MB/s eta 0:00:00
5.671   Installing build dependencies: started
7.284   Installing build dependencies: finished with status 'done'
7.285   Getting requirements to build wheel: started
7.422   Getting requirements to build wheel: finished with status 'done'
7.424   Installing backend dependencies: started
8.110   Installing backend dependencies: finished with status 'done'
8.111   Preparing metadata (pyproject.toml): started
9.703   Preparing metadata (pyproject.toml): finished with status 'error'
9.705   error: subprocess-exited-with-error
9.705
9.705   × Preparing metadata (pyproject.toml) did not run successfully.
9.705   │ exit code: 1
9.705   ╰─> [19 lines of output]
9.705       + /var/lang/bin/python /tmp/pip-install-h9z4x5k3/numpy_7fb65dbcf47341c59ade3b7c610386a5/vendored-meson/meson/meson.py setup /tmp/pip-install-h9z4x5k3/numpy_7fb65dbcf47341c59ade3b7c610386a5 /tmp/pip-install-h9z4x5k3/numpy_7fb65dbcf47341c59ade3b7c610386a5/.mesonpy-fios8xex -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-h9z4x5k3/numpy_7fb65dbcf47341c59ade3b7c610386a5/.mesonpy-fios8xex/meson-python-native-file.ini
9.705       The Meson build system
9.705       Version: 1.6.1
9.705       Source dir: /tmp/pip-install-h9z4x5k3/numpy_7fb65dbcf47341c59ade3b7c610386a5
9.705       Build dir: /tmp/pip-install-h9z4x5k3/numpy_7fb65dbcf47341c59ade3b7c610386a5/.mesonpy-fios8xex
9.705       Build type: native build
9.705       Project name: NumPy
9.705       Project version: 2.3.0
9.705       C compiler for the host machine: cc (gcc 7.3.1 "cc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
9.705       C linker for the host machine: cc ld.bfd 2.29.1-31
9.705       C++ compiler for the host machine: c++ (gcc 7.3.1 "c++ (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
9.705       C++ linker for the host machine: c++ ld.bfd 2.29.1-31
9.705       Cython compiler for the host machine: cython (cython 3.1.2)
9.705       Host machine cpu family: x86_64
9.705       Host machine cpu: x86_64
9.705
9.705       ../meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 9.3
9.705
9.705       A full log can be found at /tmp/pip-install-h9z4x5k3/numpy_7fb65dbcf47341c59ade3b7c610386a5/.mesonpy-fios8xex/meson-logs/meson-log.txt
9.705       [end of output]
9.705
9.705   note: This error originates from a subprocess, and is likely not a problem with pip.
9.710 error: metadata-generation-failed
9.710
9.710 × Encountered error while generating package metadata.
9.710 ╰─> See above for output.
9.710
9.710 note: This is an issue with the package mentioned above, not pip.
9.710 hint: See above for details.
------

 2 warnings found (use docker --debug to expand):
 - InvalidDefaultArgInFrom: Default value for ARG public.ecr.aws/lambda/python:${PYTHON_VERSION} results in empty or invalid base image name (line 2)
 - FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/amd64" (line 2)
Dockerfile:11
--------------------
   9 |
  10 |     COPY titiler-pgstac-api/runtime/requirements.txt requirements.txt
  11 | >>> RUN python -m pip install -r requirements.txt "mangum>=0.14,<0.15" -t /asset
  12 |
  13 |     # Remove system dependencies
--------------------
ERROR: failed to solve: process "/bin/sh -c python -m pip install -r requirements.txt \"mangum>=0.14,<0.15\" -t /asset" did not complete successfully: exit code: 1

[gbegkas]

Thank you @hrodmn for the confirmation.

[hrodmn]

We can actually solve this with by pinning numpy and numexpr (#154)

I will do a patch release with that change and follow up with a larger release that upgrades all of the Lambda Python runtimes to 3.12 because we keep getting tripped up by this on 3.11!

[hrodmn]

@gbegkas v8.2.3 has been released with the fix from #154 which allows us to keep using the Python 3.11 runtime for now. I have plans to bring the image runtimes up to 3.12 or 3.13 soon so we don't have to keep installing patches like this!