Please restore _Py_InitializeMain or provide an alternate API for it

[kovidgoyal]

As discussed in #129033, opening a new issue at the request of @vstinner. See #129033 (comment) for the reasons this API is needed.

In short, a way to run python code and insert a meta importer into sys.meta_path before the "main" phase of initialization runs is needed. This allows, for instance, to bundle all python code as pre-compiled bytecode into a single mmapped file and directly load it from there for improved performance and hygiene.

I am aware that the state of the interpreter after "core" initialization is not entirely specified, however, in practice it is complete enough to write a meta path importer.

This facility has been used for many years by at least two projects that I know of: bypy (link to source code) https://github.com/kovidgoyal/bypy/blob/master/bypy/freeze/bypy-freeze.h#L835 and PyOxidizer: https://github.com/indygreg/PyOxidizer/blob/1ceca8664c71f39e849ce4873e00d821504b32bd/pyembed/src/technotes.rs#L155

I dont much care about this API in particular, but I would like some way to hook into the initialization process after "core" initialization and before "main" initialization.

Some context about use case. bypy is used in the calibre and kitty programs, both of which embed a "frozen" Python interpreter. _Py_InitializeMain() allows that frozen importer to import all its python code, including stdlib code from a single mmaped file contianing pre-compiled bytecode. This has obvious performance benefits especially on windows.

Linked PRs

• gh-142417: Add PyInitConfig_SetInitCallback() function #142420

[vstinner]

If this feature is used, we should add a clean C API to use it, rather than just restore the private API to move back to the status quo.

I dislike config._init_main=0 to stop the initialization at the "core" phase, and then having to call _Py_InitializeMain() to "finish" the initialization to the main phase.

IMO a simple callback would be enough for bypy and PyOxidizer use cases: it would be called between core and main initialization phases. In practice, it would be called once the Python interpreter is initialized and before the first import. Something like:

PyStatus PyConfig_SetInitCallback(PyStatus (*callback)(void *arg), void *arg);

int PyInitConfig_SetInitCallback(PyInitConfig *config, PyStatus (*callback)(void *arg), void *arg);

@kovidgoyal: Would it fit your use cases?

cc @ncoghlan @ericsnowcurrently

[vstinner]

For the 3.14 branch, we can restore the private _Py_InitializeMain() function, but only in Python 3.14.3. It means that Python 3.14.0, 3.14.1 and 3.14.2 cannot be used.

[kovidgoyal]

Yes, I can make that work in bypy. It will require a little bit of
re-writing, but no big deal. Far more important for me is to have an
official stable API for this. Whatever shape that API takes I can adapt
to it, provided it allows me to run my callback immediately before the
first import but after the interpreter is initialized. Your proposed
PyConfig_SetInitCallback() would be fine for me.

Personally, I am anyway patching Python 3.14 currently to restore that
function (since I embed private python in my apps, that's easy to do).
So to me it does not matter what happens in 3.14 lifecycle or later. I
cant speak for other projects, of course.