[PLAN] Tracking issue for 2.10.1

[henryiii]

Known regressions regression Regression in a recent release :

• [BUG]: error: lvalue required as increment operand #4100
• [BUG]: ABI incompatibility between 2.9.2 and 2.10 #4105
• [BUG]: void value not ignored as it ought to be  #4117
• fix: add flag for overriding classic Python search values #4195 - fix: support pybind11 2.10 scikit_build_example#45
• fix: Revert pfect args make iterator #4234

Failed fixes:

• fix(cmake): support vcpkg, try 2 #4123

Planned features:

• add installation support for pkg-config dependency detection #4077
• feat: add entrypoint for cmake modules dir #4258
• Add PYBIND11_SIMPLE_GIL_MANAGEMENT option (cmake, C++ define) #4216

Fixes:

• embed.h Python 3.11 config.use_environment=1 + PYTHONPATH test #4119
• [BUG]: Python 3.12 support #4115
• fix: Python-3.12 compatibility #4168
• fix: improve bytes to str decoding error handling #4294

The next release of PyPy will help too. :)

[rwgk]

@henryiii @Skylion007 @ax3l

I've been turning over in my mind what we should do about #4105 — It leaves the 2.10 release in a doubt-full state, but unfortunately nobody provided even the slightest clue what the problem could be. I don't want to act on suspicions, as they could lead us astray badly and waste a lot of time. (Furthermore, if nobody cares enough to provide actionable clues, it puts the importance of the issue for them into doubt, too.)

In the meantime we also got:

• Add Eigen::Tensor & Eigen::TensorMap support #4201
• Add PYBIND11_SIMPLE_GIL_MANAGEMENT option (cmake, C++ define) #4216 (with Strip down gil.h to the simple version only. #4218 as the best resolution IMO)

Considering all of this together, I believe it will be best to abandon the 2.10 line b/o the doubts about ABI compatibility, and
go straight to 2.11, bumping the internals version (to resolve the doubts) but leave the #ifs of #4218 (internals.h) in place for now, so people can choose to go back to internals version 4 if it works for them and they are ok with the doubts.

[henryiii]

I'm still strongly in favor of a 2.10.1. It will be be no worse than 2.10.0, and it needs to be done before 3.11 is released, which is combine up soon. Both of those "bigger" changes above really should sit in master for a bit before release.

Also, I don't think an ABI bump with a "go back" option is all that useful. Distributions like conda-forge can't handle "selectable" ABI bumps. A version of pybind11 has a set ABI, and they can run a migration if it changes. Let's release a 2.10.1 later this week, then put in those larger changes, maybe go ahead and merge some of the other ABI bump ideas we've had in the past, then do a 2.11 properly.

[rwgk]

'm still strongly in favor of a 2.10.1. It will be be no worse than 2.10.0

I think it is: it gives 2.10 more legitimacy than it deserves. Unsuspecting users will upgrade to 2.10.1, only to discover that they will want to 2.11, i.e. we're leading people into wasting effort.

I'm happy to cut the 2.11 release, as soon as #4201 is merged.

How much longer do you think you need to fix up the scikit_build_example? All the rest looks like it could be done with a few hours of effort at most, if we don't drag our feet, but I don't know about scikit.

Also, I don't think an ABI bump with a "go back" option is all that useful.

If we explain well (a couple sentences will do here I think) we give people a choice. I don't like to force people's hands, unless there really is no other way, which is evidently (#4228) not the case here.

then do a 2.11 properly.

We cannot do 2.10 properly because of the doubts-without-clues, but I don't see why we cannot do 2.11 properly very soon, minus the scikit issue that's been lingering for 2 months already. But we could just document that that's broken since 2.10 and we still haven't been able to fix.

[henryiii]

I'd prefer not to leave a release series "broken" - some users will undoubtedly be stuck on 2.10 for a while. A 2.10.1 with better Python 3.11 support would be seen as a "safer" upgrade than 2.11.0. I've moved to 2.10 everywhere and have been very happy with it. The ABI problem doesn't affect all users.

Even if we release 2.11 a couple of weeks later, I think we should leave the 2.10 series in the best possible state.

If we explain well (a couple sentences will do here I think) we give people a choice.

Only if people read it, and only if they can. Conda-forge, for example, doesn't have a choice and this will trigger a migration. Given 2.10 is the 'last' of a very long ABI series, I wouldn't be surprised if it's a mini Python 2.

I'm happy to cut the 2.11 release, as soon as #4201 is merged.

"bigger" changes really should sit in master for a bit before release.

How much longer do you think you need to fix up the scikit_build_example?

Probably this week. By next Monday, at the latest.

[henryiii]

By "broken" I'm referring to the Python 3.11 fixes, not the ABI incompatibility, which I'm not too worried about, at the most I think it's likely quite small.

[rwgk]

How about:

• You cut the 2.10.1 release when you get to it.
• We branch 2.10 before merging Add Eigen::Tensor & Eigen::TensorMap support #4201 or Reproducer and fix for issue encountered in smart_holder update. #4228 (which ever comes first).
• I cut the 2.11.0 release as soon as possible, to unblock pytorch ([BUG]: Fatal Python error: PyThreadState_Get with c++ thread pool #4226) and get Eigen::Tensor (Add Eigen::Tensor & Eigen::TensorMap support #4201) into production.

I'd prefer not to leave a release series "broken" - some users will undoubtedly be stuck on 2.10 for a while.

If they can upgrade to 2.10.1, with the option to pin internals version 4, why would they not want to take a 2.11 instead? Especially to get rid of #4226, which is a long-standing but nevertheless serious issue.

"bigger" changes really should sit in master for a bit before release.

Why? Our testing is extremely thorough. We're not forcing anyone to upgrade. Let people choose their own pace. Holding developments back because of version numbering considerations looks to me like putting the cart in front of the horse: version numbers are meant to inform, e.g. to help staying organized in a progressing environment, not to stifle progress.

Using my own situation as an example: the Eigen::Tensor work looks super useful. Nobody has been asking me for it, but I want it deployed Google-internally asap, I bet it will get picked up quickly when people see it. — Only for completeness: Releases don't matter to me directly (I merge to smart_holder, then deploy from there) but having a close match with releases helps keeping everybody on the same page internally and externally. Therefore I'm happy to put in the effort cutting 2.11.

[henryiii]

That's fine and about what I was thinking - I'm not against a 3.11 in the near future, I just want a 3.10.1 too. I don't think we are really thinking that differently.

Our testing is extremely thorough.

It's never as thorough as real world usage, and we have quite a few projects running off of master, as evidenced by the quick issues and PRs whenever we break master. Usually breaking changes to master get noticed within a week. Our testing is on lots of systems, but only our unit testing, which isn't full coverage (in fact, we have prioritized a quick test suite over a thorough one). Google's test suite is great, but focused - some things it just doesn't catch. ABI breakages, for example, can't be caught by either. New features are often not caught by either (though a little rough edges on a new feature is better than breaking old code). Some things just aren't used at all by Google that are important to other projects. I'd recommend at least letting any large PRs sit in master for at least week before releasing. If it's entirely additive (#4201), I don't think it's as important, but #4226 is not additive.

version numbers are meant to inform

I think they are also a bit of a mark of stability. master could have bugs - we try our best, we keep tests passing, but it is fresh, un-vetted code. stable points at the last release, and doesn't update otherwise - releases are "stable", not fresh, code. If you release immediately after a big PR is merged, that's not "stable". A version number is intended to be more stable. It's a point at which we say the code is "ready for use". And if we catch a mistake, we backport fixes and release a new patch. So far, everything we have caught is enough for a combined patch release, but not so critical we need to rush a patch. (Ideally, we should have probably released two patch releases, actually).

If you live at head, then you want master, and that's fine, but it means you have thorough CI checking things. Many projects would rather stable code and are happy if it's a little stale. Many projects also live at head which is way waiting a little bit is useful. :)

We branch before merging ...

#4228 is already merged? Is it something I need to keep out of the 2.10.1 release? I'm totally okay with branching, it's not that much work and much better than holding up progress. If you want to merge something (that's been reviewed), tag it with 2.11 and I'll work in the v2.10 branch.

[henryiii]

Unless I missed something, #4228 is the NVIDIA workaround? That's fine for a patch?...

I finally understand what the problem is with scikit_build_example. Not totally sure what the best fix for it is, but I know what's going wrong.

[ax3l]

Yes, the enable_if version in that PR is great as a patch for the nvcc issue :)

[rwgk]

That's fine and about what I was thinking - I'm not against a 3.11 in the near future, I just want a 3.10.1 too. I don't think we are really thinking that differently.

Cool, thanks! (typo? 2.11, 2.10.1? mainly mentioning in case someone looks here later)

#4201 good another great push today and I hope we can merge it tomorrow or so. (I hope to merge it into smart_holder and deploy internally right after.)

What's the best way to handle the 2.10 branch?

Is see you marked master already as 2.11.0.dev1 (8d82f29). Do we need to do something special before merging #4201?

[rwgk]

@henryiii the branch point for 2.10.1 is b926396 (that commit to be included in 2.10.1)

[rwgk]

@Henry Centos7 has been failing for a whole week now. I see EOL is still 1 yr 9 mo off (https://cloud.google.com/compute/docs/eol/centos-eol-guidance), but I'm thinking we should drop this ancient platform anyway (release date was 2014-07-07, before pybind11 even existed). The centos builds are regular troublemakers, nibbling away more than their fair share from our time. Could we re-host PGI 22.3 on a less troublesome OS, ideally a modern one?