[release/10.0] Fix regression caused by special marker type optimization (#123413)

[davidwrighton]

Backport of #123413 to release/10.0

/cc @copilot

Customer Impact

• Customer reported
• Found internally

[Select one or both of the boxes. Describe how this issue impacts customers, citing the expected and actual behaviors and scope of the issue. If customer-reported, provide the issue number.]

There are 2 fixes here

1. For issue VB.NET program using .NET runtime 10.0.2 throws EntryPointNotFoundException #123254, the problem is that the implementation of the optimized path interface map logic for interfaces which can be expanded by just manually expanding the first 2 layers of interfaces was not wired up at all. It turns out that this path is almost exclusively only available to VB.NET programs due to differences in the metadata generation algorithms between the VB and C# compilers. Notably, the VB.NET compiler does not expand the full set of interfaces into the outermost type, and C# compilation does. The fix is to implement handling for all of the various combinations of type that can appear at that point. See the newly added regression test for details on how to get into those situations. All known customers hitting this issue are using VB.NET, although it is theoretically possible to encounter this problem in C# if complex binary assembly compatibility scenarios are in play. (e.g Assembly A is built against assembly B version 1 which has a significantly less complex implied interface hierarchy, but is run against assembly B version 2 which has a more complex hierarchy.) However, this is a relatively unusual scenario.
2. For issue .NET 10 - System.ExecutionEngineException in System.Reflection.MethodBaseInvoker.InvokeWithOneArg #123318 there was an issue where if there are more than MaxGenericParametersForSpecialMarkerType but we may have attempted to use a special marker type to optimize type loading. The fix is to place the logic for determining if a type can be represented in an interface map via a special marker type into a common location, and to not forget the check against MaxGenericParametersForSpecialMarkerType. This is known to impact customers with large numbers of generic parameters on their types where all the generic parameters are the same. This issue is impacting customers using C# as well as VB.

Fixes #123254 and #123318

Regression

• Yes
• No

This regressed with the change to improve the performance of process startup. See PR #120712 . That PR had several bugs which were noticed within days of 10.0.2 releasing.

Testing

The fix was verified by running the customer repro cases, and additional exploratory tests. New tests were added which stressed the problematic behavior. Notably, the new tests are written in VB.NET which due to an implementation detail of the VB.NET compiler has significantly different interface loading behavior compared to C#. The issue #123318 was simply an issue of insufficient boundary condition testing.

Risk

Low compared to risk of fix remaining in the product.

This fix fixes issues in a codepath which was severely misbehaving. At worst this fix misses some special cases, but it's certainly better than it was before.

---

[Copilot]

Pull request overview

Backport to release/10.0 addressing regressions in CoreCLR’s “special marker type” interface-map optimization, including correct handling of additional interface-expansion shapes and enforcing the MaxGenericParametersForSpecialMarkerType boundary, with new regression coverage (VB + C#).

Changes:

• Centralize “special marker type” eligibility checks and apply them in signature/type loading and interface-map construction paths.
• Fix interface-map expansion to correctly handle additional combinations involving special marker types (notably seen with VB-emitted metadata).
• Add new regression tests for #123254 (VB) and #123318 (C#), and adjust test build props to support VB DefineConstants formatting.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/coreclr/vm/typehandle.h	Add DAC contract annotation to Instantiation::ContainsAllOneType.
src/coreclr/vm/siginfo.cpp	Use centralized eligibility check and improve retry signaling when invalid special-marker scenarios are detected.
src/coreclr/vm/methodtablebuilder.cpp	Expand/adjust exact interface-map building logic to correctly handle special-marker cases and fallback conditions.
src/coreclr/vm/clsload.hpp	Declare centralized EligibleForSpecialMarkerTypeUsage helper.
src/coreclr/vm/clsload.cpp	Implement centralized EligibleForSpecialMarkerTypeUsage helper.
src/tests/Directory.Build.props	Adjust DefineConstants delimiter for VB test projects.
src/tests/Loader/classloader/regressions/GitHub_123318/GitHub_123318.csproj	Add new regression test project for #123318.
src/tests/Loader/classloader/regressions/GitHub_123318/GitHub_123318.cs	Add C# regression test exercising >8 generic parameter boundary.
src/tests/Loader/classloader/regressions/GitHub_123254/GitHub_123254.vbproj	Add new VB regression test project for #123254.
src/tests/Loader/classloader/regressions/GitHub_123254/GitHub_123254.vb	Add VB regression tests covering special-marker interface-map expansion shapes.