[proposal] New resolver configuration

docs/index.rst

@@ -39,6 +39,7 @@ those special cases directly into fromager.
    cli.rst
    glossary.rst
    develop.md
+   proposals/index.rst
 
 What's with the name?
 ---------------------

docs/proposals/index.rst

@@ -0,0 +1,7 @@
+Fromager Enhancement Proposals
+==============================
+
+.. toctree::
+   :maxdepth: 1
+
+   new-resolver-config

docs/proposals/new-resolver-config.md

@@ -0,0 +1,190 @@
+# New resolver and download configuration
+
+- Author: Christian Heimes
+- Created: 2026-02-24
+- Status: Open
+
+## What
+
+This enhancement document proposal a new approach to configure the package
+resolver and source / sdist downloader. The new settings are covering a
+wider range of use cases. Common patterns like building a package from a
+git checkout will no longer need custom Python plugins.
+
+## Why
+
+In downstream, we are encountering an increasing amount of packages that do
+not build from sdists on PyPI. Either package maintainers are not uploading
+source distributions to PyPI or sdists have issues. In some cases, packages
+use a midstream fork that is not on PyPI. The sources need to be build from
+git.
+
+Because Fromager <= 0.76 does not have declarative settings for GitHub/GitLab
+resolver or cloning git repositories, we have to write custom Python plugins.
+The plugins are a maintenance burden.
+
+## Goals
+
+- support common use cases with package settings instead of custom plugin code
+- cover most common resolver scenarios:
+  - resolve package on PyPI (sdist, wheel, or both)
+  - resolve package on GitHub or GitLab with custom tag matcher
+- cover common sdist download and build scenarios:
+  - sdist from PyPI
+  - prebuilt wheel from PyPI
+  - download tarball from URL
+  - clone git repository
+  - download an artifact from GitHub / GitLab release or tag
+  - build sdist with PEP 517 hook or plain tarball
+- support per-variant setting, e.g. one variant uses prebuilt wheel while the
+  rest uses sdist.
+- gradual migration path from old system to new configuration
+
+## Non-goals
+
+- The new system will not cover all use cases. Some specific use cases will
+  still require custom code.
+- Retrieval of additional sources is out of scope, e.g. a package `egg` that
+  needs `libegg-{version}.tar.gz`.
+- Provide SSH transport for git. The feature can be added at a later point
+  when it's needed.
+- Extra options for authentication. The `requests` library and `git` CLI can
+  use `$HOME/.netrc` for authentication.
+  > **NOTE:** `requests` also supports `NETRC` environment variable,
+    `libcurl` and `git` _only_ support `$HOME/.netrc`.
+
+## How
+
+The new system will use a new top-level configuration key `source`. The old
+`download_source` and `resolver_dist` settings will stay supported for a
+while. Eventually the old options will be deprecated and removed.
+
+The resolver and source downloader can be configuration for all variants of
+a package as well as re-defined for each variant. A package can be configured
+as prebuilt for all variants or a variant can have a different resolver and
+sources than other.
+
+Each use case is handled a provider profile. The profile name acts as a tag
+([discriminated union](https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions)).
+Each use case has a well-defined set of mandatory and optional arguments.
+
+**Example:**
+
+```yaml
+source:
+  # `pypi-sdist` is the default provider
+  provider: pypi-sdist
+variants:
+  egg:
+    source:
+      # resolve and download prebuilt wheel
+      provider: pypi-prebuilt
+      index_url: https://custom-index.example/simple
+  spam:
+    source:
+        # resolve on GitLab, clone tag over https, build an sdist with PEP 517 hook
+        provider: gitlab-git
+        url: https://gitlab.example/spam/spam
+        matcher_factory: package_plugins.matchers:midstream_matcher_factory
+        build_sdist: pep517
+  viking:
+    source:
+        # resolve on PyPI, git clone, and build as tarball
+        provider: pypi-git
+        clone_url: https://git.example/viking/viking.git
+        tag: 'v{version}'
+        build_sdist: tarball
+  camelot:
+    source:
+      # On second thought, let's not go to Camelot. It is a silly place.
+      provider: not-available
+```
+
+### Profiles
+
+- The `pypi-sdist` profile resolve versions on PyPI or PyPI-compatible index.
+  It only takes sdists into account and downloads the sdist from the index.
+  The profile is equivalent to the current default settings with
+  `include_sdists: true` and `include_wheels: false`.
+
+- The `pypi-prebuilt` profile resolve versions of platform-specific wheels
+  on PyPI and downloads the pre-built wheel. The profile is equivalent to
+  `include_sdists: false`, `include_wheels: true`, and variant setting
+  `pre_build: true`.
+
+- The `pypi-download` resolve versions of any package on PyPI and downloads
+  a tarball from an external URL (with `{version}` variable in download URL).
+  It takes any sdist and any wheel into account. The profile is equivalent
+  with `include_sdists: true`, `include_wheels: true`, `ignore_platform: true`,
+  and a `download_source.url`.
+
+- The `pypi-git` is similar to the `pypi-download` profile. Instead of
+  downloading a tarball, it clones a git repository at a specific tag.
+
+- The `gitlab-git` and `github-git` profiles use the `GitLabTagProvider` or
+  `GitHubTagProvider` to resolve versions. The profiles git clone a project
+  over `https` or `ssh` protocol.
+
+- The `gitlab-download` and `github-download` are similar to `gitlab-git` and
+  `github-git` profiles. Instead of cloning a git repository, they download
+  a git tarball or an release artifact.
+
+- The `not-available` profile raises an error. It can be used to block a
+  package and only enable it for a single variant.
+
+Like pip's VCS feature, all git clone operations automatically retrieve all
+submodules recursively.
+
+### URL schema for git
+
+The resolver and `Candidate` class do not support VCS URLs, yet. Fromager can
+adopt pip's [VCS support](https://pip.pypa.io/en/stable/topics/vcs-support/)
+syntax. The URL `git+https://git.example/viking/viking.git@v1.1.0` clones the
+git repository over HTTPS and checks out the tag `v1.1.0`.
+
+### Matcher factory
+
+The matcher factory argument is an import string. The string must resolve to
+a callable that accepts a `ctx` argument and returns a `re.Pattern`
+(recommended) or `MatchFunction`. If the return value is a pattern object,
+then it must have exactly one match group. The pattern is matched with
+`re.match`.
+
+The default matcher factory parsed the tag with `packaging.version.Version`
+and ignores any error.
+
+```python
+import re
+
+from fromager import context, resolver
+from packaging.version import Version
+
+
+def matcher_factory_pat(ctx: context.WorkContext) -> re.Pattern | resolver.MatchFunction:
+    # tag must must v1.2+midstream.1.cpu and results in Version("1.2+midstream.1")
+    variant = re.escape(ctx.variant)
+    pat = rf"^v(.*\+midstream\.\d+)\.{variant}$"
+    return re.compile(pat)
+
+
+def matcher_factory_func(ctx: context.WorkContext) -> re.Pattern | resolver.MatchFunction:
+    def pep440_matcher(identifier: str, item: str) -> Version | None:
+        try:
+            return Version(item)
+        except ValueError:
+            return None
+    return pep440_matcher
+```
+
+### Deprecations
+
+- `download_source.url` is handled by `pypi-download` profile or
+  `release_artifact` parameter of `github` or `gitlab` provider
+- `download_source.destination_filename` is not needed. All sdists use
+  standard `{dist_name}-{version}.tar.gz` file name
+- `resolver_dist.sdist_server_url` is replaced by `index_url` parameter.
+  All `pypi-*` profile support a custom index.
+- `git_options.submodules` is not needed. Like pip, Fromager will always
+  clone all submodules.
+- variant settings `wheel_server_url` and `pre_build` are replaced by
+  `pypi-prebuilt` profile