Revisit fsspec/boto3 dependency declarations

[victorlin]

Context

Lengthy comment above the following lines (pasted below for readability):

cli/setup.py

Lines 140 to 141 in 5a31a64

	"fsspec !=2023.9.1",
	"s3fs[boto3] >=2021.04.0, !=2023.9.1, <2025.12.0",

We use fsspec's S3 support, which has a runtime dep on s3fs. s3fs itself requires aiobotocore, which in turn requires very specific versions of botocore (because aiobotocore is a giant monkey-patch).

We also use boto3, which also requires botocore, usually with minimum versions closely matching the boto3 version (they're released in near lock step).

If we declare a dep on boto3 directly, this leads to conflicts during dependency resolution when a newer boto3 (from our declaration here) requires a newer botocore than is supported by s3fs → aiobotocore's declarations.

Resolve the issue by using a specially-provided package extra from s3fs (first introduced with 2021.4.0, removed in 2025.12.0) which causes them to declare an explicit dependency on aiobotocore's specially-provided package extra on boto3 so that dependency resolver can figure it out properly.

Note that the upper limit is not future-proof and will likely cause issues down the road. There may be a better combination to use here, but that needs extra digging.

More background:

• Dependency on aiobotocore renders packaging s3fs no longer possible fsspec/s3fs#357
• [batch] Unable to submit builds when installed with latest aiobotocore #133
• Directly specify optional dependencies awscli and boto3 fsspec/s3fs#994

What a mess.

Avoiding 2023.9.1 due to change in auto_mkdir parameter in fsspec/filesystem_spec#1358 that causes the error described in fsspec/s3fs#790

Possible solutions

• Declare direct dependencies on boto3 and fsspec[s3]
  • Awaiting release of fsspec/filesystem_spec@dd6945a

[jakob-keller]

A brief search of this codebase yielded several imports of boto3. Yet, that direct dependency on boto3 is not included in this project's "install_requires" list in setup.py.

If you fix this issue, dependency resolution should work out fine, including with the current release of s3fs.

[victorlin]

@jakob-keller I tried this out in #499. Unfortunately, the current release of s3fs isn't compatible with aiobotocore 3.0.0, which contains the relaxed dependency constraints that you've added. This has been addressed by fsspec/s3fs#998 so we're just waiting on that to be released.

[jakob-keller]

IMO, you should really provide version specifiers for ALL of your dependencies with lower AND upper bounds that are either known or can be expected to be compatible with your project. That would help avoid such issues in the future, especially if other projects did the same.

Fictitious example: you require a feature that the requests library introduced in release 1.20.0. The current release 2.32.5 is known to work. An appropriate version specifier would be: requests >=1.20.0 <3.0.0. Any future requests release <3.0.0 should not contain any breaking changes and should be compatible. If requests 3.0.0 introduces changes that break your project, your users would not be affected and you'd have time to make adjustments to the code AND version specifier and ship an updated release to your users.

The root cause for your issue seems to lie with fsspec. They should also add appropriate version specifiers: https://github.com/fsspec/filesystem_spec/blob/2951d1f783bbc7b7d292f3155784b9d6021b0309/pyproject.toml#L65

Maybe you could raise this with the fsspec project?

For now, besides adding the direct boto3 dependency, you mights need to keep the transient s3fs dependency as a temporary work-around, including the specified lower bound (!), to prevent ending up with a very old version. If fsspec provides a fix, you could use that release as a lower bound and drop the s3fs dependency.

[jakob-keller]

fsspec maintainer has already responded positively: aio-libs/aiobotocore#1424 (comment)

[victorlin]

@jakob-keller thanks for the link. I agree on the benefits of lower and upper bounds, but upper bounds are harder to justify with the CalVer scheme used by fsspec and s3fs, where any version could introduce breaking changes.

Given that things are moving along for fsspec/s3fs, I'll wait for the new releases before coming back to this.