Skip to content

Conversation

@clbarnes
Copy link
Contributor

Which issue does this PR close?

Closes #6984 .

Rationale for this change

Instantiating Operators from URLs greatly increases flexibility for consumers of the openDAL API as an abstract backend.

What changes are included in this PR?

Add from_uri static methods to the Operator and AsyncOperator classes in the python bindings.

Are there any user-facing changes?

Two new methods are available to users of the python library.

AI Usage Statement

None

Other notes

I am having issues with my development environment which the devcontainer is not making any easier, so I'm flying a bit blind and relying on CI.

@clbarnes
Copy link
Contributor Author

Tests pass but I'm not clear how best to test the new functionality as it would mean parsing the config provided by the setup_config fixture and then constructing a URI out of it, which is going to be different for each service.

@clbarnes clbarnes marked this pull request as ready for review December 12, 2025 16:35
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels Dec 12, 2025
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Dec 12, 2025
@Xuanwo Xuanwo changed the title Python: Add from_uri constructors for Operators feat(bindings/python): Add from_uri constructors for Operators Dec 12, 2025
@Xuanwo
Copy link
Member

Xuanwo commented Dec 12, 2025

Thank you for working on this! I will invite @chitralverma to take a review on this.

@chitralverma
Copy link
Contributor

@clbarnes can you run stubgen with just to get the python stubs as well?

@clbarnes
Copy link
Contributor Author

clbarnes commented Dec 15, 2025

I found that running stubgen locally drops a bunch of services and deletes a lot of generated code as well - possibly all the services which have been split out of core into their own crates?

--- a/bindings/python/src/services.rs
+++ b/bindings/python/src/services.rs
@@ -62,16 +62,6 @@ submit! {
 #[pyo3(rename_all = "PascalCase")]
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
 pub enum PyScheme {
-    #[cfg(feature = "services-aliyun-drive")]
-    AliyunDrive,
-    #[cfg(feature = "services-alluxio")]
-    Alluxio,
-    #[cfg(feature = "services-azblob")]
-    Azblob,
-    #[cfg(feature = "services-azdls")]
-    Azdls,
-    #[cfg(feature = "services-azfile")]
-    Azfile,
     #[cfg(feature = "services-b2")]
     B2,
     #[cfg(feature = "services-cacache")]
@@ -84,24 +74,14 @@ pub enum PyScheme {
     Dropbox,
     #[cfg(feature = "services-fs")]
     Fs,
-    #[cfg(feature = "services-ftp")]
-    Ftp,
-    #[cfg(feature = "services-gcs")]
-    Gcs,
     #[cfg(feature = "services-gdrive")]
     Gdrive,
-    #[cfg(feature = "services-ghac")]
-    Ghac,
     #[cfg(feature = "services-gridfs")]
     Gridfs,
-    #[cfg(feature = "services-hdfs-native")]
-    HdfsNative,
     #[cfg(feature = "services-http")]
     Http,
     #[cfg(feature = "services-huggingface")]
     Huggingface,
-    #[cfg(feature = "services-ipfs")]
-    Ipfs,
     #[cfg(feature = "services-ipmfs")]
     Ipmfs,
     #[cfg(feature = "services-koofr")]

etc.. That comes from the cargo run --quiet --manifest-path=../../dev/Cargo.toml -- generate -l python step.

The second step doesn't finish successfully - cargo run --quiet --bin stub_gen.

opendal/bindings/python on  py-from-uri [$!⇡] is 📦 v0.47.0 via 🐍 v3.14.0 (opendal) via 🦀 v1.91.0 took 1m3s 
❯ cargo run --quiet --bin stub_gen
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Fatal Python error: Failed to import encodings module
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001f16b6080 [main] (most recent call first):
  <no Python frame>

but python -c 'import encodings' works.

This could possibly resolved using the devcontainer, but that has issues too: #6992

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

new feature: expose Operator::from_uri to other language bindings

3 participants