thread state %p must be current when releasing" in PyGILState_Release

[Jay-ju]

scene

Multiple concurrent calls to the dataset scanner, and there are also some limits. When a certain scanner ends,

reproduce script

io_config = IOConfig(
    s3=S3Config(
        endpoint_url="https://kamui.tos-s3-cn-beijing.ivolces.com",
        force_virtual_addressing=True,
        key_id="xx",
        access_key="xx==",
        verify_ssl=True,
        region_name="cn-beijing"
    )
)


dataset = daft.read_lance(url="s3://kamui/las/lance/ds_13e2k2mwq336zi3nf3m6.lance/", io_config=io_config)

df = daft.sql("select * from dataset limit 100")
df.show(6)

error stack

__pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007f79737a6f4f in __pthread_kill_internal (signo=6, 
    threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007f7973757fb2 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#3  0x00007f7973742472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x000000000049f959 in fatal_error_exit (status=-1)
    at /usr/local/src/conda/python-3.9.19/Python/pylifecycle.c:2201
#5  fatal_error (stream=stream@entry=0x7f79738ef680 <_IO_2_1_stderr_>, 
    header=header@entry=0, prefix=prefix@entry=0x0, msg=msg@entry=0x0, 
    status=status@entry=-1)
    at /usr/local/src/conda/python-3.9.19/Python/pylifecycle.c:2219
#6  0x00000000004a0223 in _Py_FatalErrorFormat (
    func=func@entry=0x68af50 <__func__.1.lto_priv.10> "PyGILState_Release", 
    format=format@entry=0x61e1c0 "thread state %p must be current when releasing") at /usr/local/src/conda/python-3.9.19/Python/pylifecycle.c:2335
#7  0x000000000046538d in PyGILState_Release (oldstate=PyGILState_UNLOCKED)
    at /usr/local/src/conda/python-3.9.19/Python/pystate.c:1418
#8  0x00007f78c54aab44 in pyo3::gil::{impl#1}::drop (self=0x7f7916ddcbfc)
    at src/gil.rs:242
#9  0x00007f78c548f897 in core::ptr::drop_in_place<pyo3::gil::GILGuard> ()
    at /root/.rustup/toolchains/1.86.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:523
#10 0x00007f78bc9cfb3f in pyo3::marker::Python::with_gil<lance::executor::{impl#0}::spawn_impl::{closure_env#1}<lance::reader::{impl#1}::next::{async_block_env#0}>, core::result::Result<core::option::Option<core::result::Result<arrow_array::record_batch::RecordBatch, lance_core::error::Error>>, pyo3::err::PyErr>> (
    f=...)
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.24.2/src/marker.rs:412
#11 0x00007f78bcffa535 in lance::executor::BackgroundExecutor::spawn_impl<lance::reader::{impl#1}::next::{async_block_env#0}> (
    self=0x7f78c7723010 <lance::RT::h97ac8331dcd658c2>, task=...)
    at src/executor.rs:85
#12 0x00007f78bcff843d in lance::executor::{impl#0}::spawn::{closure#1}::{closure#0}<lance::reader::{impl#1}::next::{async_block_env#0}> ()
    at src/executor.rs:62
#13 0x00007f78bc9c7b9f in pyo3::marker::Python::allow_threads<core::result::Result<core::option::Option<core::result::Result<arrow_array::record_batch::RecordBatch, lance_core::error::Error>>, pyo3::err::PyErr>, lance::executor::{impl#0}::spawn::{closure#1}::{closure_env#0}<lance::reader::{impl#1}::next::{async_block_env#0}>> (f=...)
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.24.2/src/marker.rs:520
#14 0x00007f78bcff7d5a in lance::executor::{impl#0}::spawn::{closure#1}<lance::reader::{impl#1}::next::{async_block_env#0}> () at src/executor.rs:62
#15 0x00007f78bc9cee79 in pyo3::marker::Python::with_gil<lance::executor::{impl#0}::spawn::{closure_env#1}<lance::reader::{impl#1}::next::{async_block_env#0}>, core::result::Result<core::option::Option<core::result::Result<arrow_array::record_batch::RecordBatch, lance_core::error::Error>>, pyo3::err::PyErr>> (f=...)
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.24.2/src/marker.rs:411
#16 0x00007f78bcff729b in lance::executor::BackgroundExecutor::spawn<lance::reader::{impl#1}::next::{async_block_env#0}> (
    self=0x7f78c7723010 <lance::RT::h97ac8331dcd658c2>, py=..., task=...)
    at src/executor.rs:62
--Type <RET> for more, q to quit, c to continue without paging--
#17 0x00007f78bcce3785 in lance::reader::{impl#1}::next (self=0x7f77b8002a10)
    at src/reader.rs:61
#18 0x00007f78c6a6ebee in alloc::boxed::iter::{impl#0}::next<(dyn arrow_array::record_batch::RecordBatchReader<Item=core::result::Result<arrow_array::record_batch::RecordBatch, arrow_schema::error::ArrowError>> + core::marker::Send), alloc::alloc::Global> (self=0x7f77b8002140)
    at /root/.rustup/toolchains/1.86.0-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed/iter.rs:21
#19 0x00007f78c69f1074 in arrow_array::ffi_stream::ExportedArrayStream::get_next (self=0x7f7916ddd480, out=0x7f7916ddd4f0) at src/ffi_stream.rs:240
#20 0x00007f78c69f088d in arrow_array::ffi_stream::get_next (
    stream=0x7f77b8003c98, array=0x7f7916ddd4f0) at src/ffi_stream.rs:138
#21 0x00007f795ca02649 in arrow::(anonymous namespace)::ArrayStreamBatchReader<false>::ReadNext(std::shared_ptr<arrow::RecordBatch>*) ()
   from /data00/code/emr/Daft/.venv/lib/python3.9/site-packages/pyarrow/libarrow.so.1900
#22 0x00007f7964d823a9 in __pyx_pw_7pyarrow_3lib_17RecordBatchReader_7read_next_batch(_object*, _object* const*, long, _object*) ()
   from /data00/code/emr/Daft/.venv/lib/python3.9/site-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#23 0x00007f7964cd7858 in __pyx_pf_7pyarrow_3lib_17RecordBatchReader_4__next__(__pyx_obj_7pyarrow_3lib_RecordBatchReader*) ()
   from /data00/code/emr/Daft/.venv/lib/python3.9/site-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so
#24 0x00000000004e9493 in _PyEval_EvalFrameDefault (tstate=<optimized out>, 
    f=0x7f79239bccf0, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:2206
#25 0x000000000050bb1c in _PyEval_EvalFrame (throwflag=<optimized out>, 
    f=0x7f79239bccf0, tstate=0x7f77b8001ab0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#26 gen_send_ex (gen=0x7f79239d6ac0, arg=<optimized out>, exc=<optimized out>, 
    closing=<optimized out>)
    at /usr/local/src/conda/python-3.9.19/Objects/genobject.c:215
#27 0x00000000004e8232 in _PyEval_EvalFrameDefault (tstate=<optimized out>, 
    f=0x7f792b051ac0, throwflag=<optimized out>)
    at /usr/local/src/conda/python-3.9.19/Python/ceval.c:3308
#28 0x000000000050bb1c in _PyEval_EvalFrame (throwflag=<optimized out>, 
    f=0x7f792b051ac0, tstate=0x7f77b8001ab0)
    at /usr/local/src/conda/python-3.9.19/Include/internal/pycore_ceval.h:40
#29 gen_send_ex (gen=0x7f79239d6b30, arg=<optimized out>, exc=<optimized out>, 
    closing=<optimized out>)
    at /usr/local/src/conda/python-3.9.19/Objects/genobject.c:215
#30 0x000000000050a5ee in PyIter_Next (iter=<optimized out>, 
    iter=<optimized out>)
    at /usr/local/src/conda/python-3.9.19/Objects/abstract.c:2702
#31 0x00007f7971d2319e in pyo3::types::iterator::<impl pyo3::instance::Borrowed<pyo3::types::iterator::PyIterator>>::next::h00f3a9cf55f1b2c2 ()
    at src/types/iterator.rs:111
#32 0x00007f796bf6b06d in pyo3::types::iterator::<impl core::iter::traits::iterator::Iterator for pyo3::instance::Bound<pyo3::types::iterator::PyIterator>>::next::ha8ff32d6fa856fa1 ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.24.2/src/types/iterator.rs:95
#33 0x00007f796bfcc25e in daft_micropartition::python::read_pyfunc_into_table_iter::{{closure}}::{{closure}}::{{closure}}::hb32804589672be77 ()
    at src/daft-micropartition/src/python.rs:1141
#34 0x00007f796bf9f49e in pyo3::marker::Python::with_gil::hf73c35f2e123902f ()
--Type <RET> for more, q to quit, c to continue without paging--
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.24.2/src/marker.rs:411
#35 0x00007f796bfcc1de in daft_micropartition::python::read_pyfunc_into_table_iter::{{closure}}::{{closure}}::h511d1f298315bdf7 ()
    at src/daft-micropartition/src/python.rs:1140
#36 0x00007f796bf9a28e in <core::iter::sources::from_fn::FromFn<F> as core::iter::traits::iterator::Iterator>::next::h355902434cf4af70 ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/sources/from_fn.rs:71
#37 0x00007f796a33b54c in core::ops::function::FnOnce::call_once::hc930aa17f27699e4 ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250
#38 0x00007f796a49c1f9 in and_then_or_clear<core::iter::sources::from_fn::FromFn<daft_micropartition::python::read_pyfunc_into_table_iter::{closure#1}::{closure_env#0}>, core::result::Result<daft_recordbatch::RecordBatch, daft_micropartition::Error>, fn(&mut core::iter::sources::from_fn::FromFn<daft_micropartition::python::read_pyfunc_into_table_iter::{closure#1}::{closure_env#0}>) -> core::option::Option<core::result::Result<daft_recordbatch::RecordBatch, daft_micropartition::Error>>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/flatten.rs:808
#39 0x00007f796a4932a8 in next<core::iter::adapters::map::Map<alloc::vec::into_iter::IntoIter<pyo3::instance::Py<pyo3::types::any::PyAny>, alloc::alloc::Global>, daft_micropartition::python::read_pyfunc_into_table_iter::{closure_env#1}>, core::iter::sources::from_fn::FromFn<daft_micropartition::python::read_pyfunc_into_table_iter::{closure#1}::{closure_env#0}>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/flatten.rs:575
#40 0x00007f796a4910fe in <core::iter::adapters::flatten::FlatMap<I,U,F> as core::iter::traits::iterator::Iterator>::next::h7d8e077ad326f1a7 ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/flatten.rs:64
#41 0x00007f796a47dab5 in next<core::result::Result<daft_recordbatch::RecordBatch, daft_micropartition::Error>, core::iter::adapters::flatten::FlatMap<alloc::vec::into_iter::IntoIter<pyo3::instance::Py<pyo3::types::any::PyAny>, alloc::alloc::Global>, core::iter::sources::from_fn::FromFn<daft_micropartition::python::read_pyfunc_into_table_iter::{closure#1}::{closure_env#0}>, daft_micropartition::python::read_pyfunc_into_table_iter::{closure_env#1}>, usize, daft_micropartition::python::read_pyfunc_into_table_iter::{closure_env#2}> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/scan.rs:46
#42 0x00007f796a6dba71 in next<core::result::Result<daft_recordbatch::RecordBatch, common_error::error::DaftError>, core::iter::adapters::scan::Scan<core::iter::adapters::flatten::FlatMap<alloc::vec::into_iter::IntoIter<pyo3::instance::Py<pyo3::types::any::PyAny>, alloc::alloc::Global>, core::iter::sources::from_fn::FromFn<daft_micropartition::python::read_pyfunc_into_table_iter::{closure#1}::{closure_env#0}>, daft_micropartition::python::read_pyfunc_into_table_iter::{closure_env#1}>, usize, daft_micropartition::python::read_pyfunc_into_table_iter::{closure_env#2}>, daft_local_execution::sources::scan_task::stream_scan_task::{async_fn#0}::{closure_env#11}> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:107
#43 0x00007f796a79ca7b in <futures_util::stream::iter::Iter<I> as futures_core::stream::Stream>::poll_next::h28f02580e89e78c1 ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/iter.rs:60
--Type <RET> for more, q to quit, c to continue without paging--
#44 0x00007f796c06d079 in poll_next<alloc::boxed::Box<(dyn futures_core::stream::Stream<Item=core::result::Result<daft_recordbatch::RecordBatch, common_error::error::DaftError>> + core::marker::Send), alloc::alloc::Global>> ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/stream.rs:130
#45 0x00007f796a77ec27 in poll_next<core::pin::Pin<alloc::boxed::Box<(dyn futures_core::stream::Stream<Item=core::result::Result<daft_recordbatch::RecordBatch, common_error::error::DaftError>> + core::marker::Send), alloc::alloc::Global>>, daft_local_execution::sources::scan_task::stream_scan_task::{async_fn#0}::{closure_env#12}> ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/map.rs:58
#46 0x00007f796a77f4de in futures_util::stream::stream::StreamExt::poll_next_unpin::hae94c4673f2a4b4c ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/mod.rs:1638
#47 0x00007f796a5ee6f6 in <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll::h60af7bf66bb53035 ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/stream/stream/next.rs:32
#48 0x00007f796a46e494 in {async_fn#0} ()
    at src/daft-local-execution/src/sources/scan_task.rs:377
#49 0x00007f796a6f7dd9 in poll<alloc::boxed::Box<daft_local_execution::sources::scan_task::forward_scan_task_stream::{async_fn_env#0}, alloc::alloc::Global>>
    ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/future.rs:124
#50 0x00007f796a51b8c5 in tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h54fdd5c532e9bb9f ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:331
#51 0x00007f796a50fbb3 in with_mut<tokio::runtime::task::core::Stage<core::pin::Pin<alloc::boxed::Box<daft_local_execution::sources::scan_task::forward_scan_task_stream::{async_fn_env#0}, alloc::alloc::Global>>>, core::task::poll::Poll<core::result::Result<(), common_error::error::DaftError>>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<daft_local_execution::sources::scan_task::forward_scan_task_stream::{async_fn_env#0}, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>> ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/loom/std/unsafe_cell.rs:16
#52 poll<core::pin::Pin<alloc::boxed::Box<daft_local_execution::sources::scan_task::forward_scan_task_stream::{async_fn_env#0}, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>> ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:320
#53 0x00007f796a3d4336 in tokio::runtime::task::harness::poll_future::{{closure}}::h7bcde384696fe6a7 ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:532

#54 0x00007f796a6c6b9d in <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h7f856c372bacaf0e ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272
#55 0x00007f796a69524e in do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<core::pin::Pin<alloc::b--Type <RET> for more, q to quit, c to continue without paging--
oxed::Box<daft_local_execution::sources::scan_task::forward_scan_task_stream::{async_fn_env#0}, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<core::result::Result<(), common_error::error::DaftError>>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:587
#56 0x00007f796a690e6b in __rust_try ()
   from /data00/code/emr/Daft/daft/daft.abi3.so
#57 0x00007f796a6679bc in try<core::task::poll::Poll<core::result::Result<(), common_error::error::DaftError>>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<daft_local_execution::sources::scan_task::forward_scan_task_stream::{async_fn_env#0}, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:550
#58 catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<daft_local_execution::sources::scan_task::forward_scan_task_stream::{async_fn_env#0}, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<core::result::Result<(), common_error::error::DaftError>>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:358
#59 0x00007f796a3a07aa in tokio::runtime::task::harness::poll_future::h1c01ad566424318c ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:520
#60 0x00007f796a3f4336 in tokio::runtime::task::harness::Harness<T,S>::poll_inner::hda45ff9db3f34f11 ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:209
#61 0x00007f796a41d097 in tokio::runtime::task::harness::Harness<T,S>::poll::h2c3caabd677a9a4c ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:154
#62 0x00007f796a8333f0 in tokio::runtime::task::raw::poll::h8f7c8b8872b74c8e ()
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:271
#63 0x00007f79704b8acb in tokio::runtime::task::raw::RawTask::poll::hac3e83d0324bba2d () at src/runtime/task/raw.rs:201
#64 0x00007f79704ef305 in tokio::runtime::task::LocalNotified<S>::run::hfc7e2e42860f0b8e () at src/runtime/task/mod.rs:463
#65 0x00007f79704a001d in {closure#0} ()
    at src/runtime/scheduler/multi_thread/worker.rs:600
#66 0x00007f797049fe76 in with_budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> () at src/task/coop/mod.rs:167
#67 budget<core::result::Result<alloc::boxed::Box<tokio::runtime::scheduler::multi_thread::worker::Core, alloc::alloc::Global>, ()>, tokio::runtime::scheduler::multi_thread::worker::{impl#1}::run_task::{closure_env#0}> ()
    at src/task/coop/mod.rs:133
#68 run_task () at src/runtime/scheduler/multi_thread/worker.rs:594
#69 0x00007f797049f73d in tokio::runtime::scheduler::multi_thread::worker::Context::run::hfad4281d1e7b4a90 ()
    at src/runtime/scheduler/multi_thread/worker.rs:554

--Type <RET> for more, q to quit, c to continue without paging--
#70 0x00007f797049f1d4 in tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h717f0cc52bc676c9 ()
    at src/runtime/scheduler/multi_thread/worker.rs:507
#71 0x00007f79704e9ccf in tokio::runtime::context::scoped::Scoped<T>::set::h23f86e4b3550d375 () at src/runtime/context/scoped.rs:40
#72 0x00007f79704972a1 in tokio::runtime::context::set_scheduler::{{closure}}::h17d16d33d624e979 () at src/runtime/context.rs:176
#73 0x00007f79704d4a66 in try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(), tokio::runtime::scheduler::multi_thread::worker::run::{closure#0}::{closure_env#0}>, ()> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:308
#74 0x00007f79704d3107 in std::thread::local::LocalKey<T>::with::hc6cc39a6b04286af ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272
#75 0x00007f7970497232 in tokio::runtime::context::set_scheduler::hae5786726f829496 () at src/runtime/context.rs:176
#76 0x00007f797049f0e1 in tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h8b292652770d9e20 ()
    at src/runtime/scheduler/multi_thread/worker.rs:502
#77 0x00007f79704d16d8 in tokio::runtime::context::runtime::enter_runtime::hb941f9fe7be32b10 () at src/runtime/context/runtime.rs:65
#78 0x00007f797049ee84 in tokio::runtime::scheduler::multi_thread::worker::run::he111b8b8ff70c127 () at src/runtime/scheduler/multi_thread/worker.rs:494
#79 0x00007f797049eba7 in tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h88563b504ad2a519 ()
    at src/runtime/scheduler/multi_thread/worker.rs:460
#80 0x00007f79704f6064 in <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hd8454e4811d80211 ()
    at src/runtime/blocking/task.rs:42
#81 0x00007f79704ff3f6 in tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h90c09983c6747d20 () at src/runtime/task/core.rs:331
#82 0x00007f79704fed48 in with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>> ()
    at src/loom/std/unsafe_cell.rs:16
#83 poll<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule> () at src/runtime/task/core.rs:320
#84 0x00007f797047821e in tokio::runtime::task::harness::poll_future::{{closure}}::h625460777c52f152 () at src/runtime/task/harness.rs:532
#85 0x00007f79704b9c41 in <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0795f869b3e38f29 ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272
#86 0x00007f79704c2576 in do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:587
#87 0x00007f79704ea3db in __rust_try ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /data00/code/emr/Daft/daft/daft.abi3.so
#88 0x00007f79704e7bc8 in try<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:550
#89 catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<tokio::runtime::blocking::task::BlockingTask<tokio::runtime::scheduler::multi_thread::worker::{impl#0}::launch::{closure_env#0}>, tokio::runtime::blocking::schedule::BlockingSchedule>>, core::task::poll::Poll<()>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:358
#90 0x00007f7970477d48 in tokio::runtime::task::harness::poll_future::hc379eb0724c5f875 () at src/runtime/task/harness.rs:520
#91 0x00007f7970472096 in tokio::runtime::task::harness::Harness<T,S>::poll_inner::he7db7ef4093149b8 () at src/runtime/task/harness.rs:209
#92 0x00007f7970471107 in tokio::runtime::task::harness::Harness<T,S>::poll::h08fdef42bb7c4353 () at src/runtime/task/harness.rs:154
#93 0x00007f79704b8dd0 in tokio::runtime::task::raw::poll::hcebe58de2823daff ()
    at src/runtime/task/raw.rs:271
#94 0x00007f79704b8acb in tokio::runtime::task::raw::RawTask::poll::hac3e83d0324bba2d () at src/runtime/task/raw.rs:201
#95 0x00007f79704ef3bd in tokio::runtime::task::UnownedTask<S>::run::hb34c95cb87f51142 () at src/runtime/task/mod.rs:500
#96 0x00007f79704ef6b6 in tokio::runtime::blocking::pool::Task::run::h14a5d04db30a09fe () at src/runtime/blocking/pool.rs:161
#97 0x00007f79704f4d5e in run () at src/runtime/blocking/pool.rs:511
#98 0x00007f79704f4a94 in tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::hdf69d6dbce01b43b () at src/runtime/blocking/pool.rs:469
#99 0x00007f79704a68f6 in __rust_begin_short_backtrace<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/backtrace.rs:152
#100 0x00007f797047a612 in std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hd91ba102e02671dd ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:559
#101 0x00007f79704b9fb1 in <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h92625b4880d42aed ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272
#102 0x00007f79704c24a1 in do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:587
#103 0x00007f79704863db in __rust_try ()
   from /data00/code/emr/Daft/daft/daft.abi3.so
#104 0x00007f797047a2dd in try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:550
#105 catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#--Type <RET> for more, q to quit, c to continue without paging--
0}::spawn_unchecked_::{closure#1}::{closure_env#0}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()>>, ()> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:358
#106 {closure#1}<tokio::runtime::blocking::pool::{impl#6}::spawn_thread::{closure_env#0}, ()> ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:557


#107 0x00007f797047a8f7 in core::ops::function::FnOnce::call_once{{vtable.shim}}::h538be4313acfc16c ()
    at /root/.rustup/toolchains/nightly-2025-01-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250
#108 0x00007f797214a63b in alloc::boxed::{impl#28}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> ()
    at library/alloc/src/boxed.rs:2013
#109 alloc::boxed::{impl#28}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> ()
    at library/alloc/src/boxed.rs:2013
#110 std::sys::pal::unix::thread::{impl#2}::new::thread_start ()
    at library/std/src/sys/pal/unix/thread.rs:106
#111 0x00007f79737a51f5 in start_thread (arg=<optimized out>)
    at ./nptl/pthread_create.c:442
#112 0x00007f797382589c in clone3 ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[jackye1995]

The error seems to be using Daft. Is this issue specific to daft, or something more fundamental? Could you help explain more details for context?

[Jay-ju]

@jackye1995 This problem is also related to the absence of a close function in the lance dataset. The scenario is that in Daft, "select * from lance_table limit 10" is used. Due to the early termination of the limit, the main thread has finished running, but the sub - thread is still executing the lance scan iter. At this time, it seems that the state of the GIL lock becomes inconsistent.

Therefore, in the PR I submitted, I specifically used an independent thread to manage the state of the GIL lock, Ensure that the GIL is only used in the context of Python threads，and through testing, it indeed works.

crash The context shown

[Xuanwo]

Hi, thank you @Jay-ju for your work on this! #4153 is a good attempt, but I feel it may not address the issue in the best way. I'm concerned about calling std::thread::scope every time in spawn_impl.

If I understand correctly, the way to reproduce this is by forgetting to call close on the dataset and eventually letting Python's garbage collector drop the connection?

[westonpace]

Is this multiprocessing? If so, I think it might be something we can best fix upstream in pyo3-async PyO3/pyo3-async-runtimes#40

I ran into this a while back. I will see if I can find my reproducer. I used spawn-based multiprocessing to spawn a bunch of subprocesses. These were very short lived tasks that ran a delete. Unfortunately, by the time that delete returned, there was still some task hanging around in tokio. When the subprocess ended it started to shutdown python. That tokio task then woke up and tried to grab the GIL which led to this issue.

If we had some way to wait until all tokio tasks were finished before we shutdown then that would be a nice fix but I'm not sure how to do that.

[westonpace]

Here's my reproducer. I needed to run stress -c 32 on a separate shell (e.g. tie up the CPU) to get it to reproduce reliably.

import multiprocessing
import shutil
from typing import Any, Callable

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")
    # Will not error with forkserver because we don't shutdown the process
    # after each task.
    # multiprocessing.set_start_method("forkserver")    

import lancedb  # type: ignore[import-untyped]
import pandas as pd  # type: ignore[import-untyped]


class Multiprocessor:
    def __init__(self) -> None:
        self._procs: list[multiprocessing.Process] = []
        self._exit_codes: list[int | None] = []
        self._msg_queue: multiprocessing.Queue = multiprocessing.Queue()

    def run(
        self,
        target: Callable[..., None],
        args_list: list[Any],
        max_concurrency: int,
    ) -> None:
        self._procs = [
            multiprocessing.Process(
                target=target,
                args=(proc_id, self._msg_queue, *args),
            )
            for proc_id, args in enumerate(args_list)
        ]
        self._spawn(max_concurrency)

    def _spawn(self, max_concurrency: int) -> None:
        next_i = 0
        num_done = 0
        num_running = 0

        can_spawn_another = lambda: (
            num_running < max_concurrency and next_i < len(self._procs)
        )

        while num_done < len(self._procs):
            while can_spawn_another():
                self._procs[next_i].start()
                num_running += 1
                next_i += 1

            msg: int | str = self._msg_queue.get()
            if isinstance(msg, str):
                print(msg)
            elif isinstance(msg, int):
                i = msg
                proc = self._procs[i]
                proc.join()
                num_done += 1
                num_running -= 1


def delete(
    proc_id: int,
    msg_queue: multiprocessing.Queue,
    db_uri: str,
    table_index: int,
) -> None:
    msg_queue.put(f"Deleting from table{table_index}")

    dbc = lancedb.connect(db_uri)
    table = dbc.open_table(f"table{table_index}")
    table.delete(f"foo in (foo)")

    msg_queue.put(proc_id)


if __name__ == "__main__":
    shutil.rmtree("/tmp/testdb", ignore_errors=True)

    df = pd.DataFrame({"foo": ["foo"] * 1024})
    db = lancedb.connect("/tmp/testdb")
    for i in range(25):
        db.create_table(f"table{i}", df)

    delete_args = [("/tmp/testdb", table_index) for table_index in range(25)]
    Multiprocessor().run(delete, delete_args, 5)

[Jay-ju]

Hi, thank you @Jay-ju for your work on this! #4153 is a good attempt, but I feel it may not address the issue in the best way. I'm concerned about calling std::thread::scope every time in spawn_impl.

If I understand correctly, the way to reproduce this is by forgetting to call close on the dataset and eventually letting Python's garbage collector drop the connection?

@Xuanwo thank you for your reply , i think root cause is @westonpace mentioned follows . The solution explained below and mentioned by Weston in this PR at PyO3/pyo3-async-runtimes#40, I actually tried it. However, the scope of changes was relatively large, so I gave up. But I do think that the dataset needs to have a close method.

[Jay-ju]

Is this multiprocessing? If so, I think it might be something we can best fix upstream in pyo3-async PyO3/pyo3-async-runtimes#40

I ran into this a while back. I will see if I can find my reproducer. I used spawn-based multiprocessing to spawn a bunch of subprocesses. These were very short lived tasks that ran a delete. Unfortunately, by the time that delete returned, there was still some task hanging around in tokio. When the subprocess ended it started to shutdown python. That tokio task then woke up and tried to grab the GIL which led to this issue.

If we had some way to wait until all tokio tasks were finished before we shutdown then that would be a nice fix but I'm not sure how to do that.

@westonpace This situation can actually occur in multithreading as well. Indeed, the tokio task is attempting to use a released GIL. my ut: https://github.com/lancedb/lance/pull/4153/files#diff-d69bb29f0d45f73a2c8c7a4a574c85f02be5d35c07f7074453c56f735b464280

[Jay-ju]

Fragment 28 processed
Fragment 32 processed
Fragment 31 processed
Fragment 33 processed
Main thread exiting immediately
Fragment 30 processed
Fragment 34 processed
Fragment 35 processed
Fragment 37 processed
Fatal Python error: PyGILState_Release: thread state 0x22e16d0 must be current when releasing
Python runtime state: finalizing (tstate=0x1a2da40)

Thread 0x00007f97e9269740 (most recent call first):
<no Python frame>
[1]    4031693 IOT instruction (core dumped)  python lance_crash.py

Some error stacks that should be in python-c have been added.

[Kaspre]

Adding a data point from a different code path that hits the same crash signature, in case it helps with triage.

Environment

• lancedb 0.30.1 (custom-built from source with target-cpu=native for an older Westmere CPU)
• pyarrow 23.0.1, numpy 2.4.3, Python 3.12.3, Ubuntu under WSL2
• Call site: a short-lived Python CLI that opens a table, runs a single t.update(where=..., values=...), prints a line, and exits. Single-threaded on the Python side, no concurrent scanners, no daft or read_lance.

Symptom

Fatal Python error: PyGILState_Release: thread state ... must be current when releasing on interpreter shutdown. The t.update() commits successfully — re-reading the row afterward shows the new values — so the crash is strictly in finalization, not in the write path. Practical fallout is that the process exits with SIGABRT (134) and anything downstream that checks $? treats a successful update as a failed one.

Shape of the failing script (simplified from our automation wrapper):

import lancedb
db = lancedb.connect("/path/to/db.lance")
t = db.open_table("claims")
t.update(where="id = '...'", values={"source_ids": ["..."]})
print("done")
# process returns 0 from main flow, Python starts finalization, crash fires here

The original report involves concurrent dataset scanners; this path has no concurrency on the Python side at all, which suggests the race isn't tied to user-level parallelism — even a single update-then-exit can trigger it if lance's internal background threads outlive the main thread.

Local workaround

Replacing the implicit shutdown with os._exit(0) immediately after the work commits bypasses Python finalization entirely and dodges the race. Sample:

import os, sys
# ... do the update, print result ...
sys.stdout.flush()
sys.stderr.flush()
os._exit(0)

Obviously a sledgehammer — anything that depends on normal interpreter shutdown (flushing non-stdio loggers, __del__ on LanceDB handles, atexit handlers) gets skipped — but for one-shot CLIs with no meaningful cleanup it's been effective. Posting in case the workaround is useful to others while a real fix lands upstream, and as a second repro shape that a proper fix will want to cover.

Happy to provide stack traces, re-test against a candidate fix, or share more of our call graph if that would help.

[westonpace]

Thanks for adding more information. At this point I think we understand the cause. It's just not easy to fix without upstream changes (and I don't know how much bandwidth there is on pyo3 maintainers). pyo3 has a tokio runtime. This runtime must be gracefully shutdown before exit. There is no way to access this runtime to shut it down.

Alternatively, there may be some way to ensure that no background tokio tasks are running when a python call returns. I didn't investigate fully enough to really trace out what these are. Perhaps some kind of spawn_blocking async close type thing which we can avoid by properly calling close.

Looking at the upstream issue, it appears that etcd-client-py ended up solving this using a vendored copy of the upstream library. It's possible we could do something similar but it would be a bit of a maintenance burden (e.g. the vendored copy used by etcd-client-py is already 7 commits behind):

lablup/etcd-client-py#14
lablup/etcd-client-py#17