pytest crashing with ptest-xdist #14112
Description
When running pytest in parallel w/ pytest-xdist, I'm seeing the follow stack trace:
`
/opt/pyenv/versions/3.10.12/lib/python3.10/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
:1050: in _gcd_import
???
:1027: in _find_and_load
???
:1006: in _find_and_load_unlocked
???
:688: in _load_unlocked
???
/usr/local/lib/gravity/poetry/venvs/MXt4Tns3-py3.10/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:174: in exec_module
co = _read_pyc(fn, pyc, state.trace)
/usr/local/lib/gravity/poetry/venvs/MXt4Tns3-py3.10/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:374: in _read_pyc
with fp:
E OSError: [Errno 5] Input/output error
`
Looking at the code, it seems like we're missing some exception handling.
When I put the entire context in the try/except, everything works as expected.
Seems like this is a best attempt, as if the open fails, we return None.
`
def _read_pyc(
source: Path, pyc: Path, trace: Callable[[str], None] = lambda x: None
) -> types.CodeType | None:
"""Possibly read a pytest pyc containing rewritten code.
Return rewritten code if successful or None if not.
"""
try:
fp = open(pyc, "rb")
except OSError:
return None
with fp: <-------------------- shouldn't this be in the try/except?
try:
stat_result = os.stat(source)
mtime = int(stat_result.st_mtime)
size = stat_result.st_size
data = fp.read(16)
except OSError as e:
trace(f"_read_pyc({source}): OSError {e}")
return None
# Check for invalid or out of date pyc file.
if len(data) != (16):
trace(f"_read_pyc({source}): invalid pyc (too short)")
return None
if data[:4] != importlib.util.MAGIC_NUMBER:
trace(f"_read_pyc({source}): invalid pyc (bad magic number)")
return None
if data[4:8] != b"\x00\x00\x00\x00":
trace(f"_read_pyc({source}): invalid pyc (unsupported flags)")
return None
mtime_data = data[8:12]
if int.from_bytes(mtime_data, "little") != mtime & 0xFFFFFFFF:
trace(f"_read_pyc({source}): out of date")
return None
size_data = data[12:16]
if int.from_bytes(size_data, "little") != size & 0xFFFFFFFF:
trace(f"_read_pyc({source}): invalid pyc (incorrect size)")
return None
try:
co = marshal.load(fp)
except Exception as e:
trace(f"_read_pyc({source}): marshal.load error {e}")
return None
if not isinstance(co, types.CodeType):
trace(f"_read_pyc({source}): not a code object")
return None
return co
`
installed python modules:
`
dev@gvt-dev:~/repo/ws/global-config-manager$ poetry run pip list
Package Version
aniso8601 10.0.1
astroid 2.15.8
async-timeout 5.0.1
atlassian-python-api 4.0.7
attrs 25.4.0
backports.asyncio.runner 1.2.0
backports.zstd 1.3.0
beautifulsoup4 4.14.2
blinker 1.9.0
brotli 1.2.0
certifi 2025.11.12
cffi 2.0.0
cfgv 3.4.0
charset-normalizer 3.4.4
circus 0.19.0
click 8.3.0
confluent-kafka 2.8.0
coverage 7.11.3
cramjam 2.11.0
crypto 1.4.1
cryptography 45.0.7
deepdiff 8.6.1
Deprecated 1.3.1
dill 0.4.0
distlib 0.4.0
enum34 1.1.10
exceptiongroup 1.3.0
execnet 2.1.2
fakeredis 2.32.1
filelock 3.20.0
flake8 7.3.0
flake8-tidy-imports 4.12.0
Flask 3.1.2
Flask-Compress 1.23
flask-restx 1.3.2
gevent 25.9.1
greenlet 3.2.4
grpcio 1.76.0
grpcio-tools 1.71.2
hvac 2.3.0
icdiff 2.0.7
identify 2.6.15
idna 3.11
importlib_resources 6.5.2
iniconfig 2.3.0
isort 5.13.2
itsdangerous 2.2.0
Jinja2 3.1.6
jmespath 1.0.1
jsonpickle 4.1.1
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
lazy-object-proxy 1.12.0
libyang 2.99999.99999
lxml 6.0.2
MarkupSafe 3.0.3
mccabe 0.7.0
mmh3 4.1.0
Naked 0.1.32
netaddr 1.3.0
newrelic 11.2.0
nodeenv 1.9.1
oauthlib 3.3.1
orderly-set 5.5.0
packaging 25.0
pip 25.1.1
platformdirs 4.5.0
pluggy 1.6.0
pprintpp 0.4.0
pre_commit 4.4.0
protobuf 5.29.5
psutil 7.1.3
psycopg2-binary 2.9.11
py 1.11.0
pyang 2.7.1
pyangbind 0.8.7
pycodestyle 2.14.0
pycparser 2.23
pycryptodome 3.23.0
pyflakes 3.4.0
Pygments 2.19.2
pylint 2.17.7
pylint-exit 1.2.0
pyOpenSSL 25.2.0
pyroaring 1.0.3
pytest 8.4.2
pytest-asyncio 1.3.0
pytest-cov 6.3.0
pytest-flask 1.3.0
pytest-forked 1.6.0
pytest-icdiff 0.9
pytest-mock 3.15.1
pytest-sugar 1.1.1
pytest-xdist 3.8.0
python-consul 1.1.0
python-generate-mac 1.3.1
python-snappy 0.7.3
pytz 2025.2
PyYAML 6.0.3
pyzmq 27.1.0
redis 5.2.1
referencing 0.37.0
regex 2025.11.3
requests 2.32.5
requests-mock 1.12.1
requests-oauthlib 2.0.0
retrying 1.4.2
rpds-py 0.28.0
setuptools 80.9.0
shellescape 3.8.1
simplekv 0.14.1
six 1.17.0
sortedcontainers 2.4.0
soupsieve 2.8
SQLAlchemy 1.4.54
sqlalchemy-cockroachdb 1.4.4
SQLAlchemy-Utils 0.38.3
termcolor 3.2.0
tomli 2.3.0
tomlkit 0.13.3
tornado 6.5.2
typing_extensions 4.15.0
urllib3 2.5.0
uWSGI 2.0.31
virtualenv 20.35.4
Werkzeug 3.1.4
wrapt 1.17.3
zope.event 6.1
zope.interface 8.1
`
running on mac os, in docker
$ uname -a Linux gvt-dev 6.10.14-linuxkit #1 SMP Wed Sep 10 06:47:45 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible