Skip to content

chore: ligrec add explicit reproducibility tests#1132

Open
selmanozleyen wants to merge 9 commits intoscverse:mainfrom
selmanozleyen:feat/ligrec-rng-threading
Open

chore: ligrec add explicit reproducibility tests#1132
selmanozleyen wants to merge 9 commits intoscverse:mainfrom
selmanozleyen:feat/ligrec-rng-threading

Conversation

@selmanozleyen
Copy link
Member

@selmanozleyen selmanozleyen commented Mar 2, 2026

Adds file for pval reprod tests not to rely on plotting tests as it should be independent.

Also removes kwargs from ligrec to make the taken parameters more explicit.

No longer does these to be backwards compatible for now:
#1131 and creates new files to check full reproducibility.

This PR is also important because after this we can verify if #1125 works or not.

@selmanozleyen selmanozleyen changed the title Feat/ligrec rng threading fix: ligrec seed given to modern numpy rng api Mar 2, 2026
@selmanozleyen selmanozleyen marked this pull request as ready for review March 2, 2026 12:15
Copy link
Contributor

@ilan-gold ilan-gold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this PR will break reproducibility? Just judging by the _images changed

@selmanozleyen
Copy link
Member Author

selmanozleyen commented Mar 2, 2026

Backwards reproducibility: yes. Images were the only thing to test reproducibility but if matplotlib had an update it would also fail so I also added proper reprod tests

@codecov
Copy link

codecov bot commented Mar 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.96%. Comparing base (240dcfe) to head (b4a2b58).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1132   +/-   ##
=======================================
  Coverage   72.96%   72.96%           
=======================================
  Files          38       38           
  Lines        6484     6484           
  Branches     1151     1151           
=======================================
  Hits         4731     4731           
  Misses       1270     1270           
  Partials      483      483           
Files with missing lines Coverage Δ
src/squidpy/gr/_ligrec.py 77.51% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ilan-gold
Copy link
Contributor

Backwards reproducibility: yes. Images were the only thing to test reproducibility but if matplotlib had an update it would also fail so I also added proper reprod tests

I'm not sure I follow here - "Backwards reproducibility" seems like the only thing that would matter here. I don't think we would want new installations of squidpy after this PR to change results when the same function is run on the same data. Isn't that what would happen here?

Images were the only thing to test reproducibility but if matplotlib had an update it would also fail so I also added proper reprod tests

But has matplotlib had an update? Why weren't the tests proper before?

@selmanozleyen
Copy link
Member Author

I'm not sure I follow here - "Backwards reproducibility" seems like the only thing that would matter here. I don't think we would want new installations of squidpy after this PR to change results when the same function is run on the same data. Isn't that what would happen here?

Well I consider it kind of broken already so I wouldn't mind the results changing. Could also be done with RandomState but that will also change the results.

But has matplotlib had an update? Why weren't the tests proper before?

Because plotting and reprod tests should be separated. When it has an update it will prevent someone from updating the references without seeing if the contents are the same 1-1 when there is also a margin change for example. In fact I bet this happened before.

@selmanozleyen selmanozleyen changed the title fix: ligrec seed given to modern numpy rng api fix: ligrec add explicit reprod tests Mar 2, 2026
@selmanozleyen selmanozleyen changed the title fix: ligrec add explicit reprod tests fix: ligrec add explicit reproducibility tests Mar 2, 2026
@selmanozleyen selmanozleyen changed the title fix: ligrec add explicit reproducibility tests chore: ligrec add explicit reproducibility tests Mar 2, 2026
@selmanozleyen selmanozleyen requested a review from ilan-gold March 2, 2026 15:25
@timtreis
Copy link
Member

timtreis commented Mar 10, 2026

Matplotlib would need a pretty major upate for any of the visual tests to actually change. It's pretty much always the numerical stuff under the hood that actually changed given to what extend we locked down everything else. And the images can't really be reproduced locally anyway, they need to be generated on the GitHub runner. The plotting tests implicitly test the output of whatever feeds into them for consistency (to a certain precision)

@selmanozleyen
Copy link
Member Author

But I think it's nicer and more explicit this way. For example it was easier for me to check these results instead of plots

@timtreis
Copy link
Member

ah yes, if a function has a numeric output, one should specifically test that output.

Comment on lines +268 to +269
with open("tests/_data/ligrec_pvalues_reference.pickle", "rb") as fin:
return pickle.load(fin)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we store this as a zarr file or hdf5 or even just an anndata?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants