Include Maven installation of ImageJ and Fiji artifacts in postBuild for binder

[imagejan]

Maybe we can run mvn (or jgo) in a binder/postBuild file to pre-install the Fiji and ImageJ Maven artifacts (at the versions used in the notebooks) at the time of building the binder image, so that using the notebooks on mybinder.org requires less waiting.

[imagejan]

I tried to achieve this by adding:

mvn dependency:get -DremoteRepositories=imagej.public::default::https://maven.imagej.net/content/groups/public -Dartifact=net.imagej:imagej:2.0.0-rc-71
mvn dependency:get -DremoteRepositories=imagej.public::default::https://maven.imagej.net/content/groups/public -Dartifact=sc.fiji:fiji:2.0.0-pre-10

(see master...imagejan:pre-install-dependencies)

While this populates .m2/ with dependencies as expected, it's not sufficient for the notebooks to initialize ImageJ/Fiji without downloading lots of further artifacts. My observations so far (running without the above commands to test where artifacts are stored by running the notebooks):

• Running from a Python kernel:

  import imagej
  ij = imagej.init('sc.fiji:fiji:2.0.0-pre-10')
  

  • This will also put artifacts into .jgo/sc/fiji/fiji/2.0.0-pre-10+net.imglib-imglib2-imglyb-0.3.0/, so we'll have to copy them there (or use jgo to set them up?)
  • This also results in many more different versions of artifacts in .m2/ (e.g. the above commands in postBuild only get imagej-ops-0.43.2, while running the notebook cell results in versions 0.36.0, 0.39.0, 0.41.0, 0.41.1, 0.43.1, 0.43.2 all being present.

• Running from a BeakerX kernel:

  %classpath config resolver imagej.public https://maven.imagej.net/content/groups/public
  %classpath add mvn net.imagej imagej 2.0.0-rc-71
  ij = new net.imagej.ImageJ()
  

  • This does not create .m2 or .jgo subfolders. In fact, I wasn't able to locate where the dependency jars are stored.

Maybe we can improve jgo to use the artifacts in .m2 instead of copying them to .jgo?
/cc @ctrueden, @hanslovsky, @hadim

[hanslovsky]

Maybe we can improve jgo to use the artifacts in .m2 instead of copying them to .jgo

I do not think that that is necessary: by default, artifacts are not copied but rather hard-linked, sym-links can be used if preferred (e.g. if maven cache and jgo cache are on different file systems). Or maybe I do not understand what you would like to improve here?

Also, this may be relevant:
twosigma/beakerx#6340

[ctrueden]

Thanks a lot for working on this!

As @hanslovsky said, jgo uses links from .m2/repository, so as long as the needed artifact versions are already present in the local Maven repo cache, the Python notebook's ij = imagej.init(...) calls should not download anything else, and the linking should be snappy. The version numbers line up, right?

As for BeakerX, one way forward might be to add something like this to the postBuild:

jupyter nbconvert --execute --stdout ImageJ_Primer.ipynb

Where ImageJ_Primer.ipynb is a one-cell notebook with the needed %classpath magic to download stuff.

[hadim]

I don't have a strong feeling about this feature but I have a comment: the postBuild step of Conda is pretty hard to debug when something goes wrong and it is also supposed to be a fast step. Moreover, if something is wrong during that step it can mess up your Conda installation a nightmare for a beginner who does not fully understand how Conda works).

So if you want to download artifacts at that step you should really come with bullet-proof scripts that silently exit with a 0 code if something goes wrong.

By the way, I like the jupyter nbconvert --execute --stdout ImageJ_Primer.ipynb idea.

[hadim]

Arf sorry guys I mixed up the postBuild script of binder with the activate.sh script of Anaconda....

[imagejan]

I tried running nbconvert but was running into issues. Will investigate more as time permits.

For the python kernel and pyimagej, I run imagej.init() from python in postBuild, which worked, but the first notebook cell still takes about a minute or so to run. I'll follow up with some benchmarks.