Make QuPath extension installer not depend on Fiji

[NicoKiaru]

Currently installing an extension on QuPath require running a script on Fiji -> that's not optimal

[hedjour]

You can install QuPath extension just by mv or untar the jar extension file in the QuPath Extension Path.
No need of Fidji script

[hedjour]

@NicoKiaru Can help you with this Issue ?

[NicoKiaru]

If you find a way, that'd be nice (I'm also pinging @petebankhead), but I'm not sure it's super obvious.

I'll try to summarize the goal of the installer, what's being achieved here, and what's not being achieved.

Goal

Here's the goal of the installer scripts:

• 1. run the script once, it installs everything and sets-up everything, for all users (think workstation in a multi-user facility).
• 2. running the script when everything is already installed should just result in an update

What's achieved

Regarding point 2:

• Running Fiji add-update-site works as expected: it installs the update site if not activated, and update it if needed (updates existing jars, removed obsolete ones)

• To my knowledge, there's no updater for QuPath extension. Because I wanted to add an 'updater logic', and I was much more familiar with Java than with Bash, and there was going to be a Java anyway, I went to write a groovy script. So the install/update is done through a Fiji script: InstallQuPathExtension.
  InstallQuPathExtension downloads the release of the extension, at the BIOP we've decided to put in the release, either a single jar, or, if other jars are needed, a zip file which contains all needed jars in a flat hierarchy, this to simplify the update process. When the QuPath extensions are installed / updated, the presence of old jars is checked and removed if detected. That's what is done in
  

  biop-bash-scripts/InstallQuPathExtension.groovy

  Lines 162 to 196 in 66dbb09

  	listJars(extensionsDir.getAbsolutePath())
  	.stream()
  	.each{it ->
  	IJ.log(it.toString())
  	Matcher matcher = pattern.matcher(it.toString());
  	if (matcher.find()) {
  	def v1 = new RepoAndVersion(it.toString(),
  	matcher.group(1),
  	matcher.group(2),
  	matcher.group(3),
  	matcher.group(4),
  	matcher.group(5)!=null)
  	if (jarsMaxVersion.containsKey(v1.repoName)) {
  	// Fight : who has a higher version ?
  	def v2 = jarsMaxVersion.get(v1.repoName)
  	if (v1.hasHigherVersionThan(v2)) {
  	// Swap the map
  	jarsMaxVersion.put(v1.repoName, v1)
  	// Delete v2, because v1>v2
  	def result = new File(extensionsDir, v2.fileName).delete()
  	if (!result) IJ.log("Delete failed! Do you have QuPath opened ? Please close it before installing the extension")
  	IJ.log("Found duplicated jars, deleting older version "+v2.fileName)
  	} else {
  	// Delete v1, because v2>v1
  	def result = new File(extensionsDir, v1.fileName).delete()
  	if (!result) IJ.log("Delete failed! Do you have QuPath opened ? Please close it before installing the extension")
  	IJ.log("Found duplicated jars, deleting older version "+v1.fileName)
  	}
  	} else {
  	// No Fight
  	jarsMaxVersion.put(v1.repoName, v1)
  	}
  	}
  	}

All this logic is probably pretty annoying to do in bash, or at least it's far above what I can do.

• The QuPath extension folder is stored using java prefs. It's easy to set in Java, but not in bash, because Java Prefs are stored differently depending on the OS (windows registry on win, some hidden file on Mac, and probably the same in Linux).

What's not achieved

Now there are some issues:

• The qupath extension folder is set at the user level only. So each user has to set the QuPath extension folder anyway when first logging in. There's no solution to that in this repo. Each new user has to set its QuPath extension folder (unless there's a way to set it at the machine level ?)

• If, in a QuPath extension, a jar is not needed anymore and becomes obsolete, it won't be removed by the script logic: there's no way to know if a jar is needed or not. (However if there are several versions of the same jar, it's possible to remove the old ones). It's not a big deal I think, but that's in contrast with Fiji which has a more advanced updater.

[petebankhead]

The QuPath extension folder is stored using java prefs. It's easy to set in Java, but not in bash, because Java Prefs are stored differently depending on the OS (windows registry on win, some hidden file on Mac, and probably the same in Linux).

Would the ability to set it from the command line help?

I was recently thinking of adding the ability to set the extension directory as a potential command line argument, although I'm not entirely sure how straightforward that will be...

If, in a QuPath extension, a jar is not needed anymore and becomes obsolete, it won't be removed by the script logic: there's no way to know if a jar is needed or not. (However if there are several versions of the same jar, it's possible to remove the old ones). It's not a big deal I think, but that's in contrast with Fiji which has a more advanced updater.

If I remember correctly, QuPath looks for extensions in subdirectories. If you aren't doing it already, could you put all your jars into a subdirectory and then just remove that when it's not needed?

I guess conflicts could happen, but they probably could anyway...

In any case, QuPath's whole extension loading mechanism needs a rethink. Hopefully v0.5.x will have some improvements, but nothing very major or advanced.

[hedjour]

The QuPath extension folder is stored using java prefs. It's easy to set in Java, but not in bash, because Java Prefs are >>stored differently depending on the OS (windows registry on win, some hidden file on Mac, and probably the same in Linux).

Would the ability to set it from the command line help?

I was recently thinking of adding the ability to set the extension directory as a potential command line argument, although I'm not >entirely sure how straightforward that will be...
It will be usefull for person who installed QuPath by script and which have a restrictive antivirus which need all information in a specific path.

Could the QuPath extension follow a guidelines ? like make a sub-directory by extension in the QuPath Extension directory if different jar are needed ? or have is version number in it's name.

So we can make a script which analyse the QuPath subdir and replace a version by another ?

[petebankhead]

Could you not already do that with your own extensions, without anything needing to check on QuPath's side?

QuPath's extension mechanism isn't very advanced - I'm expecting that most will just be a single jar. I know the BIOP ones are more complicated, but I think you're doing more advanced stuff with QuPath extensions than anyone else I know :)

I agree QuPath ought to look for version info and try to use it when available though. That's potentially a small enough change to try to get into v0.5.0 if I find time to work on it. I think proper extension support would require QuPath to make use of ClassLoaders in a more advanced way... but that's a much bigger, longer-term change.

[NicoKiaru]

Changing the QuPath updater mechanism goes way beyond this issue I believe. So far it's ok for us, we need a few extra dependencies that hold in a zip, and as of now, to my knowledge, no conflict showed up.

After, if some conflicting situation show up, I guess it will time to dig into this nightmare of resolving dependencies and conflicts, and probably try to re-use what's existing instead of re-inventing a dependency solver (on the Fiji side, the dependencies declared in pom and the use of maven is doing fine, with a pretty large codebase, but that's also probably working thanks to @ctrueden and team constant efforts). Jgo may be worth a look as well (https://github.com/scijava/jgo)

Anyway, as far as this issue is concerned, I would go for:

• executing the groovy script with QuPath instead of Fiji, in order to make the QuPath installer script more self contained
• keeping a groovy script will allow to maintain this basic mechanism that removes old jars (keeping a flat hierarchy makes this 'easy')

@petebankhead when you write a CLI to set the extension folder, is it for setting the extension 'permanently' or just during the script execution time ?

• Also, do you think it would be possible to set an option to set the extension folder with Java system wide prefs ? Does such a thing exists (system-prefs and not user prefs)?

[NicoKiaru]

Could the QuPath extension follow a guidelines ? like make a sub-directory by extension in the QuPath Extension directory if different jar are needed ? or have is version number in it's name.

So we can make a script which analyse the QuPath subdir and replace a version by another ?

I think the problem here is that there are common dependencies for different extensions. imglib2 and imglib2-realtransform are used for both Warpy and and ABBA. Having a flat hierarchy for jars simplifies jars version checking

[NicoKiaru]

(side note I'm happy so far that the extension we provide are pretty stable, so far the Warpy extension dates from 2021 and is working great, the ABBA extension bugfix and API improvements did not lead to users complaints since its initial release in 2021. That's also thanks to Pete would did not break anything from v0.3 to v0.4 ;-)

[petebankhead]

@petebankhead when you write a CLI to set the extension folder, is it for setting the extension 'permanently' or just during the script execution time ?

That's the bit that prompted

I'm not entirely sure how straightforward that will be...

I think it makes most sense to do it just for the current execution and make no permanent changes. What do you think?

Also, do you think it would be possible to set an option to set the extension folder with Java system wide prefs ? Does such a thing exists (system-prefs and not user prefs)?

I think it exists in Java, and sounds like it could also be a command line option in QuPath (which could also be provided 'permanently' by modifying the QuPath.cfg file).

I think the problem here is that there are common dependencies for different extensions. imglib2 and imglib2-realtransform are used for both Warpy and and ABBA. Having a flat hierarchy for jars simplifies jars version checking

Would it make things better or worse if these were bundled with QuPath itself? We're just starting to explore the use of imglib2 elsewhere in the software.

[NicoKiaru]

I think it makes most sense to do it just for the current execution and make no permanent changes. What do you think?

Yes, makes sense. So far I did not use QuPath with a CLI (except maybe for this issue), but I can see how it could help to specify a subset of extension during a script execution.

which could also be provided 'permanently' by modifying the QuPath.cfg file

Is it possible ?? This would be awesome. But I could not find a way to set java preferences using Java options. It's not written on the Preferences API (https://docs.oracle.com/javase/8/docs/api/java/util/prefs/Preferences.html). And googling this question do not return any OS-independent solution. Maybe I missed it.

With the current QuPath code, I do not think it's possible to set the extension folder for all users (https://github.com/qupath/qupath/blob/b3a59e880a9feef1b0b449189d72ffab3890e38e/qupath-gui-fx/src/main/java/qupath/lib/gui/prefs/PathPrefs.java#L816-L821)

Would it make things better or worse if these were bundled with QuPath itself? We're just starting to explore the use of imglib2 elsewhere in the software.

I think it's a neutral change (except for a transition period where I should avoid duplications).

---

Regarding setting preferences in command line, @romainGuiet wrote this groovy script and pref file to set the extension folder (https://github.com/BIOP/renku-helpers/tree/main/qupath) but again, it's not system wide. Also, if I go for this direction for the bash scripts, this means creating an xml file with bash.

[hedjour]

which could also be provided 'permanently' by modifying the QuPath.cfg file

Is it possible ?? This would be awesome. But I could not find a way to set java preferences using Java options. It's not written on >the Preferences API (https://docs.oracle.com/javase/8/docs/api/java/util/prefs/Preferences.html). And googling this question do >not return any OS-independent solution. >Maybe I missed it.

Is awk and gsub function installed with git bash ? If yes you can setup the cfgfile in a single line of code.

[petebankhead]

which could also be provided 'permanently' by modifying the QuPath.cfg file
Is it possible ?? This would be awesome.

I was imagining it would require a core QuPath change to optionally use
https://docs.oracle.com/en/java/javase/17/docs/api/java.prefs/java/util/prefs/Preferences.html#systemRoot()
Although I don't know what exactly that does...

For now, the most relevant think you can do is to provide a custom preference location. The relevant code is at
https://github.com/qupath/qupath/blob/b3a59e880a9feef1b0b449189d72ffab3890e38e/qupath-gui-fx/src/main/java/qupath/lib/gui/prefs/PathPrefs.java#L111

I think it would look something like this:

-Dqupath.prefs.name=my.custom.qupath.prefs"

Not sure if that helps with anything, but I've found it useful during development.