Solver hangs indefinitely after global optimization and fails to create view transformations

[winnubstj]

On some datasets the solver initializes, performs global optimization, but then hangs indefinitely and fails to generate transformation models for the tiles.

Here is part of the log:

Starting the spark driver
Checking for /shared/scratch/spark-work/b4ccdc/.sessionId
Found /shared/scratch/spark-work/b4ccdc/.sessionId in 0 s
Use Spark IP: 10.0.1.142
[-x, /shared/scratch/nextflow/b4/fcdbe1fa668763a47ae34f263372a1/stitched-dataset.xml, -s, IP, -l, fiducials, --method, ONE_ROUND_ITERATIVE, -tm, AFFINE, --channelId, 1, --relativeThreshold, 3.5, --absoluteThreshold, 7.0, --groupChannels]
xml: /shared/scratch/nextflow/b4/fcdbe1fa668763a47ae34f263372a1/stitched-dataset.xml

...

(Thu Aug 08 20:23:38 GMT 2024): Fixing group-tile [0-1]
(Thu Aug 08 21:07:27 GMT 2024): prealigned all tiles
Shuffling took 0 ms
First apply took 763 ms
0: 4.8665754214827155 4.8665754214827155
1: 4.4960545530422475 4.8665754214827155

...


Concurrent tile optimization loop took 1273080 ms, total took 1273843 ms
(Thu Aug 08 21:28:40 GMT 2024): Global optimization of 62
(Thu Aug 08 21:28:40 GMT 2024):    Avg Error: 3.0632728220428964px
(Thu Aug 08 21:28:40 GMT 2024):    Min Error: 1.6824240273465503px
(Thu Aug 08 21:28:40 GMT 2024):    Max Error: 11.933269910172326px

The process then continues to run (single threaded) but doesn't produce additional outputs. I observe the same behavior with other methods (ONE_ROUND_SIMPLE). In this example all tiles are connected.

I tried running it locally (with ./solver) and observe the same thing.

Also a more general question, does only the optimization step run multi-threaded? I ask specifically because the prealignment step seems to take quite long (~40 mins here) and runs single threaded? Not sure if that is expected behavior.

Full log can be found here

[StephanPreibisch]

Sorry, I missed that .. is it fixed meanwhile with any of the updates? The global optimization is unfortunately not really parallelizable (it actually is, we did it for EM, but it wasn't necessary yet for light), for better performance I'd run it on a computer with high single-threaded performance.

Still, this behavior does not sound correct, I wonder where it hangs and why. If you could share the XML and interestpoints I could try to re-create and understand what it is doing ... sorry it took so long, sending an email to remind me is always appreciated :)