Precompiled MTEX workflows are not documented

[gcapes]

It looks like there is an idea to default to precompiled MTEX scripts, but only some of them have been implemented, and only for Windows. That PR has unchecked check boxes for "Compile all scripts" and "Document environment definition examples for each OS"

As such the current workflows don't run on Linux/CSF3. I have made a workaround from reading the comments, which is to add to the inputs list execution_mode: script.
I'm using MatFlow 0.3.0a226 and MTEX 6.0.0

Edit - this looks to be working with the matflow version above. Perhaps I got confused when testing the older matflow version.
While exploring this, the task visualise_orientations_pole_figure_mtex fails consistently when run after the task sample_texture_from_model_ODF_mtex i.e. the workflow sample_texture_model_ODF. I haven't worked out when it started to fail - I would have thought that I would have noticed this when developing the CIPER PR #392 if it wasn't working at that point. Anyway, it works in the following workflows

• sample_orientations_CTF_file

• sample_texture_random

• sample_texture_CRC_file

• sample_texture_ODF_mat

• Default to execution_mode: script until the precompiled code has been both implemented and documented

• Fix visualisation task/sample texture from model ODF task. Could it be that MTEX 6.0.0 has a bug here?

[aplowman]

The precompiled code is implemented on all platforms, but it requires a specific version of the Matlab (runtime), 2024b, so maybe that's the problem. I think I've run them on CSF3 previously.

https://github.com/hpcflow/matflow-data/tree/main/programs/mtex

[gcapes]

Thanks - what's the syntax for the environment definition to use the precompiled code? https://docs.matflow.io/stable/installation.html#linux-macos

Is it still possible to compile and run compiled mtex code, or are those options now obsolete?

[Gbowker]

I also want to know this as now Im on v0.3.0a226 and matflow can't use mtex becuase it doesnt have the option run_precompiled_mtex which it's trying to use for all mtex tasks by default?

[gcapes]

The workaround is to add execution_mode: script as an input for the mtex task, and it'll just run MATLAB uncompiled.

[gcapes]

It seems that if I use execution_mode: script for the first MTEX task in a workflow, this is then used for all subsequent mtex tasks. I have tried in this in a workflow that uses sample_texture_from_model_ODF_mtex followed by visualise_orientations_pole_figure_mtex. The workflow won't submit without execution_mode: script in the first task, but runs just fine without it in the second task. This seems pretty inconsistent

[Gbowker]

The precompiled code is implemented on all platforms, but it requires a specific version of the Matlab (runtime), 2024b, so maybe that's the problem. I think I've run them on CSF3 previously.

https://github.com/hpcflow/matflow-data/tree/main/programs/mtex

Can we have an update on this? If you have run this successfully can you please share the envs.yaml block we need to use this?
Related to this issue? hpcflow/matflow-environments#2
Thanks

[aplowman]

The workaround is to add execution_mode: script as an input for the mtex task, and it'll just run MATLAB uncompiled.

Yeah I think we should set the default to script now (happy to accept a PR for that). The pre-compiled scripts are more of a hassle than I originally thought, since they must use a precise version of the runtime.

It seems that if I use execution_mode: script for the first MTEX task in a workflow, this is then used for all subsequent mtex tasks. I have tried in this in a workflow that uses sample_texture_from_model_ODF_mtex followed by visualise_orientations_pole_figure_mtex. The workflow won't submit without execution_mode: script in the first task, but runs just fine without it in the second task. This seems pretty inconsistent

Not sure I fully understand, but when specifying an input parameter in the first task and not specifying that same input parameter in a second task, it is expected behaviour for the second task to inherit the input value from the earlier task. You can be explicit if you prefer.

[aplowman]

The precompiled code is implemented on all platforms, but it requires a specific version of the Matlab (runtime), 2024b, so maybe that's the problem. I think I've run them on CSF3 previously.
https://github.com/hpcflow/matflow-data/tree/main/programs/mtex

Can we have an update on this? If you have run this successfully can you please share the envs.yaml block we need to use this? Related to this issue? hpcflow/matflow-environments#2 Thanks

@Gbowker I'd recommend you switch to using execution_mode: script, unless you specifically need to run the MTEX tasks without a Matlab license.

[gcapes]

Not sure I fully understand, but when specifying an input parameter in the first task and not specifying that same input parameter in a second task, it is expected behaviour for the second task to inherit the input value from the earlier task. You can be explicit if you prefer.

That sounds like the explanation!

Yeah I think we should set the default to script now (happy to accept a PR for that). The pre-compiled scripts are more of a hassle than I originally thought, since they must use a precise version of the runtime.

Me too - I can do PR, but as far as I can tell, the syntax to configure the matflow environment to use the precompiled versions is still not documented?

[gcapes]

See here for further details matflow env setup matlab --help

[gcapes]

This doesn't currently work well enough to be worth documenting I don't think.

(.venv) lightform $ matflow env setup matlab --path /home/mbexegc2/software/matlab/R2019a/ --mtex-path /home/mbexegc2/software/matlab/toolboxes/mtex/mtex-6.0.0/
Traceback (most recent call last):
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/bin/matflow", line 8, in <module>
    sys.exit(cli())
             ~~~^^
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
    rv = self.invoke(ctx)
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/matflow/cli.py", line 106, in matlab
    env = env_configure_matlab(
        shell,
    ...<2 lines>...
        mtex_path=mtex_path,
    )
  File "/home/mbexegc2/projects/lightform/3D_EBSD_workflow/.venv/lib/python3.13/site-packages/matflow/environments.py", line 211, in env_configure_matlab
    if matlab_mcc.is_file():
       ^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'is_file'

If I just specify matflow env setup matlab --path /home/mbexegc2/software/matlab/R2019a/ it configures a run_precompiled_mtex executable only - nothing to just run MATLAB without compiling. But given it doesn't seem to work with MTEX, it's currently broken.