ADR suggestion: extending dimensions

[henrikjacobsenfys]

General

Currently, the dimensions are pretty hard-coded: a dataset is a sc.Dataarray with coordinates energy and Q. This is a good beginning, but it needs to be extended to arbitrary dimensions such as temperature, pressure, and perhaps even sample number. A more general approach is needed. This requires changes in Experiment, SampleModel (and ResolutionModel and BackgroundModel), Analysis1d, Analysis and ParameterAnalysis.

These changes will have to happen simultaneously, as the API of all these classes will have to change. I'm unsure how best to do this. The first thing that comes to mind is that I just make a giant PR, since I'm also designing as I code. But it may be better to make a new develop branch and make smaller PR's against that. I'll still need the old develop branch to make quick bug fixes.

Experiment

This is the most straightforward change: we simply allow any coordinates, and only energy is a required coordinate. I think the energy property can remian, but the Q property should be removed/replaced with a more general coordinate getter.

It is the source of truth for all coordinates, see below.

We should also consider using a DataGroup - it will be needed in some form eventually. A typical example would be users measuring the same sample at two different experiments, or with two different instrument settings (e.g. low resolution, wide energy/Q range, high resolution, narrow energy/Q range), and wanting to fit that data simultaneously. But I think it's good to build this thing in steps.

Shared changes among other classes

Currently, there is always a Q coordinate, and Analysis, Analysis1d, SampleModel, ResolutionModel, BackgroundModel all store the relevant objects (e.g. ComponentCollection or Analysis1d) as a matching list. This will need to change. Extending to nested lists is bound to go wrong, so we will instead use a dict-like system, probably based on scipp DataGroup. I will need to look a bit more at how exactly to implement things, but the idea is that users must be able to select an object both based on physical variables (e.g. all data at temperature of 300 K, with optional tolerance) and on indices/slices (e.g. the first and last temperature). I don't think EasyList is general enough.

It may make sense to make a small helper class that handles indexing, since it will be needed by so many classes. I'll need to make a prototype and try it out.

Generation of objects

SampleModel, ResolutionModel and BackgroundModel will need to generate a ComponentCollection for each coordinate. Analysis will need to create a Analysis1d for each coordinate, which will also generate a Convolution. This is a lot of generation. Generation of things is currently not standardized and a mix og eager/lazy generation. I will move to a standard solution of lazy generation, i.e. any changes marks the object as dirty in some way, and any call to calculate, fit etc. checks if it's dirty and acts accordingly. There will also be a public method to generate everything, since it may be time consuming for big data sets.

Analysis

In addition to what's mentioned above, it will have a rebin method, which uses the rebin method from the data. This will also solve #157 . It will require a confirm=True argument, since it will destroy all generated objects.

Anaysis1d

Analysis1d is the basic analysis object, and remains so. But it currently stores the full DataArray and a Q index, and allows changing the Q index. This functionality will be removed, and it will instead store only the data given to it, i.e. only the data at the particular Q index. It will not allow changing the Q index. It will store all metadata, i.e. what the Q, temperatue etc. index it corresponds to.

SampleModel

Special care needs to be taken with models of diffusion, as they will require a Q coordinate, and the data won't necessarily have one anymore. At the same time, SampleModel will not and should not require knowing what coordinates it will have eventually when attached to an Analysis object.

[damskii9992]

Be VERY careful about over-generalizing . . .
You're making your library increasingly flexible which also makes it increasingly harder to use and harder to maintain. At some point you're just a basic fitting library with some model functions, rather than a spectroscopy data analysis library.
Not saying this is necessarily wrong in this case, I don't have a good enough overview of the EasyDynamics structure to make that judgement, just saying be careful.
It is often better and more maintainable to have separate classes for separate tasks. Like another SampleModel class for pressure experiments etc.
Would users with their data in a different coordinate do exactly the same fitting tasks? And are the coordinates not directly related?

That being said. I plan to use the DataGroup as my internal data-class for Measurement also, to hold meta-data not directly relevant to the data's coordinates. So not for different measurements with different settings, those should be two different Measurement classes IMO.

In regards to fitting multiple different experiments, I plan to make my Analysis take in an arbitrary number of Measurement and SampleModels, and fit differently depending on if it gets 1 or more Measurement.

[henrikjacobsenfys]

Be VERY careful about over-generalizing . . .
You're making your library increasingly flexible which also makes it increasingly harder to use and harder to maintain. At some point you're just a basic fitting library with some model functions, rather than a spectroscopy data analysis library.
Not saying this is necessarily wrong in this case, I don't have a good enough overview of the EasyDynamics structure to make that judgement, just saying be careful.

This is a valid concern, but extending to at least temperature is mandatory. All (both) of my users need it, all examples of my own work need it, and it's needed for the analysis I did for the Halric project. Come to think of it, I don't have any real world examples where temperature is not needed.
I could stop at Q and temperature, but it seems like that would come back to bite me - a general approach is much more flexible, and multiple coordinates are built into scipp anyway.

Would users with their data in a different coordinate do exactly the same fitting tasks? And are the coordinates not directly related?

Yes, users (almost) always want to fit more or less the same model and see how it changes as function of a coordinate. The coordinate is typically temperature, but can be pressure or sample number as well. That should cover most cases, but some users may want magnetic field or electric field. The logic will be the same regardless: Experiment knows what coordinates exist, and SampleModel (etc) and Analysis will create copies of their relevant internal things (ComponentCollection or Analysis1d) corresponding to the coordinates they're given by Analysis. In fact, typing it out, a general solution will probably be easier than a hard-coded one.

Also, the multiple different experiments will need to be fitted with (somewhat) overlapping SampleModels - that's the whole point :) For example:
In reality, a system may have 3 energy scales: high, medium and low energy. A user does two experiments, one with poor resolution and large energy transfer, that probes the high and medium energy, but cannot resolve the low energy stuff, and one with high resolution but small energy transfer, that probes the medium and low energy, but cannot see the high energy stuff. They want to fit the full model to those data, i.e. the high and medium energy part of the model to the first experiment and the medium and low energy part of the model to the second experiment. The medium energy part is shared among the two models, while the rest is not.

This is not a hypothetical situation, I know it sounds like I'm overusing this phrase, but literally all quite a few users do need this functionality.