Zoom levels

[Gautzilla]

🐳 Whaaaat

The old zoom export feature appears to be wanted by users (@PCarvaillo @PaulCarvaillo).

We'll try to give it a little glow-up and integrate it nicely in the new API.

🐳 Design idea(s)

I'll add here more sections if we have other ideas:

🐬 Core zoom datasets with Public zoom relation

Here is how I first see the thing: Each zoom level would be a Core dataset, and the relationship between the different zooms would be stored in the public API Dataset.

Example:

analysis = Analysis(AnalysisType.SPECTROGRAM
    begin=Timestamp("2022-09-25 22:35:00"),
    end=Timestamp("2022-09-25 22:35:30"),
    data_duration=Timedelta(seconds=15),
    zoom_levels=(2,3),
)

The zoom_level=(2,3) parameter means that additionnaly to the analysis SpectroDataset with 15-seconds long spectrograms, 2 other SpectroDatasets with 7.5s and 5s-long spectrograms will be exported:

processed
├── analysis
│   ├── spectrogram
│   │   ├── 2022_09_25_22_35_00_000000.png
│   │   └── 2022_09_25_22_35_15_000000.png
│   └── analysis.json
├── analysis_z2
│   ├── spectrogram
│   │   ├── 2022_09_25_22_35_00_000000.png
│   │   ├── 2022_09_25_22_35_07_500000.png
│   │   ├── 2022_09_25_22_35_15_000000.png
│   │   └── 2022_09_25_22_35_22_500000.png
│   └── analysis_z2.json
└── analysis_z3
    ├── spectrogram
    │   ├── 2022_09_25_22_35_00_000000.png
    │   ├── 2022_09_25_22_35_05_000000.png
    │   ├── 2022_09_25_22_35_10_000000.png
    │   ├── 2022_09_25_22_35_15_000000.png
    │   ├── 2022_09_25_22_35_20_000000.png
    │   └── 2022_09_25_22_35_25_000000.png
    └── analysis_z3.json

And the relationship between the zoom datasets and their original-length counterpart would be stored in the public dataset, with something like that:

{
    "datasets": {
        "original": {
            "analysis": "original",
            "class": "AudioDataset",
            "zoom": {
                "level": 1,
                "zoomed_from": "original"
            },
            "json": "C:\\Users\\berthoga\\Documents\\Datasets\\small_dataset\\data\\audio\\original\\original.json"
        },
        "analysis": {
            "analysis": "analysis",
            "class": "SpectroDataset",
            "zoom": {
                "level": 1,
                "zoomed_from": "analysis"
            },
            "json": "C:\\Users\\berthoga\\Documents\\Datasets\\small_dataset\\processed\\analysis\\analysis.json"
        },
        "analysis_z2": {
            "analysis": "analysis",
            "class": "SpectroDataset",
            "zoom": {
                "level": 2,
                "zoomed_from": "analysis"
            },
            "json": "C:\\Users\\berthoga\\Documents\\Datasets\\small_dataset\\processed\\analysis_z2\\analysis_z2.json"
        },
        "analysis_z3": {
            "analysis": "analysis",
            "class": "SpectroDataset",
            "zoom": {
                "level": 3,
                "zoomed_from": "analysis"
            },
            "json": "C:\\Users\\berthoga\\Documents\\Datasets\\small_dataset\\processed\\analysis_z3\\analysis_z3.json"
        }
    },
    "depth": 0.0,
    "folder": "C:\\Users\\berthoga\\Documents\\Datasets\\small_dataset",
    "gps_coordinates": [
        0,
        0
    ],
    "instrument": null,
    "strptime_format": "%y%m%d%H%M%S",
    "timezone": null
}

What do you guys think?

[mathieudpnt]

Sounds good to me, i'm wondering if such an architecture is manageable for APLOSE, what do you think @ElodieENSTA ?

[ElodieENSTA]

It should work for APLOSE on common zoom levels.
We just have to round the float identically. For exemple when zooming z3 a 10s segment, what will be the filename? 2022_09_25_22_35_03_333333.png or 2022_09_25_22_35_03_333334.png? Rounded to the upper, lower, or nearest?

[Gautzilla]

It should work for APLOSE on common zoom levels. We just have to round the float identically. For exemple when zooming z3 a 10s segment, what will be the filename? 2022_09_25_22_35_03_333333.png or 2022_09_25_22_35_03_333334.png? Rounded to the upper, lower, or nearest?

id say we floor the values with a microsecond resolution (I guess 1 MHz frequency is high enough):

d.split(3)

Out[17]: 
[2020-01-01 00:00:00 - 2020-01-01 00:00:03.333333333,
 2020-01-01 00:00:03.333333333 - 2020-01-01 00:00:06.666666666,
 2020-01-01 00:00:06.666666666 - 2020-01-01 00:00:10]

That's the current behaviour with the OSEkit's split() method and it seems to work pretty well for keeping a continuity between the split parts 🤷

[ElodieENSTA]

The d is your data object? Maybe I can just do this d.split(zoom) to get the names.

I just run into a new question. How does it work if there is a file overlap ?

[Gautzilla]

The d is your data object? Maybe I can just do this d.split(zoom) to get the names.

Yup, d is the data object (the split() method is inherited from BaseData).

Sure, calling d.split(zoom) will lead to the same subdata (with the same names as the exported PNGs), since I'll use this same method to generate the zoomed datasets.

I just run into a new question. How does it work if there is a file overlap ?

I don't recall how it was in the previous OSEkit version, but here I guess the behaviour will be the same wether there is overlap or not. In a case where overlap and zoom would lead to a same file name (e.g. 50% overlap and a x2 zoom for a dataset starting at 00:00:00 with 10s-long spectrograms will have both a 10s-long and a 5s-long spectro starting at 00:00:05), I don't think there would be an issue because the files would not be located in the same folder (one being in the unzoomed processed folder and the other in the _z2 folder).