Error with the notebook 3a_rpw_tnr_data_tutorial.ipynb

[MohamedNedal]

Hello, I have followed the installation instructions, and I got the following error while plotting the RPW dynamic spectrum:

# 3a. Plot data "as is" (i.e., without any post-processing or filters)
fig, ax = plt.subplots(figsize=(10, 5))
# Define plot main title
#fig.suptitle("RPW Tuto")
fig.tight_layout()
cbar_ax, kw = cbar.make_axes(ax, shrink=1.4)
# plot AUTO
plot_auto(tnr_data, ax=ax, figure=fig, cbar_ax=cbar_ax)
# Define plot subtitle
ax.set_title('RPW TNR auto-correlation data from ' + filepath.name)
plt.show()

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/.conda/envs/solo8/lib/python3.9/site-packages/xarray/core/dataset.py:1340, in Dataset._construct_dataarray(self, name)
   1339 try:
-> 1340     variable = self._variables[name]
   1341 except KeyError:

KeyError: 'VOLTAGE_SPECTRAL_POWER'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
Cell In[5], line 8
      6 cbar_ax, kw = cbar.make_axes(ax, shrink=1.4)
      7 # plot AUTO
----> 8 plot_auto(tnr_data, ax=ax, figure=fig, cbar_ax=cbar_ax)
      9 # Define plot subtitle
     10 ax.set_title('RPW TNR auto-correlation data from ' + filepath.name)

File ~/.conda/envs/solo8/lib/python3.9/site-packages/maser/plot/rpw/tnr.py:21, in plot_auto(data_wrapper, ax, sensor, bands, cbar_ax, **kwargs)
     18 if cbar_ax is None:
     19     cbar_ax, kw = cbar.make_axes(ax)
---> 21 auto = data_wrapper.as_xarray()["VOLTAGE_SPECTRAL_POWER"]
     23 # keep only V1-V2 sensor
     24 v1_v2_auto = auto.where(auto.sensor == sensor, drop=True)

File ~/.conda/envs/solo8/lib/python3.9/site-packages/xarray/core/dataset.py:1431, in Dataset.__getitem__(self, key)
   1429     return self.isel(**key)
   1430 if utils.hashable(key):
-> 1431     return self._construct_dataarray(key)
   1432 if utils.iterable_of_hashable(key):
   1433     return self._copy_listed(key)

File ~/.conda/envs/solo8/lib/python3.9/site-packages/xarray/core/dataset.py:1342, in Dataset._construct_dataarray(self, name)
   1340     variable = self._variables[name]
   1341 except KeyError:
-> 1342     _, name, variable = _get_virtual_variable(self._variables, name, self.dims)
   1344 needed_dims = set(variable.dims)
   1346 coords: dict[Hashable, Variable] = {}

File ~/.conda/envs/solo8/lib/python3.9/site-packages/xarray/core/dataset.py:178, in _get_virtual_variable(variables, key, dim_sizes)
    176 split_key = key.split(".", 1)
    177 if len(split_key) != 2:
--> 178     raise KeyError(key)
    180 ref_name, var_name = split_key
    181 ref_var = variables[ref_name]

KeyError: 'VOLTAGE_SPECTRAL_POWER'

[jgieseler]

I'm not sure if the authors are notified automatically for issues in this repo. Might be a good idea to ping them via email. According to https://github.com/SolarOrbiterWorkshop/solo8_tutorials/blob/main/RPW_tutorial/README.md, this would be Antonio Vecchio (antonio.vecchio@obspm.fr) and Xavier Bonnin (xavier.bonnin@obspm.fr)

[MohamedNedal]

Thanks a lot @jgieseler :)

[xbonnin]

Dear Mohamed,
I'm indeed not notified automatically (so thanks Jan for the comment and Mohamed for the mail).
I will check the code and see if I can reproduce the error on my side. (It might be an xarray version incompatibility in the maser Python package.)

[xbonnin]

Hello @MohamedNedal ,
A new version 0.3.2 of maser-plot, which includes the plot_auto() method has been released this morning in pypi.
This new version should fix the problem you encounter, otherwise please let me know.
NOTE: if you install tutorial with poetry, you might need first to run the command poetry lock to update the poetry.lock file

[jgieseler]

Thanks for the updated version @xbonnin! I'm actually playing with this right now, too. 😇

It's now working without producing an error, but the plot looks a bit weird, as if the time resolution changed now or the plotting is not spanning over the time intervals as intended. But one can see that the overall picture is the same as in the example notebook.

Example notebook figure:

Figure I obtain:

Code to reproduce the figure:

import sunpy_soar
from astropy.time import Time
from sunpy_soar.attrs import Product as Identifier
from sunpy.net import Fido, attrs as a
import matplotlib.pyplot as plt
import matplotlib.colorbar as cbar

# Requires MASER module (https://gitlab.obspm.fr/maser/maser4py/-/tree/namespace)
from maser.data import Data
from maser.plot.rpw.tnr import plot_auto

query = Fido.search(a.Time('2021-10-09', '2021-10-09 23:59'), a.Instrument('RPW'), a.Level(2), 
                    Identifier('RPW-TNR-SURV'))
file_tnr = Fido.fetch(query[0])

tnr_data_tnr = Data(filepath=str(file_tnr[0]))

fig, ax = plt.subplots(figsize=(10, 5))
# Define plot main title
#fig.suptitle("RPW Tuto")
fig.tight_layout()
cbar_ax, kw = cbar.make_axes(ax, shrink=1.4)
# plot AUTO
plot_auto(tnr_data_tnr, ax=ax, figure=fig, cbar_ax=cbar_ax)
# Define plot subtitle
ax.set_title('RPW TNR auto-correlation data from ' + tnr_data_tnr.filepath.name)
plt.show()

[xbonnin]

Thanks for the feedback @jgieseler!

I confirm the plot_auto() method in maser-plot v3.2 works actually as expected, but for a reason which remains unclear the way pcolormesh() function displays the TNR data seems to have change.

A hotfix has been added in the plot_auto() method to interpolate the data gaps in the plot. New V3.3 of maser-plot has been just released. Let me know if it corrects the display problem on your side (on my side it does).

I give below more detailed explanations if you are interested:
As you may know SOLO/RPW TNR-HFR equipment acquires data on-board on two channels CH1 and CH2. Each channel can be connected to a given sensor (V1, V2, V1-V2, etc.), which can be changed by command. Both channels can also be used by TNR receiver only, or one channel can be used by TNR and the other one by HFR receiver.
During the date 2021-10-09, channel CH1 only acquired data with HFR (see sensors looking like "HF_*"), so no data is found for TNR on channel CH1 for this date, only FILLVAL (-1.0e31). Channel CH2 was connected to TNR, but acquired data using a pattern of different sensors repeated n-times over the day. However the plot_auto() method only takes one sensor value as an input parameter (by default is "V1-V2", which is often used on-board), so only the sensor measurements are actually displayed by plot_auto() (the measurements from other sensors appear like blank gaps in the spectrum). I'm not fully specialist of xarray and pcolormesh, but it seems that they now let these blanks spaces in the plot. Interpolating data gaps in the maser-plot 3.3 version seems to fix the problem.

[xbonnin]

@jgieseler,

By the way, do you know if it would be possible to slightly update the RPW_Tutorial code in Github repo, in order to make sure that the latest Python dependencies versions are used?
(If it is too complicated no problem, instructions given in the README file should be enough.)

[jgieseler]

Yes, with maser-plot 0.3.3 it looks now like intended. 👍 Great, thanks! And thanks for the explanation.

@xbonnin I guess it shouldn't be a problem to push a little update. Not sure if you can merge a corresponding PR yourself or if you would need to notify @hayesla about it.

[MohamedNedal]

Hi @xbonnin @jgieseler
I confirm too that it works with the new maser update! :) Thank you so much for working on it!

I tested it on another date and here's the quicklook from this website:

And here's what I got with the code after the update:

I just wonder why the plot looks different when I use the plt.imshow() function. Any thoughts?

[xbonnin]

@xbonnin I guess it shouldn't be a problem to push a little update. Not sure if you can merge a corresponding PR yourself or if you would need to notify @hayesla about it.

OK thanks.
I will see if it is really needed or not. I do not want to disturb @hayesla if there is only minor corrections to commit.

[xbonnin]

@MohamedNedal,

Summary plots provided in https://rpw.lesia.obspm.fr/rpw-data/daily-summary-plots/ are actually not generated using plot_auto() method from maser. Additionaly, the figure from rpw web site plots both TNR and HFR receiver data between approx few kHz up to ~10 MHz. The plot_auto() method onlys displays the TNR data "as is" (i.e. without any post-processing).
Nevetherless, I do agree that the visual rendering with plot_auto() method can be improved. For the moment it gives a simple example of how to display TNR data. This is something to be done in a next release.
Concerning imshow(), difficult to say but it seems that it does not taken account of the log scale in the y-axis (frequencies) for some reasons.