`OSError: ncs files have 2 different sections structures. Unsupported` for `neo.io.NeuralynxIO`

[TheChymera]

Background:
Following the previous issue #1038 regarding different sampling rates, we have decided to temporarily remove the differently sampled data (WE*.ncs) in order to bridge this solution pending #1041 .

After curtailing the original archive:

chymera@neurohost /mnt/data/manish/_vStr_phase_stim/M235/M235-2021-07-16 $ tree /mnt/data/manish/vStr_phase_stim/M235/M235-2021-07-16/
/mnt/data/manish/vStr_phase_stim/M235/M235-2021-07-16/
├── CheetahLogFile.txt
├── ConfigurationLog
│   ├── CheetahLastConfiguration.cfg
│   └── MM_ContLFP_V958_withWheel.log
├── CSC10.ncs
├── CSC11.ncs
├── CSC12.ncs
├── CSC13.ncs
├── CSC14.ncs
├── CSC15.ncs
├── CSC16.ncs
├── CSC17.ncs
├── CSC18.ncs
├── CSC19.ncs
├── CSC1.ncs
├── CSC20.ncs
├── CSC21.ncs
├── CSC22.ncs
├── CSC23.ncs
├── CSC24.ncs
├── CSC25.ncs
├── CSC26.ncs
├── CSC27.ncs
├── CSC28.ncs
├── CSC29.ncs
├── CSC2.ncs
├── CSC30.ncs
├── CSC31.ncs
├── CSC32.ncs
├── CSC3.ncs
├── CSC4.ncs
├── CSC5.ncs
├── CSC6.ncs
├── CSC7.ncs
├── CSC8.ncs
├── CSC9.ncs
├── DataProcessingErrors.nde
├── Events.nev
├── LFP28.ncs
├── LFP30.ncs
├── LFP4.ncs
├── LFP6.ncs
├── TT1.ntt
├── TT2.ntt
├── TT3.ntt
├── TT4.ntt
├── TT5.ntt
├── TT6.ntt
├── TT7.ntt
├── TT8.ntt
├── WE1.ncs
└── WE2.ncs

to:

chymera@neurohost /mnt/data/manish/_vStr_phase_stim/M235/M235-2021-07-16 $ tree
.
├── CheetahLogFile.txt
├── ConfigurationLog
│   ├── CheetahLastConfiguration.cfg
│   └── MM_ContLFP_V958_withWheel.log
├── CSC10.ncs
├── CSC11.ncs
├── CSC12.ncs
├── CSC13.ncs
├── CSC14.ncs
├── CSC15.ncs
├── CSC16.ncs
├── CSC17.ncs
├── CSC18.ncs
├── CSC19.ncs
├── CSC1.ncs
├── CSC20.ncs
├── CSC21.ncs
├── CSC22.ncs
├── CSC23.ncs
├── CSC24.ncs
├── CSC25.ncs
├── CSC26.ncs
├── CSC27.ncs
├── CSC28.ncs
├── CSC29.ncs
├── CSC2.ncs
├── CSC30.ncs
├── CSC31.ncs
├── CSC32.ncs
├── CSC3.ncs
├── CSC4.ncs
├── CSC5.ncs
├── CSC6.ncs
├── CSC7.ncs
├── CSC8.ncs
├── CSC9.ncs
├── DataProcessingErrors.nde
├── Events.nev
├── LFP28.ncs
├── LFP30.ncs
├── LFP4.ncs
├── LFP6.ncs
├── TT1.ntt
├── TT2.ntt
├── TT3.ntt
├── TT4.ntt
├── TT5.ntt
├── TT6.ntt
├── TT7.ntt
└── TT8.ntt

We get the following error:

chymera@neurohost ~/src/neuralynx_nwb/neuralynx_nwb $ python -c 'import convert; convert.reposit_data()'
/home/chymera/src/neuralynx_nwb/neuralynx_nwb/convert.py:201: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if s is 0:
/home/chymera/src/neuralynx_nwb/neuralynx_nwb/convert.py:252: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if s is 0:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/chymera/src/neuralynx_nwb/neuralynx_nwb/convert.py", line 44, in reposit_data
    reader = neo.io.NeuralynxIO(dirname=session_data, keep_original_times=False) # TODO: newer version should support: , keep_original_times=True)
  File "/home/chymera/src/python-neo/neo/io/neuralynxio.py", line 51, in __init__
    BaseFromRaw.__init__(self, dirname)
  File "/home/chymera/src/python-neo/neo/io/basefromrawio.py", line 74, in __init__
    self.parse_header()
  File "/home/chymera/src/python-neo/neo/rawio/baserawio.py", line 185, in parse_header
    self._parse_header()
  File "/home/chymera/src/python-neo/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 285, in _parse_header
    self._sigs_memmaps, ncsSegTimestampLimits = self.scan_ncs_files(self.ncs_filenames)
  File "/home/chymera/src/python-neo/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 587, in scan_ncs_files
    raise IOError('ncs files have {} different sections structures. Unsupported.'.format(
OSError: ncs files have 2 different sections structures. Unsupported.

Produced by:

...
reader = neo.io.NeuralynxIO(dirname='/mnt/data/manish/_vStr_phase_stim/M235/M235-2021-07-16', keep_original_times=False)
...

@manimoh

[bendichter]

@JuliaSprenger could you look into this?

@TheChymera it will probably be easier if we could use this data to reproduce the error. Would you be able to share it? If so, I can supply a Google Drive folder.

[TheChymera]

@JuliaSprenger although I believe this is the same data which @manimoh already sent you, just to be sure, here's the archive we're working off of: https://chymera.eu/M235.tar.xz

Using the new feature, excluding the LFP* and WE* files solves the problem, so there's probably something about these files which is unsupported:

https://github.com/TheChymera/neuralynx_nwb/blob/3db29062e99e891fcfbe4d2a3f22004e3a9a4b2c/neuralynx_nwb/convert.py#L44-L54

[bendichter]

edit: @JuliaSprenger has the data so nevermind

@TheChymera I am not able to download the data from that link. Would you be able to share a Google Drive folder?

[JuliaSprenger]

@TheChymera You are providing the data via an insecure connection. I got them now, but in principle using a secure access would be better.
I will have a look at this tomorrow.

[JuliaSprenger]

It seems like your LFP files, even though they have the same sampling rate as the CSC files, have an offset of 15 us. That's why neo can handle them in the same stream as the CSC files. If you also exclude LFP files you should be able to load the data.

[TheChymera]

@JuliaSprenger So excluding the LFP files via the new exclude_filename parameter will still load all data? That's what I described doing in the last post, but I thought that automatically exludes that part of the data from the resulting object — or am I mistaken?

[JuliaSprenger]

@JuliaSprenger So excluding the LFP files via the new exclude_filename parameter will still load all data? That's what I described doing in the last post, but I thought that automatically exludes that part of the data from the resulting object — or am I mistaken?

Hi @TheChymera! Sorry, I missed that you already excluded not only WE* files, but also LFP* files.
Of course you are only loading data files, that are not explicitly excluded. However, for now you can use the exclude mechanism to load CSC and LFP and WE files using dedicated instances of the NeuralynxIO, to still get access to all sampes. I will update #1041 to also deal with data with same sampling rates, but different t_starts and t_stops (as you have it in your CSC and LFP files).

[TheChymera]

Hi @JuliaSprenger , are there any updates on this? :)

[JuliaSprenger]

Hi @TheChymera! Sorry for the delay, I will try to finalize #1041 next week.