Neuralynx header read error

[nmarkowitz]

Hi,

I'm trying to read neuralynx (.ncs) files but I keep getting an error.

from neo.io import NeuralynxIO
nlxDir = 'mydata' # Directionary containing .ncs files
reader = NeuralynxIO(dirname=nlxDir)

Traceback (most recent call last):
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\IPython\core\interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-871ff3d292cc>", line 5, in <module>
    nlx = NeuralynxIO(dirname=dname) # dirname=dname, filename=fname1
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\io\neuralynxio.py", line 58, in __init__
    BaseFromRaw.__init__(self, dirname)
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\io\basefromrawio.py", line 74, in __init__
    self.parse_header()
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\rawio\baserawio.py", line 185, in parse_header
    self._parse_header()
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\rawio\neuralynxrawio\neuralynxrawio.py", line 185, in _parse_header
    info = NlxHeader(filename)
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\rawio\neuralynxrawio\nlxheader.py", line 166, in __init__

I've tried this on both a Windows and Mac computer and have come up with the same error. Any idea what could be causing this?

[samuelgarcia]

Hi Thanks for report.
Have you tried with current master or an updated version ?
@JuliaSprenger : this is for you ?

[JuliaSprenger]

Hi @sportnoah14,
based on your traceback I assume you get the error 'Number of channel ids does not match channel names.'. How many ncs files do you have? Did you try already loading only a subset?

[nmarkowitz]

@JuliaSprenger I reinstalled the latest version of neo using pip. I ran the following

from neo.io import NeuralynxIO
import os
import glob
nlxDir = 'mydata' # Directionary containing .ncs files
ncsFiles = glob.glob(nlxDir + os.sep + '*.ncs')
reader = NeuralynxIO(filename=ncsFiles[0])

Traceback (most recent call last):
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\IPython\core\interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-c0562903cdb8>", line 6, in <module>
    reader = NeuralynxIO(filename=ncsFiles[0]) # dirname=dname, filename=fname1
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\io\neuralynxio.py", line 56, in __init__
    BaseFromRaw.__init__(self, filename)
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\io\basefromrawio.py", line 74, in __init__
    self.parse_header()
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\rawio\baserawio.py", line 185, in parse_header
    self._parse_header()
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\rawio\neuralynxrawio\neuralynxrawio.py", line 185, in _parse_header
    info = NlxHeader(filename)
  File "C:\Users\nmarkowitz\.conda\envs\ieeg2\lib\site-packages\neo\rawio\neuralynxrawio\nlxheader.py", line 166, in __init__
    'Number of channel ids does not match channel names.'
AssertionError: Number of channel ids does not match channel names.

HOWEVER, when I tried this exact same function on my mac I got a different error:

Traceback (most recent call last):
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3457, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-6ed2d08b28ac>", line 5, in <module>
    reader = NeuralynxIO(filename=ncsFiles[0])
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg/lib/python3.7/site-packages/neo/io/neuralynxio.py", line 56, in __init__
    BaseFromRaw.__init__(self, filename)
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg/lib/python3.7/site-packages/neo/io/basefromrawio.py", line 74, in __init__
    self.parse_header()
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg/lib/python3.7/site-packages/neo/rawio/baserawio.py", line 185, in parse_header
    self._parse_header()
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg/lib/python3.7/site-packages/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 300, in _parse_header
    self._sigs_memmaps, ncsSegTimestampLimits = self.scan_ncs_files(self.ncs_filenames)
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg/lib/python3.7/site-packages/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 620, in scan_ncs_files
    lastNcsSections = NcsSectionsFactory.build_for_ncs_file(data, nlxHeader)
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg/lib/python3.7/site-packages/neo/rawio/neuralynxrawio/ncssections.py", line 425, in build_for_ncs_file
    raise TypeError("Unknown Ncs file type from header.")
TypeError: Unknown Ncs file type from header.

Any ideas? Both computers are running a conda environment using python 3.7.12 and packages installed for each are similar if not identical.

[JuliaSprenger]

The different errors on the os could be caused by glob returning the files in different orders. In any case can you try to identify if only a subset of your files is causing the issue by loading only a subset?

[nmarkowitz]

@JuliaSprenger looped through each file in the folder to see if I could open that individual .ncs file. All of them returned an error and I wasn't able to successfully open a single one. Also, you were correct about glob.

[JuliaSprenger]

Can you provide one file for each of the error types you observe? If you want you can upload them here https://amubox.univ-amu.fr/s/5Cd2BedrfJs9BQf

[nmarkowitz]

@JuliaSprenger sorry for the delay, I uploaded two .ncs files to the link you provided. Each file has a different error when I try to read them using reader = NeuralynxIO(filename=<this_file>). Thank you for the help.

[JuliaSprenger]

Hi @sportnoah14
Thanks for providing the files.
I fixed the reading of the header in #1107 Can you confirm this is also working on your side? I noticed that the two files you provided have a lot of inconsistent data block durations (+-1 sample). This leads to the generation of a number of segments when reading those files. Are you aware of that issue or did you fix this already in your recording system?

[nmarkowitz]

@JuliaSprenger I cloned and tried that branch. It works on individual files. However I'm still getting an error if I try to read the whole directory:

from neo.io import NeuralynxIO
nlxDir = 'mydata' # Directionary containing .ncs files
reader = NeuralynxIO(dirname=nlxDir)

Traceback (most recent call last):
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg38/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3361, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-12-7f30552bcfa2>", line 2, in <cell line: 2>
    reader = NeuralynxIO(dirname=nlxDir)
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg38/lib/python3.8/site-packages/neo/io/neuralynxio.py", line 58, in __init__
    BaseFromRaw.__init__(self, dirname)
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg38/lib/python3.8/site-packages/neo/io/basefromrawio.py", line 74, in __init__
    self.parse_header()
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg38/lib/python3.8/site-packages/neo/rawio/baserawio.py", line 185, in parse_header
    self._parse_header()
  File "/Users/sportsnoah14/opt/miniconda3/envs/ieeg38/lib/python3.8/site-packages/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 287, in _parse_header
    assert sampling_rate.size == 1
AssertionError

Any idea what could be causing this?

This is actually data from another group so I'll pass that info about block size to them. Thanks for catching it. Would it help if I upload the entire directory of files?

Thanks again for working on this.

[JuliaSprenger]

The sampling_rate.size == 1 error is raised because we are still assuming all ncs files are recorded with the same sampling rate (single stream). I am currently working this to extend this for multiple streams here: #1041.

[nmarkowitz]

Thanks @JuliaSprenger . Let me know if I can help with this in any way. In the meantime, I can read individual streams by specifying "filename" correct?

[JuliaSprenger]

Yes, or alternatively load all files that share their sampling rate by collecting them in a separate directory.

[JuliaSprenger]

@sportnoah14 Would it be possible to add the two test files you provided to the public database of test files for Neo on gin? If yes, whom can I reference as provider?