Keras_DeepConvNet.yml and Keras_ShallowConvNet.yml pipelines skipping datasets silently

[0-1deathtouch]

Hello,

I am a master's student at University of Washington, Bothell campus and I am doing my thesis on reproducibility in the field of EEG motor imagery classification via machine learning. I am attempting to reproduce your benchmark results and I am running into some problems that I was hoping you could help with.

Here is the environment I'm running in. I am using a container, so I can change environments if needed:
moabb[deeplearning] version: 1.2.0
python version: 3.10.20
OS: Debian 12 (Bookworm)

Note that I used an older version of MOABB because my desired pipeline was removed from newer versions.

I've pasted my python file and two log files. One shows the pipeline failing at Zhou2016 because it doesn't have enough events. The other shows the output when Zhou2016 is commented out to prevent the failure. AlexMI, BNCI2014_001 and PhysionetMI all fail silently, and only results for Schirrmeister2017 and Weibo2014 are output.

I tried deleting the results folder and rerunning from scratch to rule out cached results.

moabb_braindecode.py

import moabb
from moabb import benchmark
from moabb.pipelines.utils import parse_pipelines_from_directory
import mne
mne.set_config("MNE_DATA", "/gscratch/comdata/raw_data/mne_data")

results_multiclass = benchmark(
    pipelines="../pipelines", 
    paradigms=["MotorImagery"],
    evaluations=["WithinSession"],
    include_datasets=[
	"AlexMI",
	"BNCI2014_001",
	"PhysionetMI",
	"Schirrmeister2017",
	"Weibo2014",
#	"Zhou2016",
    ]
)

# Print accuracy per pipeline per dataset
print(results_multiclass.groupby(["dataset", "pipeline"])["score"].mean())

# Save to CSV for later use
results_multiclass.to_csv("results_motor_imagery.csv", index=False)

Relevant portion of moabb_braindecode_cpu_34934528.out

Traceback (most recent call last):
  File "/gscratch/comdata/users/akoser/moabb_braindecode/moabb_braindecode.py", line 27, in <module>
    results_multiclass = benchmark(
  File "/usr/local/lib/python3.10/site-packages/moabb/benchmark.py", line 190, in benchmark
    paradigm_results = context.process(
  File "/usr/local/lib/python3.10/site-packages/moabb/evaluations/base.py", line 222, in process
    process_pipeline = self.paradigm.make_process_pipelines(
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/base.py", line 140, in make_process_pipelines
    epochs_pipeline = self._get_epochs_pipeline(return_epochs, return_raws, dataset)
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/base.py", line 389, in _get_epochs_pipeline
    ("events", self._get_events_pipeline(dataset)),
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/base.py", line 538, in _get_events_pipeline
    event_id = self.used_events(dataset)
  File "/usr/local/lib/python3.10/site-packages/moabb/paradigms/motor_imagery.py", line 378, in used_events
    raise (
ValueError: Dataset Zhou2016 did not have enough events in None to run analysis

moabb_braindecode_gpu_35023148.out

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1777931155.227444   41268 cpu_feature_guard.cc:227] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
I0000 00:00:1777931164.144223   41268 gpu_device.cc:2043] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 43487 MB memory:  -> device: 0, name: NVIDIA L40S, pci bus id: 0000:84:00.0, compute capability: 8.9
Choosing from all possible events
/usr/local/lib/python3.10/site-packages/moabb/datasets/fake.py:93: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_FAKEDATASET-IMAGERY-10-2--60-60--120-120--FAKE1-FAKE2-FAKE3--C3-CZ-C4_PATH"
  set_config(key, temp_dir)
/usr/local/lib/python3.10/site-packages/moabb/datasets/fake.py:93: RuntimeWarning: Setting non-standard config type: "MNE_DATASETS_FAKEVIRTUALREALITYDATASET-P300-21-1--60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60-60--120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120-120--TARGET-NONTARGET--C3-CZ-C4_PATH"
  set_config(key, temp_dir)

Schirrmeister2017-WithinSession:   0%|          | 0/14 [00:00<?, ?it/s]
Schirrmeister2017-WithinSession:   0%|          | 0/14 [00:00<?, ?it/s]

Weibo2014-WithinSession:   0%|          | 0/10 [00:00<?, ?it/s]
Weibo2014-WithinSession:   0%|          | 0/10 [00:00<?, ?it/s]
             dataset     evaluation              pipeline  avg score
0  Schirrmeister2017  WithinSession     Keras_DeepConvNet   0.327083
1  Schirrmeister2017  WithinSession  Keras_ShallowConvNet   0.811967
2          Weibo2014  WithinSession     Keras_DeepConvNet   0.191071
3          Weibo2014  WithinSession  Keras_ShallowConvNet   0.457143
dataset            pipeline            
Schirrmeister2017  Keras_DeepConvNet       0.327083
                   Keras_ShallowConvNet    0.811967
Weibo2014          Keras_DeepConvNet       0.191071
                   Keras_ShallowConvNet    0.457143
Name: score, dtype: float32

[bruAristimunha]

Hey @0-1deathtouch !

To fully reproduce the moabb paper would should use the version from time of the paper. You need to check better what was the version.

Otherwise, you should use braindecode, because we depreciate the keras implementation I think one year ago, or maybe one year and half.

Here is some nice tutorial showing how to use braindecode and moabb:

https://braindecode.org/stable/auto_examples/advanced_training/plot_moabb_benchmark.html

If you need any help please let me know.

[bruAristimunha]

For the paper, largest reproduction, we have one docker image too at docker hub.

[0-1deathtouch]

Ok thank you! I am using run.py in the docker image and I will let you know if that works.

[0-1deathtouch]

The container has two MOABBs: site-packages reports 0.5.0 and lacks Resampler_Epoch/n_jobs_evaluation which are used by run.py and /workdir/pipelines_deep/Keras_*ConvNet.yml; the source tree at /workdir/moabb/ has both. Setting PYTHONPATH=/workdir/moabb does not override the installed package. Was the environment configured for the published runs so that /workdir/moabb/ takes import precedence? And what is the version/commit of the /workdir/moabb/ source tree?