Generate All.h5 data using uno_baseline_keras2.py

[zhen-xie]

Hi all,

I have an issue with the Uno application (https://github.com/ECP-CANDLE/Benchmarks/tree/master/Pilot1/Uno).

When I try to generate the pre-staged dataset file (All.h5) with the recommended command (python uno_baseline_keras2.py --train_sources all --cache cache/all --use_landmark_genes True --preprocess_rnaseq source_scale --no_feature_source True --no_response_source True -z 4096 --export_data All.h5 --shuffle True) on ThetaGPU node, it returns a error as follows:

'''
Params: {'train_sources': ['all'], 'test_sources': ['train'], 'cell_types': None, 'cell_features': ['rnaseq'], 'drug_features': ['descriptors', 'fingerprints'], 'dense': [1000, 1000, 1000], 'dense_feature_layers': [1000, 1000, 1000], 'activation': 'relu', 'loss': 'mse', 'optimizer': 'adam', 'scaling': 'std', 'dropout': 0, 'epochs': 10, 'batch_size': 4096, 'val_split': 0.2, 'cv': 1, 'max_val_loss': 1.0, 'learning_rate': None, 'base_lr': None, 'residual': False, 'reduce_lr': False, 'warmup_lr': False, 'batch_normalization': False, 'feature_subsample': 0, 'rng_seed': 2018, 'save_path': 'save/uno', 'no_gen': False, 'verbose': False, 'gpus': [0], 'timeout': 3600, 'train_bool': True, 'profiling': False, 'experiment_id': 'EXP000', 'run_id': 'RUN000', 'logfile': None, 'shuffle': True, 'ckpt_restart_mode': 'auto', 'ckpt_checksum': False, 'ckpt_skip_epochs': 0, 'ckpt_directory': './save', 'ckpt_save_best': True, 'ckpt_save_best_metric': 'val_loss', 'ckpt_save_weights_only': False, 'ckpt_save_internal': 1, 'ckpt_keep_mode': 'linear', 'ckpt_keep_limit': 1000000, 'agg_dose': None, 'by_cell': None, 'by_drug': None, 'cell_subset_path': '', 'drug_subset_path': '', 'drug_median_response_min': -1, 'drug_median_response_max': 1, 'no_feature_source': True, 'no_response_source': True, 'dense_cell_feature_layers': None, 'dense_drug_feature_layers': None, 'use_landmark_genes': True, 'use_filtered_genes': False, 'feature_subset_path': '', 'cell_feature_subset_path': '', 'drug_feature_subset_path': '', 'preprocess_rnaseq': 'source_scale', 'es': False, 'cp': False, 'tb': False, 'tb_prefix': 'tb', 'partition_by': None, 'cache': 'cache/all', 'single': False, 'export_csv': None, 'export_data': 'All.h5', 'use_exported_data': None, 'on_memory_loader': False, 'growth_bins': 0, 'initial_weights': None, 'save_weights': None, 'data_type': <class 'numpy.float32'>, 'output_dir': '/gpfs/mira-home/zhen/Benchmarks/Pilot1/Uno/Output/EXP000/RUN000'}
WARNING:tensorflow:From /gpfs/mira-home/zhen/Benchmarks/Pilot1/Uno/uno_baseline_keras2.py:240: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.
partition:train, rank:0, sharded index size:20156416, batch_size:4096, steps:4921
partition:val, rank:0, sharded index size:5144576, batch_size:4096, steps:1256
Traceback (most recent call last):
File "/gpfs/mira-home/zhen/Benchmarks/Pilot1/Uno/uno_baseline_keras2.py", line 505, in
main()
File "/gpfs/mira-home/zhen/Benchmarks/Pilot1/Uno/uno_baseline_keras2.py", line 501, in main
run(params)
File "/gpfs/mira-home/zhen/Benchmarks/Pilot1/Uno/uno_baseline_keras2.py", line 305, in run
store.append('x_{}_{}'.format(partition, j), input_feature.astype('float32'), format='table', data_column=True)
TypeError: append() got an unexpected keyword argument 'data_column'
Closing remaining open files:All.h5...done
'''

I am not sure what is this problem, can you help me to solve it? Thanks in advance.

Best,
-Zhen

[jmohdyusof]

This appears to be a typo; the store function is calling pandas.HDFStore:
grep store uno_baseline_keras2.py
store = pd.HDFStore(fname, complevel=9, complib='blosc:snappy')
store.append('x_{}{}'.format(partition, j), input_feature.astype('float32'), format='table', data_column=True)
store.append('y{}'.format(partition), y.astype({target: 'float32'}), format='table', data_column=True,

Pandas documentation shows that the correct argument should be 'data_columns' (plural). I can confirm that this change allows the code to run.

[zhen-xie]

Yes, the code can work well with this argument (data_column=None). Can I submit a pull request to make the code correct?

[jmohdyusof]

I don't understand your comment; my solution was to correct the argument name (data_column -> data_columns). You now claim that changing the argument value from 'True' to 'None' is a fix?

[zhen-xie]

Yes, changing the argument value from 'data_column = True' to 'data_columns = None' is a fix. It is a small bug.

[jmohdyusof]

Well, the bug is the name of the argument (data_column->data_columns). Changing the value of the argument from True to None is changing the actual intent of the code.