experiment: impact of other model (everything frozen)

[ArneBinder]

take the best learning rate found in #80 and conduct the following experiment:

• frozen pretrained target + frozen other task1
• frozen pretrained target + frozen other task2
• ...

as before, repeat with 5 different seeds

Generic command:

python src/train.py \
experiment=TARGET_TASK_multimodel_base \
model.pretrained_models={TARGET_TASK_ID:TARGET_TASK_MODEL_NAME_OR_PATH,OTHER_TASK_ID:OTHER_TASK_MODEL_NAME_OR_PATH} \
+model.freeze_models=[TARGET_TASK_ID,OTHER_TASK_ID] \
model.learning_rate=BEST_LR \
trainer=gpu \
seed=1,2,3,4,5 \
+hydra.callbacks.save_job_return.integrate_multirun_result=true \
--multirun

Notes:

• OTHER_TASK_ID should be one of bert-base-cased-coref-hoi, bert-base-cased-re-tacred, bert-base-cased-qa-squad2, bert-base-cased-ner-ontonotes, (or an arbitrary new id for the similarity model Tatiana mentioned). It is especially important to use the correct ids for COREF and RE as other task, because for that we need to define the config correctly in the TARGET_TASK_multimodel_base.yaml, see PR set pretrained_configs in *_multimodel_base experiment configs #87.
• adding +hydra.callbacks.save_job_return.integrate_multirun_result=true will create files with aggregated results (mean, stddev, etc.), the file location will be written to the console at the end of the multirun (not in the W&B logs), so please save that before starting the next multirun!
• This requires set pretrained_configs in *_multimodel_base experiment configs #87 to be merged.

Example command for NER (replace LOCAL_PRETRAINED_DIR!):

python src/train.py \
experiment=conll2012_ner-multimodel_base \
model.pretrained_models={bert-base-cased-ner-ontonotes:LOCAL_PRETRAINED_DIR/bert-base-cased-ner-ontonotes,bert-base-cased-coref-hoi:LOCAL_PRETRAINED_DIR/bert-base-cased-coref-hoi} \
+model.freeze_models=[bert-base-cased-ner-ontonotes,bert-base-cased-coref-hoi] \
model.learning_rate=1e-4 \
trainer=gpu \
seed=1,2,3,4,5 \
+hydra.callbacks.save_job_return.integrate_multirun_result=true \
--multirun