How can i get the answers in the dataset?

[Guozhenyuan]

The CodeMMLU is a great piece of work!

I noticed that the dataset provides task_id, question, and choices columns, but is there an answer column?

How should I handle this dataset if I want to fine-tune an LLM?

[sinsauzero]

+1

[nmd2k]

Thank you for your interest in CodeMMLU! @Guozhenyuan @sinsauzero

1. In order to evaluate with CodeMMLU, you can hand me the LLM's response via email (follow format mentioned in HERE).

2. Since CodeMMLU is intended as a benchmark, not a training dataset, its purpose is to evaluate the performance of pre-trained LLMs, so it might not correctly serve fine-tune purpose.

[sleepingcat4]

@nmd2k If test set is public, by not releasing the answers there isn't any significant good that being done here. Can you guys release the answers on HF?

[Xenomirant]

Dear developers, I've tried to send the generation results via email. However, the mail server seems to block incoming emails due to some policy rules. Is there an other way to send you the results for evaluation?

[nmd2k]

Thank you for find this work interesting @Xenomirant @sleepingcat4
We recently just release the full version including the answer of CodeMMLU at the Huggingface hub
Hope this will help