This repository was archived by the owner on Jun 17, 2025. It is now read-only.
This repository was archived by the owner on Jun 17, 2025. It is now read-only.
BERT Pre-Training #1
Description
Hi,
I would like to test this flaxformer library to pre-train a BERT from scratch.
What is necessary to create the pre-training data (mlm with duplication factor) on an own corpus with an own created wordpiece-based vocab.
How can the pre-training started.
I'm really excited to test it, any help is highly appreciated!