Question about vocabulary construction in the Potion series (based on bge-base-en-v1.5)

[hank317]

I noticed that the Potion series, derived from bge-base-en-v1.5, have vocabularies of different sizes, and these sizes also differ from the original backbone bge-base-en-v1.5. How were the vocabularies for the Potion models determined? Specifically, could you elaborate on the vocabulary construction process for potion-base-32M?

Thank you!

[Pringled]

Hi @hank317, the expanded vocabulary is automatically mined using Tokenlearn, see https://github.com/MinishLab/tokenlearn/blob/main/tokenlearn/train.py#L69. The default value here gives the exact size for potion-base-32M, in combination with our training datasets. Hope that clears it up!