Fixed a bug in FastWordpieceTokenizer where vocab sizes >= 7 would cause failures if the unknown token was not at the end of the vocabulary by ensuring the internal hash map is reserved upfront.#1470

Open

copybara-service[bot] wants to merge 1 commit intomasterfrom

Commits on Feb 16, 2026

Fixed a bug in FastWordpieceTokenizer where vocab sizes >= 7 would cause failures if the unknown token was not at the end of the vocabulary by ensuring the internal hash map is reserved upfront.
tf-text-github-robot
committed