issue/354 - add tokenizer utils to unify encode actions#355

wooway777 wants to merge 1 commit intomainInfiniTensor/InfiniLM:mainfrom

issue/354InfiniTensor/InfiniLM:issue/354Copy head branch name to clipboard