Mismatch between slow and fast tokenizer

[KaiLv69]

Hi, I trained a sentencepiece tokenizer with prefix match. After convert to HF tokenizer, the tokenization result is not consistent with slow tokenizer.

In sentencepiece, we can choose whether to use prefix match to split the input into token sequences. (https://github.com/google/sentencepiece/blob/d8f741853847553169444afc12c00f4bbff3e9ce/src/bpe_model.cc#L111) I don't find similar func in tokenizers.

Is there any plan to support prefix match for better alignment with sentencepiece?

[ArthurZucker]

cc @Narsil as I have no direct idea, would need to look into it!