Option to disable cache for FromPretrained and FromFile

[daulet]

Related to this.

Currently cache config for BPE tokenizers is set either via BpeBuilder or via config in tokenizer.json for pretrained tokenizers, hence for tokenizers loaded from a file there is no API to disable caching. This is a request to add one.

[ArthurZucker]

I am not sure when you'd want to disable caching but let's see if this is asked by the community!

[daulet]

because it grows uncontrollably unless one actively calls clear_cache. At the very least it should not be default behavior (it's on by default for configs that dont ever mention cache). If one wants to load a pretrained tokenizer there is no option to disable it unless manually modifying tokenizer config file.