Unified tokenizer type onboarding

Resolves #1536 Moves TokenizerType to a centralized place. Adds support at different places. # To do Add documentation for future tokenizer onboard