Index

• training is per token; larger tokens space is expensive
• smaller space; example one token per byte or chars, do not carry any semantics.
  • quality would suffer; larger number of tokens
• how is image/audio/video tokenzied?
• Example:
• Tokenization research evaluates trade-offs between vocabulary size and sequence length using algorithms like Byte Pair Encoding to balance compression, training costs, and multilingual efficiency.