Token (AI/NLP)

The basic unit of text processed by language models—typically a word, subword, or character. Tokenizers (BPE, SentencePiece) split text into tokens for model input. 'Solana blockchain' might tokenize as ['Sol', 'ana', ' block', 'chain']. Token count determines context window usage and API billing. Not to be confused with blockchain tokens (cryptocurrency assets).