Token
Token is the fundamental units of text processing in a Large Language Model (LLM).
Tokens can represent various forms of text, including characters, words, or sub-words which is the input to the LLM as well as generated by LLM. The specific form of a token depends on the tokenization strategy employed by the large language model selected.