The basic unit of text used by language models. It can be a word, part of a word, or a symbol. Tokens are used to process language in pieces.
« Back to Glossary IndexToken
« Back to Glossary Index
Artificial Intelligence
A Journey Through AI Shaping Tomorrow