Skip to main content
AI Boyfriends

Glossary

Token

Token — A unit of text represented internally by a language model; it may be a word, part of a word or punctuation.

In more detail

A unit of text represented internally by a language model; it may be a word, part of a word or punctuation.

Token counts do not equal character counts. Context limits and usage accounting depend on the model and provider, so a number quoted for one system may not apply to another.

Related terms

Entries that only make full sense together.

  • LLM

    A large language model is a machine-learning model trained to process and generate sequences of text tokens.

  • Context Window

    The amount of text an AI system can consider at one time.

  • Memory

    Information retained or supplied to inform a later conversation.

Where this comes up