← Zurück zum Glossar
Architektur

Attention Mechanism

The core operation in transformers that lets each token attend to all other tokens.

Self-attention computes a weighted sum of all token representations, where weights reflect how relevant each token is to each other. This gives transformers their ability to model long-range dependencies and contextual meaning. Multi-head attention runs this operation in parallel across many representation subspaces, enabling richer feature extraction.

Verwandte Begriffe

Transformer

The neural network architecture underlying virtually all modern LLMs.

Context Window

The maximum number of tokens a model can process in a single request.

LLM

Large Language Model — a neural network trained on vast text corpora to generate human-like text.