Architecture
Parameter
A learnable weight in a neural network; model size is measured in billions of parameters.
Parameters are the numerical values in a neural network that are learned during training and determine its behavior. Larger parameter counts generally correlate with greater capability but also higher inference cost. Models are commonly described by parameter count: 7B, 70B, 405B. However, MoE models have high total parameters but activate only a fraction per inference.
Related Terms
LLM
Large Language Model — a neural network trained on vast text corpora to generate human-like text.
Transformer
The neural network architecture underlying virtually all modern LLMs.
Quantization
Compressing model weights to lower numerical precision to reduce memory and speed up inference.
Mixture of Experts (MoE)
An architecture where only a subset of model parameters is activated per token.