Transformers

Overview

Transformers changed sequence modeling by making attention a central operation rather than processing tokens only in order. Their flexible architecture now supports language, images, audio, code, multimodal data, and scientific representations.

What it is

A Transformer is a neural architecture built around attention, which lets each element use information from other elements. Blocks typically combine attention, nonlinear transformations, residual connections, and normalization.

How it works

Inputs become vectors with information about position or structure. Queries, keys, and values determine which relationships matter for each representation, while stacked blocks refine the result. Training objectives vary from predicting missing or next elements to matching multiple modalities.

Key concepts

• Self-attention
• Queries, keys, and values
• Positional information
• Encoder and decoder stacks
• Causal masking
• Efficient and sparse attention

Current research questions

• How can attention handle long contexts without quadratic cost?
• When do attention patterns correspond to meaningful computation?
• How should architecture change for streams, graphs, images, or scientific data?
• What makes a Transformer robust to noisy or adversarial inputs?

Applications

• Language modeling and translation
• Vision and video understanding
• Speech and music processing
• Code generation
• Multimodal scientific models

Relevant research papers

A live selection from the ScholarLens research index. Open any result to read its full paper page and use the Reading Assistant where available.