Vision Transformers, without the ceremonial fog

A Vision Transformer (ViT) applies the architecture that reshaped natural-language processing to image recognition. Instead of sliding learned filters across pixels as a convolutional neural network does, it turns an image into a sequence and lets self-attention connect the pieces.

That sounds almost suspiciously simple. The clever part is deciding what an image “token” should be.

Turn pixels into patches #

Take a 224 × 224 image and divide it into 16 × 16 patches. You now have 196 patches. Flatten each patch, project it into an embedding, add positional information, and prepend a learned classification token.

The Transformer encoder then processes that sequence using multi-head self-attention and feed-forward layers. The final classification-token representation feeds a classification head.

class VisionTransformer(nn.Module):
    def __init__(self, image_size, patch_size, num_classes):
        super().__init__()
        self.patch_embedding = nn.Conv2d(
            in_channels=3,
            out_channels=embedding_dim,
            kernel_size=patch_size,
            stride=patch_size,
        )
        self.position_embedding = nn.Parameter(
            torch.randn(1, num_patches + 1, embedding_dim)
        )
        self.transformer_encoder = TransformerEncoder(
            num_layers=12,
            embedding_dim=embedding_dim,
        )
        self.classification_head = nn.Linear(embedding_dim, num_classes)

Using a strided convolution for patch embedding is a compact implementation trick: the kernel and stride equal the patch size, so each output position represents one non-overlapping patch.

Where ViT differs from a CNN #

A CNN begins with a strong local bias. Nearby pixels interact through small kernels; deeper layers gradually build larger receptive fields. That bias is extremely useful, particularly when training data is limited.

A ViT makes global interaction available through self-attention. Any patch can attend to any other patch from the first encoder block. Spatial order is not implicit, so positional embeddings must be added explicitly.

The upside is flexible representation learning, highly parallel training, and straightforward reuse of large pretrained models. The bill arrives as data and compute: vanilla ViTs usually need substantial pretraining and can underperform on small datasets.

The family kept evolving #

Swin Transformer restores locality and hierarchy with shifted attention windows. DeiT focuses on data-efficient training. Hybrid models combine convolutional front ends with Transformer encoders. In other words, the industry did not replace one dogma with another; it borrowed the good inductive biases back.

ViT’s enduring contribution is a change in framing. An image does not have to enter a model as a grid processed only by convolutions. It can become a sequence of visual tokens, and the relationships between those tokens can be learned directly.