In this blog post, we will briefly discuss image-based self-supervised learning (SSL). This allows one to learn a useful data representation based on automatically generating a supervisory signal from unlabeled data alone. In particular, we focus on the masked autoencoder (MAE) [He et al., 2021] and self-distillation with no labels (DINO) [Caron et al., 2021], which are two widely used schemes for pretraining vision foundation models. MAEs are trained to reconstruct missing image parts, whereas DINO matches the representations of different crops. Due to the patch-wise image processing and global context, both methods are especially well-suited to the vision transformer (ViT) [Dosovitskiy et al., 2020] as a backbone architecture.
Self-supervised learning
In representation learning, one tries to encode the raw data into a more meaningful representation. This encoding may simply be more compact, or it could be more relevant for certain downstream tasks. While there are a variety of supervised and unsupervised methods that may yield such useful features, self-supervised learning (SSL) subsumes strategies where a training signal is extracted automatically from unlabeled data [Ozbulak et al., 2023; Balestriero et al., 2023; Gui et al., 2024; Uelwer et al., 2025]. Self-supervised representation learning is particularly valuable when the data has a rich internal structure, as is typically the case for images, and labels are scarce.
There are different taxonomies of SSL approaches, which is not surprising in light of the volume and diversity of recent publications. Most often, though, one distinguishes at least the following three categories:
- Reconstruction: Restore missing or hidden image information
- Contrastive: Compare the representations of different views or crops
- Teacher-student: Distill view-invariant features from teacher to student
While teacher-student methods are mostly considered non-contrastive (because they do not use explicit negative examples) [Balestriero et al., 2023; Uelwer et al., 2025], sometimes they are seen as a subcategory of contrastive approaches (because they still match positive pairs) [Gui et al., 2024]. It is also remarked that reconstruction-based approaches are sometimes described as generative (since they operate in pixel space), whereas other methods might be seen as discriminative (due to comparing the representation similarity) [Ozbulak et al., 2023].
The denoising autoencoder (DAE) [Vincent et al., 2008; Vincent et al., 2010] is the prototypical reconstruction-based technique. Here, an autoencoder is trained to reconstruct an image from a corrupted version of it, i.e. the reconstruction error between the original and the reconstructed images is minimized. This denoising task encourages the latent representation to capture the underlying structure of the data.
Other autoassociative pretext tasks for visual representation learning can be similarly seen as image-space restoration problems. These include the recolorization of grayscale images [Larsson et al., 2017], jigsaw puzzle solving [Noroozi and Favaro, 2016] and, most notably in the context of this notebook, image inpainting [Pathak et al., 2016]. The latter idea has evolved into a vast set of methods called masked image modeling (MIM) [Hondru et al., 2025].
For obtaining a meaningful latent representation, reconstruction approaches formulate an auxiliary problem in image space. Instead, contrastive approaches compare the representation of different inputs in latent space directly. One may, for example, want to pull the embeddings of differently augmented views of the same image closer together, whereas the embeddings of genuinely different images are pushed further apart [Wu et al., 2018; Misra and van der Maaten, 2019; Chen et al., 2020]. The use of negative examples in SSL based on contrastive learning is a way to avoid representation collapse, i.e. the collapse into a redundant, non-informative or even constant representation [Jing et al., 2022].
Another line of work is based on self-distillation in conjunction with a student-teacher architecture [Grill et al., 2020; Caron et al., 2021; Zhou et al., 2022]. During training, student and teacher networks are shown differently transformed versions of the same image. The student is trained to output a representation that matches the output of the teacher using gradient descent, while the teacher is more slowly updated as a running average of the student. This is another way to prevent the dimensional collapse of the learned representation.
MAE
The masked autoencoder (MAE) is a representative reconstruction-based SSL scheme that was proposed by [He et al., 2021]. It employs a patch-wise masking strategy together with a ViT-based encoder-decoder architecture that is trained to restore the original input. This masking-then-restoring approach gives rise to an informative latent representation.
An image is first decomposed into regular non-overlapping patches. These are exactly the ones from the ViT. Patches are then randomly removed from the input, and only the non-masked patches are fed to the ViT encoder. A positional embedding is added as usual. Then, the ViT decoder reads the encodings of the visible patches and a shared learnable mask token for each invisible patch. A positional embedding is added to all tokens, including the mask tokens. The original MAE paper illustrated the architecture as follows:
After training both encoder and decoder, the latter can be simply removed. It only served the purpose of mapping the latent space back to the image space during pretraining. The encoder yields the final latent representation. It may be used for further finetuning.
There are some noteworthy design choices that reduce the computational expense of pretraining MAEs. First of all, a rather high percentage of image patches (e.g. 75%) is removed from the input image. Only a fraction of the patches therefore enters the encoder. The full set of patches is only processed by the decoder, which is, second of all, a deliberately lightweight model.
DINO
Another influential family of SSL methods is known as self-distillation with no labels (DINO) [Caron et al., 2021]. DINO employs a student-teacher SSL pretraining strategy and couples it with knowledge distillation [Hinton et al., 2015]. A representation that is consistent across different views can be learned this way. The following figure is inspired by the DINO paper and shows a slightly generalized scheme:
Student and teacher share the same network architecture. While the student parameters are updated in a gradient-based schedule, an exponential moving average (EMA) of the student weights from past iterations is used for the teacher. This addresses the representation collapse problem.
Given an input image, the first step is to create global (e.g. \(224^2\)-sized) and local (e.g. \(96^2\)-sized) crops. The teacher sees only the global crops, whereas the student receives all the crops, including the local ones. This establishes a form of local-to-global correspondence, i.e. small image regions are matched against the broader context. This helps the model to learn object-level features, rather than just texture.
DINO applies a softmax to the final value of the class token of both student and teacher. In this way, the output may be interpreted as a discrete probability distribution. Minimizing the cross entropy between these two distributions encourages the representation to be similar across different views.