Contrastive Representation Learning
📰 Lilian Weng's Blog
Contrastive representation learning aims to learn an embedding space where similar samples are close and dissimilar ones are far apart
Action Steps
- Understand the concept of contrastive representation learning
- Learn about different training objectives such as contrastive loss, triplet loss, and lifted structured loss
- Implement these loss functions in deep learning models to improve their performance
Who Needs to Know This
Data scientists and machine learning engineers can benefit from understanding contrastive representation learning to improve their models' performance in self-supervised and supervised settings
Key Insight
💡 Contrastive learning can be applied to both supervised and unsupervised settings, making it a powerful approach in self-supervised learning
Share This
🤖 Contrastive representation learning: learn to embed similar samples close and dissimilar ones far apart 💡
Key Takeaways
Contrastive representation learning aims to learn an embedding space where similar samples are close and dissimilar ones are far apart
Full Article
Published Time: 2021-05-31T00:00:00Z
Lil'Log
|
Posts
Archive
Search
Tags
FAQ
Contrastive Representation Learning
Date: May 31, 2021 | Estimated Reading Time: 39 min | Author: Lilian Weng
Table of Contents
The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.
Contrastive Training Objectives
In early versions of loss functions for contrastive learning, only one positive and one negative sample are involved. The trend in recent training objectives is to include multiple positive and negative pairs in one batch.
Contrastive Loss
Contrastive loss (Chopra et al. 2005) is one of the earliest training objectives used for deep metric learning in a contrastive fashion.
Given a list of input samples
{
𝑥
𝑖
}
, each has a corresponding label
𝑦
𝑖
∈
{
1
,
…
,
𝐿
}
among
𝐿
classes. We would like to learn a function
𝑓
𝜃
(
.
)
:
𝑋
→
𝑅
𝑑
that encodes
𝑥
𝑖
into an embedding vector such that examples from the same class have similar embeddings and samples from different classes have very different ones. Thus, contrastive loss takes a pair of inputs
(
𝑥
𝑖
,
𝑥
𝑗
)
and minimizes the embedding distance when they are from the same class but maximizes the distance otherwise.
𝟙
𝟙
𝐿
cont
(
𝑥
𝑖
,
𝑥
𝑗
,
𝜃
)
=
1
[
𝑦
𝑖
=
𝑦
𝑗
]
‖
𝑓
𝜃
(
𝑥
𝑖
)
−
𝑓
𝜃
(
𝑥
𝑗
)
‖
2
2
+
1
[
𝑦
𝑖
≠
𝑦
𝑗
]
max
(
0
,
𝜖
−
‖
𝑓
𝜃
(
𝑥
𝑖
)
−
𝑓
𝜃
(
𝑥
𝑗
)
‖
2
)
2
where
𝜖
is a hyperparameter, defining the lower bound distance between samples of different classes.
Triplet Loss
Triplet loss was originally proposed in the FaceNet (Schroff et al. 2015) paper and was used to learn face recognition of the same person at different poses and angles.
Illustration of triplet loss given one positive and one negative per anchor. (Image source: Schroff et al. 2015)
Given one anchor input
𝑥
, we select one positive sample
𝑥
+
and one negative
𝑥
−
, meaning that
𝑥
+
and
𝑥
belong to the same class and
𝑥
−
is sampled from another different class. Triplet loss learns to minimize the distance between the anchor
𝑥
and positive
𝑥
+
and maximize the distance between the anchor
𝑥
and negative
𝑥
−
at the same time with the following equation:
𝐿
triplet
(
𝑥
,
𝑥
+
,
𝑥
−
)
=
∑
𝑥
∈
𝑋
max
(
0
,
‖
𝑓
(
𝑥
)
−
𝑓
(
𝑥
+
)
‖
2
2
−
‖
𝑓
(
𝑥
)
−
𝑓
(
𝑥
−
)
‖
2
2
+
𝜖
)
where the margin parameter
𝜖
is configured as the minimum offset between distances of similar vs dissimilar pairs.
It is crucial to select challenging
𝑥
−
to truly improve the model.
Lifted Structured Loss
Lifted Structured Loss (Song et al. 2015) utilizes all the pairwise edges within one training batch for better computational efficiency.
Illustration compares contrastive loss, triplet loss and lifted structured loss. Red and blue edges connect similar and dissimilar sample pairs respectively. (Image source: Song et al. 2015)
Let
𝐷
𝑖
𝑗
=
|
𝑓
(
𝑥
𝑖
)
−
𝑓
(
𝑥
𝑗
)
|
2
, a structured loss function is defined as
𝐿
struct
=
1
2
|
𝑃
|
∑
(
𝑖
,
𝑗
)
∈
𝑃
max
(
0
,
𝐿
struct
(
𝑖
𝑗
)
)
2
where
𝐿
struct
(
𝑖
𝑗
)
=
𝐷
𝑖
𝑗
+
max
(
max
(
𝑖
,
𝑘
)
∈
𝑁
𝜖
−
𝐷
𝑖
𝑘
,
max
(
𝑗
,
𝑙
)
∈
𝑁
𝜖
−
𝐷
𝑗
𝑙
)
where
𝑃
contains the set of positive pairs and
𝑁
is the set of negative pairs. Note that the dense pairwise squared distance matrix can be easily computed per training batch.
The red part in
𝐿
struct
(
𝑖
𝑗
)
is used for mining hard negatives. However, it is not smooth and may cause the convergence to a bad local optimum in practice. Thus, it is relaxed to be:
𝐿
struct
(
𝑖
𝑗
)
=
𝐷
𝑖
𝑗
+
log
(
∑
(
𝑖
,
𝑘
)
∈
𝑁
exp
(
𝜖
−
𝐷
𝑖
𝑘
)
+
∑
(
𝑗
,
𝑙
)
∈
𝑁
exp
(
𝜖
−
𝐷
Lil'Log
|
Posts
Archive
Search
Tags
FAQ
Contrastive Representation Learning
Date: May 31, 2021 | Estimated Reading Time: 39 min | Author: Lilian Weng
Table of Contents
The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.
Contrastive Training Objectives
In early versions of loss functions for contrastive learning, only one positive and one negative sample are involved. The trend in recent training objectives is to include multiple positive and negative pairs in one batch.
Contrastive Loss
Contrastive loss (Chopra et al. 2005) is one of the earliest training objectives used for deep metric learning in a contrastive fashion.
Given a list of input samples
{
𝑥
𝑖
}
, each has a corresponding label
𝑦
𝑖
∈
{
1
,
…
,
𝐿
}
among
𝐿
classes. We would like to learn a function
𝑓
𝜃
(
.
)
:
𝑋
→
𝑅
𝑑
that encodes
𝑥
𝑖
into an embedding vector such that examples from the same class have similar embeddings and samples from different classes have very different ones. Thus, contrastive loss takes a pair of inputs
(
𝑥
𝑖
,
𝑥
𝑗
)
and minimizes the embedding distance when they are from the same class but maximizes the distance otherwise.
𝟙
𝟙
𝐿
cont
(
𝑥
𝑖
,
𝑥
𝑗
,
𝜃
)
=
1
[
𝑦
𝑖
=
𝑦
𝑗
]
‖
𝑓
𝜃
(
𝑥
𝑖
)
−
𝑓
𝜃
(
𝑥
𝑗
)
‖
2
2
+
1
[
𝑦
𝑖
≠
𝑦
𝑗
]
max
(
0
,
𝜖
−
‖
𝑓
𝜃
(
𝑥
𝑖
)
−
𝑓
𝜃
(
𝑥
𝑗
)
‖
2
)
2
where
𝜖
is a hyperparameter, defining the lower bound distance between samples of different classes.
Triplet Loss
Triplet loss was originally proposed in the FaceNet (Schroff et al. 2015) paper and was used to learn face recognition of the same person at different poses and angles.
Illustration of triplet loss given one positive and one negative per anchor. (Image source: Schroff et al. 2015)
Given one anchor input
𝑥
, we select one positive sample
𝑥
+
and one negative
𝑥
−
, meaning that
𝑥
+
and
𝑥
belong to the same class and
𝑥
−
is sampled from another different class. Triplet loss learns to minimize the distance between the anchor
𝑥
and positive
𝑥
+
and maximize the distance between the anchor
𝑥
and negative
𝑥
−
at the same time with the following equation:
𝐿
triplet
(
𝑥
,
𝑥
+
,
𝑥
−
)
=
∑
𝑥
∈
𝑋
max
(
0
,
‖
𝑓
(
𝑥
)
−
𝑓
(
𝑥
+
)
‖
2
2
−
‖
𝑓
(
𝑥
)
−
𝑓
(
𝑥
−
)
‖
2
2
+
𝜖
)
where the margin parameter
𝜖
is configured as the minimum offset between distances of similar vs dissimilar pairs.
It is crucial to select challenging
𝑥
−
to truly improve the model.
Lifted Structured Loss
Lifted Structured Loss (Song et al. 2015) utilizes all the pairwise edges within one training batch for better computational efficiency.
Illustration compares contrastive loss, triplet loss and lifted structured loss. Red and blue edges connect similar and dissimilar sample pairs respectively. (Image source: Song et al. 2015)
Let
𝐷
𝑖
𝑗
=
|
𝑓
(
𝑥
𝑖
)
−
𝑓
(
𝑥
𝑗
)
|
2
, a structured loss function is defined as
𝐿
struct
=
1
2
|
𝑃
|
∑
(
𝑖
,
𝑗
)
∈
𝑃
max
(
0
,
𝐿
struct
(
𝑖
𝑗
)
)
2
where
𝐿
struct
(
𝑖
𝑗
)
=
𝐷
𝑖
𝑗
+
max
(
max
(
𝑖
,
𝑘
)
∈
𝑁
𝜖
−
𝐷
𝑖
𝑘
,
max
(
𝑗
,
𝑙
)
∈
𝑁
𝜖
−
𝐷
𝑗
𝑙
)
where
𝑃
contains the set of positive pairs and
𝑁
is the set of negative pairs. Note that the dense pairwise squared distance matrix can be easily computed per training batch.
The red part in
𝐿
struct
(
𝑖
𝑗
)
is used for mining hard negatives. However, it is not smooth and may cause the convergence to a bad local optimum in practice. Thus, it is relaxed to be:
𝐿
struct
(
𝑖
𝑗
)
=
𝐷
𝑖
𝑗
+
log
(
∑
(
𝑖
,
𝑘
)
∈
𝑁
exp
(
𝜖
−
𝐷
𝑖
𝑘
)
+
∑
(
𝑗
,
𝑙
)
∈
𝑁
exp
(
𝜖
−
𝐷
DeepCamp AI