Understanding Cross-Entropy

What is it?

In deep learning, “cross-entropy” is a function used to measure the difference between two probability distributions, typically used in classification tasks.

Analogy: Guessing the Color of Candies

Imagine you have a bag of candies containing red, green, and blue colors. You guess the color distribution of the candies is 50% red, 30% green, and 20% blue. However, the actual distribution is 60% red, 20% green, and 20% blue.

Cross-entropy tells you how close your guess (prediction) is to the actual distribution of the candies. If your guess is far from the truth, the cross-entropy value will be large; if you guess closely, the cross-entropy value will be small.

Mathematical Explanation

The mathematical formula for cross-entropy can be written as:

$$H(p,q)=-\sum p(x)\log (q(x))$$

Where:

• $p(x)$ is the true probability distribution (e.g., the actual color distribution of the candies).
• $q(x)$ is your predicted probability distribution (e.g., your guessed color distribution).
• This formula means: for each possible candy color, the difference between the true probability $p(x)$ and your predicted probability $q(x)$ is multiplied by $\log (q(x))$, then negated, and finally summed over all possible colors.

Intuitive Explanation:

1. If you guess correctly: For example, if you guess that the red candies are 60%, and the truth is also 60%, then the cross-entropy value will be small, indicating your prediction is accurate.
2. If you guess incorrectly: For instance, if you guess that the red candies are only 20%, while the truth is 60%, then the cross-entropy value will be large, indicating a significant difference between your prediction and the actual situation.

Calculation of Cross-Entropy

Example:

Suppose we have a simple classification problem with only two categories, such as “cat” and “dog.”
The true label is “cat,” so the true probability $p(x)$ is:

• Cat: 1.0
• Dog: 0.0

The model’s predicted probabilities $q(x)$ are:

• Cat: 0.8
• Dog: 0.2

Step-by-Step Calculation:

According to the formula, the cross-entropy calculation will involve each category. We substitute the probabilities of these two categories:

$$H(p,q)=-(p(\text{cat})\cdot \log (q(\text{cat}))+p(\text{dog})\cdot \log (q(\text{dog})))$$

1. Calculate the “cat” part:

   • True probability $p(\text{cat})=1.0$
   • Predicted probability $q(\text{cat})=0.8$
   • Calculate this part: $1.0\cdot \log (0.8)=\log (0.8)\approx -0.2231$

2. Calculate the “dog” part:

   • True probability $p(\text{dog})=0.0$
   • Predicted probability $q(\text{dog})=0.2$
   • Since $p(\text{dog})=0$, this term contributes 0: $0.0\cdot \log (0.2)=0$

3. Total Sum: Adding the two parts and taking the negative:

$$H(p,q)=-(-0.2231+0)=0.2231$$

Result Explanation:

The cross-entropy value is 0.2231, indicating that the model’s prediction is relatively close to the true label, but not completely correct (if the prediction were 1.0, the cross-entropy would be 0).

Application in Deep Learning

In classification tasks in deep learning, the model outputs a probability distribution for each category. For example, the probability of a picture containing a cat is 0.7, a dog is 0.2, and a bird is 0.1. Cross-entropy is used to measure the difference between the model’s predicted distribution and the true labels, helping the model adjust its parameters to improve prediction accuracy.

By minimizing cross-entropy, we can make the model’s predicted distribution as close as possible to the true category distribution, which is why cross-entropy is often used as a loss function.