Contents

- 1 Introduction
- 2 What is Loss Function?
- 3 Types of Loss Functions in Keras
- 3.1 1. Keras Loss Function for Classification
- 3.2 2. Keras Loss Function for Regression
- 3.3 Keras Custom Loss Function
- 3.4 Keras add_loss() API

- 4 Conclusion

**Introduction**

In this tutorial, we will look at various types of Keras loss functions for training neural networks. The loss functions are an important part of any neural network training process as it helps the network to minimize the error and reach as close as possible to the expected output. Here we will go through Kera loss functions for regression, classification and also see how to create a custom loss function in Keras.

**What is Loss Function?**

Loss Functions, also known as cost functions, are used for computing the error with the aim that the model should minimize it during training.

**Loss Functions** also help in finding out the slope i.e. gradient w.r.t. weights used in the model and then these **weights are updated after each epoch** with the help of **backpropagation**.

The below animation shows how a loss function works.

Selecting a loss function is not so easy, so we’ll be going over some prominent loss functions that can be helpful in various instances.

**Types of Loss Functions in Keras**

**1. Keras Loss Function for Classification **

Let us first understand the Keras loss functions for classification which is usually calculated by using probabilistic losses.

**i) Keras Binary Cross Entropy **

Binary Cross Entropy loss function finds out the loss between the true labels and predicted labels for the binary classification models that gives the output as a probability between 0 to 1.

**Syntax of Keras Binary Cross Entropy**

Following is the syntax of Binary Cross Entropy Loss Function in Keras.

```
tf.keras.losses.BinaryCrossentropy(
from_logits=False, label_smoothing=0, reduction="auto", name="binary_crossentropy"
)
```

**Keras Binary Cross Entropy Example**

The example for Keras binary cross entropy shows how two sets of random values are used as data and then the required function from losses class is used.

```
import tensorflow as tf
```

```
y_true = [[0., 1.], [0., 0.]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.
bce = tf.keras.losses.BinaryCrossentropy()
bce(y_true, y_pred).numpy()
```

**ii) Keras Categorical Cross Entropy **

This is the second type of probabilistic loss function for classification in Keras and is a generalized version of binary cross entropy that we discussed above. Categorical Cross Entropy is used for multiclass classification where there are more than two class labels.

**Syntax of Keras Categorical Cross Entropy**

Following is the syntax of Categorical Cross Entropy Loss Function in Keras.

`tf.keras.losses.CategoricalCrossentropy(from_logits=False,label_smoothing=0, reduction="auto",name="categorical_crossentropy",)`

**Keras Categorical Cross Entropy ****Example**

The following is an example of Keras categorical cross entropy. y_true denotes the actual probability distribution of the output and y_pred denotes the probability distribution we got from the model.

```
y_true = [[0, 1, 0], [0, 0, 1]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
# Using 'auto'/'sum_over_batch_size' reduction type.
cce = tf.keras.losses.CategoricalCrossentropy()
cce(y_true, y_pred).numpy()
```

**iii) Keras KL Divergence**

**Syntax of Keras KL Divergence**

Below is the syntax of LL Divergence in Keras –

```
tf.keras.losses.KLDivergence(reduction="auto", name="kl_divergence")
```

**Keras KL Divergence Example**

The **KLDivergence()** function is used in this case. The result obtained shows that there is not a huge loss but still it is considerable.

```
y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.
kl = tf.keras.losses.KLDivergence()
kl(y_true, y_pred).numpy()
```

**iv) Keras Poisson Loss Function**

In the Poisson loss function, we calculate the Poisson loss between the actual value and predicted value. Poisson Loss Function is generally used with datasets that consists of Poisson distribution. An example of Poisson distribution is the count of calls received by the call center in an hour.

**Syntax of Keras Poisson Loss Function**

Following is the syntax of Poisson Loss Function in Keras.

```
tf.keras.losses.Poisson(reduction="auto", name="poisson")
```

**Keras Poisson Loss Function Example**

The poisson loss function is used in below example.

```
y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [0., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.
p = tf.keras.losses.Poisson()
p(y_true, y_pred).numpy()
```

**iv) Keras Hinge Loss**

The above Keras loss functions for classification were using probabilistic loss as their basis for calculation. Now we are going to see some loss functions in Keras that use Hinge Loss for maximum margin classification like in SVM.

The hinge loss function is performed by computing hinge loss of true values and predicted values.

**Syntax of Keras Hinge Loss**

Below is the syntax of Keras Hinge loss –

```
tf.keras.losses.Hinge(reduction="auto", name="hinge")
```

**Keras Hinge Loss Example**

The hinge() function from the Keras package helps in finding the hinge loss

```
y_true = [[0., 1.], [0., 0.]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.
h = tf.keras.losses.Hinge()
h(y_true, y_pred).numpy()
```

**vi) Keras Squared Hinge Loss**

The squared hinge loss is calculated using **squared_hinge()** function and is similar to Hinge Loss calculation discussed above except that the result is squared.

**Syntax of Squared Hinge Loss in Keras**

```
tf.keras.losses.squared_hinge(y_true, y_pred)
```

**Example of Squared Hinge Loss in Keras**

In this example, at first, data is generated using numpy randon function, then Keras squared hinge loss function calculates the loss.

```
import numpy as np
y_true = np.random.choice([-1, 1], size=(2, 3))
y_pred = np.random.random(size=(2, 3))
loss = tf.keras.losses.squared_hinge(y_true, y_pred)
assert loss.shape == (2,)
assert np.array_equal(loss.numpy(),np.mean(np.square(np.maximum(1. - y_true * y_pred, 0.)), axis=-1))
```

**vii) Keras Categorical Hinge Loss**

The second type of hinge loss function is the categorical hinge loss function. It can help in computing categorical hinge loss between true values and predicted values for multiclass classification.

**Syntax of Keras Categorical Hinge Loss**

Below is the syntax of Categorical Hinge Loss in Keras –

```
tf.keras.losses.CategoricalHinge(reduction="auto", name="categorical_hinge")
```

**Keras Categorical Hinge Loss Example**

**CategoricalHinge()**function we calculate the final result for categorical hinge loss.

```
y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.
h = tf.keras.losses.CategoricalHinge()
h(y_true, y_pred).numpy()
```

**2. Keras Loss Function for Regression**

Let us now see the second types of loss function in Keras for Regression models

These regression loss functions are calculated on the basis of residual or error of the actual value and predicted value. The below animation shows this concept.

Different types of Regression Loss function in Keras are as follows:

**i) Keras Mean Square Error Loss**

The *mean square error* in Keras is used for computing the mean square of errors between predicted values and actual values to get the loss.

**Syntax of Mean Square Error Loss in Keras**

Below is the syntax of Keras Mean Square in Keras –

```
tf.keras.losses.MeanSquaredError(reduction="auto", name="mean_squared_error")
```

**Keras Mean Square Error Loss Example**

The below code snippet shows how we can implement mean square error in Keras.

```
y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [1., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.
mse = tf.keras.losses.MeanSquaredError()
mse(y_true, y_pred).numpy()
```

**ii) Keras Mean Absolute Error Loss**

The mean absolute error is computed using mean of absolute difference of labels and predicted values.

**Syntax of Mean Absolute Error Loss in Keras**

Below is the syntax of mean absolute error loss in Keras –

```
tf.keras.losses.MeanAbsoluteError(
reduction="auto", name="mean_absolute_error"
)
```

**Keras Mean Absolute Error Loss Example**

With help of *losses* class of Keras, we can import mean absolute error and then apply this over a dataset to compute mean absolute error loss.

```
y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [1., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.
mae = tf.keras.losses.MeanAbsoluteError()
mae(y_true, y_pred).numpy()
```

**iii) Keras Cosine Similarity Loss**

To calculate cosine similarity loss amongst the labels and predictions, we use cosine similarity. The value for cosine similarity ranges from -1 to 1.

**Syntax of Cosine Similarity Loss in Keras**

Below is the syntax of cosine similarity loss in Keras –

```
tf.keras.losses.CosineSimilarity(
axis=-1, reduction="auto", name="cosine_similarity"
)
```

**Keras Cosine Similarity Loss Example**

In this example, for implementing cosine similarity in Keras, we are going to use **cosine_loss** function.

```
y_true = [[0., 1.], [1., 1.]]
y_pred = [[1., 0.], [1., 1.]]
# Using 'auto'/'sum_over_batch_size' reduction type.
cosine_loss = tf.keras.losses.CosineSimilarity(axis=1)
cosine_loss(y_true, y_pred).numpy()
```

**iv) Keras Huber Loss Function**

In regression related problems where data is less affected by outliers, we can use **huber loss function**.

**Syntax of Huber Loss Function in Keras**

Below is the syntax of Huber Loss function in Keras

```
tf.keras.losses.Huber(delta=1.0, reduction="auto", name="huber_loss")
```

**Huber Loss Function in Keras Example**

Keras library provides Huber function for calculating the Huber loss.

```
y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.
h = tf.keras.losses.Huber()
h(y_true, y_pred).numpy()
```

**Keras Custom Loss Function**

In spite of so many loss functions, there are cases when these loss functions do not serve the purpose. In such scenarios, we can build a custom loss function in Keras, which is especially useful for research purposes.

You can pass this custom loss function in Keras as a parameter while compiling the model. But there is a constraint here that the custom loss function should take the true value (y_true) and predicted value (y_pred) as input and return an array of loss. If your function does not match this signature then you cannot use this as a custom function in Keras.

**Keras Custom Loss function Example**

The below code snippet shows how to build a custom loss function. Once this function is created, we use it to compile the model using Keras.

```
def custom_loss_function(y_true, y_pred):
squared_difference = tf.square(y_true - y_pred)
return tf.reduce_mean(squared_difference, axis=-1)
model.compile(optimizer='adam', loss=custom_loss_function)
```

**Keras add_loss() API**

As we saw above, the custom loss function in Keras has a restriction to use a specific signature of having y_true and y_pred as arguments. Keras provides another option of add_loss() API which does not have this constraint.

**Keras add_loss() API Example**

The below cell contains an example of how **add_loss()** function is used for building loss function.

```
from keras.layers import Layer
class Custom_layer(Layer):
def __init__(self,rate=1e-2):
super(Custom_layer,self).__init__()
self.rate=rate
def call(self,inputs):
self.add_loss(self.rate*tf.square(inputs))
return inputs
```

**Also Read**– Dummies guide to Cost Functions in Machine Learning [with Animation]**Also Read**– Different Types of Keras Layers Explained for Beginners**Also Read**– Keras Dropout Layer Explained for Beginners**Also Read**– Keras Dense Layer Explained for Beginners**Also Read**– Keras vs Tensorflow vs Pytorch – No More Confusion !!**Also Read**– Keras Convolution Layer – A Beginner’s Guide**Also Read –**Beginners’s Guide to Keras Models API

**Conclusion**

In this tutorial, we looked at different types of loss functions in Keras, with their syntax and examples. We looked at loss functions for classification and regression problems and lastly, we looked at the custom loss function option of Keras.

Reference Keras Documentation