In this tutorial, we will look at various types of Keras loss functions for training neural networks. The loss functions are an important part of any neural network training process as it helps the network to minimize the error and reach as close as possible to the expected output. Here we will go through Kera loss functions for regression, classification and also see how to create a custom loss function in Keras.

What is Loss Function?

Loss Functions, also known as cost functions, are used for computing the error with the aim that the model should minimize it during training.

Loss Functions also help in finding out the slope i.e. gradient w.r.t. weights used in the model and then these weights are updated after each epoch with the help of backpropagation.

The below animation shows how a loss function works.

Read More – Dummies guide to Loss Functions in Machine Learning [with Animation]

Types of Keras Loss Functions

Selecting a loss function is not so easy, so we’ll be going over some prominent loss functions that can be helpful in various instances.

Types of Loss Functions in Keras

1. Keras Loss Function for Classification

Let us first understand the Keras loss functions for classification which is usually calculated by using probabilistic losses.

i) Keras Binary Cross Entropy

Binary Cross Entropy loss function finds out the loss between the true labels and predicted labels for the binary classification models that gives the output as a probability between 0 to 1.

Types of Keras Loss Functions for Classification

Syntax of Keras Binary Cross Entropy

Following is the syntax of Binary Cross Entropy Loss Function in Keras.

tf.keras.losses.BinaryCrossentropy(
    from_logits=False, label_smoothing=0, reduction="auto", name="binary_crossentropy"
)

The example for Keras binary cross entropy shows how two sets of random values are used as data and then the required function from losses class is used.

import tensorflow as tf

y_true = [[0., 1.], [0., 0.]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
bce = tf.keras.losses.BinaryCrossentropy()
bce(y_true, y_pred).numpy()

0.81492424

ii) Keras Categorical Cross Entropy

This is the second type of probabilistic loss function for classification in Keras and is a generalized version of binary cross entropy that we discussed above. Categorical Cross Entropy is used for multiclass classification where there are more than two class labels.

Syntax of Keras Categorical Cross Entropy

Following is the syntax of Categorical Cross Entropy Loss Function in Keras.

tf.keras.losses.CategoricalCrossentropy(from_logits=False,label_smoothing=0, reduction="auto",name="categorical_crossentropy",)

Keras Categorical Cross Entropy Example

The following is an example of Keras categorical cross entropy. y_true denotes the actual probability distribution of the output and y_pred denotes the probability distribution we got from the model.

y_true = [[0, 1, 0], [0, 0, 1]]
y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
cce = tf.keras.losses.CategoricalCrossentropy()
cce(y_true, y_pred).numpy()

1.1769392

tf.keras.losses.KLDivergence(reduction="auto", name="kl_divergence")

0.45814306

The KL Divergence or Kullback-Leibler Divergene Loss function is computed between the actual value and predicted value in the case of continuous distributions.

Syntax of Keras KL Divergence

Below is the syntax of LL Divergence in Keras –

The KL Divergence or Kullback-Leibler Divergene Loss function is computed between the actual value and predicted value in the case of continuous distributions.

tf.keras.losses.KLDivergence(reduction="auto", name="kl_divergence")

The KLDivergence() function is used in this case. The result obtained shows that there is not a huge loss but still it is considerable.

y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
kl = tf.keras.losses.KLDivergence()
kl(y_true, y_pred).numpy()

0.45814306

tf.keras.losses.Poisson(reduction="auto", name="poisson")

Keras Poisson Loss Function Example

The poisson loss function is used in below example.

y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [0., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
p = tf.keras.losses.Poisson()
p(y_true, y_pred).numpy()

0.49999997

tf.keras.losses.Hinge(reduction="auto", name="hinge")

1.3

1.4000001

iv) Keras Hinge Loss

The above Keras loss functions for classification were using probabilistic loss as their basis for calculation. Now we are going to see some loss functions in Keras that use Hinge Loss for maximum margin classification like in SVM.

The hinge loss function is performed by computing hinge loss of true values and predicted values.

Syntax of Keras Hinge Loss

Below is the syntax of Keras Hinge loss –

tf.keras.losses.Hinge(reduction="auto", name="hinge")

The hinge() function from the Keras package helps in finding the hinge loss

y_true = [[0., 1.], [0., 0.]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
h = tf.keras.losses.Hinge()
h(y_true, y_pred).numpy()

1.3

tf.keras.losses.squared_hinge(y_true, y_pred)

vi) Keras Squared Hinge Loss

The squared hinge loss is calculated using squared_hinge() function and is similar to Hinge Loss calculation discussed above except that the result is squared.

Syntax of Squared Hinge Loss in Keras

tf.keras.losses.squared_hinge(y_true, y_pred)

Example of Squared Hinge Loss in Keras

In this example, at first, data is generated using numpy randon function, then Keras squared hinge loss function calculates the loss.

import numpy as np

y_true = np.random.choice([-1, 1], size=(2, 3))
y_pred = np.random.random(size=(2, 3))
loss = tf.keras.losses.squared_hinge(y_true, y_pred)
assert loss.shape == (2,)
assert np.array_equal(loss.numpy(),np.mean(np.square(np.maximum(1. - y_true * y_pred, 0.)), axis=-1))

tf.keras.losses.CategoricalHinge(reduction="auto", name="categorical_hinge")

Keras Categorical Hinge Loss Example

With the CategoricalHinge() function we calculate the final result for categorical hinge loss.

y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
h = tf.keras.losses.CategoricalHinge()
h(y_true, y_pred).numpy()

1.4000001

Let us now see the second types of loss function in Keras for Regression models

These regression loss functions are calculated on the basis of residual or error of the actual value and predicted value. The below animation shows this concept.

Types of Keras Loss Functions for Regression

Different types of Regression Loss function in Keras are as follows:

i) Keras Mean Square Error Loss

The mean square error in Keras is used for computing the mean square of errors between predicted values and actual values to get the loss.

Syntax of Mean Square Error Loss in Keras

Below is the syntax of Keras Mean Square in Keras –

tf.keras.losses.MeanSquaredError(reduction="auto", name="mean_squared_error")

Keras Mean Square Error Loss Example

The below code snippet shows how we can implement mean square error in Keras.

y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [1., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
mse = tf.keras.losses.MeanSquaredError()
mse(y_true, y_pred).numpy()

0.5

ii) Keras Mean Absolute Error Loss

The mean absolute error is computed using mean of absolute difference of labels and predicted values.

Syntax of Mean Absolute Error Loss in Keras

Below is the syntax of mean absolute error loss in Keras –

tf.keras.losses.MeanAbsoluteError(
    reduction="auto", name="mean_absolute_error"
)

Keras Mean Absolute Error Loss Example

With help of losses class of Keras, we can import mean absolute error and then apply this over a dataset to compute mean absolute error loss.

y_true = [[0., 1.], [0., 0.]]
y_pred = [[1., 1.], [1., 0.]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
mae = tf.keras.losses.MeanAbsoluteError()
mae(y_true, y_pred).numpy()

0.5

iii) Keras Cosine Similarity Loss

To calculate cosine similarity loss amongst the labels and predictions, we use cosine similarity. The value for cosine similarity ranges from -1 to 1.

Syntax of Cosine Similarity Loss in Keras

Below is the syntax of cosine similarity loss in Keras –

tf.keras.losses.CosineSimilarity(
    axis=-1, reduction="auto", name="cosine_similarity"
)

Keras Cosine Similarity Loss Example

In this example, for implementing cosine similarity in Keras, we are going to use cosine_loss function.

y_true = [[0., 1.], [1., 1.]]
y_pred = [[1., 0.], [1., 1.]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
cosine_loss = tf.keras.losses.CosineSimilarity(axis=1)
cosine_loss(y_true, y_pred).numpy()

-0.49999997

iv) Keras Huber Loss Function

In regression related problems where data is less affected by outliers, we can use huber loss function.

Syntax of Huber Loss Function in Keras

Below is the syntax of Huber Loss function in Keras

tf.keras.losses.Huber(delta=1.0, reduction="auto", name="huber_loss")

Huber Loss Function in Keras Example

Keras library provides Huber function for calculating the Huber loss.

y_true = [[0, 1], [0, 0]]
y_pred = [[0.6, 0.4], [0.4, 0.6]]
# Using 'auto'/'sum_over_batch_size' reduction type.  
h = tf.keras.losses.Huber()
h(y_true, y_pred).numpy()

0.155

Keras Custom Loss Function

In spite of so many loss functions, there are cases when these loss functions do not serve the purpose. In such scenarios, we can build a custom loss function in Keras, which is especially useful for research purposes.

You can pass this custom loss function in Keras as a parameter while compiling the model. But there is a constraint here that the custom loss function should take the true value (y_true) and predicted value (y_pred) as input and return an array of loss. If your function does not match this signature then you cannot use this as a custom function in Keras.

Keras Custom Loss function Example

The below code snippet shows how to build a custom loss function. Once this function is created, we use it to compile the model using Keras.

def custom_loss_function(y_true, y_pred):
   squared_difference = tf.square(y_true - y_pred)
   return tf.reduce_mean(squared_difference, axis=-1)

model.compile(optimizer='adam', loss=custom_loss_function)

Keras add_loss() API

As we saw above, the custom loss function in Keras has a restriction to use a specific signature of having y_true and y_pred as arguments. Keras provides another option of add_loss() API which does not have this constraint.

Keras add_loss() API Example

The below cell contains an example of how add_loss() function is used for building loss function.

from keras.layers import Layer
class Custom_layer(Layer):
  def __init__(self,rate=1e-2):
    super(Custom_layer,self).__init__()
    self.rate=rate
  
  def call(self,inputs):
    self.add_loss(self.rate*tf.square(inputs))
return inputs

Also Read – Dummies guide to Cost Functions in Machine Learning [with Animation]

Also Read – Different Types of Keras Layers Explained for Beginners
Also Read – Keras Dropout Layer Explained for Beginners
Also Read – Keras Dense Layer Explained for Beginners

Also Read – Keras vs Tensorflow vs Pytorch – No More Confusion !!
Also Read – Keras Convolution Layer – A Beginner’s Guide
Also Read – Beginners’s Guide to Keras Models API

Conclusion

In this tutorial, we looked at different types of loss functions in Keras, with their syntax and examples. We looked at loss functions for classification and regression problems and lastly, we looked at the custom loss function option of Keras.

Reference Keras Documentation