Autoencoder

In this experiment, I will implement the famous autoencoder model that will learn the best fit function to compress image data from the Fashion MINST dataset. The model also has a decoder that can reconstruct to the original image. Although the reconstrcuted images are not identical, the error rate is acceptable and the important features are transfer completely.

Import Libraries

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import tensorflow as tf

from sklearn.metrics import accuracy_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
from tensorflow.keras import layers, losses
from tensorflow.keras.datasets import fashion_mnist
from tensorflow.keras.models import Model

WARNING:tensorflow:From C:\Users\tkvkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

Pre-processing

Like any other data, we need to pre-process each data point between 0 and 1. It will be much more easier and best practice for the learning process.

(x_train, _), (x_test, _) = fashion_mnist.load_data()

x_train = x_train.astype("float32") / 255.
x_test = x_test.astype("float32") / 255.

print(x_train.shape)
print(x_test.shape)

(60000, 28, 28)
(10000, 28, 28)

plt.imshow(x_train[0])
plt.show()

The Model

The model takes inputs as a latent vector dimension, which define the vector that the image will be compressed to. First, the encoder will flatten the image data from a 28x28 matrix to a 1x784 vector. Then it will compress the image to a 1x500 vector, 1x300 vector, and finally 1xlatent_dim vector. The decoder will construct the latent vector back to the original image.

In this example, we will compress a 28x28 image to a 1x64 vector.

class Autoencoder(Model):
    def __init__(self, latent_dim, shape):
        super(Autoencoder, self).__init__()
        self.latent_dim = latent_dim
        self.shape = shape

        self.encoder = tf.keras.Sequential([
            layers.Flatten(),
            layers.Dense(500, activation='relu'),
            layers.Dense(300, activation='relu'),
            layers.Dense(self.latent_dim, activation='relu')
        ])

        self.decoder = tf.keras.Sequential([
            layers.Dense(300, activation='relu'),
            layers.Dense(500, activation='relu'),
            layers.Dense(tf.math.reduce_prod(shape), activation='sigmoid'),
            layers.Reshape(shape)
        ])

    def call(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)

        return decoded

shape = x_test.shape[1:]
latent_dim = 64
autoencoder = Autoencoder(latent_dim, shape)

WARNING:tensorflow:From C:\Users\tkvkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\keras\src\backend.py:873: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

Optimizer and Loss function

Just like simple neural network, the forward propagation is just matrix multiplication with weigth and bias matrices. The backward probagation, we will use adam optimizer and mean square error for loss function.

autoencoder.compile(optimizer='adam', loss=losses.MeanSquaredError())
autoencoder.fit(x_train, x_train,
                epochs=10,
                shuffle=True,
                validation_data=(x_test, x_test))

Epoch 1/10
WARNING:tensorflow:From C:\Users\tkvkh\AppData\Local\Programs\Python\Python311\Lib\site-packages\keras\src\utils\tf_utils.py:492: The name tf.ragged.RaggedTensorValue is deprecated. Please use tf.compat.v1.ragged.RaggedTensorValue instead.

1875/1875 [==============================] - 12s 6ms/step - loss: 0.0205 - val_loss: 0.0142
Epoch 2/10
1875/1875 [==============================] - 10s 6ms/step - loss: 0.0124 - val_loss: 0.0116
Epoch 3/10
1875/1875 [==============================] - 11s 6ms/step - loss: 0.0109 - val_loss: 0.0107
Epoch 4/10
1875/1875 [==============================] - 10s 5ms/step - loss: 0.0100 - val_loss: 0.0099
Epoch 5/10
1875/1875 [==============================] - 10s 5ms/step - loss: 0.0095 - val_loss: 0.0094
Epoch 6/10
1875/1875 [==============================] - 10s 5ms/step - loss: 0.0091 - val_loss: 0.0094
Epoch 7/10
1875/1875 [==============================] - 10s 5ms/step - loss: 0.0088 - val_loss: 0.0088
Epoch 8/10
1875/1875 [==============================] - 10s 5ms/step - loss: 0.0085 - val_loss: 0.0085
Epoch 9/10
1875/1875 [==============================] - 10s 5ms/step - loss: 0.0083 - val_loss: 0.0086
Epoch 10/10
1875/1875 [==============================] - 10s 5ms/step - loss: 0.0081 - val_loss: 0.0084

<keras.src.callbacks.History at 0x1ce002ab550>

encoded_imgs = autoencoder.encoder(x_test).numpy()
decoded_imgs = autoencoder.decoder(encoded_imgs).numpy()

Result

Here is an example, that I use a trained model above to compress then reconstruct the original images.

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
  # display original
  ax = plt.subplot(2, n, i + 1)
  plt.imshow(x_test[i])
  plt.title("original")
  plt.gray()
  ax.get_xaxis().set_visible(False)
  ax.get_yaxis().set_visible(False)

  # display reconstruction
  ax = plt.subplot(2, n, i + 1 + n)
  plt.imshow(decoded_imgs[i])
  plt.title("reconstructed")
  plt.gray()
  ax.get_xaxis().set_visible(False)
  ax.get_yaxis().set_visible(False)
plt.show()