FIT3181: Deep Learning (2026)

CE/Lecturer (Clayton): Dr Trung Le
Lecturer (Clayton): Dr Binh Nguyen
Lecturer (Malaysia): Dr Arghya Pal, Dr Sailaja Rajanala, Dr Sicily Fung Fung Ting
Head Tutor (Clayton): Mr Minh Vo (FIT3181), Dr Ruda Nie H (FIT5215)

Faculty of Information Technology, Monash University, Australia

Student Information

Surname: Won
Firstname: Luke
Student ID: 34080481
Email: lwon0059@student.monash.edu
Your tutorial time: Tuesday 1PM - 3PM

📑 Table of Contents

• What to submit
• Part 1: Theory and Knowledge Questions (30 points)
• Question 1.1: Activation functions (ELU, GELU)
• Question 1.2: Feed-forward Network Computations & Loss
• Question 1.3: Overview (Choose Option 1 or Option 2)
• Option 1: Forward, Backward & SGD for mini-batch (10 points)
• Option 2: Manually Implement Feed-forward NN in PyTorch (20 points)
• Part 2: Deep Neural Networks (DNN - FashionMNIST) (25 points)
• Question 2.1: Mini-batch Visualization (4 points)
• Question 2.2: Feed-forward Neural Net in PyTorch (4 points)
• Question 2.3: Grid Search Hyperparameter Tuning (5 points)
• Question 2.4: Regularized Loss Function (6 points)
• Question 2.5: Sharpness-Aware Minimization (SAM) (6 points)
• Part 3: Convolutional Neural Networks and Image Classification (45 points)
• CNN Architecture & Parameter Specifications
• Question 3.1: Implement YourBlock & YourCNN (12 points)
• Question 3.2: Data Mixup Technique (4 points)
• Question 3.3: CutMix Technique (4 points)
• Question 3.4: One-Versus-All (OVA) Loss (4 points)
• Question 3.5: Unsupervised Rotation Loss (6 points)
• Question 3.6: Kaggle Competition (15 points)

Deep Neural Networks

Due: 11:55pm Sunday, 13 September 2026 (Sunday)

Important note: This is an individual assignment. It contributes 25% to your final mark. Read the assignment instructions carefully.

What to submit

This assignment is to be completed individually and submitted to Moodle unit site. By the due date, you are required to submit one single zip file, named xxx_assignment01_solution.zip where xxx is your student ID, to the corresponding Assignment (Dropbox) in Moodle. You can use Google Colab to do Assigmnent 1 but you need to save it to an *.ipynb file to submit to the unit Moodle.

More importantly, if you use Google Colab to do this assignment, you need to first make a copy of this notebook on your Google drive.

For example, if your student ID is 12356, then gather all of your assignment solution to folder, create a zip file named 123456_assignment01_solution.zip and submit this file.

Within this zip folder, you must submit the following files:

1. Assignment01_solution.ipynb: this is your Python notebook solution source file.

1. Assignment01_output.html or Assignment01_output.pdf: this is the output of your Python notebook solution exported in html or pdf format.

1. Any extra files or folder needed to complete your assignment (e.g., images used in your answers).

Since the notebook is quite big to load and work together, one recommended option is to split solution into three parts and work on them seperately. In that case, replace Assignment01_solution.ipynb by three notebooks: Assignment01_Part1_solution.ipynb, Assignment01_Part2_solution.ipynb and Assignment01_Part3_solution.ipynb

You can run your codes on Google Colab. In this case, you have to make a copy of your Google colab notebook including the traces and progresses of model training before submitting.

Part 1: Theory and Knowledge Questions

The first part of this assignment is to demonstrate your knowledge in deep learning that you have acquired from the lectures and tutorials materials. Most of the contents in this assignment are drawn from the lectures and tutorials from weeks 1 to 4. Going through these materials before attempting this part is highly recommended.

Question 1.1 Activation function plays an important role in modern Deep NNs. For each of the activation functions below, state its output range, find its derivative (show your steps), and plot the activation fuction and its derivative

(a) Exponential linear unit (ELU): $\text{ELU}(x)=\begin{cases} 0.1\left(\exp(x)-1\right) & \text{if}\,x\leq0\\ x & \text{if}\,x>0 \end{cases}$

(b) Gaussian Error Linear Unit (GELU): $\text{GELU}(x)=x\Phi(x)$ where $\Phi(x)$ is the probability cummulative function of the standard Gaussian distribution or $\Phi(x) = \mathbb{P}\left(X\leq x\right)$ where $X \sim N\left(0,1\right)$. In addition, the GELU activation fuction (the link for the main paper) has been widely used in the state-of-the-art Vision for Transformers (e.g., here is the link for the main ViT paper).

Question 1.2: Assume that we feed a data point $x$ with a ground-truth label $y=2$ to the feed-forward neural network with the ReLU activation function as shown in the following figure

Question 1.2 Feed-forward Network

(a) What is the numerical value of the latent presentation $h^1(x)$?

(b) What is the numerical value of the latent presentation $h^2(x)$?

(c) What is the numerical value of the logit $h^3(x)$?

(d) What is the corresonding prediction probabilities $p(x)$?

(e) What is the predicted label $\widehat{y}$? Is it a correct and an incorect prediction? Remind that $y=2$.

(f) What is the cross-entropy loss caused by the feed-forward neural network at $(x,y)$? Remind that $y=2$.

(g) Why is the cross-entropy loss caused by the feed-forward neural network at $(x,y)$ (i.e., $\text{CE}(1_y, p(x))$) always non-negative? When does this $\text{CE}(1_y, p(x))$ loss get the value $0$? Note that you need to answer this question for a general pair $(x,y)$ and a general feed-forward neural network with, for example $M=4$ classes?

You must show both formulas and numerical results for earning full mark. Although it is optional, it is great if you show your PyTorch code for your computation.

Question 1.3:

For Question 1.3, you have two options:

• (1) perform the forward, backward propagation, and SGD update for one mini-batch (10 points), or
• (2) manually implement a feed-forward neural network that can work on real tabular datasets (20 points).

You can choose either (1) or (2) to proceed.

Option 1

Assume that we are constructing a multilayered feed-forward neural network for a classification problem with three classes where the model parameters will be generated randomly using your student ID. The architecture of this network is $3 (Input)\rightarrow 5(ELU) \rightarrow 3(Output)$ as shown in the following figure. Note that the ELU has the same formula as the one in Q1.1.

We feed a batch $X$ with the labels $Y$ as shown in the figure. Answer the following questions.

Question 1.3 Option 1 Network

You need to show both formulas, numerical results, and your PyTorch code for your computation for earning full marks.

import torch
student_id = 1234           #insert your student id here for example 1234
torch.manual_seed(student_id)

#Code to generate random matrices and biases for W1, b1, W2, b2

Forward propagation

(a) What is the value of $\bar{h}^{1}(x)$ (the pre-activation values of $h^1$)?

(b) What is the value of $h^{1}(x)$?

(c) What is the predicted value $\widehat{y}$?

(d) Suppose that we use the cross-entropy (CE) loss. What is the value of the CE loss $l$ incurred by the mini-batch?

[0.5 point]

Backward propagation

(e) What are the derivatives $\frac{\partial l}{\partial h^{2}},\frac{\partial l}{\partial W^{2}}$, and $\frac{\partial l}{\partial b^{2}}$?

(f) What are the derivatives $\frac{\partial l}{\partial h^{1}}, \frac{\partial l}{\partial \bar{h}^{1}},\frac{\partial l}{\partial W^{1}}$, and $\frac{\partial l}{\partial b^{1}}$?

SGD update

(g) Assume that we use SGD with learning rate $\eta=0.01$ to update the model parameters. What are the values of $W^2, b^2$ and $W^1, b^1$ after updating?

Option 2

In Option 2, you need to implement a feed-forward NN manually using PyTorch and auto-differentiation of PyTorch. We then manually train the model on the MNIST dataset.

We first download the MNIST dataset and preprocess it.

transform = transforms.Compose([
    transforms.ToTensor(),  # Convert the image to a tensor with shape [C, H, W]
    transforms.Normalize((0.5,), (0.5,)),  # Normalize to [-1, 1]
    transforms.Lambda(lambda x: x.view(28*28)) # Flatten the tensor to shape [-1,HW]
])

# Load the MNIST dataset
train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)

train_data, train_labels = train_dataset.data, train_dataset.targets
test_data, test_labels = test_dataset.data, test_dataset.targets
print(train_data.shape, train_labels.shape)
print(test_data.shape, test_labels.shape)

Each data point has dimension [28,28]. We need to flatten it to a vector to input to our FFN.

train_dataset.data = train_data.data.reshape(-1, 28*28)
test_dataset.data = test_data.data.reshape(-1, 28*28)

train_data, train_labels = train_dataset.data, train_dataset.targets
test_data, test_labels = test_dataset.data, test_dataset.targets
print(train_data.shape, train_labels.shape)
print(test_data.shape, test_labels.shape)

We split the train and test sets into many mini-batches of 64.

train_loader = DataLoader(dataset=train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=64, shuffle=False)

Develop the feed-forward neural networks

(a) You need to develop the class MyLinear with the following skeleton. You need to declare the weight matrix and bias of this linear layer.

class MyLinear(torch.nn.Module):
  def __init__(self, input_size, output_size):
    """
    input_size: the size of the input
    output_size: the size of the output
    """
    super().__init__()
    #Your code here
    self.W = torch.nn.Parameter(
        torch.randn(input_size, output_size) * math.sqrt(2.0 / input_size)
    )
    self.b = torch.nn.Parameter(torch.zeros(output_size))

  #forward propagation
  def forward(self, x): #x is a mini-batch
    #Your code here
    return torch.matmul(x, self.W) + self.b

(b) You need to develop the class MyFFN with the following skeleton

class MyFFN(torch.nn.Module):
  def __init__(self, input_size, num_classes, hidden_sizes, act = torch.nn.ReLU()):
    """
    input_size: the size of the input
    num_classes: the number of classes
    act is the activation function
    hidden_sizes is the list of hidden sizes
    for example input_size = 3, hidden_sizes = [5, 7], num_classes = 4, and act = torch.nn.ReLU()
    means that we are building up a FFN with the confirguration
    (3 (Input) -> 5 (ReLU) -> 7 (ReLU) -> 4 (Output))
    """
    super(MyFFN, self).__init__()
    self.input_size = input_size
    self.num_classes = num_classes
    self.act = act
    self.hidden_sizes = hidden_sizes
    self.num_layers = len(hidden_sizes) + 1

  def create_FFN(self):
    """
    This function creates the feed-forward neural network
    We stack many MyLinear layers
    """
    hidden_sizes = [self.input_size] + self.hidden_sizes + [self.num_classes]
    self.layers = []
    #Your code here


  def forward(self,x):
    """
    This implements the forward propagation of the batch x
    This needs to return the logits of x
    """
    #Your code here

  def compute_loss(self, x, y):
    """
    This function computes the cross-entropy loss
    You can use the built-in CE loss of PyTorch
    """
    #Your code here

  def update_SGD(self, x, y, learning_rate = 0.01):
    """
    This function updates the model parameters using SGD using the batch (x,y)
    You need to implement the update rule manually and cannot rely on the built-in optimizer
    """
    #Your code here

  def update_SGDwithMomentum(self, x, y, learning_rate = 0.01, momentum = 0.9):
    """
    This function updates the model parameters using SGD with momentum using the batch (x,y)
    You need to implement the update rule manually and cannot rely on the built-in optimizer
    """
    #Your code here

  def update_AdaGrad(self, x, y, learning_rate = 0.01):
    """
    This function updates the model parameters using AdaGrad using the batch (x,y)
    You need to implement the update rule manually and cannot rely on the built-in optimizer
    """
    #Your code here

myFFN = MyFFN(input_size = 28*28, num_classes = 10, hidden_sizes = [100, 100], act = torch.nn.ReLU)
myFFN.create_FFN()
print(myFFN)

(c) Write the code to evaluate the accuracy of the current myFFN model on a data loader (e.g., train_loader or test_loader).

def compute_acc(model, data_loader):
  """
  This function computes the accuracy of the model on a data loader
  """
  #Your code here

(d) Write the code to evaluate the loss of the current myFFN model on a data loader (e.g., train_loader or test_loader).

def compute_loss(model, data_loader):
  """
  This function computes the loss of the model on a data loader
  """
  #Your code here

Train on the MNIST data with 50 epochs using updateSGD.

num_epochs = 50
for epoch in range(num_epochs):
    for i, (x, y) in enumerate(train_loader):
      myFFN.update_SGD(x, y, learning_rate = 0.01)
    train_acc = compute_acc(myFFN, train_loader)
    train_loss = compute_loss(myFFN, train_loader)
    test_acc = compute_acc(myFFN, test_loader)
    test_loss = compute_loss(myFFN, test_loader)
    print(f"Epoch {epoch+1}/{num_epochs}, Train Loss: {train_loss:.4f}, Train Acc: {train_acc*100:.2f}%, Test Loss: {test_loss:.4f}, Test Acc: {test_acc*100:.2f}%")

(e) Implement the function updateSGDMomentum in the class and train the model with this optimizer in 50 epochs. You can update the corresponding function in the MyFNN class.

(f) Implement the function updateAdagrad in the class and train the model with this optimizer in 50 epochs. You can update the corresponding function in the MyFNN class.

Part 2: Deep Neural Networks (DNN)

The second part of this assignment is to demonstrate your basis knowledge in deep learning that you have acquired from the lectures and tutorials materials. Most of the contents in this assignment are drawn from the tutorials covered from weeks 1 to 2. Going through these materials before attempting this assignment is highly recommended.

In the second part of this assignment, you are going to work with the FashionMNIST dataset for the image recognition task. It has the exact same format as MNIST (70,000 grayscale images of 28 × 28 pixels each with 10 classes), but the images represent fashion items rather than handwritten digits, so each class is more diverse, and the problem is significantly more challenging than MNIST.

Load the Fashion MNIST using torchvision

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])

train_dataset_orgin = datasets.FashionMNIST(root='./data', train=True, download=True, transform=transform)
test_dataset = datasets.FashionMNIST(root='./data', train=False, download=True, transform=transform)

print(train_dataset_orgin.data.shape, train_dataset_orgin.targets.shape)
print(test_dataset.data.shape, test_dataset.targets.shape)

# Flatten the data
train_dataset_orgin.data = train_dataset_orgin.data.reshape(-1, 28*28)
test_dataset.data = test_dataset.data.reshape(-1, 28*28)

print(train_dataset_orgin.data.shape, train_dataset_orgin.targets.shape)
print(test_dataset.data.shape, test_dataset.targets.shape)

N = len(train_dataset_orgin)
print(f"Number of training samples: {N}")
N_train = int(0.9*N)
N_val = N - N_train
print(f"Number of training samples: {N_train}")
print(f"Number of validation samples: {N_val}")

train_dataset, val_dataset = torch.utils.data.random_split(train_dataset_orgin, [N_train, N_val])


print(len(train_dataset))
print(len(val_dataset))


train_loader = DataLoader(dataset=train_dataset, batch_size=64, shuffle=True)
val_loader = DataLoader(dataset=val_dataset, batch_size=64, shuffle=False)
test_loader = DataLoader(dataset=test_dataset, batch_size=1000, shuffle=False)

Question 2.1: Write the code to visualize a mini-batch in train_loader including its images and labels.

Question 2.2: Write the code for the feed-forward neural net using PyTorch

We now develop a feed-forward neural network with the architecture $784 \rightarrow 40(ReLU) \rightarrow 30(ReLU) \rightarrow 10(softmax)$. You can choose your own way to implement your network and an optimizer of interest. You should train model in $50$ epochs and evaluate the trained model on the test set.

Question 2.3: Tuning hyper-parameters with grid search

Assume that you need to tune the number of neurons on the first and second hidden layers $n_1 \in \{20, 40\}$, $n_2 \in \{20, 40\}$ and the used activation function $act \in \{sigmoid, tanh, relu\}$. The network has the architecture pattern $784 \rightarrow n_1 (act) \rightarrow n_2(act) \rightarrow 10(softmax)$ where $n_1, n_2$, and $act$ are in their grides. Write the code to tune the hyper-parameters $n_1, n_2$, and $act$. Note that you can freely choose the optimizer and learning rate of interest for this task.

Question 2.4: Implement the loss with the form: $loss(p,y)=CE(1_{y},p)+\lambda H(p)$ where $H(p)=-\sum_{i=1}^{M}p_{i}\log p_{i}$ is the entropy of $p$, $p$ is the prediction probabilities of a data point $x$ with the ground-truth label $y$, $1_y$ is an one-hot label, and $\lambda >0$ is a trade-off parameter. Set $\lambda = 0.1$ to train a model.

Question 2.5: Experimenting with sharpness-aware minimization technique

Sharpness-aware minimization (SAM) (i.e., link for main paper from Google Deepmind) is a simple yet but efficient technique to improve the generalization ability of deep learning models on unseen data examples. In your research or your work, you might potentially use this idea. Your task is to read the paper and implement Sharpness-aware minimization (SAM). Finally, you need to apply SAM to the best architecture found in Question 2.3.

Part 3: Convolutional Neural Networks and Image Classification

The third part of this assignment is to demonstrate your basis knowledge in deep learning that you have acquired from the lectures and tutorials materials. Most of the contents in this assignment are drawn from the tutorials covered from weeks 3 to 6. Going through these materials before attempting this assignment is highly recommended.

The dataset used for this part is a specific dataset for this unit consisting of approximately $10,000$ images of $20$ classes of Animals, each of which has approximately 500 images. You can download the dataset at download here if you want to do your assignment on your machine.

import os
import requests
import tarfile
import time
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, random_split
import torchvision.models as models
import torch.nn as nn
import torch
import PIL.Image
import pathlib
from torchsummary import summary
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np

# check if CUDA is available
train_on_gpu = torch.cuda.is_available()

if not train_on_gpu:
    print('CUDA is not available.  Training on CPU ...')
else:
    print('CUDA is available!  Training on GPU ...')

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.manual_seed(1234)

Download the dataset to the folder of this Google Colab.

!gdown --fuzzy https://drive.google.com/file/d/10Y65ykpja1t6UWt3imPPbZkqhe0kOPL9/view?usp=sharing # new url v1
#!gdown --fuzzy https://drive.google.com/file/d/1uc7uQ5myz7k74ZO460Q04B35_GekQeGi/view?usp=sharing # new url v2

We unzip the dataset to the folder.

!unzip -q Animals_Dataset_v1.zip
# !unzip -q Animals_Dataset_v2.zip
# !unzip -q Animals_Dataset.zip

data_dir = "./FIT5215_Dataset"

# We resize the images to [3,64,64]
transform = transforms.Compose([transforms.Resize((64,64)),  #resises the image so it can be perfect for our model.
                                      transforms.RandomHorizontalFlip(), # FLips the image w.r.t horizontal axis
                                      #transforms.RandomRotation(4),     #Rotates the image to a specified angel
                                      #transforms.RandomAffine(0, shear=10, scale=(0.8,1.2)), #Performs actions like zooms, change shear angles.
                                      transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2), # Set the color params
                                      transforms.ToTensor(), # convert the image to tensor so that it can work with torch
                                      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),  # Normalize the images, each R,G,B value is normalized with mean=0.5 and std=0.5
                                      ])


# Load the dataset using torchvision.datasets.ImageFolder and apply transformations
dataset = datasets.ImageFolder(data_dir, transform=transform)

# Split the dataset into training and validation sets
train_size = int(0.9 * len(dataset))
valid_size = len(dataset) - train_size
train_dataset, val_dataset = random_split(dataset, [train_size, valid_size])

# Example of DataLoader creation for training and validation
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)

print("Number of instance in train_set: %s" % len(train_dataset))
print("Number of instance in val_set: %s" % len(val_dataset))

class_names = ['bird', 'bottle', 'bread', 'butterfly', 'cake', 'cat', 'chicken', 'cow', 'dog', 'duck',
                  'elephant', 'fish', 'handgun', 'horse', 'lion', 'lipstick', 'seal', 'snake', 'spider', 'vase']

import math

def imshow(img):
    img = img / 2 + 0.5  # unnormalize
    plt.imshow(np.transpose(img, (1, 2, 0)))  # convert from Tensor image

def visualize_data(images, categories, images_per_row = 8):
    class_names = ['bird', 'bottle', 'bread', 'butterfly', 'cake', 'cat', 'chicken', 'cow', 'dog', 'duck',
                  'elephant', 'fish', 'handgun', 'horse', 'lion', 'lipstick', 'seal', 'snake', 'spider', 'vase']
    n_images = len(images)
    n_rows = math.ceil(float(n_images)/images_per_row)
    fig = plt.figure(figsize=(1.5*images_per_row, 1.5*n_rows))
    fig.patch.set_facecolor('white')
    for i in range(n_images):
        plt.subplot(n_rows, images_per_row, i+1)
        plt.xticks([])
        plt.yticks([])
        imshow(images[i])
        class_index = categories[i]
        plt.xlabel(class_names[class_index])
    plt.show()

For questions 3.1 to 3.5, you'll need to write your own model in a way that makes it easy for you to experiment with different architectures and parameters. The goal is to be able to pass the parameters to initialize a new instance of YourModel to build different network architectures with different parameters. Below are descriptions of some parameters for YourModel:

1. Block confirguration: Our network consists of many blocks. Each block has the pattern [conv, batch norm, activation, conv, batch norm, activation, max pool, dropout]. All convolutional layers have filter size $(3, 3)$, strides $(1, 1)$ and padding = 1, and all max pool layers have strides $(2, 2)$, kernel size $2$, and padding = 0. The network will consists of a few blocks before applying a linear layer to output the logits for the softmax layer.

1. list_feature_maps: the number of feature maps in the blocks of the network. For example, if list_feature_maps = [16, 32, 64], our network has two blocks with the input_channels or number of feature maps are 16, 32, and 64 respectively.
2. drop_rate: the keep probability for dropout. Setting drop_rate to $0.0$ means not using dropout.
3. batch_norm: the batch normalization function is used or not. Setting batch_norm to false means not using batch normalization.
4. use_skip: the skip connection is used in the blocks or not. Setting this to true means that we use 1x1 Conv2D with strides=2 for the skip connection.
5. At the end, you need to apply global average pooling (GAP) (AdaptiveAvgPool2d((1, 1))) to flatten the 3D output tensor before defining the output linear layer for predicting the labels.

Here is the model confirguration of YourCNN if the list_feature_maps = [16, 32, 64] and batch_norm = true.

Question 3.1 YourCNN Architecture

Question 3.1: You need to implement the aforementioned CNN.

First, you need to implement the block of our CNN in the class YourBlock. You can ignore use_skip and skip connection for simplicity. However, you cannot earn full marks for this question.

#Your code here
class YourBlock(nn.Module):
  def __init__(self, in_feature_maps, out_feature_maps, drop_rate = 0.2, batch_norm = True, use_skip = True):
    super(YourBlock, self).__init__()
    self.use_skip = use_skip
    #Your code here





  def forward(self, x):
    #Write your code here

Second, you need to use the above YourBlock to implement the class YourCNN.

class YourCNN(nn.Module):
  def __init__(self, list_feature_maps = [16, 32, 64], drop_rate = 0.2, batch_norm= True, use_skip = True):
    super(YourCNN, self).__init__()
    layers = []
    #Write your code here



  def forward(self, x):
    #Write your code here

We declare my_cnn from YourCNN as follows.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
my_cnn = YourCNN(list_feature_maps = [16, 32, 64], use_skip = True)
my_cnn = my_cnn.to(device)
print(my_cnn)

We declare the optimizer and the loss function.

# Loss and optimizer
learning_rate = 0.001
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(my_cnn.parameters(), lr=learning_rate)

Here are the codes to compute the loss and accuracy.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def compute_loss(model, loss_fn, loader):
  loss = 0
  # Set model to eval mode for inference
  model.eval()
  with torch.no_grad():  # No need to track gradients for validation
    for (batchX, batchY) in loader:
      # Move data to the same device as the model
      batchX, batchY = batchX.to(device).type(torch.float32), batchY.to(device).type(torch.long)
      loss += loss_fn(model(batchX), batchY)
  # Set model back to train mode
  model.train()
  return float(loss)/len(loader)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def compute_acc(model, loader):
    correct = 0
    totals = 0
    # Set model to eval mode for inference
    model.eval()
    for (batchX, batchY) in loader:
        # Move batchX and batchY to the same device as the model
        batchX, batchY = batchX.to(device).type(torch.float32), batchY.to(device)
        outputs = model(batchX)  # feed batch to the model
        totals += batchY.size(0)  # accumulate totals with the current batch size
        predicted = torch.argmax(outputs.data, 1)  # get the predicted class
        # Move batchY to the same device as predicted for comparison
        correct += (predicted == batchY).sum().item()
    return correct / totals

Here is the code to train our model.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def fit(model= None, train_loader = None, valid_loader= None, optimizer = None,
        num_epochs = 50, verbose = True, seed= 1234):
  torch.manual_seed(seed)
  # Move the model to the device before initializing the optimizer
  model.to(device) # Move the model to the GPU

  if optimizer == None:
    optim = torch.optim.Adam(model.parameters(), lr = 0.001) # Now initialize optimizer with model on GPU
  else:
    optim = optimizer
  history = dict()
  history['val_loss'] = list()
  history['val_acc'] = list()
  history['train_loss'] = list()
  history['train_acc'] = list()

  for epoch in range(num_epochs):
    model.train()
    for (X, y) in train_loader:
      # Move input data to the same device as the model
      X,y = X.to(device), y.to(device)
      # Forward pass
      outputs = model(X.type(torch.float32)) # X is already on the correct device
      loss = loss_fn(outputs, y.type(torch.long))
      # Backward and optimize
      optim.zero_grad()
      loss.backward()
      optim.step()
      #losses and accuracies for epoch
    val_loss = compute_loss(model, loss_fn, valid_loader)
    val_acc = compute_acc(model, valid_loader)
    train_loss = compute_loss(model, loss_fn, train_loader)
    train_acc = compute_acc(model, train_loader)
    history['val_loss'].append(val_loss)
    history['val_acc'].append(val_acc)
    history['train_loss'].append(train_loss)
    history['train_acc'].append(train_acc)
    if not verbose: #verbose = True means we do show the training information during training
      print(f"Epoch {epoch+1}/{num_epochs}")
      print(f"train loss= {train_loss:.4f} - train acc= {train_acc*100:.2f}% - valid loss= {val_loss:.4f} - valid acc= {val_acc*100:.2f}%")
  return history

history = fit(model= my_cnn, train_loader=train_loader, valid_loader = val_loader, optimizer = optimizer, num_epochs= 10, verbose = False)

Please note that you struggle in implementing the aforementioned CNN. You can use the MiniVGG network in our labs for doing the following questions. However, you cannot earn any mark for 3.1 and 3.2.

Question 3.2: Exploring Data Mixup Technique for Improving Generalization Ability.

Data mixup is another super-simple technique used to boost the generalization ability of deep learning models. You need to incoroporate data mixup technique to the above deep learning model and experiment its performance. There are some papers and documents for data mixup as follows:

• Main paper for data mixup link for main paper and a good article article link.

You need to extend your model developed above, train a model using data mixup, and write your observations and comments about the result.

Question 3.3: Exploring CutMix Technique for Improving Generalization Ability.

CutMix is another super-simple technique used to boost the generalization ability of deep learning models. You need to incoroporate data CutMix technique to the above deep learning model and experiment its performance. There are some papers and documents for data mixup as follows:

• Main paper for Cutmix link for main paper and a good article article link.

You need to extend your model developed above, train a model using data CutMix, and write your observations and comments about the result.

Question 3.4: Implement the one-versus-all (OVA) loss

The details are as follows:

• You need to apply the sigmoid activation function to logits $h = [h_1, h_2,...,h_M]$ instead of the softmax activation function as usual to obtain $p = [p_1, p_2,...,p_M]$, meaning that $p_i = sigmoid(h_i), i=1,...,M$. Note that $M$ is the number of classes.
• Given a data example $x$ with the ground-truth label $y$, the idea is to maximize the likelihood $p_y$ and to minimize the likelihoods $p_i, i \neq y$. Therefore, the objective function is to find the model parameters to
• $\max\left\{ \log p_{y}+\sum_{i\neq y}\log(1-p_{i})\right\}$ or equivalently $\min\left\{ -\log p_{y}-\sum_{i\neq y}\log(1-p_{i})\right\}$.
• For example, if $M=3$ and $y=2$, you need to minimize $\min\left\{ -\log(1-p_{1})-\log p_{2}-\log(1-p_{3})\right\}$.

Compare the model trained with the OVA loss and the same model trained with the standard cross-entropy loss.

Question 3.5: Incorporate the unsupervised loss to the cross-entropy loss

The unsupervised loss is defined as follows

• For each mini-batch $x$ of images, we rotate these images by $0^{\circ}, 90^{\circ}, 180^{\circ}, 270^{\circ}$ and then predict for labels $0,1,2,3$ respectively. For example, if we rotate an image by $180^{\circ}$, we need to predict the ground-truth label $2$.
• In terms of architecture, on the top of the penultimate layer (i.e., the layer right before the current output layer) of your CNN, you add one more output layer to predict the rotation angles ($0^{\circ}$ - label 0, $90^{\circ}$ - label 1, $180^{\circ}$ - label 2, and $270^{\circ}$ - label 3).
• The final loss is $total\_loss=CE\_loss+\lambda\times unsupervised\_loss$ where $\lambda > 0$ is a trade-off parameter.
• You can set $\lambda$ (e.g., $\lambda = 0.1$) a value and train a model with this value.
• Note that you can as many cells as you want to extend the current code.

Question 3.6 (Kaggle competition)