How do I put activation in nn.Linear?

To add an activation function to nn.Linear in PyTorch, you can simply chain them together using torch.nn.Sequential or by defining them separately and applying the activation function in the forward method. Here's a breakdown of both approaches:

1. Using `torch.nn.Sequential`:

This is the most straightforward and often preferred way when building simple neural networks. You can define the linear layer and activation function sequentially within a Sequential container.

Example:

import torch.nn as nn

model = nn.Sequential(
   nn.Linear(in_features=10, out_features=20), # Linear layer
   nn.ReLU() # ReLU activation function
)


In this example:

- nn.Linear(in_features=10, out_features=20) defines a linear layer that takes 10 input features and outputs 20 features.

- nn.ReLU() applies the Rectified Linear Unit (ReLU) activation function to the output of the linear layer.

You can replace nn.ReLU() with any other activation function like nn.Sigmoid(), nn.Tanh(), nn.LeakyReLU(), etc., depending on your requirements.

2. Defining Activation in the `forward` Method:

This approach provides more flexibility, especially when you want to perform more complex operations or have conditional activations. You define the linear layer and the activation function separately and apply the activation in the forward method of your custom module.

Example:

import torch
import torch.nn as nn
import torch.nn.functional as F # for functional activations

class MyModel(nn.Module):
   def __init__(self, in_features, out_features):
      super(MyModel, self).__init__()
      self.linear = nn.Linear(in_features, out_features)

   def forward(self, x):
      x = self.linear(x)
      x = F.relu(x) # Apply ReLU activation
      return x

model = MyModel(in_features=10, out_features=20)


In this example:

- We define a custom module MyModel inheriting from nn.Module.

- In the __init__ method, we define the linear layer self.linear.

- In the forward method, we first pass the input x through the linear layer and then apply the ReLU activation function using F.relu(x).

Using torch.nn.functional (imported as F) allows you to use activation functions as functions rather than layers.

Choosing the Right Approach:

- Use nn.Sequential for simple, sequential models where you just need to chain layers and activations.

- Use the `forward` method for more complex models where you need more control over how activations are applied or when you have additional operations to perform.

Both approaches are valid and widely used in PyTorch. Choose the one that best fits your needs and coding style. Remember to import the necessary modules like torch, torch.nn, and potentially torch.nn.functional.

Learn more about this