Estimated reading time: 60'
The challenge
- Generator model trained to generate new samples
- Discriminator model that attempts to classify the new samples as real (from the original dataset) or fake (generated)
Convolutional neural blocks
- Conv2d: Convolutional layer with input, output channels, kernel, stride and padding
- Dropout: Drop-out regularization layer
- BatchNorm2d: Batch normalization module
- MaxPool2d Pooling layer
- ReLu, Sigmoid, ... Activation functions
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | class ConvNeuralBlock(nn.Module): def __init__(self, in_channels: int, out_channels: int, kernel_size: int, stride: int, padding: int, batch_norm: bool, max_pooling_kernel: int, activation: nn.Module, bias: bool, is_spectral: bool = False): super(ConvNeuralBlock, self).__init__() # Assertions are omitted # 1- initialize the input and output channels self.in_channels = in_channels self.out_channels = out_channels self.is_spectral = is_spectral modules = [] # 2- create a 2 dimension convolution layer conv_module = nn.Conv2d( self.in_channels, self.out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias) # 6- if this is a spectral norm block if self.is_spectral: conv_module = nn.utils.spectral_norm(conv_module) modules.append(conv_module) # 3- Batch normalization if batch_norm: modules.append(nn.BatchNorm2d(self.out_channels)) # 4- Activation function if activation is not None: modules.append(activation) # 5- Pooling module if max_pooling_kernel > 0: modules.append(nn.MaxPool2d(max_pooling_kernel)) self.modules = tuple(modules) |
We considering the case of a generative model for images. The first step (1) is to initialize the number of input and output channels, then create the 2-dimension convolution (2), a batch normalization module (3) an activation function (4) and finally a Max pooling module (5). The spectral norm regularization (6) is optional.
The convolutional neural network is assembled from convolutional and feedback forward neural blocks, in the following build method.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | class ConvModel(NeuralModel): def __init__(self, model_id: str, # 1 Number of input and output unites input_size: int, output_size: int, # 2- PyTorch convolutional modules conv_model: nn.Sequential, dff_model_input_size: int = -1, # 3- PyTorch fully connected dff_model: nn.Sequential = None): super(ConvModel, self).__init__(model_id) self.input_size = input_size self.output_size = output_size self.conv_model = conv_model self.dff_model_input_size = dff_model_input_size self.dff_model = dff_model @classmethod def build(cls, model_id: str, conv_neural_blocks: list, dff_neural_blocks: list) -> NeuralModel: # 4- Initialize the input and output size # for the convolutional layer input_size = conv_neural_blocks[0].in_channels output_size = conv_neural_blocks[len(conv_neural_blocks) - 1].out_channels # 5- Generate the model from the sequence # of conv. neural blocks conv_modules = [conv_module for conv_block in conv_neural_blocks for conv_module in conv_block.modules] conv_model = nn.Sequential(*conv_modules) # 6- If a fully connected RBM is included in the model .. if dff_neural_blocks is not None and not is_vae: dff_modules = [dff_module for dff_block in dff_neural_blocks for dff_module in dff_block.modules] dff_model_input_size = dff_neural_blocks[0].output_size dff_model = nn.Sequential(*tuple(dff_modules)) else: dff_model_input_size = -1 dff_model = None return cls( model_id, conv_dimension, input_size, output_size, conv_model, dff_model_input_size, dff_model) |
The default constructor (1) initializes the number of input/output channels, the PyTorch modules for the convolutional layers (2) and the fully connected layers (3).
The class method, build, instantiate the convolutional model from the convolutional neural blocks and feed forward neural blocks. It initializes the size of input and output layers from the first and last neural blocks (4), generate the PyTorch convolutional modules (5) and fully-connected layers modules (6) from the neural blocks.
Inverting a convolutional block
- Specify components (PyTorch modules) for each convolutional layer
- Assemble these modules into a convolutional neural block
- Create a generator and discriminator network by aggregating the blocks
- Wire the generator and discriminator to product a fully functional GAN
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | class DeConvNeuralBlock(nn.Module): def __init__(self, in_channels: int, out_channels: int, kernel_size: int, stride: int, padding: int, batch_norm: bool, activation: nn.Module, bias: bool) -> object: super(DeConvNeuralBlock, self).__init__() self.in_channels = in_channels self.out_channels = out_channels modules = [] # Two dimension de-convolution layer de_conv = nn.ConvTranspose2d( self.in_channels, self.out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias) # Add the deconvolution block modules.append(de_conv) # Add the batch normalization, if defined if batch_norm: modules.append(nn.BatchNorm2d(self.out_channels)) # Add activation modules.append(activation) self.modules = modules |
Note that the de-convolution block does have any pooling capabilities
The class method, auto_build, takes a convolutional neural block, number of input and output channels and an optional activation function to generate a de-convolutional neural block of type DeConvNeuralBlock. The number of input and output channels in the output deconvolution layer is computed in the private method __resize
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | @classmethod def auto_build(cls, conv_block: ConvNeuralBlock, in_channels: int, out_channels: int = None, activation: nn.Module = None) -> nn.Module: # Extract the parameters of the source convolutional block kernel_size, stride, padding, batch_norm, activation = \ DeConvNeuralBlock.__resize(conv_block, activation) # Override the number of input_tensor channels # for this block if defined next_block_in_channels = in_channels if in_channels is not None \ else conv_block.out_channels # Override the number of output-channels for # this block if specified next_block_out_channels = out_channels if out_channels is not None \ else conv_block.in_channels return cls( conv_block.conv_dimension, next_block_in_channels, next_block_out_channels, kernel_size, stride, padding, batch_norm, activation, False) |
Sizing de-convolutional layers
@staticmethod def __resize( conv_block: ConvNeuralBlock, updated_activation: nn.Module) -> (int, int, int, bool, nn.Module): conv_modules = list(conv_block.modules) # 1- Extract the various components of the # convolutional neural block _, batch_norm, activation = DeConvNeuralBlock.__de_conv_modules(conv_modules) # 2- override the activation function for the # output layer, if necessary if updated_activation is not None: activation = updated_activation # 3- Compute the parameters for the de-convolutional # layer, from the conv. block kernel_size, _ = conv_modules[0].kernel_size stride, _ = conv_modules[0].stride padding = conv_modules[0].padding return kernel_size, stride, padding, batch_norm, activation
The __resize method extracts the PyTorch modules for the de-convolutional layers from the original convolutional block (1), adds the activation function to the block (2) and finally initialize the parameters of the de-convolutional (3).
@staticmethod def __de_conv_modules(conv_modules: list) -> \ (torch.nn.Module, torch.nn.Module, torch.nn.Module): activation_function = None deconv_layer = None batch_norm_module = None # 4- Extract the PyTorch de-convolutional modules # from the convolutional ones for conv_module in conv_modules: if DeConvNeuralBlock.__is_conv(conv_module): deconv_layer = conv_module elif DeConvNeuralBlock.__is_batch_norm(conv_module): batch_norm_moduled = conv_module elif DeConvNeuralBlock.__is_activation(conv_module): activation_function = conv_module return deconv_layer, batch_norm_module, activation_function
Assembling the de-convolutional network
class DeConvModel(NeuralModel, ConvSizeParams): def __init__(self, # 1 - Default constructor model_id: str, input_size: int, # 2 - Size first layer output_size: int, # 3 - Size output layer de_conv_modules: torch.nn.Sequential): super(DeConvModel, self).__init__(model_id) self.input_size = input_size self.output_size = output_size self.de_conv_modules = de_conv_modules @classmethod def build(cls, model_id: str, conv_neural_blocks: list, # 4- Input to the builder in_channels: int, out_channels: int = None, last_block_activation: torch.nn.Module = None) -> NeuralModel: de_conv_neural_blocks = [] # 5- Need to reverse the order of convolutional neural blocks list.reverse(conv_neural_blocks) # 6- Traverse the list of convolutional neural blocks for idx in range(len(conv_neural_blocks)): conv_neural_block = conv_neural_blocks[idx] new_in_channels = None activation = None last_out_channels = None # 7- Update num. input channels for the first # de-convolutional layer if idx == 0: new_in_channels = in_channels # 8- Defined, if necessary the activation # function for the last layer elif idx == len(conv_neural_blocks) - 1: if last_block_activation is not None: activation = last_block_activation if out_channels is not None: last_out_channels = out_channels # 9- Apply transposition to the convolutional block de_conv_neural_block = DeConvNeuralBlock.auto_build( conv_neural_block, new_in_channels, last_out_channels, activation) de_conv_neural_blocks.append(de_conv_neural_block) # 10- Instantiate the Deconvolutional network # from its neural blocks de_conv_model = DeConvModel.assemble( model_id, de_conv_neural_blocks) del de_conv_neural_blocks return de_conv_model
The alternate constructor, build, creates and configures the de-convolutional model from the convolutional blocks conv_neural_blocks (4).
The order of the de-convolutional layers requires the list of convolutional blocks to be reversed (5). For each block of the convolutional network (6), the method updates the number of input channels from the number of input channels of the first layer (7).
The method updates the activation function for the output layer (8) and weaves the de-convolutional blocks (9)
@classmethod def assemble(cls, model_id: str, de_conv_neural_blocks: list): input_size = de_conv_neural_blocks[0].in_channels output_size = de_conv_neural_blocks[len(de_conv_neural_blocks) - 1].out_channels # 11- Generate the PyTorch convolutional modules used by the default constructor conv_modules = tuple([conv_module for conv_block in de_conv_neural_blocks for conv_module in conv_block.modules if conv_module is not None]) de_conv_model = torch.nn.Sequential(*conv_modules) return cls(model_id, input_size, output_size, de_conv_model)
The assemble method constructs the final de-convolutional neural network from the blocks de_conv_neural_blocks by aggregating the PyTorch modules associated with each block (11).
Environment
- Python 3.8
- PyTorch 1.7.2
References
- A Gentle Introduction to Generative Adversarial Networks
- Deep learning Chap 9 Convolutional networks - I. Goodfellow, Y. Bengio, A. Courville - 2017 - MIT Press Cambridge MA
- PyTorch
- Tutorial: DCGAN in PyTorch