**************************************

第八章 网络参数初始化

**************************************



8.1 全零初始化

==================



全零初始化在PyTorch中的调用函数为：



.. code-block:: python

   :linenos:



   # zero initialization (PyTorch)

   import torch.nn as nn



   def reset_parameters(m) -> None:

       # Ln.6-Ln.8: Zero initialization weight and bias

       nn.init.zeros_(m.weight)

       if m.bias is not None:

           nn.init.zeros_(m.bias)



在MindSpore中的调用函数为：



.. code-block:: python

   :linenos:



   # zero initialization (MindSpore)

   import numpy as np

   import mindspore.nn as nn

   import mindspore as ms

   from mindspore.common.initializer import Zero



   input_data = ms.Tensor(np.ones([1, 3, 16, 50], dtype=np.float32))



   # Ln.10: Apply zero initialization.

   net = nn.Conv2d(3, 64, 3, weight_init=Zero())

   output = net(input_data)



PyTorch中全零初始化的具体实现为：



.. code-block:: python

   :linenos:



   # implementations of zero initialization (PyTorch)

   def zeros_(tensor: Tensor) -> Tensor:

       """

       Fills the input Tensor with the scalar value `0`.



       Args:

           tensor: an n-dimensional `torch.Tensor`

       """

       # Ln. 10: Apply zero initialization.

       return _no_grad_zero_(tensor)



在MindSpore中的实现为：



.. code-block:: python

   :linenos:



   # implementations of zero initialization (MindSpore)

   @_register('zeros')

   class Zero(Initializer):

       """

       Generates an array with constant value of zero in order to initialize a tensor.

       """

       def _initialize(self, arr):

           # Ln. 9: Apply zero initialization.

           arr.fill(0)



8.2 随机初始化

==================



在PyTorch中，对参数进行服从高斯分布的参数随机初始化的调用函数为：



.. code-block:: python

   :linenos:



   # Gaussian initialization (PyTorch)

   import torch.nn as nn



   def reset_parameters(m) -> None:

       # Ln.6: Apply Gaussian initialization weight.

       nn.init.normal_(m.weight, mean=0, std=1)

       if m.bias is not None:

           nn.init.zeros_(m.bias)



MindSpore中的调用函数为：



.. code-block:: python

   :linenos:



   # Gaussian initialization (MindSpore)

   import numpy as np

   import mindspore.nn as nn

   import mindspore as ms

   from mindspore.common.initializer import Normal



   input_data = ms.Tensor(np.ones([1, 3, 16, 50], dtype=np.float32))



   # Ln.10: Apply Gaussian initialization.

   net = nn.Conv2d(3, 64, 3, weight_init=Normal(sigma=1.0, mean=0.0))

   output = net(input_data)



PyTorch中服从高斯分布的参数随机初始化的具体实现为：



.. code-block:: python

   :linenos:



   # implementations of Gaussian initialization (PyTorch)

   def normal_(tensor: Tensor, mean: float = 0., std: float = 1.) -> Tensor:

       """

       Fills the input Tensor with values drawn from the normal distribution.



       Args:

           tensor: an n-dimensional `torch.Tensor`

           mean: the mean of the normal distribution

           std: the standard deviation of the normal distribution

       """

       # Ln.12-Ln.14: Apply Gaussian initialization.

       if torch.overrides.has_torch_function_variadic(tensor):

           return torch.overrides.handle_torch_function(normal_, (tensor,), tensor=tensor, mean=mean, std=std)

       return _no_grad_normal_(tensor, mean, std)



在MindSpore中的具体实现为：



.. code-block:: python

   :linenos:



   # implementations of Gaussian initialization (MindSpore)

   @_register()

   class Normal(Initializer):

       """

       Generates an array with values sampled from Normal distribution in order to initialize a tensor.



       Args:

           sigma (float): The standard deviation of Normal distribution. Default: 0.01.

           mean (float): The mean of Normal distribution. Default: 0.0.

       """

       def __init__(self, sigma=0.01, mean=0.0):

           super(Normal, self).__init__(sigma=sigma, mean=mean)

           self.sigma = sigma

           self.mean = mean



       def _initialize(self, arr):

	   # Ln.18-Ln.19: Apply Gaussian initialization.

           data = _init_random_normal(self.mean, self.sigma, arr.shape)

           _assignment(arr, data)



但是，上述做法仍会带来一个问题，即网络输出数据分布的方差会随着输入神经元个数而改变。为解决这一问题，一般会在初始化的同时加上对方差大小的规范化。Xavier初始化就可以根据每一层输入与输出的连接数目来确定权重的初始化范围，以保持信号在前向传播过程中的稳定性，其PyTorch实现如下所示：



.. code-block:: python

   :linenos:



   # Xavier initialization (PyTorch)

   def xavier_normal_(tensor: Tensor, gain: float = 1.) -> Tensor:

       """

       Fills the input `Tensor` with values according to the method

       described in `Understanding the difficulty of training deep feedforward

       neural networks` - Glorot, X. & Bengio, Y. (2010), using a normal

       distribution.



       Also known as Glorot initialization.



       Args:

           tensor: an n-dimensional `torch.Tensor`

           gain: an optional scaling factor

       """

       # Ln.16-Ln.17: Calculate the normalized standard deviation.

       fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)

       std = gain * math.sqrt(2.0 / float(fan_in + fan_out))



       return _no_grad_normal_(tensor, 0., std)



   def _calculate_fan_in_and_fan_out(tensor):

       dimensions = tensor.dim()

       if dimensions < 2:

           raise ValueError("Fan in and fan out can not be computed for tensor with fewer than 2 dimensions")



       num_input_fmaps = tensor.size(1)

       num_output_fmaps = tensor.size(0)

       receptive_field_size = 1

       if tensor.dim() > 2:

           for s in tensor.shape[2:]:

               receptive_field_size *= s

       fan_in = num_input_fmaps * receptive_field_size

       fan_out = num_output_fmaps * receptive_field_size



       return fan_in, fan_out



在MindSpore中的实现为：



.. code-block:: python

   :linenos:



   # Xavier initialization (MindSpore)

   @_register('xavier_normal')

   class XavierNormal(Initializer):

       """

       Generates an array with values sampled from Xavier normal distribution in order to initialize a tensor.



       Args:

           gain (float): An optional scaling factor. Default: 1.

       """

       def __init__(self, gain=1):

           super().__init__(gain=gain)

           self.gain = gain



       def _initialize(self, arr):

           # Ln.16-Ln.27: Calculate the normalized standard deviation.

           fan_in, fan_out = _calculate_fan_in_and_fan_out(arr.shape)

           std = self.gain * math.sqrt(2.0 / float(fan_in + fan_out))

           data = _init_random_normal(0, std, arr.shape)



           _assignment(arr, data)



不过，细心的读者应能发现Xavier方法仍有不甚完美之处，即该方法并未考虑非线性映射函数对输入\ **s**\的影响。2015年He等人对此提出改进——将非线性映射造成的影响考虑进参数初始化中，他们提出原本Xavier方法中方差规范化的分母应为sqrt(n/2)而不是sqrt(n)。He方法在PyTorch中的调用方式为\ **kaiming_normal_()**\函数：





.. code-block:: python

   :linenos:



   # He initialization (PyTorch)

   def kaiming_normal_(

        tensor: Tensor, a: float = 0, mode: str = 'fan_in', nonlinearity: str = 'leaky_relu'

   ):

        """

	Fills the input `Tensor` with values according to the method

        described in `Delving deep into rectifiers: Surpassing human-level

        performance on ImageNet classification` - He, K. et al. (2015), using a

        normal distribution.

    

        Also known as He initialization.

    

        Args:

            tensor: an n-dimensional `torch.Tensor`

            a: the negative slope of the rectifier used after this layer (only

                used with ``'leaky_relu'``)

            mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``

                preserves the magnitude of the variance of the weights in the

                forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the

                backwards pass.

            nonlinearity: the non-linear function (`nn.functional` name),

                recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).

        """

        if 0 in tensor.shape:

            warnings.warn("Initializing zero-element tensors is a no-op")

            return tensor

	# Ln.28-Ln.32: Apply He normalization according to the `mode` and the `nonlinearity` function.

        fan = _calculate_correct_fan(tensor, mode)

        gain = calculate_gain(nonlinearity, a)

        std = gain / math.sqrt(fan)

        with torch.no_grad():

            return tensor.normal_(0, std)

        

    def calculate_gain(nonlinearity, param=None):

        """

	Return the recommended gain value for the given nonlinearity function.

    

        Args:

            nonlinearity: the non-linear function (`nn.functional` name)

            param: optional parameter for the non-linear function

        """

        linear_fns = ['linear', 'conv1d', 'conv2d', 'conv3d', 'conv_transpose1d', 'conv_transpose2d', 'conv_transpose3d']

        if nonlinearity in linear_fns or nonlinearity == 'sigmoid':

            return 1

        elif nonlinearity == 'tanh':

            return 5.0 / 3

        elif nonlinearity == 'relu':

            return math.sqrt(2.0)

        elif nonlinearity == 'leaky_relu':

            if param is None:

                negative_slope = 0.01

            elif not isinstance(param, bool) and isinstance(param, int) or isinstance(param, float):

                negative_slope = param

            else:

                raise ValueError("negative_slope {} not a valid number".format(param))

            return math.sqrt(2.0 / (1 + negative_slope ** 2))

        elif nonlinearity == 'selu':

            return 3.0 / 4  # Value found empirically (https://github.com/pytorch/pytorch/pull/50664)

        else:

            raise ValueError("Unsupported nonlinearity {}".format(nonlinearity))

    

    def _calculate_correct_fan(tensor, mode):

        mode = mode.lower()

        valid_modes = ['fan_in', 'fan_out']

        if mode not in valid_modes:

            raise ValueError("Mode {} not supported, please use one of {}".format(mode, valid_modes))

    

        fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)

        return fan_in if mode == 'fan_in' else fan_out



在MindSpore中的实现为：



.. code-block:: python

   :linenos:



   # He initialization (MindSpore)

   @_register('he_normal')

   class HeNormal(Initializer):

       """

       Generates an array with values sampled from HeKaiming Normal distribution

       in order to initialize a tensor.



       For details of HeNormal algorithm, please check `<https://arxiv.org/abs/1502.01852>`_.



       Args:

           negative_slope (int, float): The negative slope of the rectifier used after this layer

               (only used when `nonlinearity` is 'leaky_relu'). Default: 0.

           mode (str): Either 'fan_in' or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the

               variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes

               in the backwards pass. Default: 'fan_in'.

           nonlinearity (str): The non-linear function, recommended to use only with 'relu' or 'leaky_relu'.

               Default: 'leaky_relu'.

       """

       def __init__(self, negative_slope=0, mode='fan_in', nonlinearity='leaky_relu'):

           super(HeNormal, self).__init__(negative_slope=negative_slope, mode=mode, nonlinearity=nonlinearity)

           self.negative_slope = negative_slope

           self.mode = mode

           self.nonlinearity = nonlinearity



       def _initialize(self, arr):

           # Ln.27-Ln.31: Apply He normalization according to the `mode` and the `nonlinearity` function.

           fan = _calculate_correct_fan(arr.shape, self.mode)

           gain = _calculate_gain(self.nonlinearity, self.negative_slope)

           std = gain / math.sqrt(fan)

           data = _init_random_normal(0, std, arr.shape)

           _assignment(arr, data)



以上是参数初始化分布服从高斯分布的情形。 均匀分布也是一种很好的初始化分布，当参数初始化分布服从均匀分布（uniform distribution）时，由于分布性质的不同，对于均匀分布需指定其取值区间，则Xavier初始化方法和He初始化方法在PyTorch中分别修改为：



.. code-block:: python

   :linenos:



   # Xavier initialization (uniform distribution, PyTorch)

   def xavier_uniform_(tensor: Tensor, gain: float = 1.) -> Tensor:

       """

       Fills the input `Tensor` with values according to the method

       described in `Understanding the difficulty of training deep feedforward

       neural networks` - Glorot, X. & Bengio, Y. (2010), using a uniform

       distribution.



       Also known as Glorot initialization.



       Args:

           tensor: an n-dimensional `torch.Tensor`

           gain: an optional scaling factor

       """

       # Ln.16-Ln.18: Calculate uniform bounds from standard deviation.

       fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)

       std = gain * math.sqrt(2.0 / float(fan_in + fan_out))

       a = math.sqrt(3.0) * std  



       return _no_grad_uniform_(tensor, -a, a)



.. code-block:: python

   :linenos:



   # He initialization (uniform distribution, PyTorch)

   def kaiming_uniform_(

       tensor: Tensor, a: float = 0, mode: str = 'fan_in', nonlinearity: str = 'leaky_relu'

   ):

       """

       Fills the input `Tensor` with values according to the method

       described in `Delving deep into rectifiers: Surpassing human-level

       performance on ImageNet classification` - He, K. et al. (2015), using a

       uniform distribution. 



       Also known as He initialization.



       Args:

           tensor: an n-dimensional `torch.Tensor`

           a: the negative slope of the rectifier used after this layer (only

               used with ``'leaky_relu'``)

           mode: either ``'fan_in'`` (default) or ``'fan_out'``. Choosing ``'fan_in'``

               preserves the magnitude of the variance of the weights in the

               forward pass. Choosing ``'fan_out'`` preserves the magnitudes in the

               backwards pass.

           nonlinearity: the non-linear function (`nn.functional` name),

               recommended to use only with ``'relu'`` or ``'leaky_relu'`` (default).

       """

       if torch.overrides.has_torch_function_variadic(tensor):

           return torch.overrides.handle_torch_function(

               kaiming_uniform_,

               (tensor,),

               tensor=tensor,

               a=a,

               mode=mode,

               nonlinearity=nonlinearity)



       if 0 in tensor.shape:

           warnings.warn("Initializing zero-element tensors is a no-op")

           return tensor

       # Ln.37-Ln.40: Calculate uniform bounds from standard deviation.

       fan = _calculate_correct_fan(tensor, mode)

       gain = calculate_gain(nonlinearity, a)

       std = gain / math.sqrt(fan)

       bound = math.sqrt(3.0) * std

       with torch.no_grad():

           return tensor.uniform_(-bound, bound)



在MindSpore中的实现为：



.. code-block:: python

   :linenos:



   # Xavier initialization (uniform distribution, MindSpore)

   @_register('xavier_uniform')

   class XavierUniform(Initializer):

       """

       Generates an array with values sampled from Xavier uniform distribution

       in order to initialize a tensor.



       For details of XavierUniform algorithm, please check

       `<http://proceedings.mlr.press/v9/glorot10a.html>`_.



       Args:

           gain (float): An optional scaling factor. Default: 1.

       """

       def __init__(self, gain=1):

           super(XavierUniform, self).__init__(gain=gain)

           self.gain = gain



       def _initialize(self, arr):

           # Ln.20-Ln.21: Calculate uniform bounds from standard deviation.

           n_in, n_out = _calculate_fan_in_and_fan_out(arr.shape)

           boundary = self.gain * math.sqrt(6.0 / (n_in + n_out))

           data = _init_random_uniform(-boundary, boundary, arr.shape)

           _assignment(arr, data)



.. code-block:: python

   :linenos:



   # He initialization (uniform distribution, MindSpore)

   @_register('he_uniform')

   class HeUniform(Initializer):

       """

       Generates an array with values sampled from HeKaiming Uniform distribution

       in order to initialize a tensor.



       For details of HeUniform algorithm, please check

       `<https://arxiv.org/abs/1502.01852>`_.



       Args:

           negative_slope (int, float, bool): The negative slope of the rectifier used after this layer

               (only used when `nonlinearity` is 'leaky_relu'). Default: 0.

           mode (str): Either 'fan_in' or 'fan_out'. Choosing 'fan_in' preserves the magnitude of the

               variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes

               in the backwards pass. Default: 'fan_in'.

           nonlinearity (str): The non-linear function, recommended to use only with 'relu' or 'leaky_relu'.

               Default: 'leaky_relu'.

       """

       def __init__(self, negative_slope=0, mode='fan_in', nonlinearity='leaky_relu'):

           super(HeUniform, self).__init__(negative_slope=negative_slope, mode=mode, nonlinearity=nonlinearity)

           self.negative_slope = negative_slope

           self.mode = mode

           self.nonlinearity = nonlinearity



       def _initialize(self, arr):

           # Ln.28-Ln.31: Calculate uniform bounds from standard deviation.

           fan = _calculate_correct_fan(arr.shape, self.mode)

           gain = _calculate_gain(self.nonlinearity, self.negative_slope)

           std = gain / math.sqrt(fan)

           boundary = math.sqrt(3.0) * std

           data = _init_random_uniform(-boundary, boundary, arr.shape)

           _assignment(arr, data)