**************************** 第十章 目标函数 **************************** 10.1 分类任务的目标函数 ================================== ---------------------------------------------- 10.1.1 交叉熵损失函数 ---------------------------------------------- 交叉熵损失函数在PyTorch中的调用方式为： .. code-block:: python :linenos: # cross entropy loss (PyTorch) import torch import torch.nn.functional as F input = torch.randn(3, 5, requires_grad=True) target = torch.randn(3, 5).softmax(dim=1) # Ln.8: Calculate cross entropy loss. loss = F.cross_entropy(input, target) 在MindSpore中的调用方式为： .. code-block:: python :linenos: # cross entropy loss (MindSpore) import mindspore import numpy as np import mindspore.nn as nn inputs = mindspore.Tensor(np.random.randn(3, 5), mindspore.float32) target = mindspore.Tensor(np.random.randn(3, 5), mindspore.float32) # Ln.9-Ln.10: Calculate cross entropy loss. loss = nn.CrossEntropyLoss() output = loss(inputs, target) 其PyTorch具体实现细节可参考： .. code-block:: python :linenos: # implementations of cross entropy loss (PyTorch) def cross_entropy( input: Tensor, target: Tensor, weight: Optional[Tensor] = None, size_average: Optional[bool] = None, ignore_index: int = -100, reduce: Optional[bool] = None, reduction: str = "mean", label_smoothing: float = 0.0, ) -> Tensor: """ This criterion computes the cross entropy loss between input logits and target. See :class:`~torch.nn.CrossEntropyLoss` for details. Args: input (Tensor) : Predicted unnormalized logits; see Shape section below for supported shapes. target (Tensor) : Ground truth class indices or class probabilities; see Shape section below for supported shapes. weight (Tensor, optional): a manual rescaling weight given to each class. If given, has to be a Tensor of size `C` size_average (bool, optional): Deprecated (see :attr:`reduction`). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field :attr:`size_average` is set to ``False``, the losses are instead summed for each minibatch. Ignored when reduce is ``False``. Default: ``True`` ignore_index (int, optional): Specifies a target value that is ignored and does not contribute to the input gradient. When :attr:`size_average` is ``True``, the loss is averaged over non-ignored targets. Note that :attr:`ignore_index` is only applicable when the target contains class indices. Default: -100 reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the losses are averaged or summed over observations for each minibatch depending on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per batch element instead and ignores :attr:`size_average`. Default: ``True`` reduction (str, optional): Specifies the reduction to apply to the output: ``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied, ``'mean'``: the sum of the output will be divided by the number of elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average` and :attr:`reduce` are in the process of being deprecated, and in the meantime, specifying either of those two args will override :attr:`reduction`. Default: ``'mean'`` label_smoothing (float, optional): A float in [0.0, 1.0]. Specifies the amount of smoothing when computing the loss, where 0.0 means no smoothing. The targets become a mixture of the original ground truth and a uniform distribution as described in `Rethinking the Inception Architecture for Computer Vision `__. Default: 0.0. Shape: - Input: Shape (C), (N, C) or (N, C, d_1, d_2, ..., d_K) with K >= 1 in the case of K-dimensional loss. - Target: If containing class indices, shape (), (N) or (N, d_1, d_2, ..., d_K) with K >= 1 in the case of K-dimensional loss where each value should be between [0, C). If containing class probabilities, same shape as the input and each value should be between [0, 1]. """ # Ln.57-Ln.72: Calculate cross entropy loss. if has_torch_function_variadic(input, target, weight): return handle_torch_function( cross_entropy, (input, target, weight), input, target, weight=weight, size_average=size_average, ignore_index=ignore_index, reduce=reduce, reduction=reduction, label_smoothing=label_smoothing, ) if size_average is not None or reduce is not None: reduction = _Reduction.legacy_get_string(size_average, reduce) return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) 在MindSpore中的具体实现为： .. code-block:: python :linenos: # implementations of cross entropy loss (MindSpore) class CrossEntropyLoss(LossBase): """ The cross entropy loss between input and target. Args: weight (Tensor): The rescaling weight to each class. If the value is not None, the shape is (C,). The data type only supports float32 or float16. Default: None. ignore_index (int): Specifies a target value that is ignored (typically for padding value) and does not contribute to the gradient. Default: -100. reduction (str): Apply specific reduction method to the output: 'none', 'mean', or 'sum'. Default: 'mean'. label_smoothing (float): Label smoothing values, a regularization tool used to prevent the model from overfitting when calculating Loss. The value range is [0.0, 1.0]. Default value: 0.0. Inputs: - **logits** (Tensor) - Tensor of shape (C,), (N, C) or (N, C, d_1, d_2, ..., d_K), where `C = number of classes`. Data type must be float16 or float32. - **labels** (Tensor) - For class indices, tensor of shape :math:`()`, :math:`(N)` or :math:`(N, d_1, d_2, ..., d_K)` , data type must be int32. For probabilities, tensor of shape (C,), (N, C) or (N, C, d_1, d_2, ..., d_K), data type must be float16 or float32. Returns: Tensor, the computed cross entropy loss value. """ def __init__(self, weight=None, ignore_index=-100, reduction='mean', label_smoothing=0.0): super().__init__(reduction) validator.check_value_type('ignore_index', ignore_index, int, self.cls_name) validator.check_value_type('label_smoothing', label_smoothing, float, self.cls_name) validator.check_float_range(label_smoothing, 0.0, 1.0, Rel.INC_BOTH, 'label_smoothing', self.cls_name) if weight is not None: validator.check_value_type("weight", weight, [Tensor], self.cls_name) validator.check_type_name('weight', weight.dtype, [mstype.float16, mstype.float32], self.cls_name) self.weight = weight self.ignore_index = ignore_index self.reduction = reduction self.label_smoothing = label_smoothing def construct(self, logits, labels): # Ln.45-Ln.53: Calculate cross entropy loss. _check_is_tensor('logits', logits, self.cls_name) _check_is_tensor('labels', labels, self.cls_name) _check_cross_entropy_inputs(logits.shape, labels.shape, \ logits.ndim, labels.ndim, \ logits.dtype, labels.dtype, \ self.cls_name) if logits.ndim == labels.ndim and self.ignore_index > 0: _cross_entropy_ignore_index_warning(self.cls_name) return ops.cross_entropy(logits, labels, self.weight, self.ignore_index, self.reduction, self.label_smoothing) ---------------------------------------------- 10.1.2 合页损失函数 ---------------------------------------------- 合页损失函数在PyTorch中的调用方式为： .. code-block:: python :linenos: # hinge loss (PyTorch) import torch import torch.nn.functional as F input = torch.randn(3, 5, requires_grad=True) target = torch.randn(3, 5) # Ln.8: Calculate hinge loss. loss = F.hinge_embedding_loss(input, target) 在MindSpore中的调用方式为： .. code-block:: python :linenos: # hinge loss (MindSpore) import mindspore import numpy as np import mindspore.nn as nn inputs = mindspore.Tensor(np.array([0.9, -1.2, 2, 0.8, 3.9, 2, 1, 0, -1]).reshape((3, 3)), mindspore.float32) target = mindspore.Tensor(np.array([1, 1, -1, 1, -1, 1, -1, 1, 1]).reshape((3, 3)), mindspore.float32) # Ln.9-Ln.10: Calculate hinge loss. loss = nn.HingeEmbeddingLoss(reduction='mean') output = loss(inputs, target) 其PyTorch具体实现细节可参考： .. code-block:: python :linenos: # implementations of hinge loss (PyTorch) def hinge_embedding_loss( input: Tensor, target: Tensor, margin: float = 1.0, size_average: Optional[bool] = None, reduce: Optional[bool] = None, reduction: str = "mean", ) -> Tensor: """ hinge_embedding_loss(input, target, margin=1.0, size_average=None, reduce=None, reduction='mean') -> Tensor See :class:`~torch.nn.HingeEmbeddingLoss` for details. """ # Ln.16-Ln.31: Calculate hinge loss. if has_torch_function_variadic(input, target): return handle_torch_function( hinge_embedding_loss, (input, target), input, target, margin=margin, size_average=size_average, reduce=reduce, reduction=reduction, ) if size_average is not None or reduce is not None: reduction_enum = _Reduction.legacy_get_enum(size_average, reduce) else: reduction_enum = _Reduction.get_enum(reduction) return torch.hinge_embedding_loss(input, target, margin, reduction_enum) 在MindSpore中的具体实现为： .. code-block:: python :linenos: # implementations of hinge loss (MindSpore) class HingeEmbeddingLoss(LossBase): """ Hinge Embedding Loss. Compute the output according to the input elements. Measures the loss given an input tensor x and a labels tensor y (containing 1 or -1). This is usually used for measuring the similarity between two inputs. Args: margin (float): Threshold defined by Hinge Embedding Loss `margin`. Represented as `\Delta` in the formula. Default: 1.0. reduction (str): Specify the computing method to be applied to the outputs: 'none', 'mean', or 'sum'. Default: 'mean'. Inputs: - **logits** (Tensor) - Tensor of shape (*) where * means any number of dimensions. - **labels** (Tensor) - Same shape as the logits, contains -1 or 1. Returns: Tensor or Tensor scalar, the computed loss depending on `reduction`. """ def __init__(self, margin=1.0, reduction='mean'): super(HingeEmbeddingLoss, self).__init__() validator.check_value_type('margin', margin, [float], self.cls_name) validator.check_string(reduction, ['none', 'sum', 'mean'], 'reduction', self.cls_name) self.margin = margin self.reduction = reduction def construct(self, logits, labels): # Ln.30-Ln.31: Calculate hinge loss. loss = ops.hinge_embedding_loss(logits, labels, self.margin, self.reduction) return loss ---------------------------------------------- 10.1.3 坡道损失函数 ---------------------------------------------- 关于坡道损失函数在PyTorch中的具体实现细节可参考： .. code-block:: python :linenos: # implementations of ramp loss (PyTorch) import torch import torch.nn.functional as F def ramp_loss(pred, label, s): # the value in ``label`` is expected to be 0 or 1 # Ln.8-Ln.11: Calculate ramp loss. label = 2 * label - torch.ones(label.size()) h = pred * label loss = (F.relu(1 - h) - F.relu(s - h)).sum(axis=1) return loss.mean() 在MindSpore中的实现为： .. code-block:: python :linenos: # implementations of ramp loss (MindSpore) import mindspore as ms import mindspore.ops as ops def ramp_loss(pred, label, s): ones = ops.Ones() relu = ops.ReLU() # Ln.10-Ln.13: Calculate ramp loss. label = 2 * label - ones(label.shape, ms.float32) h = pred * label loss = (relu(1 - h) - relu(s - h)).sum(axis=1) return loss.mean() ---------------------------------------------- 10.1.4 大间隔交叉熵损失函数 ---------------------------------------------- 关于大间隔交叉熵函数在PyTorch中的具体实现细节可参考： .. code-block:: python :linenos: # implementations of large margin softmax loss (PyTorch) import math import numpy as np import torch import torch.nn.functional as F import torch.nn as nn from scipy.special import binom class LSoftmaxLinear(nn.Linear): def __init__(self, input_features, output_features, margin, device): super().__init__() self.input_dim = input_features # number of input feature i.e. output of the last fc layer self.output_dim = output_features # number of output = class numbers self.margin = margin # m self.beta = 100 self.beta_min = 0 self.scale = 0.99 # Ln.20-Ln.26: Initialize L-Softmax parameters. self.weight = nn.Parameter(torch.FloatTensor(input_features, output_features)) self.divisor = math.pi / self.margin # pi/m self.C_m_2n = torch.Tensor(binom(margin, range(0, margin + 1, 2))) # C_m{2n} self.cos_powers = torch.Tensor(range(self.margin, -1, -2)) # m - 2n self.sin2_powers = torch.Tensor(range(len(self.cos_powers))) # n self.signs = torch.ones(margin // 2 + 1).to(device) self.signs[1::2] = -1 # 1, -1, 1, -1, ... def calculate_cos_m_theta(self, cos_theta): # Ln.30-Ln.34: Calculate cos(m*theta). sin2_theta = 1 - cos_theta**2 cos_terms = cos_theta ** self.cos_powers # cos^{m - 2n} sin2_terms = (sin2_theta ** self.sin2_powers) # sin2^{n} cos_m_theta = (self.signs * self.C_m_2n * cos_terms * sin2_terms).sum(1) # -1^{n} * C_m{2n} * cos^{m - 2n} * sin2^{n} return cos_m_theta def find_k(self, cos): # Ln.38-Ln.42: Find k. eps = 1e-7 cos = torch.clamp(cos, -1 + eps, 1 - eps) acos = cos.acos() k = (acos / self.divisor).floor().detach() return k def forward(self, input, target=None): if self.training: assert target is not None x, w = input, self.weight beta = max(self.beta, self.beta_min) logit = x.mm(w) indexes = range(logit.size(0)) logit_target = logit[indexes, target] # Ln.54-Ln.56: Calculate cos(theta) = w * x / ||w||*||x|| w_target_norm = w[:, target].norm(p=2, dim=0) x_norm = x.norm(p=2, dim=1) cos_theta_target = logit_target / (w_target_norm * x_norm + 1e-10) # Ln.59: Calculate cos(m*theta) cos_m_theta_target = self.calculate_cos_m_theta(cos_theta_target) # Ln.62: Find k k = self.find_k(cos_theta_target) logit_target_updated = (w_target_norm * x_norm * (((-1) ** k * cos_m_theta_target) - 2 * k)) logit_target_updated_beta = (logit_target_updated + beta * logit[indexes, target]) / (1 + beta) logit[indexes, target] = logit_target_updated_beta self.beta *= self.scale return logit else: assert target is None return input.mm(self.weight) 在MindSpore中的实现为： .. code-block:: python :linenos: # implementations of large margin softmax loss (MindSpore) import math import numpy as np import mindspore import mindspore.nn as nn from mindspore import Tensor, Parameter import mindspore.ops as ops from scipy.special import binom class LSoftmaxLinear(nn.Cell): def __init__(self, in_dim, out_dim, margin): super().__init__() self.weight = Parameter(Tensor(np.random.normal(0, 0.01, (out_dim, in_dim)), mindspore.float32)) self.margin = margin # Ln.17-Ln.24: Initialize L-Softmax parameters. self.beta = Parameter(Tensor([100.]), requires_grad=False) self.beta_min = Tensor(0) self.scale = 0.99 self.C_m_2n = Tensor(binom(margin, range(0, margin + 1, 2))) self.cos_powers = Tensor(np.arange(margin, -1, -2)) self.sin2_powers = Tensor(np.arange(0, margin // 2 + 1)) self.signs = Tensor((-np.ones(margin // 2 + 1)) ** self.sin2_powers) def _calc_cos_m_theta(self, cos_theta): # Ln.28-Ln.32: Calculate cos(m*theta). sin2_theta = 1 - cos_theta ** 2 cos_terms = cos_theta.reshape(-1, 1) ** self.cos_powers sin2_terms = sin2_theta.reshape(-1, 1) ** self.sin2_powers cos_m_theta = (self.signs * self.C_m_2n * cos_terms * sin2_terms).sum(1) return cos_m_theta def construct(self, inp, target): logits = ops.MatMul()(inp, ops.Transpose()(self.weight, (1, 0))) indices = ops.Tensor(np.arange(0, target.size()[0])) target_logit = logits[indices.astype(int), target] w_norm = ops.Sqrt()(ops.ReduceSum()(self.weight[:, target] ** 2)) inp_norm = ops.Sqrt()(ops.ReduceSum()(inp ** 2)) target_cos_theta = target_logit / (w_norm * inp_norm + 1e-6) # Ln.59:Calculate cos(m*theta) target_cos_m_theta = self._calc_cos_m_theta(target_cos_theta) # Ln.62:Find k k = (ops.Acos()(target_cos_theta) / math.pi * self.margin).floor().detach() updated_target_logit = w_norm * inp_norm * ((-1) ** k * target_cos_m_theta - 2 * k) beta = max(self.beta.asnumpy(), self.beta_min.asnumpy()) updated_target_logit_beta = (updated_target_logit + beta * target_logit) / (1 + beta) logits[indices.astype(int), target] = updated_target_logit.float() self.beta.data = Tensor(self.scale * beta) return logits ---------------------------------------------- 10.1.5 中心损失函数 ---------------------------------------------- 关于中心损失函数在PyTorch中的具体实现细节可参考： .. code-block:: python :linenos: # implementations of center loss (PyTorch) import torch import torch.nn as nn class CenterLoss(nn.Module): def __init__(self, num_classes=10, feat_dim=2, use_gpu=True): super(CenterLoss, self).__init__() self.num_classes = num_classes self.feat_dim = feat_dim self.use_gpu = use_gpu if self.use_gpu: self.centers = nn.Parameter(torch.randn(self.num_classes, self.feat_dim).cuda()) else: self.centers = nn.Parameter(torch.randn(self.num_classes, self.feat_dim)) def forward(self, x, labels): # Ln.19-Ln.32: Calculate center loss by the Euclidean distance matrix between the centers and sample embeddings. batch_size = x.size(0) distmat = torch.pow(x, 2).sum(dim=1, keepdim=True).expand(batch_size, self.num_classes) + \ torch.pow(self.centers, 2).sum(dim=1, keepdim=True).expand(self.num_classes, batch_size).t() distmat.addmm_(x, self.centers.t(), beta=1, alpha=-2) classes = torch.arange(self.num_classes).long() if self.use_gpu: classes = classes.cuda() labels = labels.unsqueeze(1).expand(batch_size, self.num_classes) mask = labels.eq(classes.expand(batch_size, self.num_classes)) dist = distmat * mask.float() loss = dist.clamp(min=1e-12, max=1e+12).sum() / batch_size return loss 中心损失函数在MindSpore中的实现为： .. code-block:: python :linenos: # implementations of center loss (MindSpore) import mindspore import mindspore.nn as nn from mindspore import Tensor, Parameter import mindspore.ops as ops class CenterLoss(nn.Cell): def __init__(self, num_classes=10, feat_dim=2, use_gpu=True): super(CenterLoss, self).__init__() self.num_classes = num_classes self.feat_dim = feat_dim self.use_gpu = use_gpu self.centers = Parameter(Tensor(np.random.randn(self.num_classes, self.feat_dim), mindspore.float32)) def construct(self, x, labels): # Ln.18-Ln.30: Calculate center loss by the Euclidean distance matrix between the centers and sample embeddings. batch_size = x.shape[0] distmat = ops.Reshape()(ops.ReduceSum()(ops.Square()(x), 1), (batch_size, 1)) + \ ops.Reshape()(ops.ReduceSum()(ops.Square()(self.centers), 1), (self.num_classes, 1)) distmat = ops.MatMul()(x, ops.Transpose()(self.centers, (1, 0)), b=distmat, alpha=-2) classes = Tensor(np.arange(self.num_classes).astype(np.int32)) labels = ops.Reshape()(labels, (batch_size, 1)) mask = ops.Equal()(labels, classes) dist = distmat * ops.Cast()(mask, mindspore.float32) loss = ops.ReduceSum()(ops.ClipByValue()(dist, 1e-12, 1e+12)) / batch_size return loss 10.2 回归任务的目标函数 ================================= ---------------------------------------------- 10.2.1 l1损失函数 ---------------------------------------------- l1损失函数在PyTorch中的调用方式为： .. code-block:: python :linenos: # l1 loss (PyTorch) import torch import torch.nn.functional as F input = torch.FloatTensor([3, 3, 3, 3]) target = torch.tensor([2, 8, 6, 1]) # Ln.8: Calculate l1 loss. loss = F.l1_loss(input, target) print(loss) 其具体实现细节可参考： .. code-block:: python :linenos: # implementations of l1 loss (PyTorch) def l1_loss( input: Tensor, target: Tensor, size_average: Optional[bool] = None, reduce: Optional[bool] = None, reduction: str = "mean", ) -> Tensor: """ l1_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor Function that takes the mean element-wise absolute value difference. See :class:`~torch.nn.L1Loss` for details. """ # Ln.17-Ln.32: Calculate l1 loss. if has_torch_function_variadic(input, target): return handle_torch_function( l1_loss, (input, target), input, target, size_average=size_average, reduce=reduce, reduction=reduction ) if not (target.size() == input.size()): warnings.warn( "Using a target size ({}) that is different to the input size ({}). " "This will likely lead to incorrect results due to broadcasting. " "Please ensure they have the same size.".format(target.size(), input.size()), stacklevel=2, ) if size_average is not None or reduce is not None: reduction = _Reduction.legacy_get_string(size_average, reduce) expanded_input, expanded_target = torch.broadcast_tensors(input, target) return torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction)) 在MindSpore中的调用方式为： .. code-block:: python :linenos: # l1 loss (MindSpore) from mindspore import Tensor, ops from mindspore import dtype as mstype x = ms.Tensor([[1, 2, 3], [4, 5, 6]], mstype.float32) target = ms.Tensor([[6, 5, 4], [3, 2, 1]], mstype.float32) # Ln.8: Calculate l1 loss. output = ops.l1_loss(x, target, reduction="mean") print(output) 其在MindSpore中具体实现细节为： .. code-block:: python :linenos: # implementations of l1 loss (MindSpore) def l1_loss(input, target, reduction='mean'): """ Calculate the mean absolute error between the `input` value and the `target` value. Assuming that the `x` and `y` are 1-D Tensor, length `N`, `reduction` is set to ``'none'``, then calculate the loss of `x` and `y` without dimensionality reduction. Args: input (Tensor): Predicted value, Tensor of any dimension. target (Tensor): Target value, usually has the same shape as the `input`. If `input` and `target` have different shape, make sure they can broadcast to each other. reduction (str, optional): Apply specific reduction method to the output: ``'none'`` , ``'mean'`` , ``'sum'`` . Default: ``'mean'`` . - ``'none'``: no reduction will be applied. - ``'mean'``: compute and return the mean of elements in the output. - ``'sum'``: the output elements will be summed. Returns: Tensor or Scalar, if `reduction` is ``'none'``, return a Tensor with same shape and dtype as `input`. Otherwise, a scalar value will be returned. """ # Ln.24-Ln.29: Calculate l1 loss. _check_is_tensor('input', input, 'l1_loss') _check_is_tensor('target', target, 'l1_loss') if reduction not in ('mean', 'sum', 'none'): raise ValueError(f"For l1_loss, the 'reduction' must be in ['mean', 'sum', 'none'], but got {reduction}.") loss = _get_cache_prim(ops.Abs)()(input - target) return _get_loss(loss, reduction, 'l1_loss') ---------------------------------------------- 10.2.2 l2损失函数 ---------------------------------------------- l2损失函数在PyTorch中的具体实现细节可参考： .. code-block:: python :linenos: # implementations of l2 loss (PyTorch) def mse_loss( input: Tensor, target: Tensor, size_average: Optional[bool] = None, reduce: Optional[bool] = None, reduction: str = "mean", ) -> Tensor: """ mse_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor Measures the element-wise mean squared error. See :class:`~torch.nn.MSELoss` for details. """ # Ln.17-Ln.32: Calculate l2 loss. if has_torch_function_variadic(input, target): return handle_torch_function( mse_loss, (input, target), input, target, size_average=size_average, reduce=reduce, reduction=reduction ) if not (target.size() == input.size()): warnings.warn( "Using a target size ({}) that is different to the input size ({}). " "This will likely lead to incorrect results due to broadcasting. " "Please ensure they have the same size.".format(target.size(), input.size()), stacklevel=2, ) if size_average is not None or reduce is not None: reduction = _Reduction.legacy_get_string(size_average, reduce) expanded_input, expanded_target = torch.broadcast_tensors(input, target) return torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction)) l2损失函数在MindSpore中的实现为： .. code-block:: python :linenos: # implementations of l2 loss (MindSpore) def mse_loss(input, target, reduction='mean'): """ Calculates the mean squared error between the predicted value and the label value. For detailed information, please refer to :class:`mindspore.nn.MSELoss`. Args: input (Tensor): Tensor of any dimension. target (Tensor): The input label. Tensor of any dimension, same shape as the `input` in common cases. However, it supports that the shape of `input` is different from the shape of `target` and they should be broadcasted to each other. reduction (str, optional): Apply specific reduction method to the output: ``'none'`` , ``'mean'`` , ``'sum'`` . Default: ``'mean'`` . - ``'none'``: no reduction will be applied. - ``'mean'``: compute and return the mean of elements in the output. - ``'sum'``: the output elements will be summed. Returns: Tensor, loss of type float, the shape is zero if `reduction` is ``'mean'`` or ``'sum'`` , while the shape of output is the broadcasted shape if `reduction` is ``'none'`` . """ # Ln.25-Ln.53: Calculate l2 loss. if not isinstance(input, (Tensor, Tensor_)): raise TypeError("For ops.mse_loss, the `input` must be tensor") if not isinstance(target, (Tensor, Tensor_)): raise TypeError("For ops.mse_loss, the `target` must be tensor") if reduction not in ['mean', 'none', 'sum']: raise ValueError("For ops.mse_loss, `reduction` value should be either 'mean', 'none' or 'sum'.") x = _get_cache_prim(ops.Square)()(input - target) float_type = (mstype.float16, mstype.float32, mstype.float64) if x.dtype not in float_type: input_dtype = mstype.float32 else: input_dtype = x.dtype x = _get_cache_prim(ops.Cast)()(x, mstype.float32) average_flag = True reduce_flag = True if reduction == 'sum': average_flag = False if reduction == 'none': reduce_flag = False if reduce_flag and average_flag: x = _get_cache_prim(ops.ReduceMean)()(x, _get_axis(x)) if reduce_flag and not average_flag: x = _get_cache_prim(ops.ReduceSum)()(x, _get_axis(x)) return _get_cache_prim(ops.Cast)()(x, input_dtype) ---------------------------------------------- 10.2.3 Tukey's biweight损失函数 ---------------------------------------------- Tukey's biweight损失函数在PyTorch中的具体实现细节可参考： .. code-block:: python :linenos: # implementations of Tukey's biweight loss (PyTorch) import torch import torch.nn.functional as F def tukey_biweight_loss(pred, target, c=4.6851): # Ln.7-Ln.12: Calculate Tukey's biweight loss. l = torch.abs(pred - target) loss1 = (1 - (1 - (l / c) ** 2) ** 3) * (c ** 2) / 6 # upper half of the piecewise function loss2 = torch.tensor((c ** 2) / 6.).repeat(loss1.size()) # lower half of the piecewise function mask = l < c # condition loss = torch.where(mask, loss1 / loss1.size()[0], loss2) # the Tukey's biweight loss return loss.mean() 在MindSpore中的实现为： .. code-block:: python :linenos: # implementations of Tukey's biweight loss (MindSpore) import mindspore import mindspore.ops as ops def tukey_biweight_loss(pred, target, c=4.6851): # Ln.7-Ln.12: Calculate Tukey's biweight loss. l = ops.abs(pred - target) loss1 = (1 - (1 - (l / c) ** 2) ** 3) * (c ** 2) / 6 # upper half of the piecewise function loss2 = ops.Fill()(loss1.shape(), (c ** 2) / 6.) # lower half of the piecewise function mask = l < c # condition loss = ops.Select() (mask, loss1 / loss1.size()[0], loss2) # the Tukey's biweight loss return ops.ReduceMean(keep_dims=False)(loss)