第十章 目标函数
10.1 分类任务的目标函数
10.1.1 交叉熵损失函数
交叉熵损失函数在PyTorch中的调用方式为：
1# cross entropy loss (PyTorch)
2import torch
3import torch.nn.functional as F
4
5input = torch.randn(3, 5, requires_grad=True)
6target = torch.randn(3, 5).softmax(dim=1)
7# Ln.8: Calculate cross entropy loss.
8loss = F.cross_entropy(input, target)
在MindSpore中的调用方式为：
1# cross entropy loss (MindSpore)
2import mindspore
3import numpy as np
4import mindspore.nn as nn
5
6inputs = mindspore.Tensor(np.random.randn(3, 5), mindspore.float32)
7target = mindspore.Tensor(np.random.randn(3, 5), mindspore.float32)
8# Ln.9-Ln.10: Calculate cross entropy loss.
9loss = nn.CrossEntropyLoss()
10output = loss(inputs, target)
其PyTorch具体实现细节可参考：
1# implementations of cross entropy loss (PyTorch)
2def cross_entropy(
3 input: Tensor,
4 target: Tensor,
5 weight: Optional[Tensor] = None,
6 size_average: Optional[bool] = None,
7 ignore_index: int = -100,
8 reduce: Optional[bool] = None,
9 reduction: str = "mean",
10 label_smoothing: float = 0.0,
11) -> Tensor:
12 """
13 This criterion computes the cross entropy loss between input logits and target.
14
15 See :class:`~torch.nn.CrossEntropyLoss` for details.
16
17 Args:
18 input (Tensor) : Predicted unnormalized logits;
19 see Shape section below for supported shapes.
20 target (Tensor) : Ground truth class indices or class probabilities;
21 see Shape section below for supported shapes.
22 weight (Tensor, optional): a manual rescaling weight given to each
23 class. If given, has to be a Tensor of size `C`
24 size_average (bool, optional): Deprecated (see :attr:`reduction`). By default,
25 the losses are averaged over each loss element in the batch. Note that for
26 some losses, there multiple elements per sample. If the field :attr:`size_average`
27 is set to ``False``, the losses are instead summed for each minibatch. Ignored
28 when reduce is ``False``. Default: ``True``
29 ignore_index (int, optional): Specifies a target value that is ignored
30 and does not contribute to the input gradient. When :attr:`size_average` is
31 ``True``, the loss is averaged over non-ignored targets. Note that
32 :attr:`ignore_index` is only applicable when the target contains class indices.
33 Default: -100
34 reduce (bool, optional): Deprecated (see :attr:`reduction`). By default, the
35 losses are averaged or summed over observations for each minibatch depending
36 on :attr:`size_average`. When :attr:`reduce` is ``False``, returns a loss per
37 batch element instead and ignores :attr:`size_average`. Default: ``True``
38 reduction (str, optional): Specifies the reduction to apply to the output:
39 ``'none'`` | ``'mean'`` | ``'sum'``. ``'none'``: no reduction will be applied,
40 ``'mean'``: the sum of the output will be divided by the number of
41 elements in the output, ``'sum'``: the output will be summed. Note: :attr:`size_average`
42 and :attr:`reduce` are in the process of being deprecated, and in the meantime,
43 specifying either of those two args will override :attr:`reduction`. Default: ``'mean'``
44 label_smoothing (float, optional): A float in [0.0, 1.0]. Specifies the amount
45 of smoothing when computing the loss, where 0.0 means no smoothing. The targets
46 become a mixture of the original ground truth and a uniform distribution as described in
47 `Rethinking the Inception Architecture for Computer Vision <https://arxiv.org/abs/1512.00567>`__. Default: 0.0.
48
49 Shape:
50 - Input: Shape (C), (N, C) or (N, C, d_1, d_2, ..., d_K) with K >= 1
51 in the case of K-dimensional loss.
52 - Target: If containing class indices, shape (), (N) or (N, d_1, d_2, ..., d_K) with
53 K >= 1 in the case of K-dimensional loss where each value should be between [0, C).
54 If containing class probabilities, same shape as the input and each value should be between [0, 1].
55 """
56 # Ln.57-Ln.72: Calculate cross entropy loss.
57 if has_torch_function_variadic(input, target, weight):
58 return handle_torch_function(
59 cross_entropy,
60 (input, target, weight),
61 input,
62 target,
63 weight=weight,
64 size_average=size_average,
65 ignore_index=ignore_index,
66 reduce=reduce,
67 reduction=reduction,
68 label_smoothing=label_smoothing,
69 )
70 if size_average is not None or reduce is not None:
71 reduction = _Reduction.legacy_get_string(size_average, reduce)
72 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
在MindSpore中的具体实现为：
1# implementations of cross entropy loss (MindSpore)
2class CrossEntropyLoss(LossBase):
3 """
4 The cross entropy loss between input and target.
5
6 Args:
7 weight (Tensor): The rescaling weight to each class. If the value is not None, the shape is (C,).
8 The data type only supports float32 or float16. Default: None.
9 ignore_index (int): Specifies a target value that is ignored (typically for padding value)
10 and does not contribute to the gradient. Default: -100.
11 reduction (str): Apply specific reduction method to the output: 'none', 'mean', or 'sum'.
12 Default: 'mean'.
13 label_smoothing (float): Label smoothing values, a regularization tool used to prevent the model
14 from overfitting when calculating Loss. The value range is [0.0, 1.0]. Default value: 0.0.
15
16 Inputs:
17 - **logits** (Tensor) - Tensor of shape (C,), (N, C) or (N, C, d_1, d_2, ..., d_K),
18 where `C = number of classes`. Data type must be float16 or float32.
19 - **labels** (Tensor) - For class indices, tensor of shape :math:`()`, :math:`(N)` or
20 :math:`(N, d_1, d_2, ..., d_K)` , data type must be int32.
21 For probabilities, tensor of shape (C,), (N, C) or (N, C, d_1, d_2, ..., d_K),
22 data type must be float16 or float32.
23
24 Returns:
25 Tensor, the computed cross entropy loss value.
26 """
27 def __init__(self, weight=None, ignore_index=-100, reduction='mean',
28 label_smoothing=0.0):
29 super().__init__(reduction)
30 validator.check_value_type('ignore_index', ignore_index, int, self.cls_name)
31 validator.check_value_type('label_smoothing', label_smoothing, float, self.cls_name)
32 validator.check_float_range(label_smoothing, 0.0, 1.0, Rel.INC_BOTH, 'label_smoothing', self.cls_name)
33
34 if weight is not None:
35 validator.check_value_type("weight", weight, [Tensor], self.cls_name)
36 validator.check_type_name('weight', weight.dtype, [mstype.float16, mstype.float32], self.cls_name)
37
38 self.weight = weight
39 self.ignore_index = ignore_index
40 self.reduction = reduction
41 self.label_smoothing = label_smoothing
42
43 def construct(self, logits, labels):
44 # Ln.45-Ln.53: Calculate cross entropy loss.
45 _check_is_tensor('logits', logits, self.cls_name)
46 _check_is_tensor('labels', labels, self.cls_name)
47 _check_cross_entropy_inputs(logits.shape, labels.shape, \
48 logits.ndim, labels.ndim, \
49 logits.dtype, labels.dtype, \
50 self.cls_name)
51 if logits.ndim == labels.ndim and self.ignore_index > 0:
52 _cross_entropy_ignore_index_warning(self.cls_name)
53 return ops.cross_entropy(logits, labels, self.weight, self.ignore_index, self.reduction, self.label_smoothing)
10.1.2 合页损失函数
合页损失函数在PyTorch中的调用方式为：
1# hinge loss (PyTorch)
2import torch
3import torch.nn.functional as F
4
5input = torch.randn(3, 5, requires_grad=True)
6target = torch.randn(3, 5)
7# Ln.8: Calculate hinge loss.
8loss = F.hinge_embedding_loss(input, target)
在MindSpore中的调用方式为：
1# hinge loss (MindSpore)
2import mindspore
3import numpy as np
4import mindspore.nn as nn
5
6inputs = mindspore.Tensor(np.array([0.9, -1.2, 2, 0.8, 3.9, 2, 1, 0, -1]).reshape((3, 3)), mindspore.float32)
7target = mindspore.Tensor(np.array([1, 1, -1, 1, -1, 1, -1, 1, 1]).reshape((3, 3)), mindspore.float32)
8# Ln.9-Ln.10: Calculate hinge loss.
9loss = nn.HingeEmbeddingLoss(reduction='mean')
10output = loss(inputs, target)
其PyTorch具体实现细节可参考：
1# implementations of hinge loss (PyTorch)
2def hinge_embedding_loss(
3 input: Tensor,
4 target: Tensor,
5 margin: float = 1.0,
6 size_average: Optional[bool] = None,
7 reduce: Optional[bool] = None,
8 reduction: str = "mean",
9) -> Tensor:
10 """
11 hinge_embedding_loss(input, target, margin=1.0, size_average=None, reduce=None, reduction='mean') -> Tensor
12
13 See :class:`~torch.nn.HingeEmbeddingLoss` for details.
14 """
15 # Ln.16-Ln.31: Calculate hinge loss.
16 if has_torch_function_variadic(input, target):
17 return handle_torch_function(
18 hinge_embedding_loss,
19 (input, target),
20 input,
21 target,
22 margin=margin,
23 size_average=size_average,
24 reduce=reduce,
25 reduction=reduction,
26 )
27 if size_average is not None or reduce is not None:
28 reduction_enum = _Reduction.legacy_get_enum(size_average, reduce)
29 else:
30 reduction_enum = _Reduction.get_enum(reduction)
31 return torch.hinge_embedding_loss(input, target, margin, reduction_enum)
在MindSpore中的具体实现为：
1# implementations of hinge loss (MindSpore)
2class HingeEmbeddingLoss(LossBase):
3 """
4 Hinge Embedding Loss. Compute the output according to the input elements. Measures the loss given an input tensor x
5 and a labels tensor y (containing 1 or -1).
6 This is usually used for measuring the similarity between two inputs.
7
8 Args:
9 margin (float): Threshold defined by Hinge Embedding Loss `margin`.
10 Represented as `\Delta` in the formula. Default: 1.0.
11 reduction (str): Specify the computing method to be applied to the outputs: 'none', 'mean', or 'sum'.
12 Default: 'mean'.
13
14 Inputs:
15 - **logits** (Tensor) - Tensor of shape (*) where * means any number of dimensions.
16 - **labels** (Tensor) - Same shape as the logits, contains -1 or 1.
17
18 Returns:
19 Tensor or Tensor scalar, the computed loss depending on `reduction`.
20 """
21 def __init__(self, margin=1.0, reduction='mean'):
22 super(HingeEmbeddingLoss, self).__init__()
23 validator.check_value_type('margin', margin, [float], self.cls_name)
24 validator.check_string(reduction, ['none', 'sum', 'mean'], 'reduction', self.cls_name)
25 self.margin = margin
26 self.reduction = reduction
27
28 def construct(self, logits, labels):
29 # Ln.30-Ln.31: Calculate hinge loss.
30 loss = ops.hinge_embedding_loss(logits, labels, self.margin, self.reduction)
31 return loss
10.1.3 坡道损失函数
关于坡道损失函数在PyTorch中的具体实现细节可参考：
1# implementations of ramp loss (PyTorch)
2import torch
3import torch.nn.functional as F
4
5def ramp_loss(pred, label, s):
6 # the value in ``label`` is expected to be 0 or 1
7 # Ln.8-Ln.11: Calculate ramp loss.
8 label = 2 * label - torch.ones(label.size())
9 h = pred * label
10 loss = (F.relu(1 - h) - F.relu(s - h)).sum(axis=1)
11 return loss.mean()
在MindSpore中的实现为：
1# implementations of ramp loss (MindSpore)
2import mindspore as ms
3import mindspore.ops as ops
4
5def ramp_loss(pred, label, s):
6 ones = ops.Ones()
7 relu = ops.ReLU()
8
9 # Ln.10-Ln.13: Calculate ramp loss.
10 label = 2 * label - ones(label.shape, ms.float32)
11 h = pred * label
12 loss = (relu(1 - h) - relu(s - h)).sum(axis=1)
13 return loss.mean()
10.1.4 大间隔交叉熵损失函数
关于大间隔交叉熵函数在PyTorch中的具体实现细节可参考：
1# implementations of large margin softmax loss (PyTorch)
2import math
3import numpy as np
4import torch
5import torch.nn.functional as F
6import torch.nn as nn
7from scipy.special import binom
8
9class LSoftmaxLinear(nn.Linear):
10 def __init__(self, input_features, output_features, margin, device):
11 super().__init__()
12 self.input_dim = input_features # number of input feature i.e. output of the last fc layer
13 self.output_dim = output_features # number of output = class numbers
14 self.margin = margin # m
15 self.beta = 100
16 self.beta_min = 0
17 self.scale = 0.99
18
19 # Ln.20-Ln.26: Initialize L-Softmax parameters.
20 self.weight = nn.Parameter(torch.FloatTensor(input_features, output_features))
21 self.divisor = math.pi / self.margin # pi/m
22 self.C_m_2n = torch.Tensor(binom(margin, range(0, margin + 1, 2))) # C_m{2n}
23 self.cos_powers = torch.Tensor(range(self.margin, -1, -2)) # m - 2n
24 self.sin2_powers = torch.Tensor(range(len(self.cos_powers))) # n
25 self.signs = torch.ones(margin // 2 + 1).to(device)
26 self.signs[1::2] = -1 # 1, -1, 1, -1, ...
27
28 def calculate_cos_m_theta(self, cos_theta):
29 # Ln.30-Ln.34: Calculate cos(m*theta).
30 sin2_theta = 1 - cos_theta**2
31 cos_terms = cos_theta ** self.cos_powers # cos^{m - 2n}
32 sin2_terms = (sin2_theta ** self.sin2_powers) # sin2^{n}
33 cos_m_theta = (self.signs * self.C_m_2n * cos_terms * sin2_terms).sum(1) # -1^{n} * C_m{2n} * cos^{m - 2n} * sin2^{n}
34 return cos_m_theta
35
36 def find_k(self, cos):
37 # Ln.38-Ln.42: Find k.
38 eps = 1e-7
39 cos = torch.clamp(cos, -1 + eps, 1 - eps)
40 acos = cos.acos()
41 k = (acos / self.divisor).floor().detach()
42 return k
43
44 def forward(self, input, target=None):
45 if self.training:
46 assert target is not None
47 x, w = input, self.weight
48 beta = max(self.beta, self.beta_min)
49 logit = x.mm(w)
50 indexes = range(logit.size(0))
51 logit_target = logit[indexes, target]
52
53 # Ln.54-Ln.56: Calculate cos(theta) = w * x / ||w||*||x||
54 w_target_norm = w[:, target].norm(p=2, dim=0)
55 x_norm = x.norm(p=2, dim=1)
56 cos_theta_target = logit_target / (w_target_norm * x_norm + 1e-10)
57
58 # Ln.59: Calculate cos(m*theta)
59 cos_m_theta_target = self.calculate_cos_m_theta(cos_theta_target)
60
61 # Ln.62: Find k
62 k = self.find_k(cos_theta_target)
63
64 logit_target_updated = (w_target_norm * x_norm * (((-1) ** k * cos_m_theta_target) - 2 * k))
65 logit_target_updated_beta = (logit_target_updated + beta * logit[indexes, target]) / (1 + beta)
66
67 logit[indexes, target] = logit_target_updated_beta
68 self.beta *= self.scale
69 return logit
70 else:
71 assert target is None
72 return input.mm(self.weight)
在MindSpore中的实现为：
1# implementations of large margin softmax loss (MindSpore)
2import math
3import numpy as np
4import mindspore
5import mindspore.nn as nn
6from mindspore import Tensor, Parameter
7import mindspore.ops as ops
8from scipy.special import binom
9
10class LSoftmaxLinear(nn.Cell):
11 def __init__(self, in_dim, out_dim, margin):
12 super().__init__()
13 self.weight = Parameter(Tensor(np.random.normal(0, 0.01, (out_dim, in_dim)), mindspore.float32))
14 self.margin = margin
15
16 # Ln.17-Ln.24: Initialize L-Softmax parameters.
17 self.beta = Parameter(Tensor([100.]), requires_grad=False)
18 self.beta_min = Tensor(0)
19 self.scale = 0.99
20
21 self.C_m_2n = Tensor(binom(margin, range(0, margin + 1, 2)))
22 self.cos_powers = Tensor(np.arange(margin, -1, -2))
23 self.sin2_powers = Tensor(np.arange(0, margin // 2 + 1))
24 self.signs = Tensor((-np.ones(margin // 2 + 1)) ** self.sin2_powers)
25
26 def _calc_cos_m_theta(self, cos_theta):
27 # Ln.28-Ln.32: Calculate cos(m*theta).
28 sin2_theta = 1 - cos_theta ** 2
29 cos_terms = cos_theta.reshape(-1, 1) ** self.cos_powers
30 sin2_terms = sin2_theta.reshape(-1, 1) ** self.sin2_powers
31 cos_m_theta = (self.signs * self.C_m_2n * cos_terms * sin2_terms).sum(1)
32 return cos_m_theta
33
34 def construct(self, inp, target):
35 logits = ops.MatMul()(inp, ops.Transpose()(self.weight, (1, 0)))
36 indices = ops.Tensor(np.arange(0, target.size()[0]))
37 target_logit = logits[indices.astype(int), target]
38 w_norm = ops.Sqrt()(ops.ReduceSum()(self.weight[:, target] ** 2))
39 inp_norm = ops.Sqrt()(ops.ReduceSum()(inp ** 2))
40 target_cos_theta = target_logit / (w_norm * inp_norm + 1e-6)
41
42 # Ln.59:Calculate cos(m*theta)
43 target_cos_m_theta = self._calc_cos_m_theta(target_cos_theta)
44
45 # Ln.62:Find k
46 k = (ops.Acos()(target_cos_theta) / math.pi * self.margin).floor().detach()
47
48 updated_target_logit = w_norm * inp_norm * ((-1) ** k * target_cos_m_theta - 2 * k)
49 beta = max(self.beta.asnumpy(), self.beta_min.asnumpy())
50 updated_target_logit_beta = (updated_target_logit + beta * target_logit) / (1 + beta)
51 logits[indices.astype(int), target] = updated_target_logit.float()
52 self.beta.data = Tensor(self.scale * beta)
53
54 return logits
10.1.5 中心损失函数
关于中心损失函数在PyTorch中的具体实现细节可参考：
1# implementations of center loss (PyTorch)
2import torch
3import torch.nn as nn
4
5class CenterLoss(nn.Module):
6 def __init__(self, num_classes=10, feat_dim=2, use_gpu=True):
7 super(CenterLoss, self).__init__()
8 self.num_classes = num_classes
9 self.feat_dim = feat_dim
10 self.use_gpu = use_gpu
11
12 if self.use_gpu:
13 self.centers = nn.Parameter(torch.randn(self.num_classes, self.feat_dim).cuda())
14 else:
15 self.centers = nn.Parameter(torch.randn(self.num_classes, self.feat_dim))
16
17 def forward(self, x, labels):
18 # Ln.19-Ln.32: Calculate center loss by the Euclidean distance matrix between the centers and sample embeddings.
19 batch_size = x.size(0)
20 distmat = torch.pow(x, 2).sum(dim=1, keepdim=True).expand(batch_size, self.num_classes) + \
21 torch.pow(self.centers, 2).sum(dim=1, keepdim=True).expand(self.num_classes, batch_size).t()
22 distmat.addmm_(x, self.centers.t(), beta=1, alpha=-2)
23
24 classes = torch.arange(self.num_classes).long()
25 if self.use_gpu: classes = classes.cuda()
26 labels = labels.unsqueeze(1).expand(batch_size, self.num_classes)
27 mask = labels.eq(classes.expand(batch_size, self.num_classes))
28
29 dist = distmat * mask.float()
30 loss = dist.clamp(min=1e-12, max=1e+12).sum() / batch_size
31
32 return loss
中心损失函数在MindSpore中的实现为：
1# implementations of center loss (MindSpore)
2import mindspore
3import mindspore.nn as nn
4from mindspore import Tensor, Parameter
5import mindspore.ops as ops
6
7class CenterLoss(nn.Cell):
8 def __init__(self, num_classes=10, feat_dim=2, use_gpu=True):
9 super(CenterLoss, self).__init__()
10 self.num_classes = num_classes
11 self.feat_dim = feat_dim
12 self.use_gpu = use_gpu
13
14 self.centers = Parameter(Tensor(np.random.randn(self.num_classes, self.feat_dim), mindspore.float32))
15
16 def construct(self, x, labels):
17 # Ln.18-Ln.30: Calculate center loss by the Euclidean distance matrix between the centers and sample embeddings.
18 batch_size = x.shape[0]
19 distmat = ops.Reshape()(ops.ReduceSum()(ops.Square()(x), 1), (batch_size, 1)) + \
20 ops.Reshape()(ops.ReduceSum()(ops.Square()(self.centers), 1), (self.num_classes, 1))
21 distmat = ops.MatMul()(x, ops.Transpose()(self.centers, (1, 0)), b=distmat, alpha=-2)
22
23 classes = Tensor(np.arange(self.num_classes).astype(np.int32))
24 labels = ops.Reshape()(labels, (batch_size, 1))
25 mask = ops.Equal()(labels, classes)
26
27 dist = distmat * ops.Cast()(mask, mindspore.float32)
28 loss = ops.ReduceSum()(ops.ClipByValue()(dist, 1e-12, 1e+12)) / batch_size
29
30 return loss
10.2 回归任务的目标函数
10.2.1 l1损失函数
l1损失函数在PyTorch中的调用方式为：
1# l1 loss (PyTorch)
2import torch
3import torch.nn.functional as F
4
5input = torch.FloatTensor([3, 3, 3, 3])
6target = torch.tensor([2, 8, 6, 1])
7# Ln.8: Calculate l1 loss.
8loss = F.l1_loss(input, target)
9print(loss)
其具体实现细节可参考：
1# implementations of l1 loss (PyTorch)
2def l1_loss(
3 input: Tensor,
4 target: Tensor,
5 size_average: Optional[bool] = None,
6 reduce: Optional[bool] = None,
7 reduction: str = "mean",
8) -> Tensor:
9 """
10 l1_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor
11
12 Function that takes the mean element-wise absolute value difference.
13
14 See :class:`~torch.nn.L1Loss` for details.
15 """
16 # Ln.17-Ln.32: Calculate l1 loss.
17 if has_torch_function_variadic(input, target):
18 return handle_torch_function(
19 l1_loss, (input, target), input, target, size_average=size_average, reduce=reduce, reduction=reduction
20 )
21 if not (target.size() == input.size()):
22 warnings.warn(
23 "Using a target size ({}) that is different to the input size ({}). "
24 "This will likely lead to incorrect results due to broadcasting. "
25 "Please ensure they have the same size.".format(target.size(), input.size()),
26 stacklevel=2,
27 )
28 if size_average is not None or reduce is not None:
29 reduction = _Reduction.legacy_get_string(size_average, reduce)
30
31 expanded_input, expanded_target = torch.broadcast_tensors(input, target)
32 return torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
在MindSpore中的调用方式为：
1# l1 loss (MindSpore)
2from mindspore import Tensor, ops
3from mindspore import dtype as mstype
4
5x = ms.Tensor([[1, 2, 3], [4, 5, 6]], mstype.float32)
6target = ms.Tensor([[6, 5, 4], [3, 2, 1]], mstype.float32)
7# Ln.8: Calculate l1 loss.
8output = ops.l1_loss(x, target, reduction="mean")
9print(output)
其在MindSpore中具体实现细节为：
1# implementations of l1 loss (MindSpore)
2def l1_loss(input, target, reduction='mean'):
3 """
4 Calculate the mean absolute error between the `input` value and the `target` value.
5
6 Assuming that the `x` and `y` are 1-D Tensor, length `N`, `reduction` is set to ``'none'``,
7 then calculate the loss of `x` and `y` without dimensionality reduction.
8
9 Args:
10 input (Tensor): Predicted value, Tensor of any dimension.
11 target (Tensor): Target value, usually has the same shape as the `input`.
12 If `input` and `target` have different shape, make sure they can broadcast to each other.
13 reduction (str, optional): Apply specific reduction method to the output: ``'none'`` , ``'mean'`` , ``'sum'`` . Default: ``'mean'`` .
14
15 - ``'none'``: no reduction will be applied.
16 - ``'mean'``: compute and return the mean of elements in the output.
17 - ``'sum'``: the output elements will be summed.
18
19 Returns:
20 Tensor or Scalar, if `reduction` is ``'none'``, return a Tensor with same shape and dtype as `input`.
21 Otherwise, a scalar value will be returned.
22 """
23 # Ln.24-Ln.29: Calculate l1 loss.
24 _check_is_tensor('input', input, 'l1_loss')
25 _check_is_tensor('target', target, 'l1_loss')
26 if reduction not in ('mean', 'sum', 'none'):
27 raise ValueError(f"For l1_loss, the 'reduction' must be in ['mean', 'sum', 'none'], but got {reduction}.")
28 loss = _get_cache_prim(ops.Abs)()(input - target)
29 return _get_loss(loss, reduction, 'l1_loss')
10.2.2 l2损失函数
l2损失函数在PyTorch中的具体实现细节可参考：
1# implementations of l2 loss (PyTorch)
2def mse_loss(
3 input: Tensor,
4 target: Tensor,
5 size_average: Optional[bool] = None,
6 reduce: Optional[bool] = None,
7 reduction: str = "mean",
8) -> Tensor:
9 """
10 mse_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor
11
12 Measures the element-wise mean squared error.
13
14 See :class:`~torch.nn.MSELoss` for details.
15 """
16 # Ln.17-Ln.32: Calculate l2 loss.
17 if has_torch_function_variadic(input, target):
18 return handle_torch_function(
19 mse_loss, (input, target), input, target, size_average=size_average, reduce=reduce, reduction=reduction
20 )
21 if not (target.size() == input.size()):
22 warnings.warn(
23 "Using a target size ({}) that is different to the input size ({}). "
24 "This will likely lead to incorrect results due to broadcasting. "
25 "Please ensure they have the same size.".format(target.size(), input.size()),
26 stacklevel=2,
27 )
28 if size_average is not None or reduce is not None:
29 reduction = _Reduction.legacy_get_string(size_average, reduce)
30
31 expanded_input, expanded_target = torch.broadcast_tensors(input, target)
32 return torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
l2损失函数在MindSpore中的实现为：
1# implementations of l2 loss (MindSpore)
2def mse_loss(input, target, reduction='mean'):
3 """
4 Calculates the mean squared error between the predicted value and the label value.
5
6 For detailed information, please refer to :class:`mindspore.nn.MSELoss`.
7
8 Args:
9 input (Tensor): Tensor of any dimension.
10 target (Tensor): The input label. Tensor of any dimension, same shape as the `input` in common cases.
11 However, it supports that the shape of `input` is different from the shape of `target`
12 and they should be broadcasted to each other.
13 reduction (str, optional): Apply specific reduction method to the output: ``'none'`` , ``'mean'`` ,
14 ``'sum'`` . Default: ``'mean'`` .
15
16 - ``'none'``: no reduction will be applied.
17 - ``'mean'``: compute and return the mean of elements in the output.
18 - ``'sum'``: the output elements will be summed.
19
20 Returns:
21 Tensor, loss of type float, the shape is zero if `reduction` is ``'mean'`` or ``'sum'`` ,
22 while the shape of output is the broadcasted shape if `reduction` is ``'none'`` .
23 """
24 # Ln.25-Ln.53: Calculate l2 loss.
25 if not isinstance(input, (Tensor, Tensor_)):
26 raise TypeError("For ops.mse_loss, the `input` must be tensor")
27 if not isinstance(target, (Tensor, Tensor_)):
28 raise TypeError("For ops.mse_loss, the `target` must be tensor")
29 if reduction not in ['mean', 'none', 'sum']:
30 raise ValueError("For ops.mse_loss, `reduction` value should be either 'mean', 'none' or 'sum'.")
31
32 x = _get_cache_prim(ops.Square)()(input - target)
33 float_type = (mstype.float16, mstype.float32, mstype.float64)
34 if x.dtype not in float_type:
35 input_dtype = mstype.float32
36 else:
37 input_dtype = x.dtype
38 x = _get_cache_prim(ops.Cast)()(x, mstype.float32)
39
40 average_flag = True
41 reduce_flag = True
42 if reduction == 'sum':
43 average_flag = False
44 if reduction == 'none':
45 reduce_flag = False
46
47 if reduce_flag and average_flag:
48 x = _get_cache_prim(ops.ReduceMean)()(x, _get_axis(x))
49
50 if reduce_flag and not average_flag:
51 x = _get_cache_prim(ops.ReduceSum)()(x, _get_axis(x))
52
53 return _get_cache_prim(ops.Cast)()(x, input_dtype)
10.2.3 Tukey’s biweight损失函数
Tukey’s biweight损失函数在PyTorch中的具体实现细节可参考：
1# implementations of Tukey's biweight loss (PyTorch)
2import torch
3import torch.nn.functional as F
4
5def tukey_biweight_loss(pred, target, c=4.6851):
6 # Ln.7-Ln.12: Calculate Tukey's biweight loss.
7 l = torch.abs(pred - target)
8 loss1 = (1 - (1 - (l / c) ** 2) ** 3) * (c ** 2) / 6 # upper half of the piecewise function
9 loss2 = torch.tensor((c ** 2) / 6.).repeat(loss1.size()) # lower half of the piecewise function
10 mask = l < c # condition
11 loss = torch.where(mask, loss1 / loss1.size()[0], loss2) # the Tukey's biweight loss
12 return loss.mean()
在MindSpore中的实现为：
1# implementations of Tukey's biweight loss (MindSpore)
2import mindspore
3import mindspore.ops as ops
4
5def tukey_biweight_loss(pred, target, c=4.6851):
6 # Ln.7-Ln.12: Calculate Tukey's biweight loss.
7 l = ops.abs(pred - target)
8 loss1 = (1 - (1 - (l / c) ** 2) ** 3) * (c ** 2) / 6 # upper half of the piecewise function
9 loss2 = ops.Fill()(loss1.shape(), (c ** 2) / 6.) # lower half of the piecewise function
10 mask = l < c # condition
11 loss = ops.Select() (mask, loss1 / loss1.size()[0], loss2) # the Tukey's biweight loss
12 return ops.ReduceMean(keep_dims=False)(loss)