第九章 激活函数

9.8 激活函数实践

以经典的Sigmoid型函数为例，其在PyTorch中的调用方式为：

# Sigmoid (PyTorch)

import torch

import torch.nn as nn



sigmoid = nn.Sigmoid()

inputs = torch.randn(2)

output = sigmoid(inputs)

在MindSpore中的调用方式为：

# Sigmoid (MindSpore)

import mindspore as ms

import mindspore.nn as nn

import numpy as np



x = ms.Tensor(np.array([-1, -2, 0, 2, 1]), ms.float16)

sigmoid = nn.Sigmoid()

output = sigmoid(x)

其他常用激活函数如ReLU、PReLU和Leaky ReLU（其式中代表当x≤0时斜率的超参数一般为0.1）的PyTorch调用方式分别为：

# ReLU (PyTorch)

relu = nn.ReLU()

inputs = torch.randn(2)

output = relu(inputs)

# PReLU (PyTorch)

prelu = nn.PReLU()

inputs = torch.randn(2)

output = prelu(inputs)

# Leaky ReLU (PyTorch)

leaky_relu = nn.LeakyReLU(negative_slope=0.1)

inputs = torch.randn(2)

output = leaky_relu(inputs)

在MindSpore中的调用方式分别为：

# ReLU (MindSpore)

x = ms.Tensor(np.array([-1, -2, 0, 2, 1]), ms.float16)

relu = nn.ReLU()

output = relu(x)

# PReLU (MindSpore)

x = ms.Tensor(np.array([-1, -2, 0, 2, 1]), ms.float16)

prelu = nn.PReLU()

output = prelu(x)

# Leaky ReLU (MindSpore)

x = ms.Tensor(np.array([-1, -2, 0, 2, 1]), ms.float16)

leaky_relu = nn.LeakyReLU(0.1)

output = leaky_relu(x)