combustion.nn.functional

Extensions to torch.nn.functional.

Activation Functions

combustion.nn.functional.swish(inputs, memory_efficient=True)

The swish activation function, defined as

\[f(x) = x \cdot \text{sigmoid}(x) \]

Parameters

• inputs (Tensor) – The input tensor

• memory_efficient (bool, optional) – Whether or not to use an implementation that is more memory efficient at training time. When memory_efficient=True, this method is incompatible with TorchScript.

Return type

torch.Tensor

Warning

This method is traceable with TorchScript when memory_efficient=False, but is un-scriptable due to the use of torch.autograd.Function for a memory-efficient backward pass. Please export using torch.jit.trace() with memory_efficient=False

combustion.nn.functional.hard_swish(inputs, inplace=False)

The hard swish activation function proposed in Searching For MobileNetV3, defined as

\[f(x) = x \cdot \frac{\text{ReLU6}(x + 3)}{6} \]

Hard swish approximates the swish activation, but computationally cheaper due to the removal of \(\text{sigmoid}(x)\).

Parameters

• inputs (Tensor) – The input tensor

• inplace (bool, optional) – Whether or not to perform the operation in place.

Return type

torch.Tensor

combustion.nn.functional.hard_sigmoid(inputs, inplace=True)

The hard sigmoid activation function, defined as

\[f(x) = \frac{\text{ReLU6}(x + 3)}{6} \]

Hard sigmoid is a computationally efficient approximation to the sigmoid activation and is more suitable for quantization.

Parameters

• inputs (Tensor) – The input tensor

• inplace (bool, optional) – Whether or not to perform the operation in place.

Return type

torch.Tensor