API - NN¶
Layer list¶
|
The basic |
|
The class |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
Simplified version of |
|
Simplified version of |
|
Simplified version of |
|
Simplified version of |
|
Simplified version of |
|
Separable/Depthwise Convolutional 2D layer, see tf.nn.depthwise_conv2d. |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
Max pooling for 1D signal. |
|
Mean pooling for 1D signal. |
|
Max pooling for 2D image. |
|
Mean pooling for 2D image [batch, height, width, channel]. |
|
Max pooling for 3D volume. |
|
Mean pooling for 3D volume. |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
Corner pooling for 2D image [batch, height, width, channel], see here. |
|
It is a 1D sub-pixel up-sampling layer. |
|
It is a 2D sub-pixel up-sampling layer, usually be used for Super-Resolution applications, see SRGAN for example. |
|
The |
|
The |
|
The |
|
The |
|
It implements the function of the Layer Normalization Layer and can be applied to mini-batch input data. |
|
An Elman RNN cell with tanh or ReLU non-linearity. |
|
A long short-term memory (LSTM) cell. |
|
A gated recurrent unit (GRU) cell. |
|
Multilayer Elman network(RNN). |
|
Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence. |
|
Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence. |
|
Allows the model to jointly attend to information from different representation subspaces. |
|
A transformer model. |
|
TransformerEncoder is a stack of N encoder layers |
|
TransformerDecoder is a stack of N decoder layers |
|
TransformerEncoderLayer is made up of self-attn and feedforward network. |
|
TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network. |
|
A layer that reshapes high-dimension input into a vector. |
|
A layer that reshapes a given tensor. |
|
A layer that transposes the dimension of a tensor. |
|
A layer that shuffle a 2D image [batch, height, width, channel], see here. |
|
A layer that concats multiple tensors according to given axis. |
|
A layer that combines multiple |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
The |
|
MaskedConv3D. |
Base Layer¶
Module¶
-
class
tensorlayerx.nn.
Module
(name=None, act=None, *args, **kwargs)[source]¶ The basic
Module
class represents a single layer of a neural network. It should be subclassed when implementing new types of layers. :param name: A unique layer name. If None, a unique name will be automatically assigned. :type name: str or None-
all_weights
()¶ Return a list of Tensor which are all weights of this Layer.
-
trainable_weights
()¶ Return a list of Tensor which are all trainable weights of this Layer.
-
nontrainable_weights
()¶ Return a list of Tensor which are all nontrainable weights of this Layer.
-
build
()[source]¶ Abstract method. Build the Layer. All trainable weights should be defined in this function.
-
load_weights
()[source]¶ Load model weights from a given file, which should be previously saved by self.save_weights().
-
save_standard_weights
()[source]¶ Input file_path, save model weights into a npz_dict file. These parameters can support multiple backends.
-
Sequential Layer¶
-
class
tensorlayerx.nn.
SequentialLayer
(*args)[source]¶ The class
SequentialLayer
is a linear stack of layers. TheSequentialLayer
can be created by passing a list of layer instances. The given layer instances will be automatically connected one by one. :param layers: A list of layers. :type layers: list of Layer :param name: A unique layer name. If None, a unique name will be automatically assigned. :type name: str or None-
weights
()¶ A collection of weights of all the layer instances.
-
build
()[source]¶ Build the LayerList. The layer instances will be connected automatically one by one.
Examples
>>> conv = tlx.layers.Conv2d(3, 2, 3, pad_mode='valid') >>> bn = tlx.layers.BatchNorm2d(2) >>> seq = tlx.nn.SequentialLayer([conv, bn]) >>> x = tlx.layers.Input((1, 3, 4, 4)) >>> seq(x)
-
Input Layers¶
Input Layer¶
-
tensorlayerx.nn.
Input
(shape, init=<tensorlayerx.nn.initializers.tensorflow_initializers.Ones object>, dtype=tensorflow.float32, name=None)[source]¶ The
Input
class is the starting layer of a neural network.- Parameters
shape (tuple (int)) – Including batch size.
init (initializer or str or None) – The initializer for initializing the input matrix
dtype (dtype) – The type of input values. By default, tf.float32.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> ni = tlx.nn.Input([10, 50, 50, 32], name='input') >>> output shape : [10, 50, 50, 32]
One-hot Layer¶
-
class
tensorlayerx.nn.
OneHot
(depth=None, on_value=1.0, off_value=0.0, axis=-1, dtype=tensorflow.float32, name=None)[source]¶ The
OneHot
class is the starting layer of a neural network, seetf.one_hot
. Useful link: https://www.tensorflow.org/api_docs/python/tf/one_hot.- Parameters
depth (None or int) – If the input indices is rank N, the output will have rank N+1. The new axis is created at dimension axis (default: the new axis is appended at the end).
on_value (None or number) – The value to represnt ON. If None, it will default to the value 1.
off_value (None or number) – The value to represnt OFF. If None, it will default to the value 0.
axis (None or int) – The axis.
dtype (None or TensorFlow dtype) – The data type, None means tlx.float32.
name (str) – A unique layer name.
Examples
>>> net = tlx.nn.Input([32], dtype=tlx.int32) >>> onehot = tlx.nn.OneHot(depth=8) >>> print(onehot) OneHot(depth=8, name='onehot') >>> tensor = tlx.nn.OneHot(depth=8)(net) >>> print(tensor) Tensor([...], shape=(32, 8), dtype=float32)
Word2Vec Embedding Layer¶
-
class
tensorlayerx.nn.
Word2vecEmbedding
(vocabulary_size, embedding_size, num_sampled=64, activate_nce_loss=True, nce_loss_args=None, E_init='random_uniform', nce_W_init='truncated_normal', nce_b_init='constant', name=None)[source]¶ The
Word2vecEmbedding
class is a fully connected layer. For Word Embedding, words are input as integer index. The output is the embedded word vector.The layer integrates NCE loss by default (activate_nce_loss=True). If the NCE loss is activated, in a dynamic model, the computation of nce loss can be turned off in customised forward feeding by setting use_nce_loss=False when the layer is called. The NCE loss can be deactivated by setting activate_nce_loss=False.
- Parameters
vocabulary_size (int) – The size of vocabulary, number of words
embedding_size (int) – The number of embedding dimensions
num_sampled (int) – The number of negative examples for NCE loss
activate_nce_loss (boolean) – Whether activate nce loss or not. By default, True If True, the layer will return both outputs of embedding and nce_cost in forward feeding. If False, the layer will only return outputs of embedding. In a dynamic model, the computation of nce loss can be turned off in forward feeding by setting use_nce_loss=False when the layer is called. In a static model, once the model is constructed, the computation of nce loss cannot be changed (always computed or not computed).
nce_loss_args (dictionary) – The arguments for tf.ops.nce_loss()
E_init (initializer or str) – The initializer for initializing the embedding matrix
nce_W_init (initializer or str) – The initializer for initializing the nce decoder weight matrix
nce_b_init (initializer or str) – The initializer for initializing of the nce decoder bias vector
name (str) – A unique layer name
-
outputs
¶ The embedding layer outputs.
- Type
Tensor
-
normalized_embeddings
¶ Normalized embedding matrix.
- Type
Tensor
-
nce_weights
¶ The NCE weights only when activate_nce_loss is True.
- Type
Tensor
-
nce_biases
¶ The NCE biases only when activate_nce_loss is True.
- Type
Tensor
Examples
Word2Vec With TensorLayer (Example in examples/text_word_embedding/tutorial_word2vec_basic.py)
>>> import tensorlayerx as tlx >>> batch_size = 8 >>> embedding_size = 50 >>> inputs = tlx.nn.Input([batch_size], dtype=tlx.int32) >>> labels = tlx.nn.Input([batch_size, 1], dtype=tlx.int32) >>> emb_net = tlx.nn.Word2vecEmbedding( >>> vocabulary_size=10000, >>> embedding_size=embedding_size, >>> num_sampled=100, >>> activate_nce_loss=True, # the nce loss is activated >>> nce_loss_args={}, >>> E_init=tlx.initializers.random_uniform(minval=-1.0, maxval=1.0), >>> nce_W_init=tlx.initializers.truncated_normal(stddev=float(1.0 / np.sqrt(embedding_size))), >>> nce_b_init=tlx.initializers.constant(value=0.0), >>> name='word2vec_layer', >>> ) >>> print(emb_net) Word2vecEmbedding(vocabulary_size=10000, embedding_size=50, num_sampled=100, activate_nce_loss=True, nce_loss_args={}) >>> embed_tensor = emb_net(inputs, use_nce_loss=False) # the nce loss is turned off and no need to provide labels >>> embed_tensor = emb_net([inputs, labels], use_nce_loss=False) # the nce loss is turned off and the labels will be ignored >>> embed_tensor, embed_nce_loss = emb_net([inputs, labels]) # the nce loss is calculated >>> outputs = tlx.layers.Dense(n_units=10, name="dense")(embed_tensor) >>> model = tlx.model.Model(inputs=[inputs, labels], outputs=[outputs, embed_nce_loss], name="word2vec_model") # a static model >>> out = model([data_x, data_y], is_train=True) # where data_x is inputs and data_y is labels
References
https://www.tensorflow.org/tutorials/representation/word2vec
Embedding Layer¶
-
class
tensorlayerx.nn.
Embedding
(vocabulary_size, embedding_size, E_init='random_uniform', name=None)[source]¶ The
Embedding
class is a look-up table for word embedding.Word content are accessed using integer indexes, then the output is the embedded word vector. To train a word embedding matrix, you can used
Word2vecEmbedding
. If you have a pre-trained matrix, you can assign the parameters into it.- Parameters
vocabulary_size (int) – The size of vocabulary, number of words.
embedding_size (int) – The number of embedding dimensions.
E_init (initializer or str) – The initializer for the embedding matrix.
E_init_args (dictionary) – The arguments for embedding matrix initializer.
name (str) – A unique layer name.
-
outputs
¶ The embedding layer output is a 3D tensor in the shape: (batch_size, num_steps(num_words), embedding_size).
- Type
tensor
Examples
>>> import tensorlayerx as tlx >>> input = tlx.nn.Input([8, 100], dtype=tlx.int32) >>> embed = tlx.nn.Embedding(vocabulary_size=1000, embedding_size=50, name='embed') >>> print(embed) Embedding(vocabulary_size=1000, embedding_size=50) >>> tensor = embed(input) >>> print(tensor) Tensor([...], shape=(8, 100, 50), dtype=float32)
Average Embedding Layer¶
-
class
tensorlayerx.nn.
AverageEmbedding
(vocabulary_size, embedding_size, pad_value=0, E_init='random_uniform', name=None)[source]¶ The
AverageEmbedding
averages over embeddings of inputs. This is often used as the input layer for model like DAN[1] and FastText[2].- Parameters
vocabulary_size (int) – The size of vocabulary.
embedding_size (int) – The dimension of the embedding vectors.
pad_value (int) – The scalar padding value used in inputs, 0 as default.
E_init (initializer or str) – The initializer of the embedding matrix.
name (str) – A unique layer name.
-
outputs
¶ The embedding layer output is a 2D tensor in the shape: (batch_size, embedding_size).
- Type
tensor
References
[1] Iyyer, M., Manjunatha, V., Boyd-Graber, J., & Daum’e III, H. (2015). Deep Unordered Composition Rivals Syntactic Methods for Text Classification. In Association for Computational Linguistics.
[2] Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016). Bag of Tricks for Efficient Text Classification.
Examples
>>> import tensorlayerx as tlx >>> batch_size = 8 >>> length = 5 >>> input = tlx.nn.Input([batch_size, length], dtype=tlx.int32) >>> avgembed = tlx.nn.AverageEmbedding(vocabulary_size=1000, embedding_size=50, name='avg') >>> print(avgembed) AverageEmbedding(vocabulary_size=1000, embedding_size=50, pad_value=0) >>> tensor = avgembed(input) >>> print(tensor) Tensor([...], shape=(8, 50), dtype=float32)
Convolutional Layers¶
Convolutions¶
Conv1d¶
-
class
tensorlayerx.nn.
Conv1d
(n_filter=32, filter_size=5, stride=1, act=None, padding='SAME', data_format='channels_last', dilation_rate=1, W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ Simplified version of
Conv1dLayer
.- Parameters
n_filter (int) – The number of filters
filter_size (int) – The filter size
stride (int) – The stride step
dilation_rate (int) – Specifying the dilation rate to use for dilated convolution.
act (activation function) – The function that is applied to the layer activations
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channel_last” (NWC, default) or “channels_first” (NCW).
W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 100, 1], name='input') >>> conv1d = tlx.nn.Conv1d(n_filter=32, filter_size=5, stride=2, b_init=None, in_channels=1, name='conv1d_1') >>> print(conv1d) >>> tensor = tlx.nn.Conv1d(n_filter=32, filter_size=5, stride=2, act=tlx.ReLU, name='conv1d_2')(net) >>> print(tensor)
Conv2d¶
-
class
tensorlayerx.nn.
Conv2d
(n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ Simplified version of
Conv2dLayer
.- Parameters
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (height, width).
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
W_init (initializer or str) – The initializer for the the weight matrix.
b_init (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 400, 400, 3], name='input') >>> conv2d = tlx.nn.Conv2d(n_filter=32, filter_size=(3, 3), strides=(2, 2), b_init=None, in_channels=3, name='conv2d_1') >>> print(conv2d) >>> tensor = tlx.nn.Conv2d(n_filter=32, filter_size=(3, 3), strides=(2, 2), act=tlx.ReLU, name='conv2d_2')(net) >>> print(tensor)
Conv3d¶
-
class
tensorlayerx.nn.
Conv3d
(n_filter=32, filter_size=(3, 3, 3), strides=(1, 1, 1), act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ Simplified version of
Conv3dLayer
.- Parameters
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (height, width).
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NDHWC, default) or “channels_first” (NCDHW).
W_init (initializer or str) – The initializer for the the weight matrix.
b_init (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 20, 20, 20, 3], name='input') >>> conv3d = tlx.nn.Conv3d(n_filter=32, filter_size=(3, 3, 3), strides=(2, 2, 2), b_init=None, in_channels=3, name='conv3d_1') >>> print(conv3d) >>> tensor = tlx.nn.Conv3d(n_filter=32, filter_size=(3, 3, 3), strides=(2, 2, 2), act=tlx.ReLU, name='conv3d_2')(net) >>> print(tensor)
Deconvolutions¶
DeConv2d¶
-
class
tensorlayerx.nn.
DeConv2d
(n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ Simplified version of
Deconv2dLayer
.- Parameters
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size.
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.output_shape (A 1-D Tensor) – representing the output shape of the deconvolution op.
dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
W_init (initializer or str) – The initializer for the the weight matrix.
b_init (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 400, 400, 3], name='input') >>> conv2d_transpose = tlx.nn.DeConv2d(n_filter=32, filter_size=(3, 3), strides=(2, 2), b_init=None, in_channels=3, name='conv2d_transpose_1') >>> print(conv2d_transpose) >>> tensor = tlx.nn.DeConv2d(n_filter=32, filter_size=(3, 3), strides=(2, 2), act=tlx.ReLU, name='conv2d_transpose_2')(net) >>> print(tensor)
DeConv3d¶
-
class
tensorlayerx.nn.
DeConv3d
(n_filter=32, filter_size=(3, 3, 3), strides=(1, 1, 1), act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ Simplified version of
Deconv3dLayer
.- Parameters
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (depth, height, width).
output_shape – A 1-D Tensor representing the output shape of the deconvolution op.
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NDHWC, default) or “channels_first” (NCDHW).
W_init (initializer or str) – The initializer for the the weight matrix.
b_init (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 20, 20, 20, 3], name='input') >>> deconv3d = tlx.nn.DeConv3d(n_filter=32, filter_size=(3, 3, 3), strides=(2, 2, 2), b_init=None, in_channels=3, name='deconv3d_1') >>> print(deconv3d) >>> tensor = tlx.nn.DeConv3d(n_filter=32, filter_size=(3, 3, 3), strides=(2, 2, 2), act=tlx.ReLU, name='deconv3d_2')(net) >>> print(tensor)
Deformable Convolutions¶
DeformableConv2d¶
-
class
tensorlayerx.nn.
DeformableConv2d
(offset_layer=None, n_filter=32, filter_size=(3, 3), act=None, padding='SAME', W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
DeformableConv2d
class is a 2D Deformable Convolutional Networks.- Parameters
offset_layer (tlx.Tensor) – To predict the offset of convolution operations. The shape is (batchsize, input height, input width, 2*(number of element in the convolution kernel)) e.g. if apply a 3*3 kernel, the number of the last dimension should be 18 (2*3*3)
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (height, width).
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([5, 10, 10, 16], name='input') >>> offset1 = tlx.nn.Conv2d( ... n_filter=18, filter_size=(3, 3), strides=(1, 1), padding='SAME', name='offset1' ... )(net) >>> deformconv1 = tlx.nn.DeformableConv2d( ... offset_layer=offset1, n_filter=32, filter_size=(3, 3), name='deformable1' ... )(net) >>> offset2 = tlx.nn.Conv2d( ... n_filter=18, filter_size=(3, 3), strides=(1, 1), padding='SAME', name='offset2' ... )(deformconv1) >>> deformconv2 = tlx.nn.DeformableConv2d( ... offset_layer=offset2, n_filter=64, filter_size=(3, 3), name='deformable2' ... )(deformconv1)
References
The deformation operation was adapted from the implementation in here
Notes
The padding is fixed to ‘SAME’.
The current implementation is not optimized for memory usgae. Please use it carefully.
Depthwise Convolutions¶
DepthwiseConv2d¶
-
class
tensorlayerx.nn.
DepthwiseConv2d
(filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1), depth_multiplier=1, W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ Separable/Depthwise Convolutional 2D layer, see tf.nn.depthwise_conv2d.
- Input:
4-D Tensor (batch, height, width, in_channels).
- Output:
4-D Tensor (batch, new height, new width, in_channels * depth_multiplier).
- Parameters
filter_size (tuple of 2 int) – The filter size (height, width).
strides (tuple of 2 int) – The stride step (height, width).
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
dilation_rate (tuple of 2 int) – The dilation rate in which we sample input values across the height and width dimensions in atrous convolution. If it is greater than 1, then all values of strides must be 1.
depth_multiplier (int) – The number of channels to expand to.
W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip bias.
in_channels (int) – The number of in channels.
name (str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 200, 200, 32], name='input') >>> depthwiseconv2d = tlx.nn.DepthwiseConv2d( ... filter_size=(3, 3), strides=(1, 1), dilation_rate=(2, 2), act=tlx.ReLU, depth_multiplier=2, name='depthwise' ... )(net) >>> print(depthwiseconv2d) >>> output shape : (8, 200, 200, 64)
References
tflearn’s grouped_conv_2d
keras’s separableconv2d
Group Convolutions¶
GroupConv2d¶
-
class
tensorlayerx.nn.
GroupConv2d
(n_filter=32, filter_size=(1, 1), strides=(1, 1), n_group=1, act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
GroupConv2d
class is 2D grouped convolution, see here.- Parameters
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size.
stride (tuple of int) – The stride step.
n_group (int) – The number of groups.
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 24, 24, 32], name='input') >>> groupconv2d = tlx.nn.GroupConv2d( ... n_filter=64, filter_size=(3, 3), strides=(2, 2), n_group=2, name='group' ... )(net) >>> print(groupconv2d) >>> output shape : (8, 12, 12, 64)
Separable Convolutions¶
SeparableConv1d¶
-
class
tensorlayerx.nn.
SeparableConv1d
(n_filter=32, filter_size=1, stride=1, act=None, padding='SAME', data_format='channels_last', dilation_rate=1, depth_multiplier=1, depthwise_init='truncated_normal', pointwise_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
SeparableConv1d
class is a 1D depthwise separable convolutional layer. This layer performs a depthwise convolution that acts separately on channels, followed by a pointwise convolution that mixes channels.- Parameters
n_filter (int) – The dimensionality of the output space (i.e. the number of filters in the convolution).
filter_size (int) – Specifying the spatial dimensions of the filters. Can be a single integer to specify the same value for all spatial dimensions.
stride (int) – Specifying the stride of the convolution. Can be a single integer to specify the same value for all spatial dimensions. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.
act (activation function) – The activation function of this layer.
padding (str) – One of “valid” or “same” (case-insensitive).
data_format (str) – One of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width).
dilation_rate (int) – Specifying the dilation rate to use for dilated convolution. Can be a single integer to specify the same value for all spatial dimensions. Currently, specifying any dilation_rate value != 1 is incompatible with specifying any stride value != 1.
depth_multiplier (int) – The number of depthwise convolution output channels for each input channel. The total number of depthwise convolution output channels will be equal to num_filters_in * depth_multiplier.
depthwise_init (initializer or str) – for the depthwise convolution kernel.
pointwise_init (initializer or str) – For the pointwise convolution kernel.
b_init (initializer or str) – For the bias vector. If None, ignore bias in the pointwise part only.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 50, 64], name='input') >>> separableconv1d = tlx.nn.SeparableConv1d(n_filter=32, filter_size=3, stride=2, padding='SAME', act=tlx.ReLU, name='separable_1d')(net) >>> print(separableconv1d) >>> output shape : (8, 25, 32)
SeparableConv2d¶
-
class
tensorlayerx.nn.
SeparableConv2d
(n_filter=32, filter_size=(1, 1), strides=(1, 1), act=None, padding='VALID', data_format='channels_last', dilation_rate=(1, 1), depth_multiplier=1, depthwise_init='truncated_normal', pointwise_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
SeparableConv2d
class is a 2D depthwise separable convolutional layer. This layer performs a depthwise convolution that acts separately on channels, followed by a pointwise convolution that mixes channels.- Parameters
n_filter (int) – The dimensionality of the output space (i.e. the number of filters in the convolution).
filter_size (tuple of int) – Specifying the spatial dimensions of the filters. Can be a single integer to specify the same value for all spatial dimensions.
strides (tuple of int) – Specifying the stride of the convolution. Can be a single integer to specify the same value for all spatial dimensions. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.
act (activation function) – The activation function of this layer.
padding (str) – One of “valid” or “same” (case-insensitive).
data_format (str) – One of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width).
dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution. Can be a single integer to specify the same value for all spatial dimensions. Currently, specifying any dilation_rate value != 1 is incompatible with specifying any stride value != 1.
depth_multiplier (int) – The number of depthwise convolution output channels for each input channel. The total number of depthwise convolution output channels will be equal to num_filters_in * depth_multiplier.
depthwise_init (initializer or str) – for the depthwise convolution kernel.
pointwise_init (initializer or str) – For the pointwise convolution kernel.
b_init (initializer or str) – For the bias vector. If None, ignore bias in the pointwise part only.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 50, 50, 64], name='input') >>> separableconv2d = tlx.nn.SeparableConv2d(n_filter=32, filter_size=(3,3), strides=(2,2), depth_multiplier = 3 , padding='SAME', act=tlx.ReLU, name='separable_2d')(net) >>> print(separableconv2d) >>> output shape : (8, 24, 24, 32)
SubPixel Convolutions¶
SubpixelConv1d¶
-
class
tensorlayerx.nn.
SubpixelConv1d
(scale=2, act=None, in_channels=None, name=None)[source]¶ It is a 1D sub-pixel up-sampling layer.
Calls a TensorFlow function that directly implements this functionality. We assume input has dim (batch, width, r)
- Parameters
scale (int) – The up-scaling ratio, a wrong setting will lead to Dimension size error.
act (activation function) – The activation function of this layer.
in_channels (int) – The number of in channels.
name (str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 25, 32], name='input') >>> subpixelconv1d = tlx.nn.SubpixelConv1d(scale=2, name='subpixelconv1d')(net) >>> print(subpixelconv1d) >>> output shape : (8, 50, 16)
References
SubpixelConv2d¶
-
class
tensorlayerx.nn.
SubpixelConv2d
(scale=2, data_format='channels_last', act=None, name=None)[source]¶ It is a 2D sub-pixel up-sampling layer, usually be used for Super-Resolution applications, see SRGAN for example.
- Parameters
scale (int) – factor to increase spatial resolution.
data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
act (activation function) – The activation function of this layer.
name (str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([2, 16, 16, 4], name='input1') >>> subpixelconv2d = tlx.nn.SubpixelConv2d(scale=2, data_format='channels_last', name='subpixel_conv2d1')(net) >>> print(subpixelconv2d) >>> output shape : (2, 32, 32, 1)
>>> net = tlx.nn.Input([2, 16, 16, 40], name='input2') >>> subpixelconv2d = tlx.nn.SubpixelConv2d(scale=2, data_format='channels_last', name='subpixel_conv2d2')(net) >>> print(subpixelconv2d) >>> output shape : (2, 32, 32, 10)
>>> net = tlx.nn.Input([2, 16, 16, 250], name='input3') >>> subpixelconv2d = tlx.nn.SubpixelConv2d(scale=5, data_format='channels_last', name='subpixel_conv2d3')(net) >>> print(subpixelconv2d) >>> output shape : (2, 80, 80, 10)
References
MaskedConv3d¶
-
class
tensorlayerx.nn.
MaskedConv3d
(mask_type, n_filter, filter_size=(3, 3, 3), strides=(1, 1, 1), dilation_rate=(1, 1, 1), padding='SAME', act=None, in_channels=None, data_format='channels_last', kernel_initializer='he_normal', bias_initializer='zeros', name=None)[source]¶ MaskedConv3D. Reference: [1] Nguyen D T , Quach M , Valenzise G , et al. Lossless Coding of Point Cloud Geometry using a Deep Generative Model[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, PP(99):1-1.
- Parameters
mask_type (str) – The mask type(‘A’, ‘B’)
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (height, width).
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NDHWC, default) or “channels_first” (NCDHW).
kernel_initializer (initializer or str) – The initializer for the the weight matrix.
bias_initializer (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 20, 20, 20, 3], name='input') >>> conv3d = tlx.nn.MaskedConv3d(mask_type='A', n_filter=32, filter_size=(3, 3, 3), strides=(2, 2, 2), bias_initializer=None, in_channels=3, name='conv3d_1') >>> print(conv3d) >>> tensor = tlx.nn.MaskedConv3d(mask_type='B', n_filter=32, filter_size=(3, 3, 3), strides=(2, 2, 2), act=tlx.ReLU, name='conv3d_2')(net) >>> print(tensor)
Dense Layers¶
Dense Layer¶
-
class
tensorlayerx.nn.
Dense
(n_units, act=None, W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
Dense
class is a fully connected layer.- Parameters
n_units (int) – The number of units of this layer.
act (activation function) – The activation function of this layer.
W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip biases.
in_channels (int) – The number of channels of the previous layer. If None, it will be automatically detected when the layer is forwarded for the first time.
name (None or str) – A unique layer name. If None, a unique name will be automatically generated.
Examples
With TensorLayer
>>> net = tlx.nn.Input([100, 50], name='input') >>> dense = tlx.nn.Dense(n_units=800, act=tlx.ReLU, in_channels=50, name='dense_1') >>> print(dense) Dense(n_units=800, relu, in_channels='50', name='dense_1') >>> tensor = tlx.nn.Dense(n_units=800, act=tlx.ReLU, name='dense_2')(net) >>> print(tensor) tf.Tensor([...], shape=(100, 800), dtype=float32)
Notes
If the layer input has more than two axes, it needs to be flatten by using
Flatten
.
Drop Connect Dense Layer¶
-
class
tensorlayerx.nn.
DropconnectDense
(keep=0.5, n_units=100, act=None, W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
DropconnectDense
class isDense
with DropConnect behaviour which randomly removes connections between this layer and the previous layer according to a keeping probability.- Parameters
keep (float) – The keeping probability. The lower the probability it is, the more activations are set to zero.
n_units (int) – The number of units of this layer.
act (activation function) – The activation function of this layer.
W_init (weights initializer or str) – The initializer for the weight matrix.
b_init (biases initializer or str) – The initializer for the bias vector.
in_channels (int) – The number of channels of the previous layer. If None, it will be automatically detected when the layer is forwarded for the first time.
name (str) – A unique layer name.
Examples
>>> net = tlx.nn.Input([10, 784], name='input') >>> net = tlx.nn.DropconnectDense(keep=0.8, n_units=800, act=tlx.ReLU, name='relu1')(net) >>> output shape :(10, 800) >>> net = tlx.nn.DropconnectDense(keep=0.5, n_units=800, act=tlx.ReLU, name='relu2')(net) >>> output shape :(10, 800) >>> net = tlx.nn.DropconnectDense(keep=0.5, n_units=10, name='output')(net) >>> output shape :(10, 10)
References
Dropout Layers¶
-
class
tensorlayerx.nn.
Dropout
(keep, seed=0, name=None)[source]¶ The
Dropout
class is a noise layer which randomly set some activations to zero according to a keeping probability.- Parameters
keep (float) – The keeping probability. The lower the probability it is, the more activations are set to zero.
seed (int or None) – The seed for random dropout.
name (None or str) – A unique layer name.
Examples
>>> net = tlx.nn.Input([10, 200]) >>> net = tlx.nn.Dropout(keep=0.2)(net)
Extend Layers¶
Expand Dims Layer¶
-
class
tensorlayerx.nn.
ExpandDims
(axis, name=None)[source]¶ The
ExpandDims
class inserts a dimension of 1 into a tensor’s shape, see tf.expand_dims() .- Parameters
axis (int) – The dimension index at which to expand the shape of input.
name (str) – A unique layer name. If None, a unique name will be automatically assigned.
Examples
>>> x = tlx.nn.Input([10, 3], name='in') >>> y = tlx.nn.ExpandDims(axis=-1)(x) [10, 3, 1]
Tile layer¶
-
class
tensorlayerx.nn.
Tile
(multiples=None, name=None)[source]¶ The
Tile
class constructs a tensor by tiling a given tensor, see tf.tile() .- Parameters
multiples (tensor) – Must be one of the following types: int32, int64. 1-D Length must be the same as the number of dimensions in input.
name (None or str) – A unique layer name.
Examples
>>> x = tlx.nn.Input([10, 3], name='in') >>> y = tlx.nn.Tile(multiples=[2, 3])(x)
Image Resampling Layers¶
2D UpSampling¶
-
class
tensorlayerx.nn.
UpSampling2d
(scale, method='bilinear', antialias=False, data_format='channels_last', name=None, ksize=None)[source]¶ The
UpSampling2d
class is a up-sampling 2D layer.- Parameters
scale (int or tuple of int) – (scale_height, scale_width) scale factor. scale_height = new_height/height, scale_width = new_width/width.
method (str) –
- The resize method selected through the given string. Default ‘bilinear’.
’bilinear’, Bilinear interpolation.
’nearest’, Nearest neighbor interpolation.
’bicubic’, Bicubic interpolation.
’area’, Area interpolation.
antialias (boolean) – Whether to use an anti-aliasing filter when downsampling an image.
data_format (str) – channels_last ‘channel_last’ (default) or channels_first.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> ni = tlx.nn.Input([10, 50, 50, 32], name='input') >>> ni = tlx.nn.UpSampling2d(scale=(2, 2))(ni) >>> output shape : [10, 100, 100, 32]
2D DownSampling¶
-
class
tensorlayerx.nn.
DownSampling2d
(scale, method='bilinear', antialias=False, data_format='channels_last', name=None, ksize=None)[source]¶ The
DownSampling2d
class is down-sampling 2D layer.- Parameters
scale (int or tuple of int) – (new_height, new_width) scale factor.scale_height = new_height/height, scale_width = new_width/width.
method (str) –
- The resize method selected through the given string. Default ‘bilinear’.
’bilinear’, Bilinear interpolation.
’nearest’, Nearest neighbor interpolation.
’bicubic’, Bicubic interpolation.
’area’, Area interpolation.
antialias (boolean) – Whether to use an anti-aliasing filter when downsampling an image.
data_format (str) – channels_last ‘channel_last’ (default) or channels_first.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> ni = tlx.nn.Input([10, 50, 50, 32], name='input') >>> ni = tlx.nn.DownSampling2d(scale=(2, 2))(ni) >>> output shape : [10, 25, 25, 32]
Merge Layers¶
Concat Layer¶
-
class
tensorlayerx.nn.
Concat
(concat_dim=-1, name=None)[source]¶ A layer that concats multiple tensors according to given axis.
- Parameters
concat_dim (int) – The dimension to concatenate.
name (None or str) – A unique layer name.
Examples
>>> class CustomModel(Module): >>> def __init__(self): >>> super(CustomModel, self).__init__(name="custom") >>> self.dense1 = tlx.nn.Dense(in_channels=20, n_units=10, act=tlx.ReLU, name='relu1_1') >>> self.dense2 = tlx.nn.Dense(in_channels=20, n_units=10, act=tlx.ReLU, name='relu2_1') >>> self.concat = tlx.nn.Concat(concat_dim=1, name='concat_layer')
>>> def forward(self, inputs): >>> d1 = self.dense1(inputs) >>> d2 = self.dense2(inputs) >>> outputs = self.concat([d1, d2]) >>> return outputs
ElementWise Layer¶
-
class
tensorlayerx.nn.
Elementwise
(combine_fn=<function minimum>, act=None, name=None)[source]¶ A layer that combines multiple
Layer
that have the same output shapes according to an element-wise operation. If the element-wise operation is complicated, please consider to useElementwiseLambda
.- Parameters
combine_fn (a TensorFlow element-wise combine function) – e.g. AND is
tlx.minimum
; OR istlx.maximum
; ADD istlx.add
; MUL istlx.multiply
and so on. See TensorFlow Math API . If the combine function is more complicated, please consider to useElementwiseLambda
.act (activation function) – The activation function of this layer.
name (None or str) – A unique layer name.
Examples
>>> import tensorlayerx as tlx >>> class CustomModel(tlx.nn.Module): >>> def __init__(self): >>> super(CustomModel, self).__init__(name="custom") >>> self.dense1 = tlx.nn.Dense(in_channels=20, n_units=10, act=tlx.ReLU, name='relu1_1') >>> self.dense2 = tlx.nn.Dense(in_channels=20, n_units=10, act=tlx.ReLU, name='relu2_1') >>> self.element = tlx.nn.Elementwise(combine_fn=tlx.minimum, name='minimum', act=tlx.identity)
>>> def forward(self, inputs): >>> d1 = self.dense1(inputs) >>> d2 = self.dense2(inputs) >>> outputs = self.element([d1, d2]) >>> return outputs
Noise Layer¶
-
class
tensorlayerx.nn.
GaussianNoise
(mean=0.0, stddev=1.0, is_always=True, seed=None, name=None)[source]¶ The
GaussianNoise
class is noise layer that adding noise with gaussian distribution to the activation.- Parameters
mean (float) – The mean. Default is 0.0.
stddev (float) – The standard deviation. Default is 1.0.
is_always (boolean) – Is True, add noise for train and eval mode. If False, skip this layer in eval mode.
seed (int or None) – The seed for random noise.
name (str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([64, 200], name='input') >>> net = tlx.nn.Dense(in_channels=200, n_units=100, act=tlx.ReLU, name='dense')(net) >>> gaussianlayer = tlx.nn.GaussianNoise(name='gaussian')(net) >>> print(gaussianlayer) >>> output shape : (64, 100)
Normalization Layers¶
Batch Normalization¶
-
class
tensorlayerx.nn.
BatchNorm
(decay=0.9, epsilon=1e-05, act=None, is_train=True, beta_init='zeros', gamma_init='random_normal', moving_mean_init='zeros', moving_var_init='zeros', num_features=None, data_format='channels_last', name=None)[source]¶ The
BatchNorm
is a batch normalization layer for both fully-connected and convolution outputs. Seetf.nn.batch_normalization
andtf.nn.moments
.- Parameters
decay (float) – A decay factor for ExponentialMovingAverage. Suggest to use a large value for large dataset.
epsilon (float) – Eplison.
act (activation function) – The activation function of this layer.
is_train (boolean) – Is being used for training or inference.
beta_init (initializer or str) – The initializer for initializing beta, if None, skip beta. Usually you should not skip beta unless you know what happened.
gamma_init (initializer or str) – The initializer for initializing gamma, if None, skip gamma. When the batch normalization layer is use instead of ‘biases’, or the next layer is linear, this can be disabled since the scaling can be done by the next layer. see Inception-ResNet-v2
moving_mean_init (initializer or str) – The initializer for initializing moving mean, if None, skip moving mean.
moving_var_init (initializer or str) – The initializer for initializing moving var, if None, skip moving var.
num_features (int) – Number of features for input tensor. Useful to build layer if using BatchNorm1d, BatchNorm2d or BatchNorm3d, but should be left as None if using BatchNorm. Default None.
data_format (str) – channels_last ‘channel_last’ (default) or channels_first.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 50, 32], name='input') >>> net = tlx.nn.BatchNorm()(net)
Notes
The
BatchNorm
is universally suitable for 3D/4D/5D input in static model, but should not be used in dynamic model where layer is built upon class initialization. So the argument ‘num_features’ should only be used for subclassesBatchNorm1d
,BatchNorm2d
andBatchNorm3d
. All the three subclasses are suitable under all kinds of conditions.References
Batch Normalization 1D¶
-
class
tensorlayerx.nn.
BatchNorm1d
(decay=0.9, epsilon=1e-05, act=None, is_train=True, beta_init='zeros', gamma_init='random_normal', moving_mean_init='zeros', moving_var_init='zeros', num_features=None, data_format='channels_last', name=None)[source]¶ The
BatchNorm1d
applies Batch Normalization over 2D/3D input (a mini-batch of 1D inputs (optional) with additional channel dimension), of shape (N, C) or (N, L, C) or (N, C, L). See more details inBatchNorm
.Examples
With TensorLayer
>>> # in static model, no need to specify num_features >>> net = tlx.nn.Input([10, 50, 32], name='input') >>> net = tlx.nn.BatchNorm1d()(net) >>> # in dynamic model, build by specifying num_features >>> conv = tlx.nn.Conv1d(32, 5, 1, in_channels=3) >>> bn = tlx.nn.BatchNorm1d(num_features=32)
Batch Normalization 2D¶
-
class
tensorlayerx.nn.
BatchNorm2d
(decay=0.9, epsilon=1e-05, act=None, is_train=True, beta_init='zeros', gamma_init='random_normal', moving_mean_init='zeros', moving_var_init='zeros', num_features=None, data_format='channels_last', name=None)[source]¶ The
BatchNorm2d
applies Batch Normalization over 4D input (a mini-batch of 2D inputs with additional channel dimension) of shape (N, H, W, C) or (N, C, H, W). See more details inBatchNorm
.Examples
With TensorLayer
>>> # in static model, no need to specify num_features >>> net = tlx.nn.Input([10, 50, 50, 32], name='input') >>> net = tlx.nn.BatchNorm2d()(net) >>> # in dynamic model, build by specifying num_features >>> conv = tlx.nn.Conv2d(32, (5, 5), (1, 1), in_channels=3) >>> bn = tlx.nn.BatchNorm2d(num_features=32)
Batch Normalization 3D¶
-
class
tensorlayerx.nn.
BatchNorm3d
(decay=0.9, epsilon=1e-05, act=None, is_train=True, beta_init='zeros', gamma_init='random_normal', moving_mean_init='zeros', moving_var_init='zeros', num_features=None, data_format='channels_last', name=None)[source]¶ The
BatchNorm3d
applies Batch Normalization over 5D input (a mini-batch of 3D inputs with additional channel dimension) with shape (N, D, H, W, C) or (N, C, D, H, W). See more details inBatchNorm
.Examples
With TensorLayer
>>> # in static model, no need to specify num_features >>> net = tlx.nn.Input([10, 50, 50, 50, 32], name='input') >>> net = tlx.nn.BatchNorm3d()(net) >>> # in dynamic model, build by specifying num_features >>> conv = tlx.nn.Conv3d(32, (5, 5, 5), (1, 1), in_channels=3) >>> bn = tlx.nn.BatchNorm3d(num_features=32)
Padding Layers¶
Pad Layer (Expert API)¶
Padding layer for any modes.
-
class
tensorlayerx.nn.
PadLayer
(padding=None, mode='CONSTANT', constant_values=0, name=None)[source]¶ The
PadLayer
class is a padding layer for any mode and dimension. Please see tf.pad for usage.- Parameters
padding (list of lists of 2 ints, or a Tensor of type int32.) – The int32 values to pad.
mode (str) – “CONSTANT”, “REFLECT”, or “SYMMETRIC” (case-insensitive).
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 224, 224, 3], name='input') >>> padlayer = tlx.nn.PadLayer([[0, 0], [3, 3], [3, 3], [0, 0]], "REFLECT", name='inpad')(net) >>> print(padlayer) >>> output shape : (10, 230, 230, 3)
1D Zero padding¶
-
class
tensorlayerx.nn.
ZeroPad1d
(padding, name=None)[source]¶ The
ZeroPad1d
class is a 1D padding layer for signal [batch, length, channel].- Parameters
padding (int, or tuple of 2 ints) –
If int, zeros to add at the beginning and end of the padding dimension (axis 1).
If tuple of 2 ints, zeros to add at the beginning and at the end of the padding dimension.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 1], name='input') >>> pad1d = tlx.nn.ZeroPad1d(padding=(3, 3))(net) >>> print(pad1d) >>> output shape : (10, 106, 1)
2D Zero padding¶
-
class
tensorlayerx.nn.
ZeroPad2d
(padding, name=None)[source]¶ The
ZeroPad2d
class is a 2D padding layer for image [batch, height, width, channel].- Parameters
padding (tuple of 2 ints or int, or tuple of 2 tuples of 2 ints.) –
If int, the same symmetric padding is applied to width and height.
If tuple of 2 ints, interpreted as two different symmetric padding values for height and width as
(symmetric_height_pad, symmetric_width_pad)
.If tuple of 2 tuples of 2 ints, interpreted as
((top_pad, bottom_pad), (left_pad, right_pad))
.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 100, 3], name='input') >>> pad2d = tlx.nn.ZeroPad2d(padding=((3, 3), (4, 4)))(net) >>> print(pad2d) >>> output shape : (10, 106, 108, 3)
3D Zero padding¶
-
class
tensorlayerx.nn.
ZeroPad3d
(padding, name=None)[source]¶ The
ZeroPad3d
class is a 3D padding layer for volume [batch, depth, height, width, channel].- Parameters
padding (int, or tuple of 2 ints, or tuple of 2 tuples of 2 ints.) –
If int, the same symmetric padding is applied to width and height.
If tuple of 2 ints, interpreted as two different symmetric padding values for height and width as
(symmetric_dim1_pad, symmetric_dim2_pad, symmetric_dim3_pad)
.If tuple of 2 tuples of 2 ints, interpreted as
((left_dim1_pad, right_dim1_pad), (left_dim2_pad, right_dim2_pad), (left_dim3_pad, right_dim3_pad))
.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 100, 100, 3], name='input') >>> pad3d = tlx.nn.ZeroPad3d(padding=((3, 3), (4, 4), (5, 5)))(net) >>> print(pad3d) >>> output shape : (10, 106, 108, 110, 3)
Pooling Layers¶
Pool Layer (Expert API)¶
Pooling layer for any dimensions and any pooling functions.
-
class
tensorlayerx.nn.
PoolLayer
(filter_size=(1, 2, 2, 1), strides=(1, 2, 2, 1), padding='SAME', pool=<class 'tensorlayerx.backend.ops.tensorflow_nn.MaxPool'>, name=None)[source]¶ The
PoolLayer
class is a Pooling layer. You can choosetlx.ops.max_pool
andtlx.ops.avg_pool
for 2D input ortlx.ops.max_pool3d
andtlx.ops.avg_pool3d
for 3D input.- Parameters
filter_size (tuple of int) – The size of the window for each dimension of the input tensor. Note that: len(filter_size) >= 4.
strides (tuple of int) – The stride of the sliding window for each dimension of the input tensor. Note that: len(strides) >= 4.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
pool (pooling function) – One of
tlx.ops.max_pool
,tlx.ops.avg_pool
,tlx.ops.max_pool3d
andf.ops.avg_pool3d
. See TensorFlow pooling APIsname (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 50, 32], name='input') >>> net = tlx.nn.PoolLayer()(net) >>> output shape : [10, 25, 25, 32]
1D Max pooling¶
-
class
tensorlayerx.nn.
MaxPool1d
(filter_size=3, strides=2, padding='SAME', data_format='channels_last', name=None)[source]¶ Max pooling for 1D signal.
- Parameters
filter_size (int) – Pooling window size.
strides (int) – Stride of the pooling operation.
padding (str) – The padding method: ‘VALID’ or ‘SAME’.
data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 32], name='input') >>> net = tlx.nn.MaxPool1d(filter_size=3, strides=2, padding='SAME', name='maxpool1d')(net) >>> output shape : [10, 25, 32]
1D Mean pooling¶
-
class
tensorlayerx.nn.
MeanPool1d
(filter_size=3, strides=2, padding='SAME', data_format='channels_last', dilation_rate=1, name=None)[source]¶ Mean pooling for 1D signal.
- Parameters
filter_size (int) – Pooling window size.
strides (int) – Strides of the pooling operation.
padding (str) – The padding method: ‘VALID’ or ‘SAME’.
data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 32], name='input') >>> net = tlx.nn.MeanPool1d(filter_size=3, strides=2, padding='SAME')(net) >>> output shape : [10, 25, 32]
2D Max pooling¶
-
class
tensorlayerx.nn.
MaxPool2d
(filter_size=(3, 3), strides=(2, 2), padding='SAME', data_format='channels_last', name=None)[source]¶ Max pooling for 2D image.
- Parameters
filter_size (tuple of int) – (height, width) for filter size.
strides (tuple of int) – (height, width) for strides.
padding (str) – The padding method: ‘VALID’ or ‘SAME’.
data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 50, 32], name='input') >>> net = tlx.nn.MaxPool2d(filter_size=(3, 3), strides=(2, 2), padding='SAME')(net) >>> output shape : [10, 25, 25, 32]
2D Mean pooling¶
-
class
tensorlayerx.nn.
MeanPool2d
(filter_size=(3, 3), strides=(2, 2), padding='SAME', data_format='channels_last', name=None)[source]¶ Mean pooling for 2D image [batch, height, width, channel].
- Parameters
filter_size (tuple of int) – (height, width) for filter size.
strides (tuple of int) – (height, width) for strides.
padding (str) – The padding method: ‘VALID’ or ‘SAME’.
data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 50, 32], name='input') >>> net = tlx.nn.MeanPool2d(filter_size=(3, 3), strides=(2, 2), padding='SAME')(net) >>> output shape : [10, 25, 25, 32]
3D Max pooling¶
-
class
tensorlayerx.nn.
MaxPool3d
(filter_size=(3, 3, 3), strides=(2, 2, 2), padding='VALID', data_format='channels_last', name=None)[source]¶ Max pooling for 3D volume.
- Parameters
filter_size (tuple of int) – Pooling window size.
strides (tuple of int) – Strides of the pooling operation.
padding (str) – The padding method: ‘VALID’ or ‘SAME’.
data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
- Returns
A max pooling 3-D layer with a output rank as 5.
- Return type
tf.Tensor
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 50, 50, 32], name='input') >>> net = tlx.nn.MaxPool3d(filter_size=(3, 3, 3), strides=(2, 2, 2), padding='SAME')(net) >>> output shape : [10, 25, 25, 25, 32]
3D Mean pooling¶
-
class
tensorlayerx.nn.
MeanPool3d
(filter_size=(3, 3, 3), strides=(2, 2, 2), padding='VALID', data_format='channels_last', name=None)[source]¶ Mean pooling for 3D volume.
- Parameters
filter_size (tuple of int) – Pooling window size.
strides (tuple of int) – Strides of the pooling operation.
padding (str) – The padding method: ‘VALID’ or ‘SAME’.
data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
- Returns
A mean pooling 3-D layer with a output rank as 5.
- Return type
tf.Tensor
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 50, 50, 50, 32], name='input') >>> net = tlx.nn.MeanPool3d(filter_size=(3, 3, 3), strides=(2, 2, 2), padding='SAME')(net) >>> output shape : [10, 25, 25, 25, 32]
1D Global Max pooling¶
-
class
tensorlayerx.nn.
GlobalMaxPool1d
(data_format='channels_last', name=None)[source]¶ The
GlobalMaxPool1d
class is a 1D Global Max Pooling layer.- Parameters
data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 30], name='input') >>> net = tlx.nn.GlobalMaxPool1d()(net) >>> output shape : [10, 30]
1D Global Mean pooling¶
-
class
tensorlayerx.nn.
GlobalMeanPool1d
(data_format='channels_last', name=None)[source]¶ The
GlobalMeanPool1d
class is a 1D Global Mean Pooling layer.- Parameters
data_format (str) – One of channels_last (default, [batch, length, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 30], name='input') >>> net = tlx.nn.GlobalMeanPool1d()(net) >>> output shape : [10, 30]
2D Global Max pooling¶
-
class
tensorlayerx.nn.
GlobalMaxPool2d
(data_format='channels_last', name=None)[source]¶ The
GlobalMaxPool2d
class is a 2D Global Max Pooling layer.- Parameters
data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 100, 30], name='input') >>> net = tlx.nn.GlobalMaxPool2d()(net) >>> output shape : [10, 30]
2D Global Mean pooling¶
-
class
tensorlayerx.nn.
GlobalMeanPool2d
(data_format='channels_last', name=None)[source]¶ The
GlobalMeanPool2d
class is a 2D Global Mean Pooling layer.- Parameters
data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 100, 30], name='input') >>> net = tlx.nn.GlobalMeanPool2d()(net) >>> output shape : [10, 30]
3D Global Max pooling¶
-
class
tensorlayerx.nn.
GlobalMaxPool3d
(data_format='channels_last', name=None)[source]¶ The
GlobalMaxPool3d
class is a 3D Global Max Pooling layer.- Parameters
data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 100, 100, 30], name='input') >>> net = tlx.nn.GlobalMaxPool3d()(net) >>> output shape : [10, 30]
3D Global Mean pooling¶
-
class
tensorlayerx.nn.
GlobalMeanPool3d
(data_format='channels_last', name=None)[source]¶ The
GlobalMeanPool3d
class is a 3D Global Mean Pooling layer.- Parameters
data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 100, 100, 100, 30], name='input') >>> net = tlx.nn.GlobalMeanPool3d()(net) >>> output shape : [10, 30]
1D Adaptive Max pooling¶
-
class
tensorlayerx.nn.
AdaptiveMaxPool1d
(output_size, data_format='channels_last', name=None)[source]¶ The
AdaptiveMaxPool1d
class is a 1D Adaptive Max Pooling layer.- Parameters
output_size (int) – The target output size. It must be an integer.
data_format (str) – One of channels_last (default, [batch, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 32, 3], name='input') >>> net = tlx.nn.AdaptiveMaxPool1d(output_size=16)(net) >>> output shape : [10, 16, 3]
1D Adaptive Mean pooling¶
-
class
tensorlayerx.nn.
AdaptiveMeanPool1d
(output_size, data_format='channels_last', name=None)[source]¶ The
AdaptiveMeanPool1d
class is a 1D Adaptive Mean Pooling layer.- Parameters
output_size (int) – The target output size. It must be an integer.
data_format (str) – One of channels_last (default, [batch, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 32, 3], name='input') >>> net = tlx.nn.AdaptiveMeanPool1d(output_size=16)(net) >>> output shape : [10, 16, 3]
2D Adaptive Max pooling¶
-
class
tensorlayerx.nn.
AdaptiveMaxPool2d
(output_size, data_format='channels_last', name=None)[source]¶ The
AdaptiveMaxPool2d
class is a 2D Adaptive Max Pooling layer.- Parameters
output_size (int or list or tuple) – The target output size. It cloud be an int [int,int](int, int).
data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 32, 32, 3], name='input') >>> net = tlx.nn.AdaptiveMaxPool2d(output_size=16)(net) >>> output shape : [10, 16, 16, 3]
2D Adaptive Mean pooling¶
-
class
tensorlayerx.nn.
AdaptiveMeanPool2d
(output_size, data_format='channels_last', name=None)[source]¶ The
AdaptiveMeanPool2d
class is a 2D Adaptive Mean Pooling layer.- Parameters
output_size (int or list or tuple) – The target output size. It cloud be an int [int,int](int, int).
data_format (str) – One of channels_last (default, [batch, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10,32, 32, 3], name='input') >>> net = tlx.nn.AdaptiveMeanPool2d(output_size=16)(net) >>> output shape : [10,16, 16, 3]
3D Adaptive Max pooling¶
-
class
tensorlayerx.nn.
AdaptiveMaxPool3d
(output_size, data_format='channels_last', name=None)[source]¶ The
AdaptiveMaxPool3d
class is a 3D Adaptive Max Pooling layer.- Parameters
output_size (int or list or tuple) – The target output size. It cloud be an int [int,int,int](int, int, int).
data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10,32, 32, 32, 3], name='input') >>> net = tlx.nn.AdaptiveMaxPool3d(output_size=16)(net) >>> output shape : [10, 16, 16, 16, 3]
3D Adaptive Mean pooling¶
-
class
tensorlayerx.nn.
AdaptiveMeanPool3d
(output_size, data_format='channels_last', name=None)[source]¶ The
AdaptiveMeanPool3d
class is a 3D Adaptive Mean Pooling layer.- Parameters
output_size (int or list or tuple) – The target output size. It cloud be an int [int,int,int](int, int, int).
data_format (str) – One of channels_last (default, [batch, depth, height, width, channel]) or channels_first. The ordering of the dimensions in the inputs.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10,32, 32, 32, 3], name='input') >>> net = tlx.nn.AdaptiveMeanPool3d(output_size=16)(net) >>> output shape : [10, 16, 16, 16, 3]
2D Corner pooling¶
-
class
tensorlayerx.nn.
CornerPool2d
(mode='TopLeft', name=None)[source]¶ Corner pooling for 2D image [batch, height, width, channel], see here.
- Parameters
mode (str) – TopLeft for the top left corner, Bottomright for the bottom right corner.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([10, 32, 32, 8], name='input') >>> net = tlx.nn.CornerPool2d(mode='TopLeft',name='cornerpool2d')(net) >>> output shape : [10, 32, 32, 8]
Quantized Nets¶
This is an experimental API package for building Quantized Neural Networks. We are using matrix multiplication rather than add-minus and bit-count operation at the moment. Therefore, these APIs would not speed up the inferencing, for production, you can train model via TensorLayer and deploy the model into other customized C/C++ implementation (We probably provide users an extra C/C++ binary net framework that can load model from TensorLayer).
Note that, these experimental APIs can be changed in the future.
Scale¶
-
class
tensorlayerx.nn.
Scale
(init_scale=0.05, name='scale')[source]¶ The
Scale
class is to multiple a trainable scale value to the layer outputs. Usually be used on the output of binary net.- Parameters
init_scale (float) – The initial value for the scale factor.
name (a str) – A unique layer name.
Examples
>>> inputs = tlx.nn.Input([8, 3]) >>> dense = tlx.nn.Dense(n_units=10, in_channels=3)(inputs) >>> outputs = tlx.nn.Scale(init_scale=0.5)(dense)
Binary Dense Layer¶
-
class
tensorlayerx.nn.
BinaryDense
(n_units=100, act=None, use_gemm=False, W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
BinaryDense
class is a binary fully connected layer, which weights are either -1 or 1 while inferencing.Note that, the bias vector would not be binarized.
- Parameters
n_units (int) – The number of units of this layer.
act (activation function) – The activation function of this layer, usually set to
tf.act.sign
or applySign
afterBatchNorm
.use_gemm (boolean) – If True, use gemm instead of
tf.matmul
for inference. (TODO).W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip biases.
in_channels (int) – The number of channels of the previous layer. If None, it will be automatically detected when the layer is forwarded for the first time.
name (None or str) – A unique layer name.
Examples
>>> net = tlx.nn.Input([10, 784], name='input') >>> net = tlx.nn.BinaryDense(n_units=800, act=tlx.ReLU, name='relu1')(net) >>> output shape :(10, 800) >>> net = tlx.nn.BinaryDense(n_units=10, name='output')(net) >>> output shape : (10, 10)
Binary (De)Convolutions¶
BinaryConv2d¶
-
class
tensorlayerx.nn.
BinaryConv2d
(n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='VALID', data_format='channels_last', dilation_rate=(1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
BinaryConv2d
class is a 2D binary CNN layer, which weights are either -1 or 1 while inference.Note that, the bias vector would not be binarized.
- Parameters
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (height, width).
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
W_init (initializer or str) – The initializer for the the weight matrix.
b_init (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 100, 100, 32], name='input') >>> binaryconv2d = tlx.nn.BinaryConv2d( ... n_filter=64, filter_size=(3, 3), strides=(2, 2), act=tlx.ReLU, in_channels=32, name='binaryconv2d')(net) >>> print(binaryconv2d) >>> output shape : (8, 50, 50, 64)
Ternary Dense Layer¶
TernaryDense¶
-
class
tensorlayerx.nn.
TernaryDense
(n_units=100, act=None, use_gemm=False, W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
TernaryDense
class is a ternary fully connected layer, which weights are either -1 or 1 or 0 while inference. # TODO The TernaryDense only supports TensorFlow backend.Note that, the bias vector would not be tenaried.
- Parameters
n_units (int) – The number of units of this layer.
act (activation function) – The activation function of this layer, usually set to
tf.act.sign
or applySignLayer
afterBatchNormLayer
.use_gemm (boolean) – If True, use gemm instead of
tf.matmul
for inference. (TODO).W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip biases.
in_channels (int) – The number of channels of the previous layer. If None, it will be automatically detected when the layer is forwarded for the first time.
name (None or str) – A unique layer name.
Ternary Convolutions¶
TernaryConv2d¶
-
class
tensorlayerx.nn.
TernaryConv2d
(n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', use_gemm=False, data_format='channels_last', dilation_rate=(1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
TernaryConv2d
class is a 2D ternary CNN layer, which weights are either -1 or 1 or 0 while inference.Note that, the bias vector would not be tenarized.
- Parameters
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (height, width).
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
use_gemm (boolean) – If True, use gemm instead of
tf.matmul
for inference. TODO: support gemmdata_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
W_init (initializer or str) – The initializer for the the weight matrix.
b_init (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 12, 12, 32], name='input') >>> ternaryconv2d = tlx.nn.TernaryConv2d( ... n_filter=64, filter_size=(5, 5), strides=(1, 1), act=tlx.ReLU, padding='SAME', name='ternaryconv2d' ... )(net) >>> print(ternaryconv2d) >>> output shape : (8, 12, 12, 64)
DorefaDense¶
-
class
tensorlayerx.nn.
DorefaDense
(bitW=1, bitA=3, n_units=100, act=None, use_gemm=False, W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
DorefaDense
class is a binary fully connected layer, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.Note that, the bias vector would not be binarized.
- Parameters
bitW (int) – The bits of this layer’s parameter
bitA (int) – The bits of the output of previous layer
n_units (int) – The number of units of this layer.
act (activation function) – The activation function of this layer, usually set to
tf.act.sign
or applySign
afterBatchNorm
.use_gemm (boolean) – If True, use gemm instead of
tf.matmul
for inferencing. (TODO).W_init (initializer or str) – The initializer for the weight matrix.
b_init (initializer or None or str) – The initializer for the bias vector. If None, skip biases.
in_channels (int) – The number of channels of the previous layer. If None, it will be automatically detected when the layer is forwarded for the first time.
name (a str) – A unique layer name.
Examples
>>> net = tlx.nn.Input([10, 784], name='input') >>> net = tlx.nn.DorefaDense(n_units=800, act=tlx.ReLU, name='relu1')(net) >>> output shape :(10, 800) >>> net = tlx.nn.DorefaDense(n_units=10, name='output')(net) >>> output shape :(10, 10)
DoReFa Convolutions¶
DorefaConv2d¶
-
class
tensorlayerx.nn.
DorefaConv2d
(bitW=1, bitA=3, n_filter=32, filter_size=(3, 3), strides=(1, 1), act=None, padding='SAME', data_format='channels_last', dilation_rate=(1, 1), W_init='truncated_normal', b_init='constant', in_channels=None, name=None)[source]¶ The
DorefaConv2d
class is a 2D quantized convolutional layer, which weights are ‘bitW’ bits and the output of the previous layer are ‘bitA’ bits while inferencing.Note that, the bias vector would not be binarized.
- Parameters
bitW (int) – The bits of this layer’s parameter
bitA (int) – The bits of the output of previous layer
n_filter (int) – The number of filters.
filter_size (tuple of int) – The filter size (height, width).
strides (tuple of int) – The sliding window strides of corresponding input dimensions. It must be in the same order as the
shape
parameter.act (activation function) – The activation function of this layer.
padding (str) – The padding algorithm type: “SAME” or “VALID”.
data_format (str) – “channels_last” (NHWC, default) or “channels_first” (NCHW).
dilation_rate (tuple of int) – Specifying the dilation rate to use for dilated convolution.
W_init (initializer or str) – The initializer for the the weight matrix.
b_init (initializer or None or str) – The initializer for the the bias vector. If None, skip biases.
in_channels (int) – The number of in channels.
name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> net = tlx.nn.Input([8, 12, 12, 32], name='input') >>> dorefaconv2d = tlx.nn.DorefaConv2d( ... n_filter=32, filter_size=(5, 5), strides=(1, 1), act=tlx.ReLU, padding='SAME', name='dorefaconv2d' ... )(net) >>> print(dorefaconv2d) >>> output shape : (8, 12, 12, 32)
Recurrent Layers¶
Common Recurrent layer¶
RNNCell layer¶
-
class
tensorlayerx.nn.
RNNCell
(input_size, hidden_size, bias=True, act='tanh', name=None)[source]¶ An Elman RNN cell with tanh or ReLU non-linearity.
- Parameters
input_size (int) – The number of expected features in the input x
hidden_size (int) – The number of features in the hidden state h
bias (bool) – If
False
, then the layer does not use bias weights b_ih and b_hh. Default:True
act (activation function) – The non-linearity to use. Can be either ‘tanh’ or ‘relu’. Default: ‘tanh’
name (None or str) – A unique layer name
- Returns
outputs (tensor) – A tensor with shape [batch_size, hidden_size].
states (tensor) – A tensor with shape [batch_size, hidden_size]. Tensor containing the next hidden state for each element in the batch
-
forward
(inputs, states=None)[source]¶ - Parameters
inputs (tensor) – A tensor with shape [batch_size, input_size].
states (tensor or None) – A tensor with shape [batch_size, hidden_size]. When states is None, zero state is used. Defaults to None.
Examples
With TensorLayerx
>>> input = tlx.nn.Input([4, 16], name='input') >>> prev_h = tlx.nn.Input([4,32]) >>> cell = tlx.nn.RNNCell(input_size=16, hidden_size=32, bias=True, act='tanh', name='rnncell_1') >>> y, h = cell(input, prev_h) >>> print(y.shape)
LSTMCell layer¶
-
class
tensorlayerx.nn.
LSTMCell
(input_size, hidden_size, bias=True, name=None)[source]¶ A long short-term memory (LSTM) cell.
- Parameters
input_size (int) – The number of expected features in the input x
hidden_size (int) – The number of features in the hidden state h
bias (bool) – If
False
, then the layer does not use bias weights b_ih and b_hh. Default:True
name (None or str) – A unique layer name
- Returns
outputs (tensor) – A tensor with shape [batch_size, hidden_size].
states (tensor) – A tuple of two tensor (h, c), each of shape [batch_size, hidden_size]. Tensors containing the next hidden state and next cell state for each element in the batch.
-
forward
(inputs, states=None)[source]¶ - Parameters
inputs (tensor) – A tensor with shape [batch_size, input_size].
states (tuple or None) – A tuple of two tensor (h, c), each of shape [batch_size, hidden_size]. When states is None, zero state is used. Defaults: None.
Examples
With TensorLayerx
>>> input = tlx.nn.Input([4, 16], name='input') >>> prev_h = tlx.nn.Input([4,32]) >>> prev_c = tlx.nn.Input([4,32]) >>> cell = tlx.nn.LSTMCell(input_size=16, hidden_size=32, bias=True, name='lstmcell_1') >>> y, (h, c)= cell(input, (prev_h, prev_c)) >>> print(y.shape)
GRUCell layer¶
-
class
tensorlayerx.nn.
GRUCell
(input_size, hidden_size, bias=True, name=None)[source]¶ A gated recurrent unit (GRU) cell.
- Parameters
input_size (int) – The number of expected features in the input x
hidden_size (int) – The number of features in the hidden state h
bias (bool) – If
False
, then the layer does not use bias weights b_ih and b_hh. Default:True
name (None or str) – A unique layer name
- Returns
outputs (tensor) – A tensor with shape [batch_size, hidden_size].
states (tensor) – A tensor with shape [batch_size, hidden_size]. Tensor containing the next hidden state for each element in the batch
-
forward
(inputs, states=None)[source]¶ - Parameters
inputs (tensor) – A tensor with shape [batch_size, input_size].
states (tensor or None) – A tensor with shape [batch_size, hidden_size]. When states is None, zero state is used. Defaults: None.
Examples
With TensorLayerx
>>> input = tlx.nn.Input([4, 16], name='input') >>> prev_h = tlx.nn.Input([4,32]) >>> cell = tlx.nn.GRUCell(input_size=16, hidden_size=32, bias=True, name='grucell_1') >>> y, h= cell(input, prev_h) >>> print(y.shape)
RNN layer¶
-
class
tensorlayerx.nn.
RNN
(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, act='tanh', name=None)[source]¶ Multilayer Elman network(RNN). It takes input sequences and initial states as inputs, and returns the output sequences and the final states.
- Parameters
input_size (int) – The number of expected features in the input x
hidden_size (int) – The number of features in the hidden state h
num_layers (int) – Number of recurrent layers. Default: 1
bias (bool) – If
False
, then the layer does not use bias weights b_ih and b_hh. Default:True
batch_first (bool) – If
True
, then the input and output tensors are provided as [batch_size, seq, input_size], Default:False
dropout (float) – If non-zero, introduces a Dropout layer on the outputs of each RNN layer except the last layer, with dropout probability equal to dropout. Default: 0
bidirectional (bool) – If
True
, becomes a bidirectional RNN. Default:False
act (activation function) – The non-linearity to use. Can be either ‘tanh’ or ‘relu’. Default: ‘tanh’
name (None or str) – A unique layer name
- Returns
outputs (tensor) – the output sequence. if batch_first is True, the shape is [batch_size, seq, num_directions * hidden_size], else, the shape is [seq, batch_size, num_directions * hidden_size].
final_states (tensor) – final states. The shape is [num_layers * num_directions, batch_size, hidden_size]. Note that if the RNN is Bidirectional, the forward states are (0,2,4,6,…) and the backward states are (1,3,5,7,….).
-
forward
(input, states=None)[source]¶ - Parameters
inputs (tensor) – the input sequence. if batch_first is True, the shape is [batch_size, seq, input_size], else, the shape is [seq, batch_size, input_size].
initial_states (tensor or None) – the initial states. The shape is [num_layers * num_directions, batch_size, hidden_size].If initial_state is not given, zero initial states are used. If the RNN is Bidirectional, num_directions should be 2, else it should be 1. Default: None.
Examples
With TensorLayer
>>> input = tlx.nn.Input([23, 32, 16], name='input') >>> prev_h = tlx.nn.Input([4, 32, 32]) >>> cell = tlx.nn.RNN(input_size=16, hidden_size=32, bias=True, num_layers=2, bidirectional = True, act='tanh', batch_first=False, dropout=0, name='rnn_1') >>> y, h= cell(input, prev_h) >>> print(y.shape)
LSTM layer¶
-
class
tensorlayerx.nn.
LSTM
(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, name=None)[source]¶ Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence.
- Parameters
input_size (int) – The number of expected features in the input x
hidden_size (int) – The number of features in the hidden state h
num_layers (int) – Number of recurrent layers. Default: 1
bias (bool) – If
False
, then the layer does not use bias weights b_ih and b_hh. Default:True
batch_first (bool) – If
True
, then the input and output tensors are provided as [batch_size, seq, input_size], Default:False
dropout (float) – If non-zero, introduces a Dropout layer on the outputs of each LSTM layer except the last layer, with dropout probability equal to dropout. Default: 0
bidirectional (bool) – If
True
, becomes a bidirectional LSTM. Default:False
name (None or str) – A unique layer name
- Returns
outputs (tensor) – the output sequence. if batch_first is True, the shape is [batch_size, seq, num_directions * hidden_size], else, the shape is [seq, batch_size, num_directions * hidden_size].
final_states (tensor) – final states. A tuple of two tensor. The shape of each is [num_layers * num_directions, batch_size, hidden_size]. Note that if the LSTM is Bidirectional, the forward states are (0,2,4,6,…) and the backward states are (1,3,5,7,….).
-
forward
(input, states=None)[source]¶ - Parameters
inputs (tensor) – the input sequence. if batch_first is True, the shape is [batch_size, seq, input_size], else, the shape is [seq, batch_size, input_size].
initial_states (tensor or None) – the initial states. A tuple of tensor (h, c), the shape of each is [num_layers * num_directions, batch_size, hidden_size].If initial_state is not given, zero initial states are used. If the LSTM is Bidirectional, num_directions should be 2, else it should be 1. Default: None.
Examples
With TensorLayerx
>>> input = tlx.nn.Input([23, 32, 16], name='input') >>> prev_h = tlx.nn.Input([4, 32, 32]) >>> prev_c = tlx.nn.Input([4, 32, 32]) >>> cell = tlx.nn.LSTM(input_size=16, hidden_size=32, bias=True, num_layers=2, bidirectional = True, batch_first=False, dropout=0, name='lstm_1') >>> y, (h, c)= cell(input, (prev_h, prev_c)) >>> print(y.shape)
GRU layer¶
-
class
tensorlayerx.nn.
GRU
(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, name=None)[source]¶ Applies a multi-layer gated recurrent unit (GRU) RNN to an input sequence.
- Parameters
input_size (int) – The number of expected features in the input x
hidden_size (int) – The number of features in the hidden state h
num_layers (int) – Number of recurrent layers. Default: 1
bias (bool) – If
False
, then the layer does not use bias weights b_ih and b_hh. Default:True
batch_first (bool) – If
True
, then the input and output tensors are provided as [batch_size, seq, input_size], Default:False
dropout (float) – If non-zero, introduces a Dropout layer on the outputs of each GRU layer except the last layer, with dropout probability equal to dropout. Default: 0
bidirectional (bool) – If
True
, becomes a bidirectional LSTM. Default:False
name (None or str) – A unique layer name
- Returns
outputs (tensor) – the output sequence. if batch_first is True, the shape is [batch_size, seq, num_directions * hidden_size], else, the shape is [seq, batch_size, num_directions * hidden_size].
final_states (tensor) – final states. A tuple of two tensor. The shape of each is [num_layers * num_directions, batch_size, hidden_size]. Note that if the GRU is Bidirectional, the forward states are (0,2,4,6,…) and the backward states are (1,3,5,7,….).
-
forward
(input, states=None)[source]¶ - Parameters
inputs (tensor) – the input sequence. if batch_first is True, the shape is [batch_size, seq, input_size], else, the shape is [seq, batch_size, input_size].
initial_states (tensor or None) – the initial states. A tuple of tensor (h, c), the shape of each is [num_layers * num_directions, batch_size, hidden_size].If initial_state is not given, zero initial states are used. If the GRU is Bidirectional, num_directions should be 2, else it should be 1. Default: None.
Examples
With TensorLayerx
>>> input = tlx.nn.Input([23, 32, 16], name='input') >>> prev_h = tlx.nn.Input([4, 32, 32]) >>> cell = tlx.nn.GRU(input_size=16, hidden_size=32, bias=True, num_layers=2, bidirectional = True, batch_first=False, dropout=0, name='GRU_1') >>> y, h= cell(input, prev_h) >>> print(y.shape)
Transformer Layers¶
Transformer layer¶
MultiheadAttention layer¶
-
class
tensorlayerx.nn.
MultiheadAttention
(embed_dim, num_heads, dropout=0.0, kdim=None, vdim=None, bias=True, batch_first=False, need_weights=True, name=None)[source]¶ Allows the model to jointly attend to information from different representation subspaces.
- Parameters
embed_dim (int) – total dimension of the model.
num_heads (int) – The number of heads in multi-head attention.
dropout (float) – a Dropout layer on attn_output_weights. Default: 0.0.
kdim (int) – total number of features in key. Default: None.
vdim (int) – total number of features in value. Default: None.
bias (bool) – add bias as module parameter. Default: True.
batch_first (bool) – If
True
, then the input and output tensors are provided as [batch, seq, feature]. Default:False
[seq, batch, feature].need_weights (bool) – Indicate whether to return the attention weights. Default
False
.name (None or str) – A unique layer name.
Examples
With TensorLayer
>>> q = tlx.nn.Input(shape=(4,2,128),init=tlx.initializers.ones()) >>> attn_mask = tlx.convert_to_tensor(np.zeros((4,4)),dtype='bool') >>> layer = MultiheadAttention(embed_dim=128, num_heads=4) >>> output = layer(q, attn_mask=attn_mask)
References
-
forward
(q, k=None, v=None, attn_mask=None, key_padding_mask=None)[source]¶ - Parameters
q (Tensor) – The queries for multi-head attention. If batch_first is
True
, it is a tensor with shape [batch_size, query_length, embed_dim]. If batch_first isFalse
, it is a tensor with shape [query_length, batch_size, embed_dim]. The data type should be float32 or float64.k (Tensor) – The keys for multi-head attention. It is a tensor with shape [batch_size, key_length, kdim]. If batch_first is
False
, it is a tensor with shape [key_length, batch_size, kdim]. The data type should be float32 or float64. If None, use query as key. Default is None.v (Tensor) – The values for multi-head attention. It is a tensor with shape [batch_size, value_length, vdim]. If batch_first is
False
, it is a tensor with shape [value_length, batch_size, vdim]. The data type should be float32 or float64. If None, use value as key. Default is None.attn_mask (Tensor) – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch. if a 2D mask: \((L, S)\) where L is the target sequence length, S is the source sequence length. if a 3D mask: \((N\cdot ext{num\_heads}, L, S)\). Where N is the batch size, L is the target sequence length, S is the source sequence length.
attn_mask
ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions withTrue
is not allowed to attend whileFalse
values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight.key_padding_mask (Tensor) – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored \((N, S)\) where N is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of
True
will be ignored while the position with the value ofFalse
will be unchanged.
- Returns
attn_output (Tensor) – \((L, N, E)\) where L is the target sequence length, N is the batch size, E is the embedding dimension. \((N, L, E)\) if
batch_first
isTrue
.attn_output_weights – \((N, L, S)\) where N is the batch size, L is the target sequence length, S is the source sequence length.
Transformer layer¶
-
class
tensorlayerx.nn.
Transformer
(d_model=512, nhead=8, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=2048, dropout=0.1, act='relu', custom_encoder=None, custom_decoder=None, layer_norm_eps=1e-05, batch_first=False)[source]¶ A transformer model. User is able to modify the attributes as needed.
- Parameters
d_model (int) – the number of expected features in the encoder/decoder inputs.
nhead (int) – the number of heads in the multiheadattention model.
num_encoder_layers – the number of sub-encoder-layers in the encoder.
num_decoder_layers – the number of sub-decoder-layers in the decoder.
dim_feedforward (int) – the dimension of the feedforward network model.
dropout (float) – a Dropout layer on attn_output_weights. Default: 0.0.
act (str) – the activation function of encoder/decoder intermediate layer, ‘relu’ or ‘gelu’. Default: ‘relu’.
custom_encoder (Module or None) – custom encoder.
custom_decoder (Module or None) – custom decoder
layer_norm_eps (float) – the eps value in layer normalization components. Default: 1e-5.
batch_first (bool) – If
True
, then the input and output tensors are provided as [batch, seq, feature]. Default:False
[seq, batch, feature].
Examples
With TensorLayer
>>> src = tlx.nn.Input(shape=(4,2,128),init=tlx.initializers.ones()) >>> tgt = tlx.nn.Input(shape=(4,2,128),init=tlx.initializers.ones()) >>> layer = Transformer(d_model=128, nhead=4) >>> output = layer(src, tgt)
References
-
forward
(src, tgt, src_mask=None, tgt_mask=None, memory_mask=None, src_key_padding_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)[source]¶ - Parameters
src (Tensor) – the sequence to the encoder.
tgt (Tensor) – the sequence to the decoder.
src_mask (Tensor) – the additive mask for the src sequence.
tgt_mask (Tensor) – the additive mask for the tgt sequence.
memory_mask (Tensor) – the additive mask for the encoder output.
src_key_padding_mask (Tensor) – mask for src keys per batch.
tgt_key_padding_mask (Tensor) – mask for tgt keys per batch.
memory_key_padding_mask (Tensor) – mask for memory keys per batch.
-
generate_square_subsequent_mask
(length)[source]¶ Generate a square mask for the sequence. The masked positions are filled with float(‘-inf’). Unmasked positions are filled with float(0.0).
- Parameters
length (int) – The length of sequence.
Examples
With TensorLayer
>>> length = 5 >>> mask = transformer.generate_square_subsequent_mask(length) >>> print(mask) >>> [[ 0. -inf -inf -inf -inf] >>> [ 0. 0. -inf -inf -inf] >>> [ 0. 0. 0. -inf -inf] >>> [ 0. 0. 0. 0. -inf] >>> [ 0. 0. 0. 0. 0.]]
TransformerEncoder layer¶
-
class
tensorlayerx.nn.
TransformerEncoder
(encoder_layer, num_layers, norm=None)[source]¶ TransformerEncoder is a stack of N encoder layers
- Parameters
encoder_layer (Module) – an instance of the TransformerEncoderLayer() class.
num_layers (int) – the number of sub-encoder-layers in the encoder.
norm (None) – the layer normalization component.
Examples
With TensorLayer
>>> q = tlx.nn.Input(shape=(4,2,128),init=tlx.initializers.ones()) >>> attn_mask = tlx.convert_to_tensor(np.zeros((4,4)),dtype='bool') >>> encoder = TransformerEncoderLayer(128, 2, 256) >>> encoder = TransformerEncoder(encoder, num_layers=3) >>> output = encoder(q, mask=attn_mask)
TransformerDecoder layer¶
-
class
tensorlayerx.nn.
TransformerDecoder
(decoder_layer, num_layers, norm=None)[source]¶ TransformerDecoder is a stack of N decoder layers
- Parameters
decoder_layer (Module) – an instance of the TransformerDecoderLayer() class.
num_layers (int) – the number of sub-decoder-layers in the decoder.
norm (None) – the layer normalization component.
Examples
With TensorLayer
>>> q = tlx.nn.Input(shape=(4,2,128),init=tlx.initializers.ones()) >>> decoder = TransformerDecoderLayer(128, 2, 256) >>> decoder = TransformerDecoder(decoder, num_layers=3) >>> output = decoder(q, q)
-
forward
(tgt, memory, tgt_mask=None, memory_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)[source]¶ - Parameters
tgt (Tensor) – the sequence to the decoder.
memory (Tensor) – the sequence from the last layer of the encoder.
tgt_mask (Tensor) – the mask for the tgt sequence.
memory_mask (Tensor) – the mask for the memory sequence.
tgt_key_padding_mask (Tensor) – the mask for the tgt keys per batch.
memory_key_padding_mask (Tensor) – the mask for the memory keys per batch.
TransformerEncoderLayer layer¶
-
class
tensorlayerx.nn.
TransformerEncoderLayer
(d_model, nhead, dim_feedforward, dropout=0.1, act='relu', layer_norm_eps=1e-05, batch_first=False)[source]¶ TransformerEncoderLayer is made up of self-attn and feedforward network. This standard encoder layer is based on the paper “Attention Is All You Need”.
- Parameters
d_model (int) – total dimension of the model.
nhead (int) – The number of heads in multi-head attention.
dim_feedforward (int) – the dimension of the feedforward network model.
dropout (float) – a Dropout layer on attn_output_weights. Default: 0.1.
act (str) – The activation function in the feedforward network. ‘relu’ or ‘gelu’. Default ‘relu’.
layer_norm_eps (float) – the eps value in layer normalization components. Default 1e-5.
batch_first (bool) – If
True
, then the input and output tensors are provided as [batch, seq, feature]. Default:False
[seq, batch, feature].
Examples
With TensorLayer
>>> q = tlx.nn.Input(shape=(4,2,128),init=tlx.initializers.ones()) >>> attn_mask = tlx.convert_to_tensor(np.zeros((4,4)),dtype='bool') >>> encoder = TransformerEncoderLayer(128, 2, 256) >>> output = encoder(q, src_mask=attn_mask)
TransformerDecoderLayer layer¶
-
class
tensorlayerx.nn.
TransformerDecoderLayer
(d_model, nhead, dim_feedforward, dropout=0.1, act='relu', layer_norm_eps=1e-05, batch_first=False)[source]¶ TransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network. This standard decoder layer is based on the paper “Attention Is All You Need”.
- Parameters
d_model (int) – total dimension of the model.
nhead (int) – The number of heads in multi-head attention.
dim_feedforward (int) – the dimension of the feedforward network model.
dropout (float) – a Dropout layer on attn_output_weights. Default: 0.1.
act (str) – The activation function in the feedforward network. ‘relu’ or ‘gelu’. Default ‘relu’.
layer_norm_eps (float) – the eps value in layer normalization components. Default 1e-5.
batch_first (bool) – If
True
, then the input and output tensors are provided as [batch, seq, feature]. Default:False
[seq, batch, feature].
Examples
With TensorLayer
>>> q = tlx.nn.Input(shape=(4,2,128),init=tlx.initializers.ones()) >>> encoder = TransformerDecoderLayer(128, 2, 256) >>> output = encoder(q, q)
-
forward
(tgt, memory, tgt_mask=None, memory_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)[source]¶ - Parameters
tgt (Tensor) – the sequence to the decoder layer.
memory – the sequence from the last layer of the encoder.
tgt_mask – the mask for the tgt sequence.
memory_mask – the mask for the memory sequence.
tgt_key_padding_mask – the mask for the tgt keys per batch.
memory_key_padding_mask – the mask for the memory keys per batch.
Shape Layers¶
Flatten Layer¶
-
class
tensorlayerx.nn.
Flatten
(name=None)[source]¶ A layer that reshapes high-dimension input into a vector.
Then we often apply Dense, RNN, Concat and etc on the top of a flatten layer. [batch_size, mask_row, mask_col, n_mask] —> [batch_size, mask_row * mask_col * n_mask]
- Parameters
name (None or str) – A unique layer name.
Examples
>>> x = tlx.nn.Input([8, 4, 3], name='input') >>> y = tlx.nn.Flatten(name='flatten')(x) [8, 12]
Reshape Layer¶
-
class
tensorlayerx.nn.
Reshape
(shape, name=None)[source]¶ A layer that reshapes a given tensor.
- Parameters
shape (tuple of int) – The output shape, see
tf.reshape
.name (str) – A unique layer name.
Examples
>>> x = tlx.nn.Input([8, 4, 3], name='input') >>> y = tlx.nn.Reshape(shape=[-1, 12], name='reshape')(x) (8, 12)
Transpose Layer¶
-
class
tensorlayerx.nn.
Transpose
(perm=None, conjugate=False, name=None)[source]¶ A layer that transposes the dimension of a tensor.
See tf.transpose() .
- Parameters
perm (list of int or None) – The permutation of the dimensions, similar with
numpy.transpose
. If None, it is set to (n-1…0), where n is the rank of the input tensor.conjugate (bool) – By default False. If True, returns the complex conjugate of complex numbers (and transposed) For example [[1+1j, 2+2j]] –> [[1-1j], [2-2j]]
name (str) – A unique layer name.
Examples
>>> x = tlx.nn.Input([8, 4, 3], name='input') >>> y = tlx.nn.Transpose(perm=[0, 2, 1], conjugate=False, name='trans')(x) (8, 3, 4)
Shuffle Layer¶
-
class
tensorlayerx.nn.
Shuffle
(group, in_channels=None, name=None)[source]¶ A layer that shuffle a 2D image [batch, height, width, channel], see here.
- Parameters
group (int) – The number of groups.
name (str) – A unique layer name.
Examples
>>> x = tlx.nn.Input([1, 16, 16, 8], name='input') >>> y = tlx.nn.Shuffle(group=2, name='shuffle')(x) (1, 16, 16, 8)
Stack Layer¶
Stack Layer¶
-
class
tensorlayerx.nn.
Stack
(axis=1, name=None)[source]¶ The
Stack
class is a layer for stacking a list of rank-R tensors into one rank-(R+1) tensor, see tf.stack().- Parameters
axis (int) – New dimension along which to stack.
name (str) – A unique layer name.
Examples
>>> import tensorlayerx as tlx >>> ni = tlx.nn.Input([10, 784], name='input') >>> net1 = tlx.nn.Dense(10, name='dense1')(ni) >>> net2 = tlx.nn.Dense(10, name='dense2')(ni) >>> net3 = tlx.nn.Dense(10, name='dense3')(ni) >>> net = tlx.nn.Stack(axis=1, name='stack')([net1, net2, net3]) (10, 3, 10)
Unstack Layer¶
-
class
tensorlayerx.nn.
UnStack
(num=None, axis=0, name=None)[source]¶ The
UnStack
class is a layer for unstacking the given dimension of a rank-R tensor into rank-(R-1) tensors., see tf.unstack().- Parameters
num (int or None) – The length of the dimension axis. Automatically inferred if None (the default).
axis (int) – Dimension along which axis to concatenate.
name (str) – A unique layer name.
- Returns
The list of layer objects unstacked from the input.
- Return type
list of
Layer
Examples
>>> ni = tlx.nn.Input([4, 10], name='input') >>> nn = tlx.nn.Dense(n_units=5)(ni) >>> nn = tlx.nn.UnStack(axis=1)(nn) # unstack in channel axis >>> len(nn) # 5 >>> nn[0].shape # (4,)