Core¶

Cells¶

default_rnn_cell_hparams¶

texar.tf.core.default_rnn_cell_hparams()[source]¶

Returns a dict of RNN cell hyperparameters and their default values.

{
    "type": "LSTMCell",
    "kwargs": {
        "num_units": 256
    },
    "num_layers": 1,
    "dropout": {
        "input_keep_prob": 1.0,
        "output_keep_prob": 1.0,
        "state_keep_prob": 1.0,
        "variational_recurrent": False,
        "input_size": []
    },
    "residual": False,
    "highway": False,
}

Here:

“type”: str or cell class or cell instance

The RNN cell type. This can be

The string name or full module path of a cell class. If class name is provided, the class must be in module tf.nn.rnn_cell, tf.contrib.rnn, or texar.tf.custom.
A cell class.
An instance of a cell class. This is not valid if “num_layers” > 1.

For example

"type": "LSTMCell" # class name
"type": "tensorflow.contrib.rnn.Conv1DLSTMCell" # module path
"type": "my_module.MyCell" # module path
"type": tf.nn.rnn_cell.GRUCell # class
"type": BasicRNNCell(num_units=100) # cell instance
"type": MyCell(...) # cell instance

“kwargs”: dict

Keyword arguments for the constructor of the cell class. A cell is created by cell_class(**kwargs), where cell_class is specified in “type” above.

Ignored if “type” is a cell instance.

“num_layers”: int

Number of cell layers. Each layer is a cell created as above, with the same hyperparameters specified in “kwargs”.

“dropout”: dict

Dropout applied to the cell in each layer. See DropoutWrapper for details of the hyperparameters. If all “*_keep_prob” = 1, no dropout is applied.

Specifically, if “variational_recurrent” = True, the same dropout mask is applied across all time steps per run call. If True, “input_size” is required, which is a list of input size of each cell layer. The input size of a cell layer is the last dimension size of its input tensor. For example, the input size of the first layer is usually the dimension of word embeddings, while the input size of subsequent layers are usually the num_units of the preceding-layer cell. E.g.,

# Assume embedding_dim = 100
"type": "LSTMCell",
"kwargs": { "num_units": 123 },
"num_layers": 3,
"dropout": {
    "output_keep_prob": 0.5,
    "variational_recurrent": True,
    "input_size": [100, 123, 123]
}

“residual”: bool

If True, apply residual connection on the inputs and outputs of cell in each layer except the first layer. Ignored if “num_layers” = 1.

“highway”: bool

If True, apply highway connection on the inputs and outputs of cell in each layer except the first layer. Ignored if “num_layers” = 1.

get_rnn_cell¶

texar.tf.core.get_rnn_cell(hparams=None, mode=None)[source]¶

Creates an RNN cell.

See default_rnn_cell_hparams() for all hyperparameters and default values.

Parameters:	hparams (dict or HParams, optional) – Cell hyperparameters. Missing hyperparameters are set to default values. mode (optional) – A Tensor taking value in tf.estimator.ModeKeys, including TRAIN, EVAL, and PREDICT. If None, dropout will be controlled by `texar.tf.global_mode()`.
Returns:	A cell instance.
Raises:	`ValueError` – If hparams[“num_layers”]>1 and hparams[“type”] is a class instance. `ValueError` – The cell is not an RNNCell instance.

get_rnn_cell_trainable_variables¶

texar.tf.core.get_rnn_cell_trainable_variables(cell)[source]¶

Returns the list of trainable variables of an RNN cell.

Parameters:	cell – an instance of RNNCell.
Returns:	trainable variables of the cell.
Return type:	list

Layers¶

get_layer¶

texar.tf.core.get_layer(hparams)[source]¶

Makes a layer instance.

The layer must be an instance of tf.layers.Layer.

Parameters:

hparams (dict or HParams) –

Hyperparameters of the layer, with structure:

{
    "type": "LayerClass",
    "kwargs": {
        # Keyword arguments of the layer class
        # ...
    }
}

Here:

”type”: str or layer class or layer instance

The layer type. This can be

The string name or full module path of a layer class. If the class name is provided, the class must be in module tf.layers, texar.tf.core, or texar.tf.custom.
A layer class.
An instance of a layer class.

For example

"type": "Conv1D" # class name
"type": "texar.tf.core.MaxReducePooling1D" # module path
"type": "my_module.MyLayer" # module path
"type": tf.layers.Conv2D # class
"type": Conv1D(filters=10, kernel_size=2) # cell instance
"type": MyLayer(...) # cell instance

”kwargs”: dict

A dictionary of keyword arguments for constructor of the layer class. Ignored if "type" is a layer instance.

Arguments named “activation” can be a callable, or a str of the name or module path to the activation function.
Arguments named “*_regularizer” and “*_initializer” can be a class instance, or a dict of hyperparameters of respective regularizers and initializers. See
Arguments named “*_constraint” can be a callable, or a str of the name or full path to the constraint function.

Returns:

A layer instance. If hparams["type"] is a layer instance, returns it directly.

Raises:

ValueError – If hparams is None.
ValueError – If the resulting layer is not an instance of tf.layers.Layer.

MaxReducePooling1D¶

class texar.tf.core.MaxReducePooling1D(data_format='channels_last', name=None, **kwargs)[source]¶: A subclass of tf.layers.Layer. Max Pooling layer for 1D inputs. The same as MaxPooling1D except that the pooling dimension is entirely reduced (i.e., pool_size=input_length).

AverageReducePooling1D¶

class texar.tf.core.AverageReducePooling1D(data_format='channels_last', name=None, **kwargs)[source]¶: A subclass of tf.layers.Layer. Average Pooling layer for 1D inputs. The same as AveragePooling1D except that the pooling dimension is entirely reduced (i.e., pool_size=input_length).

get_pooling_layer_hparams¶

texar.tf.core.get_pooling_layer_hparams(hparams)[source]¶

Creates pooling layer hparams dict usable for get_layer().

If the hparams sets ‘pool_size’ to None, the layer will be changed to the respective reduce-pooling layer. For example, tf.layers.MaxPooling1D is replaced with MaxReducePooling1D.

MergeLayer¶

class texar.tf.core.MergeLayer(layers=None, mode='concat', axis=1, trainable=True, name=None, **kwargs)[source]¶

A subclass of tf.layers.Layer. A layer that consists of multiple layers in parallel. Input is fed to each of the parallel layers, and the outputs are merged with a specified mode.

Parameters:

layers (list, optional) –
A list of tf.layers.Layer instances, or a list of hyperparameter dicts each of which specifies type and kwargs of each layer (see the hparams argument of get_layer()).

If None, this layer degenerates to a merging operator that merges inputs directly.
mode (str) –
Mode of the merge op. This can be:
- 'concat': Concatenates layer outputs along one axis. Tensors must have the same shape except for the dimension specified in axis, which can have different sizes.
- 'elemwise_sum': Outputs element-wise sum.
- 'elemwise_mul': Outputs element-wise product.
- 'sum': Computes the sum of layer outputs along the dimension given by axis. E.g., given axis=1, two tensors of shape [a, b] and [a, c] respectively will result in a merged tensor of shape [a].
- 'mean': Computes the mean of layer outputs along the dimension given in axis.
- 'prod': Computes the product of layer outputs along the dimension given in axis.
- 'max': Computes the maximum of layer outputs along the dimension given in axis.
- 'min': Computes the minimum of layer outputs along the dimension given in axis.
- 'and': Computes the logical and of layer outputs along the dimension given in axis.
- 'or': Computes the logical or of layer outputs along the dimension given in axis.
- 'logsumexp': Computes log(sum(exp(elements across the dimension of layer outputs)))
axis (int) – The axis to use in merging. Ignored in modes 'elemwise_sum' and 'elemwise_mul'.
trainable (bool) – Whether the layer should be trained.
name (str, optional) – Name of the layer.

compute_output_shape(input_shape)[source]¶

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Parameters:	input_shape – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:	An input shape tuple.

call(inputs)[source]¶

This is where the layer’s logic lives.

Parameters:	inputs – Input tensor, or list/tuple of input tensors. **kwargs – Additional keyword arguments.
Returns:	A tensor or list/tuple of tensors.

layers¶: The list of parallel layers.

SequentialLayer¶

class texar.tf.core.SequentialLayer(layers, trainable=True, name=None, **kwargs)[source]¶

A subclass of tf.layers.Layer. A layer that consists of multiple layers connected sequentially.

Parameters:	layers (list) – A list of tf.layers.Layer instances, or a list of hyperparameter dicts each of which specifying type and kwargs of each layer (see the hparams argument of `get_layer()`). The layers are connected sequentially.

compute_output_shape(input_shape)[source]¶

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Parameters:	input_shape – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:	An input shape tuple.

call(inputs, mode=None)[source]¶

This is where the layer’s logic lives.

Parameters:	inputs – Input tensor, or list/tuple of input tensors. **kwargs – Additional keyword arguments.
Returns:	A tensor or list/tuple of tensors.

layers¶: The list of layers connected sequentially.

default_regularizer_hparams¶

texar.tf.core.default_regularizer_hparams()[source]¶

Returns the hyperparameters and their default values of a variable regularizer:

{
    "type": "L1L2",
    "kwargs": {
        "l1": 0.,
        "l2": 0.
    }
}

The default value corresponds to L1L2 and, with (l1=0, l2=0), disables regularization.

get_regularizer¶

texar.tf.core.get_regularizer(hparams=None)[source]¶

Returns a variable regularizer instance.

See default_regularizer_hparams() for all hyperparameters and default values.

The “type” field can be a subclass of Regularizer, its string name or module path, or a class instance.

Parameters:	hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values.
Returns:	A Regularizer instance. None if `hparams` is None or taking the default hyperparameter value.
Raises:	`ValueError` – The resulting regularizer is not an instance of Regularizer.

get_initializer¶

texar.tf.core.get_initializer(hparams=None)[source]¶

Returns an initializer instance.

Parameters:

hparams (dict or HParams, optional) –

Hyperparameters with the structure

{
    "type": "initializer_class_or_function",
    "kwargs": {
        #...
    }
}

The “type” field can be a initializer class, its name or module path, or class instance. If class name is provided, the class must be from one the following modules: tf.initializers, tf.keras.initializers, tf, and texar.tf.custom. The class is created by initializer_class(**kwargs). If a class instance is given, “kwargs” is ignored and can be omitted.

Besides, the “type” field can also be an initialization function called with initialization_fn(**kwargs). In this case “type” can be the function, or its name or module path. If function name is provided, the function must be from one of the above modules or module tf.contrib.layers. If no keyword argument is required, “kwargs” can be omitted.

Returns: An initializer instance. None if hparams is None.

get_activation_fn¶

texar.tf.core.get_activation_fn(fn_name='identity', kwargs=None)[source]¶

Returns an activation function fn with the signature output = fn(input).

If the function specified by fn_name has more than one arguments without default values, then all these arguments except the input feature argument must be specified in kwargs. Arguments with default values can also be specified in kwargs to take values other than the defaults. In this case a partial function is returned with the above signature.

Parameters:

fn_name (str or callable) –
An activation function, or its name or module path. The function can be:
- Built-in function defined in tf or tf.nn, e.g., tf.identity.
- User-defined activation functions in module texar.tf.custom.
- External activation functions. Must provide the full module path, e.g., “my_module.my_activation_fn”.
kwargs (optional) – A dict or instance of HParams containing the keyword arguments of the activation function.

Returns:

An activation function. None if fn_name is None.

get_constraint_fn¶

texar.tf.core.get_constraint_fn(fn_name='NonNeg')[source]¶

Returns a constraint function.

The function must follow the signature: w_ = constraint_fn(w).

Parameters:

fn_name (str or callable) –

The name or full path to a constraint function, or the function itself.

The function can be:

Built-in constraint functions defined in modules tf.keras.constraints (e.g., NonNeg) or tf or tf.nn (e.g., activation functions).
User-defined function in texar.tf.custom.
Externally defined function. Must provide the full path, e.g., “my_module.my_constraint_fn”.

If a callable is provided, then it is returned directly.

Returns: The constraint function. None if fn_name is None.

default_conv1d_kwargs¶

texar.tf.core.default_conv1d_kwargs()[source]¶

Returns the default keyword argument values of the constructor of 1D-convolution layer class tf.layers.Conv1D.

{
    "filters": 100,
    "kernel_size": 3,
    "strides": 1,
    "padding": 'valid',
    "data_format": 'channels_last',
    "dilation_rate": 1
    "activation": "identity",
    "use_bias": True,
    "kernel_initializer": {
        "type": "glorot_uniform_initializer",
        "kwargs": {}
    },
    "bias_initializer": {
        "type": "zeros_initializer",
        "kwargs": {}
    },
    "kernel_regularizer": {
        "type": "L1L2",
        "kwargs": {
            "l1": 0.,
            "l2": 0.
        }
    },
    "bias_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "activity_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "kernel_constraint": None,
    "bias_constraint": None,
    "trainable": True,
    "name": None
}

default_dense_kwargs¶

texar.tf.core.default_dense_kwargs()[source]¶

Returns the default keyword argument values of the constructor of the dense layer class tf.layers.Dense.

{
    "units": 256,
    "activation": "identity",
    "use_bias": True,
    "kernel_initializer": {
        "type": "glorot_uniform_initializer",
        "kwargs": {}
    },
    "bias_initializer": {
        "type": "zeros_initializer",
        "kwargs": {}
    },
    "kernel_regularizer": {
        "type": "L1L2",
        "kwargs": {
            "l1": 0.,
            "l2": 0.
        }
    },
    "bias_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "activity_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "kernel_constraint": None,
    "bias_constraint": None,
    "trainable": True,
    "name": None
}

Optimization¶

default_optimization_hparams¶

texar.tf.core.default_optimization_hparams()[source]¶

Returns a dict of default hyperparameters of training op and their default values

{
    "optimizer": {
        "type": "AdamOptimizer",
        "kwargs": {
            "learning_rate": 0.001
        }
    },
    "learning_rate_decay": {
        "type": "",
        "kwargs": {},
        "min_learning_rate": 0.,
        "start_decay_step": 0,
        "end_decay_step": inf
    },
    "gradient_clip": {
        "type": "",
        "kwargs": {}
    },
    "gradient_noise_scale": None,
    "name": None
}

Here:

“optimizer”: dict

Hyperparameters of a tf.train.Optimizer.

“type” specifies the optimizer class. This can be
- The string name or full module path of an optimizer class. If the class name is provided, the class must be in module tf.train, tf.contrib.opt or texar.tf.custom , texar.tf.core.optimization
- An optimizer class.
- An instance of an optimizer class.
For example
"type": "AdamOptimizer" # class name "type": "my_module.MyOptimizer" # module path "type": tf.contrib.opt.AdamWOptimizer # class "type": my_module.MyOptimizer # class "type": GradientDescentOptimizer(learning_rate=0.1) # instance "type": MyOptimizer(...) # instance
“kwargs” is a dict specifying keyword arguments for creating the optimizer class instance, with opt_class(**kwargs). Ignored if “type” is a class instance.

“learning_rate_decay”: dict

Hyperparameters of learning rate decay function. The learning rate starts decay from "start_decay_step" and keeps unchanged after "end_decay_step" or reaching "min_learning_rate".

The decay function is specified in “type” and “kwargs”.

“type” can be a decay function or its name or module path. If function name is provided, it must be from module tf.train or texar.tf.custom, texar.tf.core.optimization.

“kwargs” is a dict of keyword arguments for the function excluding arguments named “global_step” and “learning_rate”.

The function is called with lr = decay_fn(learning_rate=lr, global_step=offset_step, **kwargs), where offset_step is the global step offset as above. The only exception is tf.train.piecewise_constant which is called with lr = piecewise_constant(x=offset_step, **kwargs).

“gradient_clip”: dict

Hyperparameters of gradient clipping. The gradient clipping function takes a list of (gradients, variables) tuples and returns a list of (clipped_gradients, variables) tuples. Typical examples include tf.clip_by_global_norm, tf.clip_by_value, tf.clip_by_norm, tf.clip_by_average_norm, etc.

“type” specifies the gradient clip function, and can be a function, or its name or mudule path. If function name is provided, the function must be from module tf or texar.tf.custom, texar.tf.core.optimization.

“kwargs” specifies keyword arguments to the function, except arguments named “t” or “t_list”.

The function is called with clipped_grads(, _) = clip_fn(t_list=grads, **kwargs) (e.g., for tf.clip_by_global_norm) or clipped_grads = [clip_fn(t=grad, **kwargs) for grad in grads] (e.g., for tf.clip_by_value).

“gradient_noise_scale”: float, optional

Adds 0-mean normal noise scaled by this value to gradient.

get_train_op¶

texar.tf.core.get_train_op(loss, variables=None, optimizer=None, learning_rate=None, global_step=None, increment_global_step=True, hparams=None)[source]¶

Creates a training op.

This is a wrapper of tf.contrib.layers.optimize_loss.

Parameters:	loss – A scalar Tensor representing the loss to minimize. variables (optional) – A list of Variables to optimize. If None, all trainable variables are used. optimizer (optional) – An tf.train.Optimizer instance. If None, use the setting in hparams to create the optimizer. learning_rate (float or Tensor, optional) – If None, learning rate specified in `hparams`, or the default learning rate of the optimizer will be used (if exists). global_step (optional) – A scalar int Tensor. Step counter to update on each step unless `increment_global_step` is False. Learning rate decay uses `global_step`. If None, it will be fetched from the default graph (see tf.train.get_global_step for more details). If it has not been created, no step will be incremented with each weight update. increment_global_step (bool) – Whether to increment `global_step`. This is useful if the `global_step` is used in multiple training ops per training step (e.g. to optimize different parts of the model) to avoid incrementing `global_step` more times than necessary. hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically. See `default_optimization_hparams()` for all hyperparameters and default values.
Returns:	the operator used for variables optimization.
Return type:	train_op

get_optimizer_fn¶

texar.tf.core.get_optimizer_fn(hparams=None)[source]¶

Returns a function optimizer_fn of making optimizer instance, along with the optimizer class.

The function has the signiture optimizer_fn(learning_rate=None) -> optimizer class instance

See the "optimizer" field of default_optimization_hparams() for all hyperparameters and default values.

The optimizer class must be a subclass of tf.train.Optimizer.

Parameters:	hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically.
Returns:	If hparams[“type”] is a string or optimizer class, returns (optimizer_fn, optimizer class), If hparams[“type”] is an optimizer instance, returns (the optimizer instance, optimizer class)

get_optimizer¶

texar.tf.core.get_optimizer(learning_rate=None, global_step=None, hparams=None)[source]¶

Creates a optimizer instance.

Parameters:	learning_rate (float or Tensor, optional) – If None, learning rate specified in `hparams`, or the default learning rate of the optimizer (if exists) is used. global_step (optional) – A scalar int Tensor. Step counter to update on each step unless `increment_global_step` is False. Learning rate decay uses `global_step`. If None, it will be fetched from the default graph (see tf.train.get_global_step for more details). If it has not been created, no step will be incremented with each weight update. hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically. See `default_optimization_hparams()` for all hyperparameters and default values.
Returns:	the tf.train.Optimizer instance specified in hparams.
Return type:	optimizer

get_learning_rate_decay_fn¶

texar.tf.core.get_learning_rate_decay_fn(hparams=None)[source]¶

Creates learning rate decay function based on the hyperparameters.

See the learning_rate_decay field in default_optimization_hparams() for all hyperparameters and default values.

Parameters:	hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically.
Returns:	If hparams[“type”] is specified, returns a function that takes (learning_rate, step, **kwargs) and returns a decayed learning rate. If hparams[“type”] is empty, returns None.
Return type:	function or None

get_gradient_clip_fn¶

texar.tf.core.get_gradient_clip_fn(hparams=None)[source]¶

Creates a gradient clipping function based on the hyperparameters.

See the gradient_clip field in default_optimization_hparams() for all hyperparameters and default values.

The gradient clipping function takes a list of (gradients, variables) tuples and returns a list of (clipped_gradients, variables) tuples. Typical examples include tf.clip_by_global_norm, tf.clip_by_value, tf.clip_by_norm, tf.clip_by_average_norm, etc.

Parameters:	hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically.
Returns:	If hparams[“type”] is specified, returns the respective function. If hparams[“type”] is empty, returns None.
Return type:	function or None

Exploration¶

EpsilonLinearDecayExploration¶

class texar.tf.core.EpsilonLinearDecayExploration(hparams=None)[source]¶

Decays epsilon linearly.

Parameters:	hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See `default_hparams()` for the defaults.

static default_hparams()[source]¶

Returns a dict of hyperparameters and their default values.

{
    'initial_epsilon': 0.1,
    'final_epsilon': 0.0,
    'decay_timesteps': 20000,
    'start_timestep': 0,
    'name': 'epsilon_linear_decay_exploration',
}

This specifies the decay process that starts at “start_timestep” with the value “initial_epsilon”, and decays for steps “decay_timesteps” to reach the final epsilon value “final_epsilon”.

get_epsilon(timestep)[source]¶

Returns the epsilon value.

Parameters:	timestep (int) – The time step.
Returns:	the epsilon value.
Return type:	float

ExplorationBase¶

class texar.tf.core.ExplorationBase(hparams=None)[source]¶

Base class inherited by all exploration classes.

Parameters:	hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See `default_hparams()` for the defaults.

static default_hparams()[source]¶

Returns a dict of hyperparameters and their default values.

{
    'name': 'exploration_base'
}

get_epsilon(timestep)[source]¶

Returns the epsilon value.

Parameters:	timestep (int) – The time step.
Returns:	the epsilon value.
Return type:	float

hparams¶: The hyperparameter.

Replay Memories¶

DequeReplayMemory¶

class texar.tf.core.DequeReplayMemory(hparams=None)[source]¶

A deque based replay memory that accepts new memory entry and deletes oldest memory entry if exceeding the capacity. Memory entries are accessed in random order.

Parameters:	hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See `default_hparams()` for the defaults.

static default_hparams()[source]¶

Returns a dict of hyperparameters and their default values.

{
    'capacity': 80000,
    'name': 'deque_replay_memory',
}

Here:

“capacity”: int: Maximum size of memory kept. Deletes oldest memories if exceeds the capacity.

add(element)[source]¶: Appends element to the memory and deletes old memory if exceeds the capacity.

get(size)[source]¶: Randomly samples size entries from the memory. Returns a list.

last()[source]¶: Returns the latest element in the memeory.

size()[source]¶: Returns the current size of the memory.

ReplayMemoryBase¶

class texar.tf.core.ReplayMemoryBase(hparams=None)[source]¶

Base class of replay memory inheritted by all replay memory classes.

Parameters:	hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See `default_hparams()` for the defaults.

static default_hparams()[source]¶

Returns a dict of hyperparameters and their default values.

{
    'name': 'replay_memory'
}

add(element)[source]¶: Inserts a memory entry

get(size)[source]¶: Pops a memory entry.

last()[source]¶: Returns the latest element in the memeory.

size()[source]¶: Returns the current size of the memory.