Core

Cells

default_rnn_cell_hparams

texar.core.default_rnn_cell_hparams()[source]

Returns a dict of RNN cell hyperparameters and their default values.

{
    "type": "LSTMCell",
    "kwargs": {
        "num_units": 256
    },
    "num_layers": 1,
    "dropout": {
        "input_keep_prob": 1.0,
        "output_keep_prob": 1.0,
        "state_keep_prob": 1.0,
        "variational_recurrent": False,
        "input_size": []
    },
    "residual": False,
    "highway": False,
}

Here:

“type” : str or cell class or cell instance

The RNN cell type. This can be

  • The string name or full module path of a cell class. If class name is provided, the class must be in module tf.nn.rnn_cell, tf.contrib.rnn, or texar.custom.
  • A cell class.
  • An instance of a cell class. This is not valid if “num_layers” > 1.

For example

"type": "LSTMCell" # class name
"type": "tensorflow.contrib.rnn.Conv1DLSTMCell" # module path
"type": "my_module.MyCell" # module path
"type": tf.nn.rnn_cell.GRUCell # class
"type": BasicRNNCell(num_units=100) # cell instance
"type": MyCell(...) # cell instance
“kwargs” : dict

Keyword arguments for the constructor of the cell class. A cell is created by cell_class(**kwargs), where cell_class is specified in “type” above.

Ignored if “type” is a cell instance.

“num_layers” : int
Number of cell layers. Each layer is a cell created as above, with the same hyperparameters specified in “kwargs”.
“dropout” : dict

Dropout applied to the cell in each layer. See DropoutWrapper for details of the hyperparameters. If all “*_keep_prob” = 1, no dropout is applied.

Specifically, if “variational_recurrent” = True, the same dropout mask is applied across all time steps per run call. If True, “input_size” is required, which is a list of input size of each cell layer. The input size of a cell layer is the last dimension size of its input tensor. For example, the input size of the first layer is usually the dimension of word embeddings, while the input size of subsequent layers are usually the num_units of the preceding-layer cell. E.g.,

# Assume embedding_dim = 100
"type": "LSTMCell",
"kwargs": { "num_units": 123 },
"num_layers": 3,
"dropout": {
    "output_keep_prob": 0.5,
    "variational_recurrent": True,
    "input_size": [100, 123, 123]
}
“residual” : bool
If True, apply residual connection on the inputs and outputs of cell in each layer except the first layer. Ignored if “num_layers” = 1.
“highway” : bool
If True, apply highway connection on the inputs and outputs of cell in each layer except the first layer. Ignored if “num_layers” = 1.

get_rnn_cell

texar.core.get_rnn_cell(hparams=None, mode=None)[source]

Creates an RNN cell.

See default_rnn_cell_hparams() for all hyperparameters and default values.

Parameters:
  • hparams (dict or HParams, optional) – Cell hyperparameters. Missing hyperparameters are set to default values.
  • mode (optional) – A Tensor taking value in tf.estimator.ModeKeys, including TRAIN, EVAL, and PREDICT. If None, dropout will be controlled by texar.global_mode().
Returns:

A cell instance.

Raises:
  • ValueError – If hparams[“num_layers”]>1 and hparams[“type”] is a class instance.
  • ValueError – The cell is not an RNNCell instance.

get_rnn_cell_trainable_variables

texar.core.get_rnn_cell_trainable_variables(cell)[source]

Returns the list of trainable variables of an RNN cell.

Parameters:cell – an instance of RNNCell.
Returns:trainable variables of the cell.
Return type:list

Layers

get_layer

texar.core.get_layer(hparams)[source]

Makes a layer instance.

The layer must be an instance of tf.layers.Layer.

Parameters:

hparams (dict or HParams) –

Hyperparameters of the layer, with structure:

{
    "type": "LayerClass",
    "kwargs": {
        # Keyword arguments of the layer class
        # ...
    }
}

Here:

”type” : str or layer class or layer instance

The layer type. This can be

  • The string name or full module path of a layer class. If the class name is provided, the class must be in module tf.layers, texar.core, or texar.custom.
  • A layer class.
  • An instance of a layer class.

For example

"type": "Conv1D" # class name
"type": "texar.core.MaxReducePooling1D" # module path
"type": "my_module.MyLayer" # module path
"type": tf.layers.Conv2D # class
"type": Conv1D(filters=10, kernel_size=2) # cell instance
"type": MyLayer(...) # cell instance
”kwargs” : dict

A dictionary of keyword arguments for constructor of the layer class. Ignored if "type" is a layer instance.

  • Arguments named “activation” can be a callable, or a str of the name or module path to the activation function.
  • Arguments named “*_regularizer” and “*_initializer” can be a class instance, or a dict of hyperparameters of respective regularizers and initializers. See
  • Arguments named “*_constraint” can be a callable, or a str of the name or full path to the constraint function.

Returns:

A layer instance. If hparams[“type”] is a layer instance, returns it directly.

Raises:
  • ValueError – If hparams is None.
  • ValueError – If the resulting layer is not an instance of tf.layers.Layer.

MaxReducePooling1D

class texar.core.MaxReducePooling1D(data_format='channels_last', name=None, **kwargs)[source]

A subclass of tf.layers.Layer. Max Pooling layer for 1D inputs. The same as MaxPooling1D except that the pooling dimension is entirely reduced (i.e., pool_size=input_length).

AverageReducePooling1D

class texar.core.AverageReducePooling1D(data_format='channels_last', name=None, **kwargs)[source]

A subclass of tf.layers.Layer. Average Pooling layer for 1D inputs. The same as AveragePooling1D except that the pooling dimension is entirely reduced (i.e., pool_size=input_length).

get_pooling_layer_hparams

texar.core.get_pooling_layer_hparams(hparams)[source]

Creates pooling layer hparams dict usable for get_layer().

If the hparams sets ‘pool_size’ to None, the layer will be changed to the respective reduce-pooling layer. For example, tf.layers.MaxPooling1D is replaced with MaxReducePooling1D.

MergeLayer

class texar.core.MergeLayer(layers=None, mode='concat', axis=1, trainable=True, name=None, **kwargs)[source]

A subclass of tf.layers.Layer. A layer that consists of multiple layers in parallel. Input is fed to each of the parallel layers, and the outputs are merged with a specified mode.

Parameters:
  • layers (list, optional) –

    A list of tf.layers.Layer instances, or a list of hyperparameter dicts each of which specifies type and kwargs of each layer (see the hparams argument of get_layer()).

    If None, this layer degenerates to a merging operator that merges inputs directly.

  • mode (str) –

    Mode of the merge op. This can be:

    • 'concat': Concatenates layer outputs along one axis. Tensors must have the same shape except for the dimension specified in axis, which can have different sizes.
    • 'elemwise_sum': Outputs element-wise sum.
    • 'elemwise_mul': Outputs element-wise product.
    • 'sum': Computes the sum of layer outputs along the dimension given by axis. E.g., given axis=1, two tensors of shape [a, b] and [a, c] respectively will result in a merged tensor of shape [a].
    • 'mean': Computes the mean of layer outputs along the dimension given in axis.
    • 'prod': Computes the product of layer outputs along the dimension given in axis.
    • 'max': Computes the maximum of layer outputs along the dimension given in axis.
    • 'min': Computes the minimum of layer outputs along the dimension given in axis.
    • 'and': Computes the logical and of layer outputs along the dimension given in axis.
    • 'or': Computes the logical or of layer outputs along the dimension given in axis.
    • 'logsumexp': Computes log(sum(exp(elements across the dimension of layer outputs)))
  • axis (int) – The axis to use in merging. Ignored in modes 'elemwise_sum' and 'elemwise_mul'.
  • trainable (bool) – Whether the layer should be trained.
  • name (str, optional) – Name of the layer.
compute_output_shape(input_shape)[source]

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Parameters:input_shape – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:An input shape tuple.
call(inputs)[source]

This is where the layer’s logic lives.

Parameters:
  • inputs – Input tensor, or list/tuple of input tensors.
  • **kwargs – Additional keyword arguments.
Returns:

A tensor or list/tuple of tensors.

layers

The list of parallel layers.

SequentialLayer

class texar.core.SequentialLayer(layers, trainable=True, name=None, **kwargs)[source]

A subclass of tf.layers.Layer. A layer that consists of multiple layers connected sequentially.

Parameters:layers (list) – A list of tf.layers.Layer instances, or a list of hyperparameter dicts each of which specifying type and kwargs of each layer (see the hparams argument of get_layer()). The layers are connected sequentially.
compute_output_shape(input_shape)[source]

Computes the output shape of the layer.

Assumes that the layer will be built to match that input shape provided.

Parameters:input_shape – Shape tuple (tuple of integers) or list of shape tuples (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:An input shape tuple.
call(inputs, mode=None)[source]

This is where the layer’s logic lives.

Parameters:
  • inputs – Input tensor, or list/tuple of input tensors.
  • **kwargs – Additional keyword arguments.
Returns:

A tensor or list/tuple of tensors.

layers

The list of layers connected sequentially.

default_regularizer_hparams

texar.core.default_regularizer_hparams()[source]

Returns the hyperparameters and their default values of a variable regularizer:

{
    "type": "L1L2",
    "kwargs": {
        "l1": 0.,
        "l2": 0.
    }
}

The default value corresponds to L1L2 and, with (l1=0, l2=0), disables regularization.

get_regularizer

texar.core.get_regularizer(hparams=None)[source]

Returns a variable regularizer instance.

See default_regularizer_hparams() for all hyperparameters and default values.

The “type” field can be a subclass of Regularizer, its string name or module path, or a class instance.

Parameters:hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values.
Returns:A Regularizer instance. None if hparams is None or taking the default hyperparameter value.
Raises:ValueError – The resulting regularizer is not an instance of Regularizer.

get_initializer

texar.core.get_initializer(hparams=None)[source]

Returns an initializer instance.

Parameters:hparams (dict or HParams, optional) –

Hyperparameters with the structure

{
    "type": "initializer_class_or_function",
    "kwargs": {
        #...
    }
}

The “type” field can be a initializer class, its name or module path, or class instance. If class name is provided, the class must be from one the following modules: tf.initializers, tf.keras.initializers, tf, and texar.custom. The class is created by initializer_class(**kwargs). If a class instance is given, “kwargs” is ignored and can be omitted.

Besides, the “type” field can also be an initialization function called with initialization_fn(**kwargs). In this case “type” can be the function, or its name or module path. If function name is provided, the function must be from one of the above modules or module tf.contrib.layers. If no keyword argument is required, “kwargs” can be omitted.

Returns:An initializer instance. None if hparams is None.

get_activation_fn

texar.core.get_activation_fn(fn_name='identity', kwargs=None)[source]

Returns an activation function fn with the signature output = fn(input).

If the function specified by fn_name has more than one arguments without default values, then all these arguments except the input feature argument must be specified in kwargs. Arguments with default values can also be specified in kwargs to take values other than the defaults. In this case a partial function is returned with the above signature.

Parameters:
  • fn_name (str or callable) –

    An activation function, or its name or module path. The function can be:

    • Built-in function defined in tf or tf.nn, e.g., tf.identity.
    • User-defined activation functions in module texar.custom.
    • External activation functions. Must provide the full module path, e.g., “my_module.my_activation_fn”.
  • kwargs (optional) – A dict or instance of HParams containing the keyword arguments of the activation function.
Returns:

An activation function. None if fn_name is None.

get_constraint_fn

texar.core.get_constraint_fn(fn_name='NonNeg')[source]

Returns a constraint function.

The function must follow the signature: w_ = constraint_fn(w).

Parameters:fn_name (str or callable) –

The name or full path to a constraint function, or the function itself.

The function can be:

  • Built-in constraint functions defined in modules tf.keras.constraints (e.g., NonNeg) or tf or tf.nn (e.g., activation functions).
  • User-defined function in texar.custom.
  • Externally defined function. Must provide the full path, e.g., “my_module.my_constraint_fn”.

If a callable is provided, then it is returned directly.

Returns:The constraint function. None if fn_name is None.

default_conv1d_kwargs

texar.core.default_conv1d_kwargs()[source]

Returns the default keyword argument values of the constructor of 1D-convolution layer class tf.layers.Conv1D.

{
    "filters": 100,
    "kernel_size": 3,
    "strides": 1,
    "padding": 'valid',
    "data_format": 'channels_last',
    "dilation_rate": 1
    "activation": "identity",
    "use_bias": True,
    "kernel_initializer": {
        "type": "glorot_uniform_initializer",
        "kwargs": {}
    },
    "bias_initializer": {
        "type": "zeros_initializer",
        "kwargs": {}
    },
    "kernel_regularizer": {
        "type": "L1L2",
        "kwargs": {
            "l1": 0.,
            "l2": 0.
        }
    },
    "bias_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "activity_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "kernel_constraint": None,
    "bias_constraint": None,
    "trainable": True,
    "name": None
}

default_dense_kwargs

texar.core.default_dense_kwargs()[source]

Returns the default keyword argument values of the constructor of the dense layer class tf.layers.Dense.

{
    "units": 256,
    "activation": "identity",
    "use_bias": True,
    "kernel_initializer": {
        "type": "glorot_uniform_initializer",
        "kwargs": {}
    },
    "bias_initializer": {
        "type": "zeros_initializer",
        "kwargs": {}
    },
    "kernel_regularizer": {
        "type": "L1L2",
        "kwargs": {
            "l1": 0.,
            "l2": 0.
        }
    },
    "bias_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "activity_regularizer": {
        # same as in "kernel_regularizer"
        # ...
    },
    "kernel_constraint": None,
    "bias_constraint": None,
    "trainable": True,
    "name": None
}

Optimization

default_optimization_hparams

texar.core.default_optimization_hparams()[source]

Returns a dict of default hyperparameters of training op and their default values

{
    "optimizer": {
        "type": "AdamOptimizer",
        "kwargs": {
            "learning_rate": 0.001
        }
    },
    "learning_rate_decay": {
        "type": "",
        "kwargs": {},
        "min_learning_rate": 0.,
        "start_decay_step": 0,
        "end_decay_step": inf
    },
    "gradient_clip": {
        "type": "",
        "kwargs": {}
    },
    "gradient_noise_scale": None,
    "name": None
}

Here:

“optimizer” : dict

Hyperparameters of a tf.train.Optimizer.

  • “type” specifies the optimizer class. This can be

    • The string name or full module path of an optimizer class. If the class name is provided, the class must be in module tf.train, tf.contrib.opt or texar.custom , texar.core.optimization
    • An optimizer class.
    • An instance of an optimizer class.

    For example

    "type": "AdamOptimizer" # class name
    "type": "my_module.MyOptimizer" # module path
    "type": tf.contrib.opt.AdamWOptimizer # class
    "type": my_module.MyOptimizer # class
    "type": GradientDescentOptimizer(learning_rate=0.1) # instance
    "type": MyOptimizer(...) # instance
    
  • “kwargs” is a dict specifying keyword arguments for creating the optimizer class instance, with opt_class(**kwargs). Ignored if “type” is a class instance.

“learning_rate_decay” : dict

Hyperparameters of learning rate decay function. The learning rate starts decay from "start_decay_step" and keeps unchanged after "end_decay_step" or reaching "min_learning_rate".

The decay function is specified in “type” and “kwargs”.

  • “type” can be a decay function or its name or module path. If function name is provided, it must be from module tf.train or texar.custom, texar.core.optimization.
  • “kwargs” is a dict of keyword arguments for the function excluding arguments named “global_step” and “learning_rate”.

The function is called with lr = decay_fn(learning_rate=lr, global_step=offset_step, **kwargs), where offset_step is the global step offset as above. The only exception is tf.train.piecewise_constant which is called with lr = piecewise_constant(x=offset_step, **kwargs).

“gradient_clip” : dict

Hyperparameters of gradient clipping. The gradient clipping function takes a list of (gradients, variables) tuples and returns a list of (clipped_gradients, variables) tuples. Typical examples include tf.clip_by_global_norm, tf.clip_by_value, tf.clip_by_norm, tf.clip_by_average_norm, etc.

“type” specifies the gradient clip function, and can be a function, or its name or mudule path. If function name is provided, the function must be from module tf or texar.custom, texar.core.optimization.

“kwargs” specifies keyword arguments to the function, except arguments named “t” or “t_list”.

The function is called with clipped_grads(, _) = clip_fn(t_list=grads, **kwargs) (e.g., for tf.clip_by_global_norm) or clipped_grads = [clip_fn(t=grad, **kwargs) for grad in grads] (e.g., for tf.clip_by_value).

“gradient_noise_scale” : float, optional
Adds 0-mean normal noise scaled by this value to gradient.

get_train_op

texar.core.get_train_op(loss, variables=None, learning_rate=None, global_step=None, increment_global_step=True, hparams=None)[source]

Creates a training op.

This is a wrapper of tf.contrib.layers.optimize_loss.

Parameters:
  • loss – A scalar Tensor representing the loss to minimize.
  • variables (optional) – A list of Variables to optimize. If None, all trainable variables are used.
  • learning_rate (float or Tensor, optional) – If None, learning rate specified in hparams, or the default learning rate of the optimizer will be used (if exists).
  • global_step (optional) – A scalar int Tensor. Step counter to update on each step unless increment_global_step is False. Learning rate decay uses global_step. If None, it will be fetched from the default graph (see tf.train.get_global_step for more details). If it has not been created, no step will be incremented with each weight update.
  • increment_global_step (bool) – Whether to increment global_step. This is useful if the global_step is used in multiple training ops per training step (e.g. to optimize different parts of the model) to avoid incrementing global_step more times than necessary.
  • hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically. See default_optimization_hparams() for all hyperparameters and default values.
Returns:

(train_op, global_step). If global_step is provided, the same global_step variable is returned, otherwise a new global step is created and returned.

Return type:

tuple

get_optimizer_fn

texar.core.get_optimizer_fn(hparams=None)[source]

Returns a function optimizer_fn of making optimizer instance, along with the optimizer class.

The function has the signiture optimizer_fn(learning_rate=None) -> optimizer class instance

See the "optimizer" field of default_optimization_hparams() for all hyperparameters and default values.

The optimizer class must be a subclass of tf.train.Optimizer.

Parameters:hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically.
Returns:
  • If hparams[“type”] is a string or optimizer class, returns (optimizer_fn, optimizer class),
  • If hparams[“type”] is an optimizer instance, returns (the optimizer instance, optimizer class)

get_learning_rate_decay_fn

texar.core.get_learning_rate_decay_fn(hparams=None)[source]

Creates learning rate decay function based on the hyperparameters.

See the learning_rate_decay field in default_optimization_hparams() for all hyperparameters and default values.

Parameters:hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically.
Returns:If hparams[“type”] is specified, returns a function that takes (learning_rate, step, **kwargs) and returns a decayed learning rate. If hparams[“type”] is empty, returns None.
Return type:function or None

get_gradient_clip_fn

texar.core.get_gradient_clip_fn(hparams=None)[source]

Creates a gradient clipping function based on the hyperparameters.

See the gradient_clip field in default_optimization_hparams() for all hyperparameters and default values.

The gradient clipping function takes a list of (gradients, variables) tuples and returns a list of (clipped_gradients, variables) tuples. Typical examples include tf.clip_by_global_norm, tf.clip_by_value, tf.clip_by_norm, tf.clip_by_average_norm, etc.

Parameters:hparams (dict or HParams, optional) – hyperparameters. Missing hyperparameters are set to default values automatically.
Returns:If hparams[“type”] is specified, returns the respective function. If hparams[“type”] is empty, returns None.
Return type:function or None

Exploration

EpsilonLinearDecayExploration

class texar.core.EpsilonLinearDecayExploration(hparams=None)[source]

Decays epsilon linearly.

Parameters:hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See default_hparams() for the defaults.
static default_hparams()[source]

Returns a dict of hyperparameters and their default values.

{
    'initial_epsilon': 0.1,
    'final_epsilon': 0.0,
    'decay_timesteps': 20000,
    'start_timestep': 0,
    'name': 'epsilon_linear_decay_exploration',
}

This specifies the decay process that starts at “start_timestep” with the value “initial_epsilon”, and decays for steps “decay_timesteps” to reach the final epsilon value “final_epsilon”.

get_epsilon(timestep)[source]

Returns the epsilon value.

Parameters:timestep (int) – The time step.
Returns:the epsilon value.
Return type:float

ExplorationBase

class texar.core.ExplorationBase(hparams=None)[source]

Base class inherited by all exploration classes.

Parameters:hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See default_hparams() for the defaults.
static default_hparams()[source]

Returns a dict of hyperparameters and their default values.

{
    'name': 'exploration_base'
}
get_epsilon(timestep)[source]

Returns the epsilon value.

Parameters:timestep (int) – The time step.
Returns:the epsilon value.
Return type:float
hparams

The hyperparameter.

Replay Memories

DequeReplayMemory

class texar.core.DequeReplayMemory(hparams=None)[source]

A deque based replay memory that accepts new memory entry and deletes oldest memory entry if exceeding the capacity. Memory entries are accessed in random order.

Parameters:hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See default_hparams() for the defaults.
static default_hparams()[source]

Returns a dict of hyperparameters and their default values.

{
    'capacity': 80000,
    'name': 'deque_replay_memory',
}

Here:

“capacity” : int
Maximum size of memory kept. Deletes oldest memories if exceeds the capacity.
add(element)[source]

Appends element to the memory and deletes old memory if exceeds the capacity.

get(size)[source]

Randomly samples size entries from the memory. Returns a list.

last()[source]

Returns the latest element in the memeory.

size()[source]

Returns the current size of the memory.

ReplayMemoryBase

class texar.core.ReplayMemoryBase(hparams=None)[source]

Base class of replay memory inheritted by all replay memory classes.

Parameters:hparams (dict or HParams, optional) – Hyperparameters. Missing hyperparameters are set to default values. See default_hparams() for the defaults.
static default_hparams()[source]

Returns a dict of hyperparameters and their default values.

{
    'name': 'replay_memory'
}
add(element)[source]

Inserts a memory entry

get(size)[source]

Pops a memory entry.

last()[source]

Returns the latest element in the memeory.

size()[source]

Returns the current size of the memory.