Pipe operator

Description

See magrittr::%>% for details.

Usage

lhs %>% rhs

Class representing the context.

Description

Class representing the context.

Class representing the context.

Public fields

Active bindings

Methods

Public methods

• AutogradContext$new()

• AutogradContext$save_for_backward()

• AutogradContext$mark_non_differentiable()

• AutogradContext$mark_dirty()

• AutogradContext$clone()

Method new()

(Dev related) Initializes the context. Not user related.

Usage

AutogradContext$new(
  ptr,
  env,
  argument_names = NULL,
  argument_needs_grad = NULL
)

Arguments

Method save_for_backward()

Saves given objects for a future call to backward().

This should be called at most once, and only from inside the forward() method.

Later, saved objects can be accessed through the saved_variables attribute. Before returning them to the user, a check is made to ensure they weren’t used in any in-place operation that modified their content.

Arguments can also be any kind of R object.

Usage

AutogradContext$save_for_backward(...)

Arguments

Method mark_non_differentiable()

Marks outputs as non-differentiable.

This should be called at most once, only from inside the forward() method, and all arguments should be outputs.

This will mark outputs as not requiring gradients, increasing the efficiency of backward computation. You still need to accept a gradient for each output in backward(), but it’s always going to be a zero tensor with the same shape as the shape of a corresponding output.

This is used e.g. for indices returned from a max Function.

Usage

AutogradContext$mark_non_differentiable(...)

Arguments

Method mark_dirty()

Marks given tensors as modified in an in-place operation.

This should be called at most once, only from inside the forward() method, and all arguments should be inputs.

Every tensor that’s been modified in-place in a call to forward() should be given to this function, to ensure correctness of our checks. It doesn’t matter whether the function is called before or after modification.

Usage

AutogradContext$mark_dirty(...)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

AutogradContext$clone(deep = FALSE)

Arguments

Abstract base class for constraints.

Description

Abstract base class for constraints.

Abstract base class for constraints.

Details

A constraint object represents a region over which a variable is valid, e.g. within which a variable can be optimized.

Methods

Public methods

• Constraint$check()

• Constraint$print()

• Constraint$clone()

Method check()

Returns a byte tensor of sample_shape + batch_shape indicating whether each event in value satisfies this constraint.

Usage

Constraint$check(value)

Arguments

Method print()

Define the print method for constraints,

Usage

Constraint$print()

Method clone()

The objects of this class are cloneable with this method.

Usage

Constraint$clone(deep = FALSE)

Arguments

Generic R6 class representing distributions

Description

Distribution is the abstract base class for probability distributions. Note: in Python, adding torch.Size objects works as concatenation Try for example: torch.Size((2, 1)) + torch.Size((1,))

Public fields

Active bindings

Methods

Public methods

• Distribution$new()

• Distribution$expand()

• Distribution$sample()

• Distribution$rsample()

• Distribution$log_prob()

• Distribution$cdf()

• Distribution$icdf()

• Distribution$enumerate_support()

• Distribution$entropy()

• Distribution$perplexity()

• Distribution$.extended_shape()

• Distribution$.validate_sample()

• Distribution$print()

• Distribution$clone()

Method new()

Initializes a distribution class.

Usage

Distribution$new(batch_shape = NULL, event_shape = NULL, validate_args = NULL)

Arguments

Method expand()

Returns a new distribution instance (or populates an existing instance provided by a derived class) with batch dimensions expanded to batch_shape. This method calls expand on the distribution’s parameters. As such, this does not allocate new memory for the expanded distribution instance. Additionally, this does not repeat any args checking or parameter broadcasting in initialize, when an instance is first created.

Usage

Distribution$expand(batch_shape, .instance = NULL)

Arguments

Method sample()

Generates a sample_shape shaped sample or sample_shape shaped batch of samples if the distribution parameters are batched.

Usage

Distribution$sample(sample_shape = NULL)

Arguments

Method rsample()

Generates a sample_shape shaped reparameterized sample or sample_shape shaped batch of reparameterized samples if the distribution parameters are batched.

Usage

Distribution$rsample(sample_shape = NULL)

Arguments

Method log_prob()

Returns the log of the probability density/mass function evaluated at value.

Usage

Distribution$log_prob(value)

Arguments

Method cdf()

Returns the cumulative density/mass function evaluated at value.

Usage

Distribution$cdf(value)

Arguments

Method icdf()

Returns the inverse cumulative density/mass function evaluated at value.

@description Returns tensor containing all values supported by a discrete distribution. The result will enumerate over dimension 0, so the shape of the result will be ⁠(cardinality,) + batch_shape + event_shape (where ⁠event_shape = ()⁠for univariate distributions). Note that this enumerates over all batched tensors in lock-step⁠list(c(0, 0), c(1, 1), ...)⁠. With ⁠expand=FALSE⁠, enumeration happens along dim 0, but with the remaining batch dimensions being singleton dimensions, ⁠list(c(0), c(1), ...)'.

Usage

Distribution$icdf(value)

Arguments

Method enumerate_support()

Usage

Distribution$enumerate_support(expand = TRUE)

Arguments

Returns

Tensor iterating over dimension 0.

Method entropy()

Returns entropy of distribution, batched over batch_shape.

Usage

Distribution$entropy()

Returns

Tensor of shape batch_shape.

Method perplexity()

Returns perplexity of distribution, batched over batch_shape.

Usage

Distribution$perplexity()

Returns

Tensor of shape batch_shape.

Method .extended_shape()

Returns the size of the sample returned by the distribution, given a sample_shape. Note, that the batch and event shapes of a distribution instance are fixed at the time of construction. If this is empty, the returned shape is upcast to (1,).

Usage

Distribution$.extended_shape(sample_shape = NULL)

Arguments

Method .validate_sample()

Argument validation for distribution methods such as log_prob, cdf and icdf. The rightmost dimensions of a value to be scored via these methods must agree with the distribution's batch and event shapes.

Usage

Distribution$.validate_sample(value)

Arguments

Method print()

Prints the distribution instance.

Usage

Distribution$print()

Method clone()

The objects of this class are cloneable with this method.

Usage

Distribution$clone(deep = FALSE)

Arguments

Abstract Base Class for LibTorch Optimizers

Description

Abstract base class for wrapping LibTorch C++ optimizers.

Super class

torch::torch_optimizer -> OptimizerIgnite

Methods

Public methods

• OptimizerIgnite$new()

• OptimizerIgnite$state_dict()

• OptimizerIgnite$load_state_dict()

• OptimizerIgnite$step()

• OptimizerIgnite$zero_grad()

• OptimizerIgnite$add_param_group()

• OptimizerIgnite$clone()

Method new()

Initializes the optimizer with the specified parameters and defaults.

Usage

OptimizerIgnite$new(params, defaults)

Arguments

Method state_dict()

Returns the state dictionary containing the current state of the optimizer. The returned list() contains two lists:

• param_groups: The parameter groups of the optimizer (lr, ...) as well as to which parameters they are applied (params, integer indices)

• state: The states of the optimizer. The names are the indices of the parameters to which they belong, converted to character.

Usage

OptimizerIgnite$state_dict()

Returns

(list())

Method load_state_dict()

Loads the state dictionary into the optimizer.

Usage

OptimizerIgnite$load_state_dict(state_dict)

Arguments

Method step()

Performs a single optimization step.

Usage

OptimizerIgnite$step(closure = NULL)

Arguments

Returns

(numeric())
The loss.

Method zero_grad()

Zeros out the gradients of the parameters.

Usage

OptimizerIgnite$zero_grad()

Method add_param_group()

Adds a new parameter group to the optimizer.

Usage

OptimizerIgnite$add_param_group(param_group)

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

OptimizerIgnite$clone(deep = FALSE)

Arguments

Converts to array

Description

Converts to array

Usage

as_array(x)

Arguments

Computes the sum of gradients of given tensors w.r.t. graph leaves.

Description

The graph is differentiated using the chain rule. If any of tensors are non-scalar (i.e. their data has more than one element) and require gradient, then the Jacobian-vector product would be computed, in this case the function additionally requires specifying grad_tensors. It should be a sequence of matching length, that contains the “vector” in the Jacobian-vector product, usually the gradient of the differentiated function w.r.t. corresponding tensors (None is an acceptable value for all tensors that don’t need gradient tensors).

Usage

autograd_backward(
  tensors,
  grad_tensors = NULL,
  retain_graph = create_graph,
  create_graph = FALSE
)

Arguments

Details

This function accumulates gradients in the leaves - you might need to zero them before calling it.

Examples

if (torch_is_installed()) {
x <- torch_tensor(1, requires_grad = TRUE)
y <- 2 * x

a <- torch_tensor(1, requires_grad = TRUE)
b <- 3 * a

autograd_backward(list(y, b))
}

Records operation history and defines formulas for differentiating ops.

Description

Every operation performed on Tensor's creates a new function object, that performs the computation, and records that it happened. The history is retained in the form of a DAG of functions, with edges denoting data dependencies (input <- output). Then, when backward is called, the graph is processed in the topological ordering, by calling backward() methods of each Function object, and passing returned gradients on to next Function's.

Usage

autograd_function(forward, backward)

Arguments

Examples

if (torch_is_installed()) {

exp2 <- autograd_function(
  forward = function(ctx, i) {
    result <- i$exp()
    ctx$save_for_backward(result = result)
    result
  },
  backward = function(ctx, grad_output) {
    list(i = grad_output * ctx$saved_variable$result)
  }
)
}

Computes and returns the sum of gradients of outputs w.r.t. the inputs.

Description

grad_outputs should be a list of length matching output containing the “vector” in Jacobian-vector product, usually the pre-computed gradients w.r.t. each of the outputs. If an output doesn’t require_grad, then the gradient can be None).

Usage

autograd_grad(
  outputs,
  inputs,
  grad_outputs = NULL,
  retain_graph = create_graph,
  create_graph = FALSE,
  allow_unused = FALSE
)

Arguments

Details

If only_inputs is TRUE, the function will only return a list of gradients w.r.t the specified inputs. If it’s FALSE, then gradient w.r.t. all remaining leaves will still be computed, and will be accumulated into their .grad attribute.

Examples

if (torch_is_installed()) {
w <- torch_tensor(0.5, requires_grad = TRUE)
b <- torch_tensor(0.9, requires_grad = TRUE)
x <- torch_tensor(runif(100))
y <- 2 * x + 1
loss <- (y - (w * x + b))^2
loss <- loss$mean()

o <- autograd_grad(loss, list(w, b))
o
}

Set grad mode

Description

Sets or disables gradient history.

Usage

autograd_set_grad_mode(enabled)

Arguments

CuDNN is available

Description

CuDNN is available

Usage

backends_cudnn_is_available()

CuDNN version

Description

CuDNN version

Usage

backends_cudnn_version()

MKL is available

Description

MKL is available

Usage

backends_mkl_is_available()

Value

Returns whether LibTorch is built with MKL support.

MKLDNN is available

Description

MKLDNN is available

Usage

backends_mkldnn_is_available()

Value

Returns whether LibTorch is built with MKL-DNN support.

MPS is available

Description

MPS is available

Usage

backends_mps_is_available()

Value

Returns whether LibTorch is built with MPS support.

OpenMP is available

Description

OpenMP is available

Usage

backends_openmp_is_available()

Value

Returns whether LibTorch is built with OpenMP support.

Given a list of values (possibly containing numbers), returns a list where each value is broadcasted based on the following rules:

Description

Raises value_error: if any of the values is not a numeric instance, a torch.*Tensor instance, or an instance implementing torch_function TODO: add has_torch_function((v,)) See: https://github.com/pytorch/pytorch/blob/master/torch/distributions/utils.py

Usage

broadcast_all(values)

Arguments

Call a (Potentially Unexported) Torch Function

Description

This function allows calling a function prefixed with torch_, including unexported functions which could have potentially valuable uses but which do not yet have a user-friendly R wrapper function. Therefore, this function should be used with extreme caution. Make sure you understand what the function expects as input. It may be helpful to read the torch source code for help with this, as well as the documentation for the corresponding function in the Pytorch C++ API. Generally for development and advanced use only.

Usage

call_torch_function(name, ..., quiet = FALSE)

Arguments

Value

The return value from calling the function name with arguments ...

Examples

if (torch_is_installed()) {
## many unexported functions do 'backward' calculations (e.g. derivatives)
## These could be used as a part of custom autograd functions for example.
x <- torch_randn(10, requires_grad = TRUE)
y <- torch_tanh(x)
## calculate backwards gradient using standard torch method
y$backward(torch_ones_like(x))
x$grad
## we can get the same result by calling the unexported `torch_tanh_backward()`
## function. The first argument is 1 to setup the Jacobian-vector product.
## see https://pytorch.org/blog/overview-of-pytorch-autograd-engine/ for details.
call_torch_function("torch_tanh_backward", 1, y)
all.equal(call_torch_function("torch_tanh_backward", 1, y, quiet = TRUE), x$grad)

}

Clone a torch module.

Description

Clones a module.

Usage

clone_module(module, deep = FALSE, ..., replace_values = TRUE)

Arguments

Examples

if (torch_is_installed()) {
clone_module(nn_linear(1, 1), deep = TRUE)
# is the same as
nn_linear(1, 1)$clone(deep = TRUE)
}

Contrib sort vertices

Description

Based on the implementation from Rotated_IoU

Usage

contrib_sort_vertices(vertices, mask, num_valid)

Arguments

Details

All tensors should be on a CUDA device so this function can be used.

Note

This function does not make part of the official torch API.

Examples

if (torch_is_installed()) {
if (cuda_is_available()) {
  v <- torch_randn(8, 1024, 24, 2)$cuda()
  mean <- torch_mean(v, dim = 2, keepdim = TRUE)
  v <- v - mean
  m <- (torch_rand(8, 1024, 24) > 0.8)$cuda()
  nv <- torch_sum(m$to(dtype = torch_int()), dim = -1)$to(dtype = torch_int())$cuda()
  result <- contrib_sort_vertices(v, m, nv)
}
}

Creates a gradient scaler

Description

A gradient scaler instance is used to perform dynamic gradient scaling to avoid gradient underflow when training with mixed precision.

Usage

cuda_amp_grad_scaler(
  init_scale = 2^16,
  growth_factor = 2,
  backoff_factor = 0.5,
  growth_interval = 2000,
  enabled = TRUE
)

Arguments

Value

A gradient scaler object.

Returns the index of a currently selected device.

Description

Returns the index of a currently selected device.

Usage

cuda_current_device()

Returns the number of GPUs available.

Description

Returns the number of GPUs available.

Usage

cuda_device_count()

Save CUDA Memory State Snapshot to File

Description

Calls cuda_memory_snapshot() and saves the resulting binary snapshot to a specified file using writeBin. The resulting file can be visualized using the interactive snapshot viewer available at pytorch.org/memory_viz.

Usage

cuda_dump_memory_snapshot(filepath)

Arguments

Value

None; snapshot is saved directly to the file.

Examples

if (torch_is_installed()) {
## Not run: 
cuda_dump_memory_snapshot("snapshot.bin")

## End(Not run)
}

Empty cache

Description

Releases all unoccupied cached memory currently held by the caching allocator so that those can be used in other GPU application and visible in nvidia-smi.

Usage

cuda_empty_cache()

Note

cuda_empty_cache() doesn’t increase the amount of GPU memory available for torch. However, it may help reduce fragmentation of GPU memory in certain cases. See Memory management article for more details about GPU memory management.

Returns the major and minor CUDA capability of device

Description

Returns the major and minor CUDA capability of device

Usage

cuda_get_device_capability(device = cuda_current_device())

Arguments

Returns a bool indicating if CUDA is currently available.

Description

Returns a bool indicating if CUDA is currently available.

Usage

cuda_is_available()

Capture CUDA Memory State Snapshot

Description

Saves a snapshot of the CUDA memory state at the time it was called. The resulting binary output is in pickle format and can be visualized using the interactive snapshot viewer available at pytorch.org/memory_viz.

Usage

cuda_memory_snapshot()

Value

Raw binary data representing the snapshot in pickle format.

Examples

if (torch_is_installed()) {
## Not run: 
snapshot <- cuda_memory_snapshot()

## End(Not run)
}

Returns a dictionary of CUDA memory allocator statistics for a given device.

Description

The return value of this function is a dictionary of statistics, each of which is a non-negative integer.

Usage

cuda_memory_stats(device = cuda_current_device())

cuda_memory_summary(device = cuda_current_device())

Arguments

Core statistics

• "allocated.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of allocation requests received by the memory allocator.

• "allocated_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of allocated memory.

• "segment.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of reserved segments from cudaMalloc().

• "reserved_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of reserved memory.

• "active.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of active memory blocks.

• "active_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of active memory.

• "inactive_split.{all,large_pool,small_pool}.{current,peak,allocated,freed}": number of inactive, non-releasable memory blocks.

• "inactive_split_bytes.{all,large_pool,small_pool}.{current,peak,allocated,freed}": amount of inactive, non-releasable memory.

For these core statistics, values are broken down as follows.

Pool type:

• all: combined statistics across all memory pools.

• large_pool: statistics for the large allocation pool (as of October 2019, for size >= 1MB allocations).

• small_pool: statistics for the small allocation pool (as of October 2019, for size < 1MB allocations).

Metric type:

• current: current value of this metric.

• peak: maximum value of this metric.

• allocated: historical total increase in this metric.

• freed: historical total decrease in this metric.

Additional metrics

• "num_alloc_retries": number of failed cudaMalloc calls that result in a cache flush and retry.

• "num_ooms": number of out-of-memory errors thrown.

Enable Recording of Memory Allocation Stack Traces

Description

Enables recording of stack traces associated with memory allocations, allowing users to identify the source of memory allocation in CUDA snapshots.

Usage

cuda_record_memory_history(
  enabled,
  context = "all",
  stacks = "all",
  max_entries = 1
)

Arguments

Details

Alongside tracking stack traces for each current allocation and free event, this function can also keep a historical log of all allocation and free events.

Use cuda_memory_snapshot() to retrieve recorded information. Visualization can be performed using pytorch.org/memory_viz.

Value

None; function invoked for side effects.

Examples

if (torch_is_installed()) {
## Not run: 
cuda_record_memory_history(enabled = 'all', context = 'all', stacks = 'all', max_entries = 10000)

## End(Not run)
}

Returns the CUDA runtime version

Description

Returns the CUDA runtime version

Usage

cuda_runtime_version()

Waits for all kernels in all streams on a CUDA device to complete.

Description

Waits for all kernels in all streams on a CUDA device to complete.

Usage

cuda_synchronize(device = NULL)

Arguments

Data loader. Combines a dataset and a sampler, and provides single- or multi-process iterators over the dataset.

Description

Data loader. Combines a dataset and a sampler, and provides single- or multi-process iterators over the dataset.

Usage

dataloader(
  dataset,
  batch_size = 1,
  shuffle = FALSE,
  sampler = NULL,
  batch_sampler = NULL,
  num_workers = 0,
  collate_fn = NULL,
  pin_memory = FALSE,
  drop_last = FALSE,
  timeout = -1,
  worker_init_fn = NULL,
  worker_globals = NULL,
  worker_packages = NULL
)

Arguments

Parallel data loading

When using num_workers > 0 data loading will happen in parallel for each worker. Note that batches are taken in parallel and not observations.

The worker initialization process happens in the following order:

• num_workers R sessions are initialized.

Then in each worker we perform the following actions:

• the torch library is loaded.

• a random seed is set both using set.seed() and using torch_manual_seed.

• packages passed to the worker_packages argument are loaded.

• objects passed trough the worker_globals parameters are copied into the global environment.

• the worker_init function is ran with an id argument.

• the dataset fetcher is copied to the worker.

See Also

dataset(), sampler()

Creates an iterator from a DataLoader

Description

Creates an iterator from a DataLoader

Usage

dataloader_make_iter(dataloader)

Arguments

Get the next element of a dataloader iterator

Description

Get the next element of a dataloader iterator

Usage

dataloader_next(iter, completed = NULL)

Arguments

Helper function to create an function that generates R6 instances of class dataset

Description

All datasets that represent a map from keys to data samples should subclass this class. All subclasses should overwrite the .getitem() method, which supports fetching a data sample for a given key. Subclasses could also optionally overwrite .length(), which is expected to return the size of the dataset (e.g. number of samples) used by many sampler implementations and the default options of dataloader().

Usage

dataset(
  name = NULL,
  inherit = Dataset,
  ...,
  private = NULL,
  active = NULL,
  parent_env = parent.frame()
)

Arguments

Value

The output is a function f with class dataset_generator. Calling f() creates a new instance of the R6 class dataset. The R6 class is stored in the enclosing environment of f and can also be accessed through fs attribute Dataset.

Get a batch of observations

By default datasets are iterated by returning each observation/item individually. Often it's possible to have an optimized implementation to take a batch of observations (eg, subsetting a tensor by multiple indexes at once is faster than subsetting once for each index), in this case you can implement a .getbatch method that will be used instead of .getitem when getting a batch of observations within the dataloader. .getbatch must work for batches of size larger or equal to 1 and care must be taken so it doesn't drop the batch dimension when it's queried with a length 1 batch index - for instance by using drop=FALSE. .getitem() is expected to not include the batch dimension as it's added by the datalaoder. For more on this see the the vignette("loading-data").

Note

dataloader() by default constructs a index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided.

Dataset Subset

Description

Subset of a dataset at specified indices.

Usage

dataset_subset(dataset, indices)

Arguments

Creates a Bernoulli distribution parameterized by probs or logits (but not both). Samples are binary (0 or 1). They take the value 1 with probability p and 0 with probability 1 - p.

Description

Creates a Bernoulli distribution parameterized by probs or logits (but not both). Samples are binary (0 or 1). They take the value 1 with probability p and 0 with probability 1 - p.

Usage

distr_bernoulli(probs = NULL, logits = NULL, validate_args = NULL)

Arguments

See Also

Distribution for details on the available methods.

Other distributions: distr_chi2(), distr_gamma(), distr_multivariate_normal(), distr_normal(), distr_poisson()

Examples

if (torch_is_installed()) {
m <- distr_bernoulli(0.3)
m$sample() # 30% chance 1; 70% chance 0
}

Creates a categorical distribution parameterized by either probs or logits (but not both).

Description

Creates a categorical distribution parameterized by either probs or logits (but not both).

Usage

distr_categorical(probs = NULL, logits = NULL, validate_args = NULL)

Arguments

Note

It is equivalent to the distribution that torch_multinomial() samples from.

Samples are integers from \{0, \ldots, K-1\} where K is probs$size(-1).

If probs is 1-dimensional with length-K, each element is the relative probability of sampling the class at that index.

If probs is N-dimensional, the first N-1 dimensions are treated as a batch of relative probability vectors.

The probs argument must be non-negative, finite and have a non-zero sum, and it will be normalized to sum to 1 along the last dimension. attr:probs will return this normalized value. The logits argument will be interpreted as unnormalized log probabilities and can therefore be any real number. It will likewise be normalized so that the resulting probabilities sum to 1 along the last dimension. attr:logits will return this normalized value.

See also: torch_multinomial()

Examples

if (torch_is_installed()) {
m <- distr_categorical(torch_tensor(c(0.25, 0.25, 0.25, 0.25)))
m$sample() # equal probability of 1,2,3,4
}

Creates a Chi2 distribution parameterized by shape parameter df. This is exactly equivalent to distr_gamma(alpha=0.5*df, beta=0.5)

Description

Creates a Chi2 distribution parameterized by shape parameter df. This is exactly equivalent to distr_gamma(alpha=0.5*df, beta=0.5)

Usage

distr_chi2(df, validate_args = NULL)

Arguments

See Also

Distribution for details on the available methods.

Other distributions: distr_bernoulli(), distr_gamma(), distr_multivariate_normal(), distr_normal(), distr_poisson()

Examples

if (torch_is_installed()) {
m <- distr_chi2(torch_tensor(1.0))
m$sample() # Chi2 distributed with shape df=1
torch_tensor(0.1046)
}

Creates a Gamma distribution parameterized by shape concentration and rate.

Description

Creates a Gamma distribution parameterized by shape concentration and rate.

Usage

distr_gamma(concentration, rate, validate_args = NULL)

Arguments

See Also

Distribution for details on the available methods.

Other distributions: distr_bernoulli(), distr_chi2(), distr_multivariate_normal(), distr_normal(), distr_poisson()

Examples

if (torch_is_installed()) {
m <- distr_gamma(torch_tensor(1.0), torch_tensor(1.0))
m$sample() # Gamma distributed with concentration=1 and rate=1
}

Mixture of components in the same family

Description

The MixtureSameFamily distribution implements a (batch of) mixture distribution where all component are from different parameterizations of the same distribution type. It is parameterized by a Categorical selecting distribution" (over k component) and a component distribution, i.e., a Distribution with a rightmost batch shape (equal to ⁠[k]⁠) which indexes each (batch of) component.

Usage

distr_mixture_same_family(
  mixture_distribution,
  component_distribution,
  validate_args = NULL
)

Arguments

Examples

if (torch_is_installed()) {
# Construct Gaussian Mixture Model in 1D consisting of 5 equally
# weighted normal distributions
mix <- distr_categorical(torch_ones(5))
comp <- distr_normal(torch_randn(5), torch_rand(5))
gmm <- distr_mixture_same_family(mix, comp)
}

Gaussian distribution

Description

Creates a multivariate normal (also called Gaussian) distribution parameterized by a mean vector and a covariance matrix.

Usage

distr_multivariate_normal(
  loc,
  covariance_matrix = NULL,
  precision_matrix = NULL,
  scale_tril = NULL,
  validate_args = NULL
)

Arguments

Details

The multivariate normal distribution can be parameterized either in terms of a positive definite covariance matrix \mathbf{\Sigma} or a positive definite precision matrix \mathbf{\Sigma}^{-1} or a lower-triangular matrix \mathbf{L} with positive-valued diagonal entries, such that \mathbf{\Sigma} = \mathbf{L}\mathbf{L}^\top. This triangular matrix can be obtained via e.g. Cholesky decomposition of the covariance.

Note

Only one of covariance_matrix or precision_matrix or scale_tril can be specified. Using scale_tril will be more efficient: all computations internally are based on scale_tril. If covariance_matrix or precision_matrix is passed instead, it is only used to compute the corresponding lower triangular matrices using a Cholesky decomposition.

See Also

Distribution for details on the available methods.

Other distributions: distr_bernoulli(), distr_chi2(), distr_gamma(), distr_normal(), distr_poisson()

Examples

if (torch_is_installed()) {
m <- distr_multivariate_normal(torch_zeros(2), torch_eye(2))
m$sample() # normally distributed with mean=`[0,0]` and covariance_matrix=`I`
}

Creates a normal (also called Gaussian) distribution parameterized by loc and scale.

Description

Creates a normal (also called Gaussian) distribution parameterized by loc and scale.

Usage

distr_normal(loc, scale, validate_args = NULL)

Arguments

Value

Object of torch_Normal class

See Also

Distribution for details on the available methods.

Other distributions: distr_bernoulli(), distr_chi2(), distr_gamma(), distr_multivariate_normal(), distr_poisson()

Examples

if (torch_is_installed()) {
m <- distr_normal(loc = 0, scale = 1)
m$sample() # normally distributed with loc=0 and scale=1
}

Creates a Poisson distribution parameterized by rate, the rate parameter.

Description

Samples are nonnegative integers, with a pmf given by

\mbox{rate}^{k} \frac{e^{-\mbox{rate}}}{k!}

Usage

distr_poisson(rate, validate_args = NULL)

Arguments

See Also

Distribution for details on the available methods.

Other distributions: distr_bernoulli(), distr_chi2(), distr_gamma(), distr_multivariate_normal(), distr_normal()

Examples

if (torch_is_installed()) {
m <- distr_poisson(torch_tensor(4))
m$sample()
}

Enumerate an iterator

Description

Enumerate an iterator

Usage

enumerate(x, ...)

Arguments

Enumerate an iterator

Description

Enumerate an iterator

Usage

## S3 method for class 'dataloader'
enumerate(x, max_len = 1e+06, ...)

Arguments

Install Torch from files

Description

List the Torch and Lantern libraries URLs to download as local files in order to proceed with install_torch_from_file().

Installs Torch and its dependencies from files.

Usage

get_install_libs_url(version = NA, type = NA)

install_torch_from_file(version = NA, type = NA, libtorch, liblantern, ...)

Arguments

Details

When "install_torch()" initiated download is not possible, but installation archive files are present on local filesystem, "install_torch_from_file()" can be used as a workaround to installation issue. "libtorch" is the archive containing all torch modules, and "liblantern" is the C interface to libtorch that is used for the R package. Both are highly dependent, and should be checked through "get_install_libs_url()"

Examples

if (torch_is_installed()) {
## Not run: 
# on a linux CPU platform 
get_install_libs_url()
# then after making both files available into /tmp/
Sys.setenv(TORCH_URL="/tmp/libtorch-v1.13.1.zip")
Sys.setenv(LANTERN_URL="/tmp/lantern-0.9.1.9001+cpu+arm64-Darwin.zip")
torch::install_torch()

## End(Not run)
}

Install Torch

Description

Installs Torch and its dependencies.

Usage

install_torch(reinstall = FALSE, ..., .inform_restart = TRUE)

Arguments

Details

This function is mainly controlled by environment variables that can be used to override the defaults:

• TORCH_HOME: the installation path. By default dependencies are installed within the package directory. Eg what's given by system.file(package="torch").

• TORCH_URL: A URL, path to a ZIP file or a directory containing a LibTorch version. Files will be installed/copied to the TORCH_HOME directory.

• LANTERN_URL: Same as TORCH_URL but for the Lantern library.

• TORCH_INSTALL_DEBUG: Setting it to 1, shows debug log messages during installation.

• PRECXX11ABI: Setting it to 1 will will trigger the installation of a Pre-cxx11 ABI installation of LibTorch. This can be useful in environments with older versions of GLIBC like CentOS7 and older Debian/Ubuntu versions.

• LANTERN_BASE_URL: The base URL for lantern files. This allows passing a directory where lantern binaries are located. The filename is then constructed as usual.

• TORCH_COMMIT_SHA: torch repository commit sha to be used when querying lantern uploads. Set it to 'none' to avoid looking for build for that commit and use the latest build for the branch.

• CUDA: We try to automatically detect the CUDA version installed in your system, but you might want to manually set it here. You can also disable CUDA installation by setting it to 'cpu'.

• TORCH_R_VERSION: The R torch version. It's unlikely that you need to change it, but it can be useful if you don't have the R package installed, but want to install the dependencies.

The TORCH_INSTALL environment variable can be set to 0 to prevent auto-installing torch and TORCH_LOAD set to 0 to avoid loading dependencies automatically. These environment variables are meant for advanced use cases and troubleshooting only. When timeout error occurs during library archive download, or length of downloaded files differ from reported length, an increase of the timeout value should help.

Checks if the object is a dataloader

Description

Checks if the object is a dataloader

Usage

is_dataloader(x)

Arguments

Checks if the object is a nn_buffer

Description

Checks if the object is a nn_buffer

Usage

is_nn_buffer(x)

Arguments

Checks if the object is an nn_module

Description

Checks if the object is an nn_module

Usage

is_nn_module(x)

Arguments

Checks if an object is a nn_parameter

Description

Checks if an object is a nn_parameter

Usage

is_nn_parameter(x)

Arguments

Checks if the object is a torch optimizer

Description

Checks if the object is a torch optimizer

Usage

is_optimizer(x)

Arguments

Checks if object is a device

Description

Checks if object is a device

Usage

is_torch_device(x)

Arguments

Check if object is a torch data type

Description

Check if object is a torch data type

Usage

is_torch_dtype(x)

Arguments

Check if an object is a torch layout.

Description

Check if an object is a torch layout.

Usage

is_torch_layout(x)

Arguments

Check if an object is a memory format

Description

Check if an object is a memory format

Usage

is_torch_memory_format(x)

Arguments

Checks if an object is a QScheme

Description

Checks if an object is a QScheme

Usage

is_torch_qscheme(x)

Arguments

Checks if a tensor is undefined

Description

Checks if a tensor is undefined

Usage

is_undefined_tensor(x)

Arguments

Creates an iterable dataset

Description

Creates an iterable dataset

Usage

iterable_dataset(
  name,
  inherit = IterableDataset,
  ...,
  private = NULL,
  active = NULL,
  parent_env = parent.frame()
)

Arguments

Examples

if (torch_is_installed()) {
ids <- iterable_dataset(
  name = "hello",
  initialize = function(n = 5) {
    self$n <- n
    self$i <- 0
  },
  .iter = function() {
    i <- 0
    function() {
      i <<- i + 1
      if (i > self$n) {
        coro::exhausted()
      } else {
        i
      }
    }
  }
)
coro::collect(ids()$.iter())
}

Compile TorchScript code into a graph

Description

See the TorchScript language reference for documentation on how to write TorchScript code.

Usage

jit_compile(source)

Arguments

Examples

if (torch_is_installed()) {
comp <- jit_compile("
def fn (x):
  return torch.abs(x)

def foo (x):
  return torch.sum(x)

")

comp$fn(torch_tensor(-1))
comp$foo(torch_randn(10))
}

Loads a script_function or script_module previously saved with jit_save

Description

Loads a script_function or script_module previously saved with jit_save

Usage

jit_load(path, ...)

Arguments

Enable idiomatic access to JIT operators from R.

Description

Call JIT operators directly from R, keeping the familiar argument types and argument order. Note, however, that:

• all arguments are required (no defaults)

• axis numbering (as well as position numbers overall) starts from 0

• scalars have to be wrapped in jit_scalar()

Usage

jit_ops

Format

An object of class torch_ops of length 0.

Examples

if (torch_is_installed()) {
t1 <- torch::torch_rand(4, 5)
t2 <- torch::torch_ones(5, 4)
# same as torch::torch_matmul(t1, t2)
jit_ops$aten$matmul(t1, t2)

# same as torch_split(torch::torch_arange(0, 3), 2, 1)
jit_ops$aten$split(torch::torch_arange(0, 3), torch::jit_scalar(2L), torch::jit_scalar(0L))

}

Saves a script_function to a path

Description

Saves a script_function to a path

Usage

jit_save(obj, path, ...)

Arguments

Examples

if (torch_is_installed()) {
fn <- function(x) {
  torch_relu(x)
}

input <- torch_tensor(c(-1, 0, 1))
tr_fn <- jit_trace(fn, input)

tmp <- tempfile("tst", fileext = "pt")
jit_save(tr_fn, tmp)
}

Saves a script_function or script_module in bytecode form, to be loaded on a mobile device

Description

Saves a script_function or script_module in bytecode form, to be loaded on a mobile device

Usage

jit_save_for_mobile(obj, path, ...)

Arguments

Examples

if (torch_is_installed()) {
fn <- function(x) {
  torch_relu(x)
}

input <- torch_tensor(c(-1, 0, 1))
tr_fn <- jit_trace(fn, input)

tmp <- tempfile("tst", fileext = "pt")
jit_save_for_mobile(tr_fn, tmp)
}

Adds the 'jit_scalar' class to the input

Description

Allows disambiguating length 1 vectors from scalars when passing them to the jit.

Usage

jit_scalar(x)

Arguments

Serialize a Script Module

Description

Serializes a script module and returns it as a raw vector. You can read the object again using jit_unserialize.

Usage

jit_serialize(obj)

Arguments

Value

raw()

Examples

if (torch_is_installed()) {
model <- jit_trace(nn_linear(1, 1), torch_randn(1))
serialized <- jit_serialize(model)
}

Trace a function and return an executable script_function.

Description

Using jit_trace, you can turn an existing R function into a TorchScript script_function. You must provide example inputs, and we run the function, recording the operations performed on all the tensors.

Usage

jit_trace(func, ..., strict = TRUE, respect_mode = TRUE)

Arguments

Details

The resulting recording of a standalone function produces a script_function.

Value

An script_function if func is a function and script_module if func is a nn_module().

Warning

Tracing only correctly records functions and modules which are not data dependent (e.g., do not have conditionals on data in tensors) and do not have any untracked external dependencies (e.g., perform input/output or access global variables). Tracing only records operations done when the given function is run on the given tensors. Therefore, the returned script_function will always run the same traced graph on any input. This has some important implications when your module is expected to run different sets of operations, depending on the input and/or the module state. For example,

• Tracing will not record any control-flow like if-statements or loops. When this control-flow is constant across your module, this is fine and it often inlines the control-flow decisions. But sometimes the control-flow is actually part of the model itself. For instance, a recurrent network is a loop over the (possibly dynamic) length of an input sequence.

• In the returned script_function, operations that have different behaviors in training and eval modes will always behave as if it is in the mode it was in during tracing, no matter which mode the script_function is in.

In cases like these, tracing would not be appropriate and scripting is a better choice. If you trace such models, you may silently get incorrect results on subsequent invocations of the model. The tracer will try to emit warnings when doing something that may cause an incorrect trace to be produced. For scripting, see jit_compile.

Examples

if (torch_is_installed()) {
fn <- function(x) {
  torch_relu(x)
}
input <- torch_tensor(c(-1, 0, 1))
tr_fn <- jit_trace(fn, input)
tr_fn(input)
}

Trace a module

Description

Trace a module and return an executable ScriptModule that will be optimized using just-in-time compilation. When a module is passed to jit_trace(), only the forward method is run and traced. With jit_trace_module(), you can specify a named list of method names to example inputs to trace (see the inputs) argument below.

Usage

jit_trace_module(mod, ..., strict = TRUE, respect_mode = TRUE)

Arguments

Details

See jit_trace for more information on tracing.

Examples

if (torch_is_installed()) {
linear <- nn_linear(10, 1)
tr_linear <- jit_trace_module(linear, forward = list(torch_randn(10, 10)))

x <- torch_randn(10, 10)
torch_allclose(linear(x), tr_linear(x))
}

Adds the 'jit_tuple' class to the input

Description

Allows specifying that an output or input must be considered a jit tuple and instead of a list or dictionary when tracing.

Usage

jit_tuple(x)

Arguments

Unserialize a Script Module

Description

Unserializes a script module from a raw vector (generated with jit_serialize').

Usage

jit_unserialize(obj)

Arguments

Value

script_module model <- jit_trace(nn_linear(1, 1), torch_randn(1)) serialized <- jit_serialize(model) model2 <- jit_unserialize(serialized)

Computes the Cholesky decomposition of a complex Hermitian or real symmetric positive-definite matrix.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the Cholesky decomposition of a complex Hermitian or real symmetric positive-definite matrix A \in \mathbb{K}^{n \times n} is defined as

Usage

linalg_cholesky(A)

Arguments

Details

where L is a lower triangular matrix and L^{H} is the conjugate transpose when L is complex, and the transpose when L is real-valued.

Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

See Also

• linalg_cholesky_ex() for a version of this operation that skips the (slow) error checking by default and instead returns the debug information. This makes it a faster way to check if a matrix is positive-definite. linalg_eigh() for a different decomposition of a Hermitian matrix. The eigenvalue decomposition gives more information about the matrix but it slower to compute than the Cholesky decomposition.

Other linalg: linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_eye(10)
linalg_cholesky(a)
}

Computes the Cholesky decomposition of a complex Hermitian or real symmetric positive-definite matrix.

Description

This function skips the (slow) error checking and error message construction of linalg_cholesky(), instead directly returning the LAPACK error codes as part of a named tuple ⁠(L, info)⁠. This makes this function a faster way to check if a matrix is positive-definite, and it provides an opportunity to handle decomposition errors more gracefully or performantly than linalg_cholesky() does. Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions. If A is not a Hermitian positive-definite matrix, or if it's a batch of matrices and one or more of them is not a Hermitian positive-definite matrix, then info stores a positive integer for the corresponding matrix. The positive integer indicates the order of the leading minor that is not positive-definite, and the decomposition could not be completed. info filled with zeros indicates that the decomposition was successful. If check_errors=TRUE and info contains positive integers, then a RuntimeError is thrown.

Usage

linalg_cholesky_ex(A, check_errors = FALSE)

Arguments

Note

If A is on a CUDA device, this function may synchronize that device with the CPU.

This function is "experimental" and it may change in a future PyTorch release.

See Also

linalg_cholesky() is a NumPy compatible variant that always checks for errors.

Other linalg: linalg_cholesky(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(2, 2)
out <- linalg_cholesky_ex(A)
out
}

Computes the condition number of a matrix with respect to a matrix norm.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the condition number \kappa of a matrix A \in \mathbb{K}^{n \times n} is defined as

Usage

linalg_cond(A, p = NULL)

Arguments

Details

The condition number of A measures the numerical stability of the linear system AX = B with respect to a matrix norm.

Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

p defines the matrix norm that is computed. See the table in 'Details' to find the supported norms.

For p is one of ⁠('fro', 'nuc', inf, -inf, 1, -1)⁠, this function uses linalg_norm() and linalg_inv().

As such, in this case, the matrix (or every matrix in the batch) A has to be square and invertible.

For p in ⁠(2, -2)⁠, this function can be computed in terms of the singular values \sigma_1 \geq \ldots \geq \sigma_n

In these cases, it is computed using linalg_svd(). For these norms, the matrix (or every matrix in the batch) A may have any shape.

Value

A real-valued tensor, even when A is complex.

Note

When inputs are on a CUDA device, this function synchronizes that device with the CPU if if p is one of ⁠('fro', 'nuc', inf, -inf, 1, -1)⁠.

Examples

if (torch_is_installed()) {
a <- torch_tensor(rbind(c(1., 0, -1), c(0, 1, 0), c(1, 0, 1)))
linalg_cond(a)
linalg_cond(a, "fro")
}

Computes the determinant of a square matrix.

Description

Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Usage

linalg_det(A)

Arguments

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_randn(3, 3)
linalg_det(a)

a <- torch_randn(3, 3, 3)
linalg_det(a)
}

Computes the eigenvalue decomposition of a square matrix if it exists.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the eigenvalue decomposition of a square matrix A \in \mathbb{K}^{n \times n} (if it exists) is defined as

Usage

linalg_eig(A)

Arguments

Details

This decomposition exists if and only if A is diagonalizable_. This is the case when all its eigenvalues are different. Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Value

A list ⁠(eigenvalues, eigenvectors)⁠ which corresponds to \Lambda and V above. eigenvalues and eigenvectors will always be complex-valued, even when A is real. The eigenvectors will be given by the columns of eigenvectors.

Warning

• This function assumes that A is diagonalizable_ (for example, when all the eigenvalues are different). If it is not diagonalizable, the returned eigenvalues will be correct but A \neq V \operatorname{diag}(\Lambda)V^{-1}.

• The eigenvectors of a matrix are not unique, nor are they continuous with respect to A. Due to this lack of uniqueness, different hardware and software may compute different eigenvectors. This non-uniqueness is caused by the fact that multiplying an eigenvector by a non-zero number produces another set of valid eigenvectors of the matrix. In this implmentation, the returned eigenvectors are normalized to have norm 1 and largest real component.

• Gradients computed using V will only be finite when A does not have repeated eigenvalues. Furthermore, if the distance between any two eigenvalues is close to zero, the gradient will be numerically unstable, as it depends on the eigenvalues \lambda_i through the computation of \frac{1}{\min_{i \neq j} \lambda_i - \lambda_j}.

Note

The eigenvalues and eigenvectors of a real matrix may be complex.

See Also

• linalg_eigvals() computes only the eigenvalues. Unlike linalg_eig(), the gradients of linalg_eigvals() are always numerically stable.

• linalg_eigh() for a (faster) function that computes the eigenvalue decomposition for Hermitian and symmetric matrices.

• linalg_svd() for a function that computes another type of spectral decomposition that works on matrices of any shape.

• linalg_qr() for another (much faster) decomposition that works on matrices of any shape.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_randn(2, 2)
wv <- linalg_eig(a)
}

Computes the eigenvalue decomposition of a complex Hermitian or real symmetric matrix.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the eigenvalue decomposition of a complex Hermitian or real symmetric matrix A \in \mathbb{K}^{n \times n} is defined as

Usage

linalg_eigh(A, UPLO = "L")

Arguments

Details

where Q^{H} is the conjugate transpose when Q is complex, and the transpose when Q is real-valued. Q is orthogonal in the real case and unitary in the complex case.

Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

A is assumed to be Hermitian (resp. symmetric), but this is not checked internally, instead:

• If UPLO\ ⁠= 'L'⁠ (default), only the lower triangular part of the matrix is used in the computation.

• If UPLO\ ⁠= 'U'⁠, only the upper triangular part of the matrix is used. The eigenvalues are returned in ascending order.

Value

A list ⁠(eigenvalues, eigenvectors)⁠ which corresponds to \Lambda and Q above. eigenvalues will always be real-valued, even when A is complex.

It will also be ordered in ascending order. eigenvectors will have the same dtype as A and will contain the eigenvectors as its columns.

Warning

• The eigenvectors of a symmetric matrix are not unique, nor are they continuous with respect to A. Due to this lack of uniqueness, different hardware and software may compute different eigenvectors. This non-uniqueness is caused by the fact that multiplying an eigenvector by -1 in the real case or by e^{i \phi}, \phi \in \mathbb{R} in the complex case produces another set of valid eigenvectors of the matrix. This non-uniqueness problem is even worse when the matrix has repeated eigenvalues. In this case, one may multiply the associated eigenvectors spanning the subspace by a rotation matrix and the resulting eigenvectors will be valid eigenvectors.

• Gradients computed using the eigenvectors tensor will only be finite when A has unique eigenvalues. Furthermore, if the distance between any two eigvalues is close to zero, the gradient will be numerically unstable, as it depends on the eigenvalues \lambda_i through the computation of \frac{1}{\min_{i \neq j} \lambda_i - \lambda_j}.

Note

The eigenvalues of real symmetric or complex Hermitian matrices are always real.

See Also

• linalg_eigvalsh() computes only the eigenvalues values of a Hermitian matrix. Unlike linalg_eigh(), the gradients of linalg_eigvalsh() are always numerically stable.

• linalg_cholesky() for a different decomposition of a Hermitian matrix. The Cholesky decomposition gives less information about the matrix but is much faster to compute than the eigenvalue decomposition.

• linalg_eig() for a (slower) function that computes the eigenvalue decomposition of a not necessarily Hermitian square matrix.

• linalg_svd() for a (slower) function that computes the more general SVD decomposition of matrices of any shape.

• linalg_qr() for another (much faster) decomposition that works on general matrices.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_randn(2, 2)
linalg_eigh(a)
}

Computes the eigenvalues of a square matrix.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the eigenvalues of a square matrix A \in \mathbb{K}^{n \times n} are defined as the roots (counted with multiplicity) of the polynomial p of degree n given by

Usage

linalg_eigvals(A)

Arguments

Details

where \mathrm{I}_n is the n-dimensional identity matrix. Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Note

The eigenvalues of a real matrix may be complex, as the roots of a real polynomial may be complex. The eigenvalues of a matrix are always well-defined, even when the matrix is not diagonalizable.

See Also

linalg_eig() computes the full eigenvalue decomposition.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_randn(2, 2)
w <- linalg_eigvals(a)
}

Computes the eigenvalues of a complex Hermitian or real symmetric matrix.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the eigenvalues of a complex Hermitian or real symmetric matrix A \in \mathbb{K}^{n \times n} are defined as the roots (counted with multiplicity) of the polynomial p of degree n given by

Usage

linalg_eigvalsh(A, UPLO = "L")

Arguments

Details

where \mathrm{I}_n is the n-dimensional identity matrix.

The eigenvalues of a real symmetric or complex Hermitian matrix are always real. Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions. The eigenvalues are returned in ascending order.

A is assumed to be Hermitian (resp. symmetric), but this is not checked internally, instead:

• If UPLO\ ⁠= 'L'⁠ (default), only the lower triangular part of the matrix is used in the computation.

• If UPLO\ ⁠= 'U'⁠, only the upper triangular part of the matrix is used.

Value

A real-valued tensor cointaining the eigenvalues even when A is complex. The eigenvalues are returned in ascending order.

See Also

• linalg_eigh() computes the full eigenvalue decomposition.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_randn(2, 2)
linalg_eigvalsh(a)
}

Computes the first n columns of a product of Householder matrices.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, for a matrix V \in \mathbb{K}^{m \times n} with columns v_i \in \mathbb{K}^m with m \geq n and a vector \tau \in \mathbb{K}^k with k \leq n, this function computes the first n columns of the matrix

Usage

linalg_householder_product(A, tau)

Arguments

Details

where \mathrm{I}_m is the m-dimensional identity matrix and v^{H} is the conjugate transpose when v is complex, and the transpose when v is real-valued. See Representation of Orthogonal or Unitary Matrices for further details.

Supports inputs of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if the inputs are batches of matrices then the output has the same batch dimensions.

Note

This function only uses the values strictly below the main diagonal of A. The other values are ignored.

See Also

• torch_geqrf() can be used together with this function to form the Q from the linalg_qr() decomposition.

• torch_ormqr() is a related function that computes the matrix multiplication of a product of Householder matrices with another matrix. However, that function is not supported by autograd.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(2, 2)
h_tau <- torch_geqrf(A)
Q <- linalg_householder_product(h_tau[[1]], h_tau[[2]])
torch_allclose(Q, linalg_qr(A)[[1]])
}

Computes the inverse of a square matrix if it exists.

Description

Throws a runtime_error if the matrix is not invertible.

Usage

linalg_inv(A)

Arguments

Details

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, for a matrix A \in \mathbb{K}^{n \times n}, its inverse matrix A^{-1} \in \mathbb{K}^{n \times n} (if it exists) is defined as

where \mathrm{I}_n is the n-dimensional identity matrix.

The inverse matrix exists if and only if A is invertible. In this case, the inverse is unique. Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Consider using linalg_solve() if possible for multiplying a matrix on the left by the inverse, as linalg_solve(A, B) == A$inv() %*% B It is always prefered to use linalg_solve() when possible, as it is faster and more numerically stable than computing the inverse explicitly.

See Also

linalg_pinv() computes the pseudoinverse (Moore-Penrose inverse) of matrices of any shape. linalg_solve() computes A$inv() %*% B with a numerically stable algorithm.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(4, 4)
linalg_inv(A)
}

Computes the inverse of a square matrix if it is invertible.

Description

Returns a namedtuple ⁠(inverse, info)⁠. inverse contains the result of inverting A and info stores the LAPACK error codes. If A is not an invertible matrix, or if it's a batch of matrices and one or more of them is not an invertible matrix, then info stores a positive integer for the corresponding matrix. The positive integer indicates the diagonal element of the LU decomposition of the input matrix that is exactly zero. info filled with zeros indicates that the inversion was successful. If check_errors=TRUE and info contains positive integers, then a RuntimeError is thrown. Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Usage

linalg_inv_ex(A, check_errors = FALSE)

Arguments

Note

If A is on a CUDA device then this function may synchronize that device with the CPU.

This function is "experimental" and it may change in a future PyTorch release.

See Also

linalg_inv() is a NumPy compatible variant that always checks for errors.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(3, 3)
out <- linalg_inv_ex(A)
}

Computes a solution to the least squares problem of a system of linear equations.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the least squares problem for a linear system AX = B with A \in \mathbb{K}^{m \times n}, B \in \mathbb{K}^{m \times k} is defined as

Usage

linalg_lstsq(A, B, rcond = NULL, ..., driver = NULL)

Arguments

Details

where \|-\|_F denotes the Frobenius norm. Supports inputs of float, double, cfloat and cdouble dtypes.

Also supports batches of matrices, and if the inputs are batches of matrices then the output has the same batch dimensions. driver chooses the LAPACK/MAGMA function that will be used.

For CPU inputs the valid values are 'gels', 'gelsy', ⁠'gelsd⁠, 'gelss'. For CUDA input, the only valid driver is 'gels', which assumes that A is full-rank.

To choose the best driver on CPU consider:

• If A is well-conditioned (its condition number is not too large), or you do not mind some precision loss.

• For a general matrix: 'gelsy' (QR with pivoting) (default)

• If A is full-rank: 'gels' (QR)

• If A is not well-conditioned.

• 'gelsd' (tridiagonal reduction and SVD)

• But if you run into memory issues: 'gelss' (full SVD).

See also the full description of these drivers

rcond is used to determine the effective rank of the matrices in A when driver is one of ('gelsy', 'gelsd', 'gelss'). In this case, if \sigma_i are the singular values of A in decreasing order, \sigma_i will be rounded down to zero if \sigma_i \leq rcond \cdot \sigma_1. If rcond = NULL (default), rcond is set to the machine precision of the dtype of A.

This function returns the solution to the problem and some extra information in a list of four tensors ⁠(solution, residuals, rank, singular_values)⁠. For inputs A, B of shape ⁠(*, m, n)⁠, ⁠(*, m, k)⁠ respectively, it cointains

• solution: the least squares solution. It has shape ⁠(*, n, k)⁠.

• residuals: the squared residuals of the solutions, that is, \|AX - B\|_F^2. It has shape equal to the batch dimensions of A. It is computed when m > n and every matrix in A is full-rank, otherwise, it is an empty tensor. If A is a batch of matrices and any matrix in the batch is not full rank, then an empty tensor is returned. This behavior may change in a future PyTorch release.

• rank: tensor of ranks of the matrices in A. It has shape equal to the batch dimensions of A. It is computed when driver is one of ('gelsy', 'gelsd', 'gelss'), otherwise it is an empty tensor.

• singular_values: tensor of singular values of the matrices in A. It has shape ⁠(*, min(m, n))⁠. It is computed when driver is one of ('gelsd', 'gelss'), otherwise it is an empty tensor.

Value

A list ⁠(solution, residuals, rank, singular_values)⁠.

Warning

The default value of rcond may change in a future PyTorch release. It is therefore recommended to use a fixed value to avoid potential breaking changes.

Note

This function computes X = A$pinverse() %*% B in a faster and more numerically stable way than performing the computations separately.

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_tensor(rbind(c(10, 2, 3), c(3, 10, 5), c(5, 6, 12)))$unsqueeze(1) # shape (1, 3, 3)
B <- torch_stack(list(
  rbind(c(2, 5, 1), c(3, 2, 1), c(5, 1, 9)),
  rbind(c(4, 2, 9), c(2, 0, 3), c(2, 5, 3))
), dim = 1) # shape (2, 3, 3)
X <- linalg_lstsq(A, B)$solution # A is broadcasted to shape (2, 3, 3)
}

Computes a matrix norm.

Description

If A is complex valued, it computes the norm of A$abs() Support input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices: the norm will be computed over the dimensions specified by the 2-tuple dim and the other dimensions will be treated as batch dimensions. The output will have the same batch dimensions.

Usage

linalg_matrix_norm(
  A,
  ord = "fro",
  dim = c(-2, -1),
  keepdim = FALSE,
  dtype = NULL
)

Arguments

Details

ord defines the norm that is computed. The following norms are supported:

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_arange(0, 8, dtype = torch_float())$reshape(c(3, 3))
linalg_matrix_norm(a)
linalg_matrix_norm(a, ord = -1)
b <- a$expand(c(2, -1, -1))
linalg_matrix_norm(b)
linalg_matrix_norm(b, dim = c(1, 3))
}

Computes the n-th power of a square matrix for an integer n.

Description

Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Usage

linalg_matrix_power(A, n)

Arguments

Details

If n=0, it returns the identity matrix (or batch) of the same shape as A. If n is negative, it returns the inverse of each matrix (if invertible) raised to the power of abs(n).

See Also

linalg_solve() computes A$inverse() %*% B with a numerically stable algorithm.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(3, 3)
linalg_matrix_power(A, 0)
}

Computes the numerical rank of a matrix.

Description

The matrix rank is computed as the number of singular values (or eigenvalues in absolute value when hermitian = TRUE) that are greater than the specified tol threshold.

Usage

linalg_matrix_rank(
  A,
  ...,
  atol = NULL,
  rtol = NULL,
  tol = NULL,
  hermitian = FALSE
)

Arguments

Details

Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

If hermitian = TRUE, A is assumed to be Hermitian if complex or symmetric if real, but this is not checked internally. Instead, just the lower triangular part of the matrix is used in the computations.

If tol is not specified and A is a matrix of dimensions ⁠(m, n)⁠, the tolerance is set to be

where \sigma_1 is the largest singular value (or eigenvalue in absolute value when hermitian = TRUE), and \varepsilon is the epsilon value for the dtype of A (see torch_finfo()).

If A is a batch of matrices, tol is computed this way for every element of the batch.

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_eye(10)
linalg_matrix_rank(a)
}

Efficiently multiplies two or more matrices

Description

Efficiently multiplies two or more matrices by reordering the multiplications so that the fewest arithmetic operations are performed.

Usage

linalg_multi_dot(tensors)

Arguments

Details

Supports inputs of float, double, cfloat and cdouble dtypes. This function does not support batched inputs.

Every tensor in tensors must be 2D, except for the first and last which may be 1D. If the first tensor is a 1D vector of shape ⁠(n,)⁠ it is treated as a row vector of shape ⁠(1, n)⁠, similarly if the last tensor is a 1D vector of shape ⁠(n,)⁠ it is treated as a column vector of shape ⁠(n, 1)⁠.

If the first and last tensors are matrices, the output will be a matrix. However, if either is a 1D vector, then the output will be a 1D vector.

Note

This function is implemented by chaining torch_mm() calls after computing the optimal matrix multiplication order.

The cost of multiplying two matrices with shapes ⁠(a, b)⁠ and ⁠(b, c)⁠ is a * b * c. Given matrices A, B, C with shapes ⁠(10, 100)⁠, ⁠(100, 5)⁠, ⁠(5, 50)⁠ respectively, we can calculate the cost of different multiplication orders as follows:

In this case, multiplying A and B first followed by C is 10 times faster.

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {

linalg_multi_dot(list(torch_tensor(c(1, 2)), torch_tensor(c(2, 3))))
}

Computes a vector or matrix norm.

Description

If A is complex valued, it computes the norm of A$abs() Supports input of float, double, cfloat and cdouble dtypes. Whether this function computes a vector or matrix norm is determined as follows:

Usage

linalg_norm(A, ord = NULL, dim = NULL, keepdim = FALSE, dtype = NULL)

Arguments

Details

• If dim is an int, the vector norm will be computed.

• If dim is a 2-tuple, the matrix norm will be computed.

• If dim=NULL and ord=NULL, A will be flattened to 1D and the 2-norm of the resulting vector will be computed.

• If dim=NULL and ord!=NULL, A must be 1D or 2D.

ord defines the norm that is computed. The following norms are supported:

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_arange(0, 8, dtype = torch_float()) - 4
a
b <- a$reshape(c(3, 3))
b

linalg_norm(a)
linalg_norm(b)
}

Computes the pseudoinverse (Moore-Penrose inverse) of a matrix.

Description

The pseudoinverse may be ⁠defined algebraically⁠_ but it is more computationally convenient to understand it ⁠through the SVD⁠_ Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Usage

linalg_pinv(A, rcond = NULL, hermitian = FALSE, atol = NULL, rtol = NULL)

Arguments

Details

If hermitian= TRUE, A is assumed to be Hermitian if complex or symmetric if real, but this is not checked internally. Instead, just the lower triangular part of the matrix is used in the computations. The singular values (or the norm of the eigenvalues when hermitian= TRUE) that are below the specified rcond threshold are treated as zero and discarded in the computation.

Note

This function uses linalg_svd() if hermitian= FALSE and linalg_eigh() if hermitian= TRUE. For CUDA inputs, this function synchronizes that device with the CPU.

Consider using linalg_lstsq() if possible for multiplying a matrix on the left by the pseudoinverse, as linalg_lstsq(A, B)$solution == A$pinv() %*% B

It is always prefered to use linalg_lstsq() when possible, as it is faster and more numerically stable than computing the pseudoinverse explicitly.

See Also

• linalg_inv() computes the inverse of a square matrix.

• linalg_lstsq() computes A$pinv() %*% B with a numerically stable algorithm.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(3, 5)
linalg_pinv(A)
}

Computes the QR decomposition of a matrix.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the full QR decomposition of a matrix A \in \mathbb{K}^{m \times n} is defined as

Usage

linalg_qr(A, mode = "reduced")

Arguments

Details

where Q is orthogonal in the real case and unitary in the complex case, and R is upper triangular. When m > n (tall matrix), as R is upper triangular, its last m - n rows are zero. In this case, we can drop the last m - n columns of Q to form the reduced QR decomposition:

The reduced QR decomposition agrees with the full QR decomposition when n >= m (wide matrix). Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions. The parameter mode chooses between the full and reduced QR decomposition.

If A has shape ⁠(*, m, n)⁠, denoting k = min(m, n)

• mode = 'reduced' (default): Returns ⁠(Q, R)⁠ of shapes ⁠(*, m, k)⁠, ⁠(*, k, n)⁠ respectively.

• mode = 'complete': Returns ⁠(Q, R)⁠ of shapes ⁠(*, m, m)⁠, ⁠(*, m, n)⁠ respectively.

• mode = 'r': Computes only the reduced R. Returns ⁠(Q, R)⁠ with Q empty and R of shape ⁠(*, k, n)⁠.

Value

A list ⁠(Q, R)⁠.

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_tensor(rbind(c(12., -51, 4), c(6, 167, -68), c(-4, 24, -41)))
qr <- linalg_qr(a)

torch_mm(qr[[1]], qr[[2]])$round()
torch_mm(qr[[1]]$t(), qr[[1]])$round()
}

Computes the sign and natural logarithm of the absolute value of the determinant of a square matrix.

Description

For complex A, it returns the angle and the natural logarithm of the modulus of the determinant, that is, a logarithmic polar decomposition of the determinant. Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

Usage

linalg_slogdet(A)

Arguments

Value

A list ⁠(sign, logabsdet)⁠. logabsdet will always be real-valued, even when A is complex. sign will have the same dtype as A.

Notes

• The determinant can be recovered as sign * exp(logabsdet).

• When a matrix has a determinant of zero, it returns ⁠(0, -Inf)⁠.

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
a <- torch_randn(3, 3)
linalg_slogdet(a)
}

Computes the solution of a square system of linear equations with a unique solution.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, this function computes the solution X \in \mathbb{K}^{n \times k} of the linear system associated to A \in \mathbb{K}^{n \times n}, B \in \mathbb{K}^{m \times k}, which is defined as

Usage

linalg_solve(A, B)

Arguments

Details

AX = B

This system of linear equations has one solution if and only if A is invertible_. This function assumes that A is invertible. Supports inputs of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if the inputs are batches of matrices then the output has the same batch dimensions.

Letting * be zero or more batch dimensions,

• If A has shape ⁠(*, n, n)⁠ and B has shape ⁠(*, n)⁠ (a batch of vectors) or shape ⁠(*, n, k)⁠ (a batch of matrices or "multiple right-hand sides"), this function returns X of shape ⁠(*, n)⁠ or ⁠(*, n, k)⁠ respectively.

• Otherwise, if A has shape ⁠(*, n, n)⁠ and B has shape ⁠(n,)⁠ or ⁠(n, k)⁠, B is broadcasted to have shape ⁠(*, n)⁠ or ⁠(*, n, k)⁠ respectively.

This function then returns the solution of the resulting batch of systems of linear equations.

Note

This function computes X = A$inverse() @ B in a faster and more numerically stable way than performing the computations separately.

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(3, 3)
b <- torch_randn(3)
x <- linalg_solve(A, b)
torch_allclose(torch_matmul(A, x), b)
}

Triangular solve

Description

Triangular solve

Usage

linalg_solve_triangular(A, B, ..., upper, left = TRUE, unitriangular = FALSE)

Arguments

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Computes the singular value decomposition (SVD) of a matrix.

Description

Letting \mathbb{K} be \mathbb{R} or \mathbb{C}, the full SVD of a matrix A \in \mathbb{K}^{m \times n}, if k = min(m,n), is defined as

Usage

linalg_svd(A, full_matrices = TRUE)

Arguments

Details

where \operatorname{diag}(S) \in \mathbb{K}^{m \times n}, V^{H} is the conjugate transpose when V is complex, and the transpose when V is real-valued.

The matrices U, V (and thus V^{H}) are orthogonal in the real case, and unitary in the complex case. When m > n (resp. m < n) we can drop the last m - n (resp. n - m) columns of U (resp. V) to form the reduced SVD:

where \operatorname{diag}(S) \in \mathbb{K}^{k \times k}.

In this case, U and V also have orthonormal columns. Supports input of float, double, cfloat and cdouble dtypes.

Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions.

The returned decomposition is a named tuple ⁠(U, S, V)⁠ which corresponds to U, S, V^{H} above.

The singular values are returned in descending order. The parameter full_matrices chooses between the full (default) and reduced SVD.

Value

A list ⁠(U, S, V)⁠ which corresponds to U, S, V^{H} above. S will always be real-valued, even when A is complex. It will also be ordered in descending order. U and V will have the same dtype as A. The left / right singular vectors will be given by the columns of U and the rows of V respectively.

Warnings

The returned tensors U and V are not unique, nor are they continuous with respect to A. Due to this lack of uniqueness, different hardware and software may compute different singular vectors. This non-uniqueness is caused by the fact that multiplying any pair of singular vectors u_k, v_k by -1 in the real case or by e^{i \phi}, \phi \in \mathbb{R} in the complex case produces another two valid singular vectors of the matrix. This non-uniqueness problem is even worse when the matrix has repeated singular values. In this case, one may multiply the associated singular vectors of U and V spanning the subspace by a rotation matrix and the resulting vectors will span the same subspace.

Gradients computed using U or V will only be finite when A does not have zero as a singular value or repeated singular values. Furthermore, if the distance between any two singular values is close to zero, the gradient will be numerically unstable, as it depends on the singular values \sigma_i through the computation of \frac{1}{\min_{i \neq j} \sigma_i^2 - \sigma_j^2}. The gradient will also be numerically unstable when A has small singular values, as it also depends on the computaiton of \frac{1}{\sigma_i}.

Note

When full_matrices=TRUE, the gradients with respect to ⁠U[..., :, min(m, n):]⁠ and ⁠Vh[..., min(m, n):, :]⁠ will be ignored, as those vectors can be arbitrary bases of the corresponding subspaces.

See Also

• linalg_svdvals() computes only the singular values. Unlike linalg_svd(), the gradients of linalg_svdvals() are always numerically stable.

• linalg_eig() for a function that computes another type of spectral decomposition of a matrix. The eigendecomposition works just on on square matrices.

• linalg_eigh() for a (faster) function that computes the eigenvalue decomposition for Hermitian and symmetric matrices.

• linalg_qr() for another (much faster) decomposition that works on general matrices.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {

a <- torch_randn(5, 3)
linalg_svd(a, full_matrices = FALSE)
}

Computes the singular values of a matrix.

Description

Supports input of float, double, cfloat and cdouble dtypes. Also supports batches of matrices, and if A is a batch of matrices then the output has the same batch dimensions. The singular values are returned in descending order.

Usage

linalg_svdvals(A)

Arguments

Value

A real-valued tensor, even when A is complex.

See Also

linalg_svd() computes the full singular value decomposition.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_tensorinv(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_randn(5, 3)
S <- linalg_svdvals(A)
S
}

Computes the multiplicative inverse of torch_tensordot()

Description

If m is the product of the first ind dimensions of A and n is the product of the rest of the dimensions, this function expects m and n to be equal. If this is the case, it computes a tensor X such that tensordot(A, X, ind) is the identity matrix in dimension m.

Usage

linalg_tensorinv(A, ind = 3L)

Arguments

Details

Supports input of float, double, cfloat and cdouble dtypes.

Note

Consider using linalg_tensorsolve() if possible for multiplying a tensor on the left by the tensor inverse as ⁠linalg_tensorsolve(A, B) == torch_tensordot(linalg_tensorinv(A), B))⁠

It is always prefered to use linalg_tensorsolve() when possible, as it is faster and more numerically stable than computing the pseudoinverse explicitly.

See Also

• linalg_tensorsolve() computes ⁠torch_tensordot(linalg_tensorinv(A), B))⁠.

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorsolve(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_eye(4 * 6)$reshape(c(4, 6, 8, 3))
Ainv <- linalg_tensorinv(A, ind = 3)
Ainv$shape
B <- torch_randn(4, 6)
torch_allclose(torch_tensordot(Ainv, B), linalg_tensorsolve(A, B))

A <- torch_randn(4, 4)
Atensorinv <- linalg_tensorinv(A, 2)
Ainv <- linalg_inv(A)
torch_allclose(Atensorinv, Ainv)
}

Computes the solution X to the system torch_tensordot(A, X) = B.

Description

If m is the product of the first B\ .ndim dimensions of A and n is the product of the rest of the dimensions, this function expects m and n to be equal. The returned tensor x satisfies tensordot(A, x, dims=x$ndim) == B.

Usage

linalg_tensorsolve(A, B, dims = NULL)

Arguments

Details

If dims is specified, A will be reshaped as A = movedim(A, dims, seq(len(dims) - A$ndim + 1, 0))

Supports inputs of float, double, cfloat and cdouble dtypes.

See Also

• linalg_tensorinv() computes the multiplicative inverse of torch_tensordot().

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_vector_norm()

Examples

if (torch_is_installed()) {
A <- torch_eye(2 * 3 * 4)$reshape(c(2 * 3, 4, 2, 3, 4))
B <- torch_randn(2 * 3, 4)
X <- linalg_tensorsolve(A, B)
X$shape
torch_allclose(torch_tensordot(A, X, dims = X$ndim), B)

A <- torch_randn(6, 4, 4, 3, 2)
B <- torch_randn(4, 3, 2)
X <- linalg_tensorsolve(A, B, dims = c(1, 3))
A <- A$permute(c(2, 4, 5, 1, 3))
torch_allclose(torch_tensordot(A, X, dims = X$ndim), B, atol = 1e-6)
}

Computes a vector norm.

Description

If A is complex valued, it computes the norm of A$abs() Supports input of float, double, cfloat and cdouble dtypes. This function does not necessarily treat multidimensonal A as a batch of vectors, instead:

Usage

linalg_vector_norm(A, ord = 2, dim = NULL, keepdim = FALSE, dtype = NULL)

Arguments

Details

• If dim=NULL, A will be flattened before the norm is computed.

• If dim is an int or a tuple, the norm will be computed over these dimensions and the other dimensions will be treated as batch dimensions.

This behavior is for consistency with linalg_norm().

ord defines the norm that is computed. The following norms are supported:

See Also

Other linalg: linalg_cholesky(), linalg_cholesky_ex(), linalg_det(), linalg_eig(), linalg_eigh(), linalg_eigvals(), linalg_eigvalsh(), linalg_householder_product(), linalg_inv(), linalg_inv_ex(), linalg_lstsq(), linalg_matrix_norm(), linalg_matrix_power(), linalg_matrix_rank(), linalg_multi_dot(), linalg_norm(), linalg_pinv(), linalg_qr(), linalg_slogdet(), linalg_solve(), linalg_solve_triangular(), linalg_svd(), linalg_svdvals(), linalg_tensorinv(), linalg_tensorsolve()

Examples

if (torch_is_installed()) {
a <- torch_arange(0, 8, dtype = torch_float()) - 4
a
b <- a$reshape(c(3, 3))
b

linalg_vector_norm(a, ord = 3.5)
linalg_vector_norm(b, ord = 3.5)
}

Load a state dict file

Description

This function should only be used to load models saved in python. For it to work correctly you need to use torch.save with the flag: ⁠_use_new_zipfile_serialization=True⁠ and also remove all nn.Parameter classes from the tensors in the dict.

Usage

load_state_dict(path, ..., legacy_stream = FALSE)

Arguments

Details

The above might change with development of this in pytorch's C++ api.

Value

a named list of tensors.

Autocast context manager

Description

Allow regions of your code to run in mixed precision. In these regions, ops run in an op-specific dtype chosen by autocast to improve performance while maintaining accuracy.

Usage

local_autocast(
  device_type,
  dtype = NULL,
  enabled = TRUE,
  cache_enabled = NULL,
  ...,
  .env = parent.frame()
)

with_autocast(
  code,
  ...,
  device_type,
  dtype = NULL,
  enabled = TRUE,
  cache_enabled = NULL
)

set_autocast(device_type, dtype = NULL, enabled = TRUE, cache_enabled = NULL)

unset_autocast(context)

Arguments

Details

When entering an autocast-enabled region, Tensors may be any type. You should not call half() or bfloat16() on your model(s) or inputs when using autocasting.

autocast should only be enabled during the forward pass(es) of your network, including the loss computation(s). Backward passes under autocast are not recommended. Backward ops run in the same type that autocast used for corresponding forward ops.

Functions

• with_autocast(): A with context for automatic mixed precision.

• set_autocast(): Set the autocast context. For advanced users only.

• unset_autocast(): Unset the autocast context.

See Also

cuda_amp_grad_scaler() to perform dynamic gradient scaling.

Examples

if (torch_is_installed()) {
x <- torch_randn(5, 5, dtype = torch_float32())
y <- torch_randn(5, 5, dtype = torch_float32())

foo <- function(x, y) {
  local_autocast(device = "cpu")
  z <- torch_mm(x, y)
  w <- torch_mm(z, x)
  w
}

out <- foo(x, y)
}

Device contexts

Description

Device contexts

Usage

local_device(device, ..., .env = parent.frame())

with_device(code, ..., device)

Arguments

Functions

• with_device(): Modifies the default device for the selected context.

Set the learning rate of each parameter group using a cosine annealing schedule

Description

Set the learning rate of each parameter group using a cosine annealing schedule

Usage

lr_cosine_annealing(
  optimizer,
  T_max,
  eta_min = 0,
  last_epoch = -1,
  verbose = FALSE
)

Arguments

Sets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr.

Description

Sets the learning rate of each parameter group to the initial lr times a given function. When last_epoch=-1, sets initial lr as lr.

Usage

lr_lambda(optimizer, lr_lambda, last_epoch = -1, verbose = FALSE)

Arguments

Examples

if (torch_is_installed()) {
# Assuming optimizer has two groups.
lambda1 <- function(epoch) epoch %/% 30
lambda2 <- function(epoch) 0.95^epoch
## Not run: 
scheduler <- lr_lambda(optimizer, lr_lambda = list(lambda1, lambda2))
for (epoch in 1:100) {
  train(...)
  validate(...)
  scheduler$step()
}

## End(Not run)

}

Multiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.

Description

Multiply the learning rate of each parameter group by the factor given in the specified function. When last_epoch=-1, sets initial lr as lr.

Usage

lr_multiplicative(optimizer, lr_lambda, last_epoch = -1, verbose = FALSE)

Arguments

Examples

if (torch_is_installed()) {
## Not run: 
lmbda <- function(epoch) 0.95
scheduler <- lr_multiplicative(optimizer, lr_lambda = lmbda)
for (epoch in 1:100) {
  train(...)
  validate(...)
  scheduler$step()
}

## End(Not run)

}

Once cycle learning rate

Description

Sets the learning rate of each parameter group according to the 1cycle learning rate policy. The 1cycle policy anneals the learning rate from an initial learning rate to some maximum learning rate and then from that maximum learning rate to some minimum learning rate much lower than the initial learning rate.

Usage

lr_one_cycle(
  optimizer,
  max_lr,
  total_steps = NULL,
  epochs = NULL,
  steps_per_epoch = NULL,
  pct_start = 0.3,
  anneal_strategy = "cos",
  cycle_momentum = TRUE,
  base_momentum = 0.85,
  max_momentum = 0.95,
  div_factor = 25,
  final_div_factor = 10000,
  last_epoch = -1,
  verbose = FALSE
)

Arguments

Details

This policy was initially described in the paper Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates.

The 1cycle learning rate policy changes the learning rate after every batch. step should be called after a batch has been used for training. This scheduler is not chainable.

Note also that the total number of steps in the cycle can be determined in one of two ways (listed in order of precedence):

• A value for total_steps is explicitly provided.

• A number of epochs (epochs) and a number of steps per epoch (steps_per_epoch) are provided.

In this case, the number of total steps is inferred by total_steps = epochs * steps_per_epoch

You must either provide a value for total_steps or provide a value for both epochs and steps_per_epoch.

Examples

if (torch_is_installed()) {
## Not run: 
data_loader <- dataloader(...)
optimizer <- optim_sgd(model$parameters, lr = 0.1, momentum = 0.9)
scheduler <- lr_one_cycle(optimizer,
  max_lr = 0.01, steps_per_epoch = length(data_loader),
  epochs = 10
)

for (i in 1:epochs) {
  coro::loop(for (batch in data_loader) {
    train_batch(...)
    scheduler$step()
  })
}

## End(Not run)

}

Reduce learning rate on plateau

Description

Reduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This scheduler reads a metrics quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced.

Usage

lr_reduce_on_plateau(
  optimizer,
  mode = "min",
  factor = 0.1,
  patience = 10,
  threshold = 1e-04,
  threshold_mode = "rel",
  cooldown = 0,
  min_lr = 0,
  eps = 1e-08,
  verbose = FALSE
)

Arguments

Examples

if (torch_is_installed()) {
## Not run:  
optimizer <- optim_sgd(model$parameters(), lr=0.1, momentum=0.9)
scheduler <- lr_reduce_on_plateau(optimizer, 'min')
for (epoch in 1:10) {
 train(...)
 val_loss <- validate(...)
 # note that step should be called after validate
 scheduler$step(val_loss)
}

## End(Not run)
}

Creates learning rate schedulers

Description

Creates learning rate schedulers

Usage

lr_scheduler(
  classname = NULL,
  inherit = LRScheduler,
  ...,
  parent_env = parent.frame()
)

Arguments

Step learning rate decay

Description

Decays the learning rate of each parameter group by gamma every step_size epochs. Notice that such decay can happen simultaneously with other changes to the learning rate from outside this scheduler. When last_epoch=-1, sets initial lr as lr.

Usage

lr_step(optimizer, step_size, gamma = 0.1, last_epoch = -1)

Arguments

Examples

if (torch_is_installed()) {
## Not run: 
# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.05     if epoch < 30
# lr = 0.005    if 30 <= epoch < 60
# lr = 0.0005   if 60 <= epoch < 90
# ...
scheduler <- lr_step(optimizer, step_size = 30, gamma = 0.1)
for (epoch in 1:100) {
  train(...)
  validate(...)
  scheduler$step()
}

## End(Not run)

}

Applies a 1D adaptive average pooling over an input signal composed of several input planes.

Description

The output size is H, for any input size. The number of output features is equal to the number of input planes.

Usage

nn_adaptive_avg_pool1d(output_size)

Arguments

Examples

if (torch_is_installed()) {
# target output size of 5
m <- nn_adaptive_avg_pool1d(5)
input <- torch_randn(1, 64, 8)
output <- m(input)
}

Applies a 2D adaptive average pooling over an input signal composed of several input planes.

Description

The output is of size H x W, for any input size. The number of output features is equal to the number of input planes.

Usage

nn_adaptive_avg_pool2d(output_size)

Arguments

Examples

if (torch_is_installed()) {
# target output size of 5x7
m <- nn_adaptive_avg_pool2d(c(5, 7))
input <- torch_randn(1, 64, 8, 9)
output <- m(input)
# target output size of 7x7 (square)
m <- nn_adaptive_avg_pool2d(7)
input <- torch_randn(1, 64, 10, 9)
output <- m(input)
}

Applies a 3D adaptive average pooling over an input signal composed of several input planes.

Description

The output is of size D x H x W, for any input size. The number of output features is equal to the number of input planes.

Usage

nn_adaptive_avg_pool3d(output_size)

Arguments

Examples

if (torch_is_installed()) {
# target output size of 5x7x9
m <- nn_adaptive_avg_pool3d(c(5, 7, 9))
input <- torch_randn(1, 64, 8, 9, 10)
output <- m(input)
# target output size of 7x7x7 (cube)
m <- nn_adaptive_avg_pool3d(7)
input <- torch_randn(1, 64, 10, 9, 8)
output <- m(input)
}

AdaptiveLogSoftmaxWithLoss module

Description

Efficient softmax approximation as described in Efficient softmax approximation for GPUs by Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou

Usage

nn_adaptive_log_softmax_with_loss(
  in_features,
  n_classes,
  cutoffs,
  div_value = 4,
  head_bias = FALSE
)

Arguments

Details

Adaptive softmax is an approximate strategy for training models with large output spaces. It is most effective when the label distribution is highly imbalanced, for example in natural language modelling, where the word frequency distribution approximately follows the Zipf's law.

Adaptive softmax partitions the labels into several clusters, according to their frequency. These clusters may contain different number of targets each.

Additionally, clusters containing less frequent labels assign lower dimensional embeddings to those labels, which speeds up the computation. For each minibatch, only clusters for which at least one target is present are evaluated.

The idea is that the clusters which are accessed frequently (like the first one, containing most frequent labels), should also be cheap to compute – that is, contain a small number of assigned labels. We highly recommend taking a look at the original paper for more details.

• cutoffs should be an ordered Sequence of integers sorted in the increasing order. It controls number of clusters and the partitioning of targets into clusters. For example setting cutoffs = c(10, 100, 1000) means that first 10 targets will be assigned to the 'head' of the adaptive softmax, targets ⁠11, 12, ..., 100⁠ will be assigned to the first cluster, and targets ⁠101, 102, ..., 1000⁠ will be assigned to the second cluster, while targets ⁠1001, 1002, ..., n_classes - 1⁠ will be assigned to the last, third cluster.

• div_value is used to compute the size of each additional cluster, which is given as \left\lfloor\frac{\mbox{in\_features}}{\mbox{div\_value}^{idx}}\right\rfloor, where idx is the cluster index (with clusters for less frequent words having larger indices, and indices starting from 1).

• head_bias if set to True, adds a bias term to the 'head' of the adaptive softmax. See paper for details. Set to False in the official implementation.

Value

NamedTuple with output and loss fields:

• output is a Tensor of size N containing computed target log probabilities for each example

• loss is a Scalar representing the computed negative log likelihood loss

Warning

Labels passed as inputs to this module should be sorted according to their frequency. This means that the most frequent label should be represented by the index 0, and the least frequent label should be represented by the index n_classes - 1.

Shape

• input: (N, \mbox{in\_features})

• target: (N) where each value satisfies 0 <= \mbox{target[i]} <= \mbox{n\_classes}

• output1: (N)

• output2: Scalar

Note

This module returns a NamedTuple with output and loss fields. See further documentation for details.

To compute log-probabilities for all classes, the log_prob method can be used.

Applies a 1D adaptive max pooling over an input signal composed of several input planes.

Description

The output size is H, for any input size. The number of output features is equal to the number of input planes.

Usage

nn_adaptive_max_pool1d(output_size, return_indices = FALSE)

Arguments

Examples

if (torch_is_installed()) {
# target output size of 5
m <- nn_adaptive_max_pool1d(5)
input <- torch_randn(1, 64, 8)
output <- m(input)
}

Applies a 2D adaptive max pooling over an input signal composed of several input planes.

Description

The output is of size H x W, for any input size. The number of output features is equal to the number of input planes.

Usage

nn_adaptive_max_pool2d(output_size, return_indices = FALSE)

Arguments

Examples

if (torch_is_installed()) {
# target output size of 5x7
m <- nn_adaptive_max_pool2d(c(5, 7))
input <- torch_randn(1, 64, 8, 9)
output <- m(input)
# target output size of 7x7 (square)
m <- nn_adaptive_max_pool2d(7)
input <- torch_randn(1, 64, 10, 9)
output <- m(input)
}

Applies a 3D adaptive max pooling over an input signal composed of several input planes.

Description

The output is of size D x H x W, for any input size. The number of output features is equal to the number of input planes.

Usage

nn_adaptive_max_pool3d(output_size, return_indices = FALSE)

Arguments

Examples

if (torch_is_installed()) {
# target output size of 5x7x9
m <- nn_adaptive_max_pool3d(c(5, 7, 9))
input <- torch_randn(1, 64, 8, 9, 10)
output <- m(input)
# target output size of 7x7x7 (cube)
m <- nn_adaptive_max_pool3d(7)
input <- torch_randn(1, 64, 10, 9, 8)
output <- m(input)
}

AUM loss

Description

Creates a criterion that measures the Area under the Min(FPR, FNR) (AUM) between each element in the input pred_tensor and target label_tensor.

Usage

nn_aum_loss()

Details

This is used for measuring the error of a binary reconstruction within highly unbalanced dataset, where the goal is optimizing the ROC curve. Note that the targets label_tensor should be factor level of the binary outcome, i.e. with values 1L and 2L.

References

J. Hillman, T.D. Hocking: Optimizing ROC Curves with a Sort-Based Surrogate Loss for Binary Classification and Changepoint Detection https://jmlr.org/papers/volume24/21-0751/21-0751.pdf

Examples

if (torch_is_installed()) {

loss <- nn_aum_loss()
input <- torch_randn(4, 6, requires_grad = TRUE)
target <- input > 1.5
output <- loss(input, target)
output$backward()

}

Applies a 1D average pooling over an input signal composed of several input planes.

Description

In the simplest case, the output value of the layer with input size (N, C, L), output (N, C, L_{out}) and kernel_size k can be precisely described as:

Usage

nn_avg_pool1d(
  kernel_size,
  stride = NULL,
  padding = 0,
  ceil_mode = FALSE,
  count_include_pad = TRUE
)

Arguments

Details

\mbox{out}(N_i, C_j, l) = \frac{1}{k} \sum_{m=0}^{k-1} \mbox{input}(N_i, C_j, \mbox{stride} \times l + m)

If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points.

The parameters kernel_size, stride, padding can each be an int or a one-element tuple.

Shape

• Input: (N, C, L_{in})

• Output: (N, C, L_{out}), where

L_{out} = \left\lfloor \frac{L_{in} + 2 \times \mbox{padding} - \mbox{kernel\_size}}{\mbox{stride}} + 1\right\rfloor

Examples

if (torch_is_installed()) {

# pool with window of size=3, stride=2
m <- nn_avg_pool1d(3, stride = 2)
m(torch_randn(1, 1, 8))
}

Applies a 2D average pooling over an input signal composed of several input planes.

Description

In the simplest case, the output value of the layer with input size (N, C, H, W), output (N, C, H_{out}, W_{out}) and kernel_size (kH, kW) can be precisely described as:

Usage

nn_avg_pool2d(
  kernel_size,
  stride = NULL,
  padding = 0,
  ceil_mode = FALSE,
  count_include_pad = TRUE,
  divisor_override = NULL
)

Arguments

Details

out(N_i, C_j, h, w) = \frac{1}{kH * kW} \sum_{m=0}^{kH-1} \sum_{n=0}^{kW-1} input(N_i, C_j, stride[0] \times h + m, stride[1] \times w + n)

If padding is non-zero, then the input is implicitly zero-padded on both sides for padding number of points.

The parameters kernel_size, stride, padding can either be:

• a single int – in which case the same value is used for the height and width dimension

• a tuple of two ints – in which case, the first int is used for the height dimension, and the second int for the width dimension

Shape

• Input: (N, C, H_{in}, W_{in})

• Output: (N, C, H_{out}, W_{out}), where

H_{out} = \left\lfloor\frac{H_{in} + 2 \times \mbox{padding}[0] - \mbox{kernel\_size}[0]}{\mbox{stride}[0]} + 1\right\rfloor

W_{out} = \left\lfloor\frac{W_{in} + 2 \times \mbox{padding}[1] - \mbox{kernel\_size}[1]}{\mbox{stride}[1]} + 1\right\rfloor

Examples

if (torch_is_installed()) {

# pool of square window of size=3, stride=2
m <- nn_avg_pool2d(3, stride = 2)
# pool of non-square window
m <- nn_avg_pool2d(c(3, 2), stride = c(2, 1))
input <- torch_randn(20, 16, 50, 32)
output <- m(input)
}

Applies a 3D average pooling over an input signal composed of several input planes.

Description

In the simplest case, the output value of the layer with input size (N, C, D, H, W), output (N, C, D_{out}, H_{out}, W_{out}) and kernel_size (kD, kH, kW) can be precisely described as:

Usage

nn_avg_pool3d(
  kernel_size,
  stride = NULL,
  padding = 0,
  ceil_mode = FALSE,
  count_include_pad = TRUE,
  divisor_override = NULL
)

Arguments

Details

\begin{array}{ll} \mbox{out}(N_i, C_j, d, h, w) = & \sum_{k=0}^{kD-1} \sum_{m=0}^{kH-1} \sum_{n=0}^{kW-1} \\ & \frac{\mbox{input}(N_i, C_j, \mbox{stride}[0] \times d + k, \mbox{stride}[1] \times h + m, \mbox{stride}[2] \times w + n)}{kD \times kH \times kW} \end{array}

If padding is non-zero, then the input is implicitly zero-padded on all three sides for padding number of points.

The parameters kernel_size, stride can either be:

• a single int – in which case the same value is used for the depth, height and width dimension

• a tuple of three ints – in which case, the first int is used for the depth dimension, the second int for the height dimension and the third int for the width dimension

Shape

• Input: (N, C, D_{in}, H_{in}, W_{in})

• Output: (N, C, D_{out}, H_{out}, W_{out}), where

D_{out} = \left\lfloor\frac{D_{in} + 2 \times \mbox{padding}[0] - \mbox{kernel\_size}[0]}{\mbox{stride}[0]} + 1\right\rfloor

H_{out} = \left\lfloor\frac{H_{in} + 2 \times \mbox{padding}[1] - \mbox{kernel\_size}[1]}{\mbox{stride}[1]} + 1\right\rfloor

W_{out} = \left\lfloor\frac{W_{in} + 2 \times \mbox{padding}[2] - \mbox{kernel\_size}[2]}{\mbox{stride}[2]} + 1\right\rfloor

Examples

if (torch_is_installed()) {

# pool of square window of size=3, stride=2
m <- nn_avg_pool3d(3, stride = 2)
# pool of non-square window
m <- nn_avg_pool3d(c(3, 2, 2), stride = c(2, 1, 2))
input <- torch_randn(20, 16, 50, 44, 31)
output <- m(input)
}

BatchNorm1D module

Description

Applies Batch Normalization over a 2D or 3D input (a mini-batch of 1D inputs with optional additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Usage

nn_batch_norm1d(
  num_features,
  eps = 1e-05,
  momentum = 0.1,
  affine = TRUE,
  track_running_stats = TRUE
)

Arguments

Details

y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta

The mean and standard-deviation are calculated per-dimension over the mini-batches and \gamma and \beta are learnable parameter vectors of size C (where C is the input size). By default, the elements of \gamma are set to 1 and the elements of \beta are set to 0.

Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:momentum of 0.1. If track_running_stats is set to FALSE, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.

Note

This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is \hat{x}_{\mbox{new}} = (1 - \mbox{momentum}) \times \hat{x} + \mbox{momentum} \times x_t, where \hat{x} is the estimated statistic and x_t is the new observed value.

Because the Batch Normalization is done over the C dimension, computing statistics on ⁠(N, L)⁠ slices, it's common terminology to call this Temporal Batch Normalization.

Shape

• Input: (N, C) or (N, C, L)

• Output: (N, C) or (N, C, L) (same shape as input)

Examples

if (torch_is_installed()) {
# With Learnable Parameters
m <- nn_batch_norm1d(100)
# Without Learnable Parameters
m <- nn_batch_norm1d(100, affine = FALSE)
input <- torch_randn(20, 100)
output <- m(input)
}

BatchNorm2D

Description

Applies Batch Normalization over a 4D input (a mini-batch of 2D inputs additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.

Usage

nn_batch_norm2d(
  num_features,
  eps = 1e-05,
  momentum = 0.1,
  affine = TRUE,
  track_running_stats = TRUE
)

Arguments

Details

y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta

The mean and standard-deviation are calculated per-dimension over the mini-batches and \gamma and \beta are learnable parameter vectors of size C (where C is the input size). By default, the elements of \gamma are set to 1 and the elements of \beta are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to torch_var(input, unbiased=FALSE). Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.

If track_running_stats is set to FALSE, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.

Shape

• Input: (N, C, H, W)

• Output: (N, C, H, W) (same shape as input)

Note

This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is \hat{x}_{\mbox{new}} = (1 - \mbox{momentum}) \times \hat{x} + \mbox{momentum} \times x_t, where \hat{x} is the estimated statistic and x_t is the new observed value. Because the Batch Normalization is done over the C dimension, computing statistics on ⁠(N, H, W)⁠ slices, it's common terminology to call this Spatial Batch Normalization.

Examples

if (torch_is_installed()) {
# With Learnable Parameters
m <- nn_batch_norm2d(100)
# Without Learnable Parameters
m <- nn_batch_norm2d(100, affine = FALSE)
input <- torch_randn(20, 100, 35, 45)
output <- m(input)
}

BatchNorm3D

Description

Applies Batch Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.

Usage

nn_batch_norm3d(
  num_features,
  eps = 1e-05,
  momentum = 0.1,
  affine = TRUE,
  track_running_stats = TRUE
)

Arguments

Details

y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta

The mean and standard-deviation are calculated per-dimension over the mini-batches and \gamma and \beta are learnable parameter vectors of size C (where C is the input size). By default, the elements of \gamma are set to 1 and the elements of \beta are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to torch_var(input, unbiased = FALSE).

Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default momentum of 0.1.

If track_running_stats is set to FALSE, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.

Shape

• Input: (N, C, D, H, W)

• Output: (N, C, D, H, W) (same shape as input)

Note

This momentum argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is: \hat{x}_{\mbox{new}} = (1 - \mbox{momentum}) \times \hat{x} + \mbox{momentum} \times x_t, where \hat{x} is the estimated statistic and x_t is the new observed value.

Because the Batch Normalization is done over the C dimension, computing statistics on ⁠(N, D, H, W)⁠ slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization.

Examples

if (torch_is_installed()) {
# With Learnable Parameters
m <- nn_batch_norm3d(100)
# Without Learnable Parameters
m <- nn_batch_norm3d(100, affine = FALSE)
input <- torch_randn(20, 100, 35, 45, 55)
output <- m(input)
}

Binary cross entropy loss

Description

Creates a criterion that measures the Binary Cross Entropy between the target and the output:

Usage

nn_bce_loss(weight = NULL, reduction = "mean")

Arguments

Details

The unreduced (i.e. with reduction set to 'none') loss can be described as:

\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad l_n = - w_n \left[ y_n \cdot \log x_n + (1 - y_n) \cdot \log (1 - x_n) \right]

where N is the batch size. If reduction is not 'none' (default 'mean'), then

\ell(x, y) = \left\{ \begin{array}{ll} \mbox{mean}(L), & \mbox{if reduction} = \mbox{'mean';}\\ \mbox{sum}(L), & \mbox{if reduction} = \mbox{'sum'.} \end{array} \right.

This is used for measuring the error of a reconstruction in for example an auto-encoder. Note that the targets y should be numbers between 0 and 1.

Notice that if x_n is either 0 or 1, one of the log terms would be mathematically undefined in the above loss equation. PyTorch chooses to set \log (0) = -\infty, since \lim_{x\to 0} \log (x) = -\infty.

However, an infinite term in the loss equation is not desirable for several reasons. For one, if either y_n = 0 or (1 - y_n) = 0, then we would be multiplying 0 with infinity. Secondly, if we have an infinite loss value, then we would also have an infinite term in our gradient, since \lim_{x\to 0} \frac{d}{dx} \log (x) = \infty.

This would make BCELoss's backward method nonlinear with respect to x_n, and using it for things like linear regression would not be straight-forward. Our solution is that BCELoss clamps its log function outputs to be greater than or equal to -100. This way, we can always have a finite loss value and a linear backward method.

Shape

• Input: (N, *) where * means, any number of additional dimensions

• Target: (N, *), same shape as the input

• Output: scalar. If reduction is 'none', then (N, *), same shape as input.

Examples

if (torch_is_installed()) {
m <- nn_sigmoid()
loss <- nn_bce_loss()
input <- torch_randn(3, requires_grad = TRUE)
target <- torch_rand(3)
output <- loss(m(input), target)
output$backward()
}

BCE with logits loss

Description

This loss combines a Sigmoid layer and the BCELoss in one single class. This version is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability.

Usage

nn_bce_with_logits_loss(weight = NULL, reduction = "mean", pos_weight = NULL)

Arguments

Details

The unreduced (i.e. with reduction set to 'none') loss can be described as:

\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad l_n = - w_n \left[ y_n \cdot \log \sigma(x_n) + (1 - y_n) \cdot \log (1 - \sigma(x_n)) \right],

where N is the batch size. If reduction is not 'none' (default 'mean'), then

\ell(x, y) = \begin{array}{ll} \mbox{mean}(L), & \mbox{if reduction} = \mbox{'mean';}\\ \mbox{sum}(L), & \mbox{if reduction} = \mbox{'sum'.} \end{array}

This is used for measuring the error of a reconstruction in for example an auto-encoder. Note that the targets t[i] should be numbers between 0 and 1. It's possible to trade off recall and precision by adding weights to positive examples. In the case of multi-label classification the loss can be described as:

\ell_c(x, y) = L_c = \{l_{1,c},\dots,l_{N,c}\}^\top, \quad l_{n,c} = - w_{n,c} \left[ p_c y_{n,c} \cdot \log \sigma(x_{n,c}) + (1 - y_{n,c}) \cdot \log (1 - \sigma(x_{n,c})) \right],

where c is the class number (c > 1 for multi-label binary classification,

c = 1 for single-label binary classification), n is the number of the sample in the batch and p_c is the weight of the positive answer for the class c. p_c > 1 increases the recall, p_c < 1 increases the precision. For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to \frac{300}{100}=3. The loss would act as if the dataset contains 3\times 100=300 positive examples.

Shape

• Input: (N, *) where * means, any number of additional dimensions

• Target: (N, *), same shape as the input

• Output: scalar. If reduction is 'none', then (N, *), same shape as input.

Examples

if (torch_is_installed()) {
loss <- nn_bce_with_logits_loss()
input <- torch_randn(3, requires_grad = TRUE)
target <- torch_empty(3)$random_(1, 2)
output <- loss(input, target)
output$backward()

target <- torch_ones(10, 64, dtype = torch_float32()) # 64 classes, batch size = 10
output <- torch_full(c(10, 64), 1.5) # A prediction (logit)
pos_weight <- torch_ones(64) # All weights are equal to 1
criterion <- nn_bce_with_logits_loss(pos_weight = pos_weight)
criterion(output, target) # -log(sigmoid(1.5))
}

Bilinear module

Description

Applies a bilinear transformation to the incoming data y = x_1^T A x_2 + b

Usage

nn_bilinear(in1_features, in2_features, out_features, bias = TRUE)

Arguments

Shape

• Input1: (N, *, H_{in1}) H_{in1}=\mbox{in1\_features} and * means any number of additional dimensions. All but the last dimension of the inputs should be the same.

• Input2: (N, *, H_{in2}) where H_{in2}=\mbox{in2\_features}.

• Output: (N, *, H_{out}) where H_{out}=\mbox{out\_features} and all but the last dimension are the same shape as the input.

Attributes

• weight: the learnable weights of the module of shape (\mbox{out\_features}, \mbox{in1\_features}, \mbox{in2\_features}). The values are initialized from \mathcal{U}(-\sqrt{k}, \sqrt{k}), where k = \frac{1}{\mbox{in1\_features}}

• bias: the learnable bias of the module of shape (\mbox{out\_features}). If bias is TRUE, the values are initialized from \mathcal{U}(-\sqrt{k}, \sqrt{k}), where k = \frac{1}{\mbox{in1\_features}}

Examples

if (torch_is_installed()) {
m <- nn_bilinear(20, 30, 50)
input1 <- torch_randn(128, 20)
input2 <- torch_randn(128, 30)
output <- m(input1, input2)
print(output$size())
}

Creates a nn_buffer

Description

Indicates that a tensor is a buffer in a nn_module

Usage

nn_buffer(x, persistent = TRUE)

Arguments

CELU module

Description

Applies the element-wise function:

Usage

nn_celu(alpha = 1, inplace = FALSE)

Arguments

Details

\mbox{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1))

More details can be found in the paper Continuously Differentiable Exponential Linear Units.

Shape

• Input: (N, *) where * means, any number of additional dimensions

• Output: (N, *), same shape as the input

Examples

if (torch_is_installed()) {
m <- nn_celu()
input <- torch_randn(2)
output <- m(input)
}

Sparsemax activation

Description

Sparsemax activation module.

Usage

nn_contrib_sparsemax(dim = -1)

Arguments

Details

The SparseMax activation is described in 'From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification' The implementation is based on aced125/sparsemax

Conv1D module

Description

Applies a 1D convolution over an input signal composed of several input planes. In the simplest case, the output value of the layer with input size (N, C_{\mbox{in}}, L) and output (N, C_{\mbox{out}}, L_{\mbox{out}}) can be precisely described as:

Usage

nn_conv1d(
  in_channels,
  out_channels,
  kernel_size,
  stride = 1,
  padding = 0,
  dilation = 1,
  groups = 1,
  bias = TRUE,
  padding_mode = "zeros"
)

Arguments

Details

\mbox{out}(N_i, C_{\mbox{out}_j}) = \mbox{bias}(C_{\mbox{out}_j}) + \sum_{k = 0}^{C_{in} - 1} \mbox{weight}(C_{\mbox{out}_j}, k) \star \mbox{input}(N_i, k)

where \star is the valid cross-correlation operator, N is a batch size, C denotes a number of channels, L is a length of signal sequence.

• stride controls the stride for the cross-correlation, a single number or a one-element tuple.

• padding controls the amount of implicit zero-paddings on both sides for padding number of points.

• dilation controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this link has a nice visualization of what dilation does.

• groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,

  • At groups=1, all inputs are convolved to all outputs.

  • At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.

  • At groups= in_channels, each input channel is convolved with its own set of filters, of size \left\lfloor\frac{out\_channels}{in\_channels}\right\rfloor.

Note

Depending of the size of your kernel, several (of the last) columns of the input might be lost, because it is a valid cross-correlation, and not a full cross-correlation. It is up to the user to add proper padding.

When groups == in_channels and out_channels == K * in_channels, where K is a positive integer, this operation is also termed in literature as depthwise convolution. In other words, for an input of size (N, C_{in}, L_{in}), a depthwise convolution with a depthwise multiplier K, can be constructed by arguments (C_{\mbox{in}}=C_{in}, C_{\mbox{out}}=C_{in} \times K, ..., \mbox{groups}=C_{in}).

Shape

• Input: (N, C_{in}, L_{in})

• Output: (N, C_{out}, L_{out}) where

L_{out} = \left\lfloor\frac{L_{in} + 2 \times \mbox{padding} - \mbox{dilation} \times (\mbox{kernel\_size} - 1) - 1}{\mbox{stride}} + 1\right\rfloor

Attributes

• weight (Tensor): the learnable weights of the module of shape (\mbox{out\_channels}, \frac{\mbox{in\_channels}}{\mbox{groups}}, \mbox{kernel\_size}). The values of these weights are sampled from \mathcal{U}(-\sqrt{k}, \sqrt{k}) where k = \frac{groups}{C_{\mbox{in}} * \mbox{kernel\_size}}

• bias (Tensor): the learnable bias of the module of shape (out_channels). If bias is TRUE, then the values of these weights are sampled from \mathcal{U}(-\sqrt{k}, \sqrt{k}) where k = \frac{groups}{C_{\mbox{in}} * \mbox{kernel\_size}}

Examples

if (torch_is_installed()) {
m <- nn_conv1d(16, 33, 3, stride = 2)
input <- torch_randn(20, 16, 50)
output <- m(input)
}

Conv2D module

Description

Applies a 2D convolution over an input signal composed of several input planes.

Usage

nn_conv2d(
  in_channels,
  out_channels,
  kernel_size,
  stride = 1,
  padding = 0,
  dilation = 1,
  groups = 1,
  bias = TRUE,
  padding_mode = "zeros"
)

Arguments

Details

In the simplest case, the output value of the layer with input size (N, C_{\mbox{in}}, H, W) and output (N, C_{\mbox{out}}, H_{\mbox{out}}, W_{\mbox{out}}) can be precisely described as:

\mbox{out}(N_i, C_{\mbox{out}_j}) = \mbox{bias}(C_{\mbox{out}_j}) + \sum_{k = 0}^{C_{\mbox{in}} - 1} \mbox{weight}(C_{\mbox{out}_j}, k) \star \mbox{input}(N_i, k)

where \star is the valid 2D cross-correlation operator, N is a batch size, C denotes a number of channels, H is a height of input planes in pixels, and W is width in pixels.

• stride controls the stride for the cross-correlation, a single number or a tuple.

• padding controls the amount of implicit zero-paddings on both sides for padding number of points for each dimension.

• dilation controls the spacing between the kernel points; also known as the à trous algorithm. It is harder to describe, but this link_ has a nice visualization of what dilation does.

• groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,

  • At groups=1, all inputs are convolved to all outputs.

  • At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.

  • At groups= in_channels, each input channel is convolved with its own set of filters, of size: \left\lfloor\frac{out\_channels}{in\_channels}\right\rfloor.