Tensor Differential Calculus Example: Backpropogation¶
By Eric Wong
This third and last notebook on the tensor differential calculus is a showcase example demonstrating how tensor differential calculus can be used to derive the update states for backpropogation in an elegant manner, without resorting to element-wise partial derivatives.
Problem setup¶
Consider a neural network with $k$ dense hidden layers and a softmax output layer. We'll use the following notation:
Tensor Differential Calculus, Part 2
Differentials 101 Extended to Tensors¶
By Eric Wong
Introduction¶
This notebook serves as a basic introduction to differential calculus, with a tensor twist. While there are plenty of resources on differential calculus (e.g., Matrix Differential Calculus by Magnus and Neudecker), the way they address tensor derivatives feels like a cop-out: the current solution is to vectorize the inputs and outputs, and use normal matrix differential calculus. This makes derivations involving higher order tensor derivatives to have convoluted forms: in order to use matrix differential calculus, often a number of unnecessary additional vec and kronecker product operations are required to finagle the problem into proper form.
Tensor Differential Calculus, Part 1
A Quick Practitioner's Guide to Tensors¶
by Eric Wong
Introduction¶
Two approaches to teaching linear algebra¶
This notebook serves as a basic introduction to tensors from a practitioners perspective. What does this mean? In many math departments, there often exists multiple levels of linear algebra. For example, at CMU there is
- 21-241 "Matrices and Linear Transformations" - typically aimed at undergraduates that need a working level of linear algebra for their application field, e.g. engineering. This usually takes the approach of teaching linear algebra from the perspective of matrices and vectors of numbers, explaining their operations, properties, and their decompositions.
Page 1 / 1