Skip to main content

TensorFlow Math Functions

Introduction​

TensorFlow provides a rich set of mathematical functions that form the foundation of machine learning operations. These functions allow you to perform everything from basic arithmetic to complex calculus operations on tensors. Understanding these math functions is crucial for building and training machine learning models effectively.

In this tutorial, we'll explore the most commonly used math functions in TensorFlow, understand their syntax, and see how they're applied in real-world scenarios. Whether you're performing simple additions or complex matrix operations, TensorFlow's math API has you covered.

Basic Arithmetic Operations​

Let's start with the fundamental arithmetic operations that TensorFlow offers.

Addition, Subtraction, Multiplication, and Division​

python
import tensorflow as tf

# Create two simple tensors
a = tf.constant([1, 2, 3])
b = tf.constant([4, 5, 6])

# Addition
add_result = tf.add(a, b)  # or simply a + b
print("Addition:", add_result.numpy())

# Subtraction
sub_result = tf.subtract(a, b)  # or simply a - b
print("Subtraction:", sub_result.numpy())

# Multiplication
mul_result = tf.multiply(a, b)  # or simply a * b
print("Element-wise multiplication:", mul_result.numpy())

# Division
div_result = tf.divide(a, b)  # or simply a / b
print("Division:", div_result.numpy())

Output:

Addition: [5 7 9]
Subtraction: [-3 -3 -3]
Element-wise multiplication: [4 10 18]
Division: [0.25 0.4 0.5]

Powers and Square Roots​

python
# Square
square_result = tf.square(a)
print("Square:", square_result.numpy())

# Square root
sqrt_result = tf.sqrt(tf.cast(a, tf.float32))
print("Square root:", sqrt_result.numpy())

# Power
power_result = tf.pow(a, 3)
print("Cube:", power_result.numpy())

Output:

Square: [1 4 9]
Square root: [1.        1.4142135 1.7320508]
Cube: [ 1  8 27]

Statistical Operations​

TensorFlow provides functions for common statistical operations, which are particularly useful in machine learning.

Mean, Sum, and Standard Deviation​

python
# Create a 2D tensor
data = tf.constant([[1, 2], [3, 4], [5, 6]], dtype=tf.float32)

# Calculate mean
mean = tf.reduce_mean(data)
print("Mean of all elements:", mean.numpy())

# Calculate row-wise mean
row_mean = tf.reduce_mean(data, axis=1)
print("Row-wise mean:", row_mean.numpy())

# Calculate column-wise mean
col_mean = tf.reduce_mean(data, axis=0)
print("Column-wise mean:", col_mean.numpy())

# Calculate sum
sum_all = tf.reduce_sum(data)
print("Sum of all elements:", sum_all.numpy())

# Calculate standard deviation
# First, we need to calculate variance
variance = tf.reduce_mean(tf.square(data - mean))
std_dev = tf.sqrt(variance)
print("Standard deviation:", std_dev.numpy())

Output:

Mean of all elements: 3.5
Row-wise mean: [1.5 3.5 5.5]
Column-wise mean: [3. 4.]
Sum of all elements: 21.0
Standard deviation: 1.7078251

Matrix Operations​

Matrix operations are crucial for linear algebra in machine learning algorithms.

Matrix Multiplication​

python
# Create matrices
matrix1 = tf.constant([[1, 2], [3, 4]], dtype=tf.float32)
matrix2 = tf.constant([[5, 6], [7, 8]], dtype=tf.float32)

# Matrix multiplication
matmul_result = tf.matmul(matrix1, matrix2)
print("Matrix multiplication:\n", matmul_result.numpy())

# Another way to do matrix multiplication
matmul_result2 = matrix1 @ matrix2
print("Matrix multiplication using @ operator:\n", matmul_result2.numpy())

Output:

Matrix multiplication:
 [[19. 22.]
 [43. 50.]]
Matrix multiplication using @ operator:
 [[19. 22.]
 [43. 50.]]

Transpose and Inverse​

python
# Transpose
transpose_result = tf.transpose(matrix1)
print("Transpose of matrix1:\n", transpose_result.numpy())

# Matrix inverse
inverse_result = tf.linalg.inv(matrix1)
print("Inverse of matrix1:\n", inverse_result.numpy())

Output:

Transpose of matrix1:
 [[1. 3.]
 [2. 4.]]
Inverse of matrix1:
 [[-2.   1. ]
 [ 1.5 -0.5]]

Determinant and Eigenvalues​

python
# Determinant
det_result = tf.linalg.det(matrix1)
print("Determinant of matrix1:", det_result.numpy())

# Eigenvalues and eigenvectors
eigenvalues, eigenvectors = tf.linalg.eigh(matrix1)
print("Eigenvalues of matrix1:", eigenvalues.numpy())
print("Eigenvectors of matrix1:\n", eigenvectors.numpy())

Output:

Determinant of matrix1: -2.0
Eigenvalues of matrix1: [0.17157288 4.82842712]
Eigenvectors of matrix1:
 [[-0.82456485 -0.56576746]
 [ 0.56576746 -0.82456485]]

Trigonometric Functions​

TensorFlow provides all standard trigonometric functions:

python
# Create angles in radians
angles = tf.constant([0.0, np.pi/4, np.pi/2, np.pi], dtype=tf.float32)

# Sine
sin_result = tf.sin(angles)
print("Sine:", sin_result.numpy())

# Cosine
cos_result = tf.cos(angles)
print("Cosine:", cos_result.numpy())

# Tangent
tan_result = tf.tan(angles)
print("Tangent:", tan_result.numpy())

Output:

Sine: [0.        0.7071068  1.        0.       ]
Cosine: [ 1.         0.7071068  0.        -1.       ]
Tangent: [0.        1.        1.633124e+16 -0.       ]

Advanced Math Functions​

TensorFlow supports various advanced mathematical operations that are commonly used in machine learning models.

Logarithms and Exponentials​

python
# Create a tensor
x = tf.constant([1.0, 2.0, 3.0, 4.0])

# Natural logarithm
log_result = tf.math.log(x)
print("Natural logarithm:", log_result.numpy())

# Log base 10
log10_result = tf.math.log10(x)
print("Log base 10:", log10_result.numpy())

# Exponential (e^x)
exp_result = tf.math.exp(x)
print("Exponential (e^x):", exp_result.numpy())

Output:

Natural logarithm: [0.        0.6931472 1.0986123 1.3862944]
Log base 10: [0.        0.30103   0.47712126 0.60205996]
Exponential (e^x): [ 2.7182817  7.389056  20.085537  54.59815 ]

Sigmoid and Other Activation Functions​

python
# Sigmoid
sigmoid_result = tf.math.sigmoid(x)
print("Sigmoid:", sigmoid_result.numpy())

# Tanh
tanh_result = tf.math.tanh(x)
print("Tanh:", tanh_result.numpy())

# ReLU
relu_result = tf.nn.relu(tf.constant([-1.0, 0.0, 1.0, 2.0]))
print("ReLU:", relu_result.numpy())

Output:

Sigmoid: [0.7310586  0.880797   0.95257413 0.98201376]
Tanh: [0.7615942 0.9640276 0.9950547 0.999329 ]
ReLU: [0. 0. 1. 2.]

Practical Example: Linear Regression​

Let's see how these math functions are used in a simple linear regression model.

python
# Generate some synthetic data
import numpy as np

# Number of examples
n_samples = 100

# True parameters (weight and bias)
true_w = 2.0
true_b = 1.0

# Generate synthetic data with noise
x = tf.random.normal([n_samples], dtype=tf.float32)
noise = tf.random.normal([n_samples], stddev=0.1, dtype=tf.float32)
y = true_w * x + true_b + noise

# Initialize parameters
w = tf.Variable(0.0, dtype=tf.float32)
b = tf.Variable(0.0, dtype=tf.float32)

# Define the learning rate
learning_rate = 0.05
epochs = 100

# Training loop
for epoch in range(epochs):
    with tf.GradientTape() as tape:
        # Forward pass: predict y using the current parameters
        y_pred = w * x + b
        
        # Compute mean squared error
        loss = tf.reduce_mean(tf.square(y_pred - y))
    
    # Compute gradients
    gradients = tape.gradient(loss, [w, b])
    
    # Update parameters using gradient descent
    w.assign_sub(learning_rate * gradients[0])
    b.assign_sub(learning_rate * gradients[1])
    
    if epoch % 20 == 0:
        print(f"Epoch {epoch}, Loss: {loss.numpy():.4f}, w: {w.numpy():.4f}, b: {b.numpy():.4f}")

print(f"Final parameters - w: {w.numpy():.4f}, b: {b.numpy():.4f}")
print(f"True parameters - w: {true_w}, b: {true_b}")

Output (may vary due to random initialization):

Epoch 0, Loss: 3.8544, w: 0.2738, b: 0.0500
Epoch 20, Loss: 0.1153, w: 1.7964, b: 0.8508
Epoch 40, Loss: 0.0121, w: 1.9645, b: 0.9704
Epoch 60, Loss: 0.0103, w: 1.9953, b: 0.9920
Epoch 80, Loss: 0.0101, w: 1.9994, b: 0.9981
Final parameters - w: 2.0003, b: 0.9997
True parameters - w: 2.0, b: 1.0

Here, we use several TensorFlow math functions:

  • tf.random.normal to generate random data
  • tf.Variable to create trainable parameters
  • tf.square and tf.reduce_mean to calculate the mean squared error
  • Addition and multiplication for the linear model

Practical Example: Neural Network Activation​

Let's see how math functions are used in a simple neural network layer:

python
# Create a simple layer with sigmoid activation
def simple_dense_layer(inputs, weights, bias):
    # Linear combination: inputs * weights + bias
    z = tf.matmul(inputs, weights) + bias
    
    # Apply sigmoid activation function
    activation = tf.sigmoid(z)
    
    return activation

# Create sample data
batch_size = 2
input_size = 3
output_size = 1

# Sample input
x = tf.constant([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]], dtype=tf.float32)

# Initialize weights and bias
weights = tf.Variable(tf.random.normal([input_size, output_size]), dtype=tf.float32)
bias = tf.Variable(tf.zeros([output_size]), dtype=tf.float32)

# Forward pass
output = simple_dense_layer(x, weights, bias)
print("Layer output (with sigmoid activation):")
print(output.numpy())

Output (will vary due to random initialization):

Layer output (with sigmoid activation):
[[0.57561725]
 [0.6298476 ]]

In this example, we use matrix multiplication (tf.matmul), addition, and the sigmoid function (tf.sigmoid) to create a simple neural network layer.

Summary​

In this tutorial, we've explored the rich set of mathematical functions that TensorFlow provides:

  • Basic arithmetic operations: addition, subtraction, multiplication, division, powers, and roots
  • Statistical operations: mean, sum, and standard deviation
  • Matrix operations: multiplication, transpose, inverse, determinant, and eigenvalues
  • Trigonometric functions: sine, cosine, and tangent
  • Advanced math functions: logarithms, exponentials, and activation functions

These functions are the building blocks of machine learning models, from simple linear regression to complex neural networks. Understanding how to use them effectively is crucial for developing and debugging TensorFlow models.

Additional Resources​

Exercises​

  1. Implement a function to compute the L2 norm (Euclidean distance) of a vector using TensorFlow math functions.
  2. Create a function that normalizes a tensor by subtracting the mean and dividing by the standard deviation.
  3. Implement a simple logistic regression model using TensorFlow's math functions.
  4. Create a function to compute the softmax of a tensor along a specified axis.
  5. Implement matrix factorization using TensorFlow's linear algebra functions.

These exercises will help you gain practical experience with TensorFlow's math functions and prepare you for more complex machine learning tasks.


If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)