Skip to main content

PyTorch Tensors

Introduction​

In PyTorch, tensors are the fundamental data structures that power all computations. Similar to NumPy arrays, tensors are multi-dimensional matrices containing elements of a single data type. However, PyTorch tensors offer additional benefits—they can run on GPUs for accelerated computing and are designed to integrate seamlessly with neural networks.

Understanding tensors is essential for anyone looking to work with PyTorch, as they form the backbone of all deep learning operations—from simple linear algebra to complex neural network architectures.

What Are Tensors?​

Tensors are multi-dimensional arrays that can represent different types of data:

  • A scalar (0-dimensional tensor) is a single number
  • A vector (1-dimensional tensor) is a list of numbers
  • A matrix (2-dimensional tensor) is a grid of numbers
  • A tensor (n-dimensional tensor) extends beyond 3 dimensions

Let's start by importing PyTorch and exploring how to create tensors.

Creating PyTorch Tensors​

Basic Tensor Creation​

python
import torch

# Creating a tensor from a Python list
x = torch.tensor([1, 2, 3, 4])
print(x)
print(f"Dimensions: {x.dim()}")
print(f"Shape: {x.shape}")

Output:

tensor([1, 2, 3, 4])
Dimensions: 1
Shape: torch.Size([4])

Creating Multi-dimensional Tensors​

python
# Creating a 2D tensor (matrix)
matrix = torch.tensor([[1, 2], [3, 4], [5, 6]])
print(matrix)
print(f"Dimensions: {matrix.dim()}")
print(f"Shape: {matrix.shape}")

Output:

tensor([[1, 2],
        [3, 4],
        [5, 6]])
Dimensions: 2
Shape: torch.Size([3, 2])

Special Tensor Creation Methods​

PyTorch provides several functions to create tensors with specific patterns:

python
# Zeros tensor
zeros = torch.zeros(3, 4)
print(f"Zeros tensor:\n{zeros}\n")

# Ones tensor
ones = torch.ones(2, 3)
print(f"Ones tensor:\n{ones}\n")

# Random tensor (values from uniform distribution between 0 and 1)
random = torch.rand(2, 2)
print(f"Random tensor:\n{random}\n")

# Tensor with uninitialized memory (values depend on what was in memory)
empty = torch.empty(2, 3)
print(f"Empty tensor:\n{empty}\n")

# Creating a range tensor
range_tensor = torch.arange(0, 10, step=2)
print(f"Range tensor:\n{range_tensor}\n")

# Creating a linearly spaced tensor
linear = torch.linspace(0, 10, steps=5)
print(f"Linear space tensor:\n{linear}\n")

Output:

Zeros tensor:
tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]])

Ones tensor:
tensor([[1., 1., 1.],
        [1., 1., 1.]])

Random tensor:
tensor([[0.1234, 0.5678],
        [0.2468, 0.3579]])

Empty tensor:
tensor([[4.5911e-09, 4.5911e-09, 1.6812e-44],
        [0.0000e+00, 0.0000e+00, 0.0000e+00]])

Range tensor:
tensor([0, 2, 4, 6, 8])

Linear space tensor:
tensor([ 0.0000,  2.5000,  5.0000,  7.5000, 10.0000])

Tensor Attributes and Properties​

Tensors have several important attributes that help us understand their structure:

python
# Create a 3D tensor
tensor_3d = torch.rand(3, 4, 5)

print(f"Dimensions: {tensor_3d.dim()}")
print(f"Shape: {tensor_3d.shape}")
print(f"Total elements: {tensor_3d.numel()}")
print(f"Data type: {tensor_3d.dtype}")
print(f"Device: {tensor_3d.device}")

Output:

Dimensions: 3
Shape: torch.Size([3, 4, 5])
Total elements: 60
Data type: torch.float32
Device: cpu

Tensor Types​

PyTorch supports different data types for tensors:

python
# Integer tensor
int_tensor = torch.tensor([1, 2, 3], dtype=torch.int32)
print(f"Integer tensor: {int_tensor}, dtype: {int_tensor.dtype}")

# Float tensor
float_tensor = torch.tensor([1.0, 2.0, 3.0], dtype=torch.float32)
print(f"Float tensor: {float_tensor}, dtype: {float_tensor.dtype}")

# Boolean tensor
bool_tensor = torch.tensor([True, False, True], dtype=torch.bool)
print(f"Boolean tensor: {bool_tensor}, dtype: {bool_tensor.dtype}")

# Converting between types
converted_tensor = float_tensor.to(torch.int64)
print(f"Converted tensor: {converted_tensor}, dtype: {converted_tensor.dtype}")

Output:

Integer tensor: tensor([1, 2, 3], dtype=torch.int32), dtype: torch.int32
Float tensor: tensor([1., 2., 3.]), dtype: torch.float32
Boolean tensor: tensor([ True, False,  True]), dtype: torch.bool
Converted tensor: tensor([1, 2, 3], dtype=torch.int64), dtype: torch.int64

Tensor Operations​

Basic Mathematical Operations​

python
# Create two tensors
a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])

# Addition
print(f"a + b = {a + b}")
print(f"torch.add(a, b) = {torch.add(a, b)}")

# Subtraction
print(f"a - b = {a - b}")
print(f"torch.sub(a, b) = {torch.sub(a, b)}")

# Multiplication (element-wise)
print(f"a * b = {a * b}")
print(f"torch.mul(a, b) = {torch.mul(a, b)}")

# Division (element-wise)
print(f"b / a = {b / a}")
print(f"torch.div(b, a) = {torch.div(b, a)}")

Output:

a + b = tensor([5, 7, 9])
torch.add(a, b) = tensor([5, 7, 9])
a - b = tensor([-3, -3, -3])
torch.sub(a, b) = tensor([-3, -3, -3])
a * b = tensor([4, 10, 18])
torch.mul(a, b) = tensor([4, 10, 18])
b / a = tensor([4.0000, 2.5000, 2.0000])
torch.div(b, a) = tensor([4.0000, 2.5000, 2.0000])

In-place Operations​

PyTorch operations that modify tensors in-place end with an underscore (_):

python
x = torch.tensor([1, 2, 3])
print(f"Original x: {x}")

# Add 5 to each element, in-place
x.add_(5)
print(f"After x.add_(5): {x}")

# Multiply each element by 2, in-place
x.mul_(2)
print(f"After x.mul_(2): {x}")

Output:

Original x: tensor([1, 2, 3])
After x.add_(5): tensor([6, 7, 8])
After x.mul_(2): tensor([12, 14, 16])

Matrix Operations​

python
# Matrix multiplication
mat1 = torch.tensor([[1, 2], [3, 4]])
mat2 = torch.tensor([[5, 6], [7, 8]])

# Matrix multiplication (dot product)
mat_mul = torch.matmul(mat1, mat2)
print(f"Matrix multiplication:\n{mat_mul}")

# Alternative matrix multiplication notation
mat_mul_alt = mat1 @ mat2
print(f"Alternative matrix multiplication:\n{mat_mul_alt}")

# Transpose
print(f"Original matrix:\n{mat1}")
print(f"Transposed matrix:\n{mat1.T}")

Output:

Matrix multiplication:
tensor([[19, 22],
        [43, 50]])
Alternative matrix multiplication:
tensor([[19, 22],
        [43, 50]])
Original matrix:
tensor([[1, 2],
        [3, 4]])
Transposed matrix:
tensor([[1, 3],
        [2, 4]])

Reshaping Tensors​

Changing the shape of tensors is a common operation:

python
# Create a tensor
x = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8])
print(f"Original tensor: {x}, shape: {x.shape}")

# Reshape to a 2x4 matrix
x_reshaped = x.reshape(2, 4)
print(f"Reshaped tensor:\n{x_reshaped}")

# View provides another way to reshape (but shares memory with the original tensor)
x_viewed = x.view(4, 2)
print(f"Viewed tensor:\n{x_viewed}")

# Changing the viewed tensor changes the original tensor
x_viewed[0, 0] = 99
print(f"Original tensor after modifying view: {x}")

# Flatten a tensor to 1D
flattened = x_reshaped.flatten()
print(f"Flattened tensor: {flattened}")

Output:

Original tensor: tensor([1, 2, 3, 4, 5, 6, 7, 8]), shape: torch.Size([8])
Reshaped tensor:
tensor([[1, 2, 3, 4],
        [5, 6, 7, 8]])
Viewed tensor:
tensor([[1, 2],
        [3, 4],
        [5, 6],
        [7, 8]])
Original tensor after modifying view: tensor([99, 2, 3, 4, 5, 6, 7, 8])
Flattened tensor: tensor([1, 2, 3, 4, 5, 6, 7, 8])

Tensor Indexing and Slicing​

PyTorch tensors can be indexed and sliced like NumPy arrays:

python
# Create a 3x3 tensor
tensor = torch.tensor([[1, 2, 3], 
                       [4, 5, 6], 
                       [7, 8, 9]])

# Get a single element
print(f"Element at position [1,2]: {tensor[1, 2]}")

# Get a row
print(f"Second row: {tensor[1]}")

# Get a column
print(f"Third column: {tensor[:, 2]}")

# Slicing
print(f"2x2 slice:\n{tensor[0:2, 1:3]}")

# Advanced indexing
indices = torch.tensor([0, 2])
print(f"First and third rows:\n{tensor[indices]}")

# Boolean indexing
mask = tensor > 5
print(f"Mask (elements > 5):\n{mask}")
print(f"Elements > 5: {tensor[mask]}")

Output:

Element at position [1,2]: 6
Second row: tensor([4, 5, 6])
Third column: tensor([3, 6, 9])
2x2 slice:
tensor([[2, 3],
        [5, 6]])
First and third rows:
tensor([[1, 2, 3],
        [7, 8, 9]])
Mask (elements > 5):
tensor([[False, False, False],
        [False, False, True],
        [True, True, True]])
Elements > 5: tensor([6, 7, 8, 9])

GPU Acceleration with Tensors​

One of PyTorch's key features is its seamless support for GPU acceleration:

python
# Check if CUDA (GPU support) is available
if torch.cuda.is_available():
    device = torch.device("cuda")
    print(f"GPU is available: {torch.cuda.get_device_name(0)}")
else:
    device = torch.device("cpu")
    print("GPU is not available, using CPU instead")

# Create a tensor on the CPU
cpu_tensor = torch.tensor([1, 2, 3, 4])
print(f"CPU tensor device: {cpu_tensor.device}")

# Move tensor to the specified device (GPU if available)
device_tensor = cpu_tensor.to(device)
print(f"Tensor moved to: {device_tensor.device}")

# Create a tensor directly on the specified device
new_tensor = torch.zeros(3, 4, device=device)
print(f"New tensor created on: {new_tensor.device}")

# Move tensor back to CPU if needed
if device.type == 'cuda':
    back_to_cpu = device_tensor.cpu()
    print(f"Tensor moved back to: {back_to_cpu.device}")

Output (if CUDA is available):

GPU is available: NVIDIA GeForce RTX 3080
CPU tensor device: cpu
Tensor moved to: cuda:0
New tensor created on: cuda:0
Tensor moved back to: cpu

Real-World Example: Image Processing​

Let's look at a practical example of using tensors to process an image:

python
import torch
import matplotlib.pyplot as plt
from PIL import Image
import numpy as np
import requests
from io import BytesIO

# Download a sample image
response = requests.get('https://images.unsplash.com/photo-1600596542815-ffad4c1539a9?ixlib=rb-4.0.3&w=640&q=80')
img = Image.open(BytesIO(response.content))

# Convert image to tensor
img_np = np.array(img)
img_tensor = torch.from_numpy(img_np).float()
print(f"Image tensor shape: {img_tensor.shape}")  # [height, width, channels]

# Normalize pixel values to [0, 1]
img_tensor = img_tensor / 255.0

# Transpose dimensions to [channels, height, width] for PyTorch convention
img_tensor = img_tensor.permute(2, 0, 1)
print(f"Transposed tensor shape: {img_tensor.shape}")

# Apply simple image processing: increase brightness
brightness_factor = 1.5
brightened_img = torch.clamp(img_tensor * brightness_factor, 0, 1)

# Convert back to numpy for displaying
brightened_np = brightened_img.permute(1, 2, 0).numpy()
original_np = img_tensor.permute(1, 2, 0).numpy()

# Display results
fig, axes = plt.subplots(1, 2, figsize=(10, 5))
axes[0].imshow(original_np)
axes[0].set_title('Original')
axes[0].axis('off')

axes[1].imshow(brightened_np)
axes[1].set_title('Brightened')
axes[1].axis('off')

plt.tight_layout()
plt.show()

# We can also extract and manipulate individual color channels
red_channel = img_tensor[0]  # First channel is red
print(f"Red channel shape: {red_channel.shape}")
print(f"Red channel min: {red_channel.min().item()}, max: {red_channel.max().item()}")

This code demonstrates how to:

  1. Convert an image to a PyTorch tensor
  2. Normalize and rearrange the dimensions according to PyTorch conventions
  3. Perform a simple image processing operation (brightness adjustment)
  4. Access individual color channels

Summary​

In this tutorial, we've covered the fundamental concepts of PyTorch tensors:

  • Creating tensors using various methods
  • Understanding tensor properties and attributes
  • Performing mathematical operations on tensors
  • Reshaping, indexing, and slicing tensors
  • Moving tensors between CPU and GPU
  • Using tensors for real-world applications like image processing

PyTorch tensors are the foundation upon which all deep learning models are built. They provide an efficient way to represent and manipulate data while leveraging GPU acceleration for faster computations.

Exercises​

To solidify your understanding, try these exercises:

  1. Create a 3x3 identity matrix using PyTorch tensors
  2. Perform matrix multiplication between two 2x3 and 3x2 tensors
  3. Create a tensor on GPU (if available) and calculate its square root
  4. Build a tensor representing a simple grayscale image (e.g., a 10x10 checkerboard pattern)
  5. Write a function that takes a tensor and applies a "negative" effect to it (inverting all values)

Additional Resources​


If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)