C# Parallel Programming

Introduction

Parallel programming is a technique that allows your applications to perform multiple operations simultaneously, taking advantage of multi-core processors to improve performance. Unlike asynchronous programming, which helps manage I/O-bound operations efficiently, parallel programming is designed to optimize CPU-bound tasks by distributing work across available processor cores.

In this tutorial, we'll explore how C# and .NET provide powerful tools for implementing parallel programming through the Task Parallel Library (TPL) and Parallel LINQ (PLINQ). By the end, you'll understand how to write code that can execute multiple operations concurrently, significantly improving performance for computationally intensive tasks.

Understanding Parallel Programming

Before diving into implementation details, let's understand some key concepts:

Sequential vs. Parallel Execution: Traditional code runs sequentially, executing one operation after another. Parallel code divides work into parts that can run simultaneously.
CPU-Bound Tasks: Operations that primarily use processor resources (calculations, algorithms, data processing) benefit most from parallelism.
Concurrency vs. Parallelism: Concurrency means handling multiple operations during overlapping time periods. Parallelism specifically means executing multiple operations simultaneously.

Task Parallel Library (TPL) Basics

The Task Parallel Library is the foundation of parallel programming in C#. It provides high-level abstractions that make parallel code easier to write and understand.

The `Parallel` Class

The Parallel class offers simple methods to parallelize loops and other operations.

Parallel.For

Parallel.For executes a for loop in which iterations can run in parallel:

csharp
using System;
using System.Threading.Tasks;
using System.Diagnostics;

class Program
{
    static void Main()
    {
        Console.WriteLine("Sequential processing:");
        ProcessSequentially();
        
        Console.WriteLine("\nParallel processing:");
        ProcessInParallel();
        
        Console.ReadKey();
    }
    
    static void ProcessSequentially()
    {
        Stopwatch stopwatch = Stopwatch.StartNew();
        
        for (int i = 0; i < 10; i++)
        {
            ProcessItem(i);
        }
        
        stopwatch.Stop();
        Console.WriteLine($"Sequential processing took: {stopwatch.ElapsedMilliseconds} ms");
    }
    
    static void ProcessInParallel()
    {
        Stopwatch stopwatch = Stopwatch.StartNew();
        
        Parallel.For(0, 10, i =>
        {
            ProcessItem(i);
        });
        
        stopwatch.Stop();
        Console.WriteLine($"Parallel processing took: {stopwatch.ElapsedMilliseconds} ms");
    }
    
    static void ProcessItem(int index)
    {
        Console.WriteLine($"Processing item {index} on thread {System.Threading.Thread.CurrentThread.ManagedThreadId}");
        // Simulate work
        System.Threading.Thread.Sleep(100);
    }
}

Output (Example - actual thread IDs may vary):

Sequential processing:
Processing item 0 on thread 1
Processing item 1 on thread 1
Processing item 2 on thread 1
...
Sequential processing took: 1013 ms

Parallel processing:
Processing item 0 on thread 4
Processing item 3 on thread 5
Processing item 1 on thread 6
Processing item 2 on thread 7
...
Parallel processing took: 312 ms

Notice how parallel processing completes the work significantly faster and uses multiple threads.

Parallel.ForEach

Parallel.ForEach works on collections rather than numeric ranges:

csharp
List<string> items = new List<string> { "item1", "item2", "item3", "item4", "item5" };

Parallel.ForEach(items, item =>
{
    Console.WriteLine($"Processing {item} on thread {Thread.CurrentThread.ManagedThreadId}");
    // Process item
    Thread.Sleep(100);
});

Parallel.Invoke

Parallel.Invoke executes multiple actions in parallel:

csharp
Parallel.Invoke(
    () => ProcessData("Data 1"),
    () => ProcessData("Data 2"),
    () => ProcessData("Data 3")
);

static void ProcessData(string data)
{
    Console.WriteLine($"Processing {data} on thread {Thread.CurrentThread.ManagedThreadId}");
    // Simulate work
    Thread.Sleep(1000);
}

Controlling Parallel Execution

Setting the Degree of Parallelism

You can control how many operations run concurrently:

csharp
var options = new ParallelOptions
{
    MaxDegreeOfParallelism = Environment.ProcessorCount / 2  // Use half of the available cores
};

Parallel.For(0, 100, options, i =>
{
    // Process item i
    Console.WriteLine($"Processing item {i} on thread {Thread.CurrentThread.ManagedThreadId}");
    Thread.Sleep(10);
});

Handling Exceptions

Parallel operations require special exception handling:

csharp
try
{
    Parallel.For(0, 10, i =>
    {
        if (i == 5)
            throw new Exception($"Error processing item {i}");
            
        Console.WriteLine($"Processing item {i}");
    });
}
catch (AggregateException ae)
{
    foreach (var ex in ae.InnerExceptions)
    {
        Console.WriteLine($"Exception: {ex.Message}");
    }
}

Parallel LINQ (PLINQ)

PLINQ extends LINQ to provide parallel query execution:

csharp
using System;
using System.Linq;
using System.Diagnostics;

class Program
{
    static void Main()
    {
        // Create a large array to process
        var numbers = Enumerable.Range(1, 10000000).ToArray();
        
        // Sequential LINQ
        Stopwatch sw = Stopwatch.StartNew();
        var evenNumbersSequential = numbers.Where(n => n % 2 == 0).Count();
        sw.Stop();
        Console.WriteLine($"Sequential LINQ took: {sw.ElapsedMilliseconds} ms");
        
        // Parallel LINQ
        sw.Restart();
        var evenNumbersParallel = numbers.AsParallel().Where(n => n % 2 == 0).Count();
        sw.Stop();
        Console.WriteLine($"Parallel LINQ took: {sw.ElapsedMilliseconds} ms");
    }
}

Output (Example):

Sequential LINQ took: 78 ms
Parallel LINQ took: 21 ms

Controlling PLINQ Execution

csharp
var result = numbers
    .AsParallel()
    .WithDegreeOfParallelism(4)  // Use 4 cores
    .WithExecutionMode(ParallelExecutionMode.ForceParallelism)  // Force parallelism even for small data sets
    .Where(n => IsPrime(n))
    .ToList();

Real-World Example: Parallel Image Processing

Here's a more practical example that processes multiple images in parallel:

csharp
using System;
using System.Collections.Generic;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Threading.Tasks;

class ImageProcessor
{
    public static void ProcessImagesSequentially(string[] imagePaths)
    {
        foreach (var path in imagePaths)
        {
            ProcessImage(path);
        }
    }

    public static void ProcessImagesInParallel(string[] imagePaths)
    {
        Parallel.ForEach(imagePaths, path =>
        {
            ProcessImage(path);
        });
    }

    private static void ProcessImage(string path)
    {
        try
        {
            Console.WriteLine($"Processing image {Path.GetFileName(path)} on thread {Thread.CurrentThread.ManagedThreadId}");
            
            // Load the image
            using (Bitmap bitmap = new Bitmap(path))
            {
                // Apply a grayscale filter (simplified)
                Bitmap result = ApplyGrayscaleFilter(bitmap);
                
                // Save with new filename
                string outputPath = Path.Combine(
                    Path.GetDirectoryName(path),
                    "processed_" + Path.GetFileName(path)
                );
                result.Save(outputPath);
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error processing {path}: {ex.Message}");
        }
    }

    private static Bitmap ApplyGrayscaleFilter(Bitmap original)
    {
        Bitmap result = new Bitmap(original.Width, original.Height);

        // Apply grayscale filter
        for (int x = 0; x < original.Width; x++)
        {
            for (int y = 0; y < original.Height; y++)
            {
                Color pixelColor = original.GetPixel(x, y);
                int grayValue = (int)(pixelColor.R * 0.3 + pixelColor.G * 0.59 + pixelColor.B * 0.11);
                Color grayColor = Color.FromArgb(pixelColor.A, grayValue, grayValue, grayValue);
                result.SetPixel(x, y, grayColor);
            }
        }

        return result;
    }
}

To use this class:

csharp
string[] imagePaths = Directory.GetFiles(@"C:\Images", "*.jpg");

Console.WriteLine("Processing images sequentially...");
var sw = Stopwatch.StartNew();
ImageProcessor.ProcessImagesSequentially(imagePaths);
sw.Stop();
Console.WriteLine($"Sequential processing took: {sw.ElapsedMilliseconds} ms");

Console.WriteLine("\nProcessing images in parallel...");
sw.Restart();
ImageProcessor.ProcessImagesInParallel(imagePaths);
sw.Stop();
Console.WriteLine($"Parallel processing took: {sw.ElapsedMilliseconds} ms");

Best Practices for Parallel Programming

Use Parallel Processing for CPU-Bound Tasks: Parallelism works best for computationally intensive operations.
Consider Task Granularity: Each parallel task should have enough work to justify the overhead of parallelization.
Be Careful with Shared State: Avoid or properly synchronize access to shared variables.
Measure Performance: Always benchmark to confirm that parallelism actually improves performance.
Handle Exceptions Properly: Use try/catch with AggregateException handling.
Consider Thread Safety: Make sure your code is thread-safe when running in parallel.
Limit the Degree of Parallelism: On heavily loaded systems, limit parallel execution to avoid resource exhaustion.

Potential Issues and Solutions

Race Conditions

Race conditions occur when multiple threads access shared data simultaneously:

csharp
// Problematic code with race condition
int total = 0;
Parallel.For(0, 10000, i =>
{
    total++; // Race condition!
});
Console.WriteLine($"Total: {total}"); // Will likely be less than 10000

// Fixed using Interlocked
int safeTotal = 0;
Parallel.For(0, 10000, i =>
{
    Interlocked.Increment(ref safeTotal);
});
Console.WriteLine($"Safe total: {safeTotal}"); // Will always be 10000

Deadlocks

Deadlocks can occur when parallel operations wait for each other:

csharp
// Potential deadlock scenario
object lock1 = new object();
object lock2 = new object();

Parallel.Invoke(
    () =>
    {
        lock (lock1)
        {
            Thread.Sleep(100); // Increases chance of deadlock for demonstration
            lock (lock2)
            {
                // Do work
            }
        }
    },
    () =>
    {
        lock (lock2)
        {
            Thread.Sleep(100);
            lock (lock1) // Deadlock! Both tasks are waiting for locks held by the other
            {
                // Do work
            }
        }
    }
);

// Solution: Always acquire locks in the same order
// Both should lock lock1 first, then lock2

When Not to Use Parallel Programming

Not all code benefits from parallelism:

I/O-Bound Operations: Use asynchronous programming instead.
Small Workloads: The overhead of creating threads may exceed benefits.
Heavily Interdependent Tasks: Tasks that need frequent synchronization.
Sequential Algorithms: Some algorithms are inherently sequential.

Summary

Parallel programming in C# allows you to take full advantage of multi-core processors to improve the performance of CPU-bound operations. The Task Parallel Library (TPL) and PLINQ provide high-level abstractions that make it relatively easy to implement parallelism in your applications.

Key takeaways:

Use Parallel.For, Parallel.ForEach, and Parallel.Invoke for basic parallelization
Use PLINQ with the .AsParallel() extension method for parallel data processing
Be careful with shared state to avoid race conditions
Always measure performance to ensure parallelism actually helps
Consider thread safety and proper exception handling

By applying these techniques wisely, you can significantly improve the performance of computationally intensive applications while maintaining code readability and reliability.

Exercises

Create a parallel program that finds all prime numbers between 1 and 10,000,000. Compare its performance with a sequential version.
Write a parallel image processing application that applies different filters to a set of images.
Implement matrix multiplication using both sequential and parallel approaches, then compare their performance.
Create a parallel text processing application that counts word frequencies in multiple text files concurrently.

Additional Resources

Microsoft Documentation on Parallel Programming
Parallel Programming in .NET
Patterns for Parallel Programming
C# in a Nutshell - Contains excellent chapters on threading and parallel programming

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)

Introduction​

Understanding Parallel Programming​

Task Parallel Library (TPL) Basics​

The Parallel Class​

Parallel.For​

Parallel.ForEach​

Parallel.Invoke​

Controlling Parallel Execution​

Setting the Degree of Parallelism​

Handling Exceptions​

Parallel LINQ (PLINQ)​

Controlling PLINQ Execution​

Real-World Example: Parallel Image Processing​

Best Practices for Parallel Programming​

Potential Issues and Solutions​

Race Conditions​

Deadlocks​

When Not to Use Parallel Programming​

Summary​

Exercises​

Additional Resources​