Handling Long-Running Background Tasks in .NET Core

Introduction

In web applications, there are situations where you need to perform tasks that take a significant amount of time. Examples include generating reports, processing large datasets, or sending bulk emails. Handling such long running background tasks directly within an API request can lead to timeouts, performance bottlenecks, and a poor user experience. Long-running tasks can also affect server performance by consuming valuable resources needed for other requests.

So how can we handle long-running background tasks effectively in a .NET Core web application?

The solution lies in offloading these tasks from the main thread to a background service. .NET Core provides a built-in way to handle background tasks through Hosted Services. Combined with a Task Queue, you can manage and execute long-running background tasks without blocking your API or impacting user experience.

In this blog, we will explore how to implement long-running tasks using IHostedService and task queues in .NET Core. We’ll break it down step-by-step with code examples so even beginners can follow along.

The Problem: Running Long-Running Background Tasks in HTTP Requests

Let’s consider a scenario where your API needs to process a large report for a user. If you were to run this task synchronously within your HTTP request, the request could time out due to its long duration. Moreover, your API’s thread pool would be blocked while processing the report, meaning it couldn’t handle other incoming requests efficiently.

Here’s an example of such a problem in code:

[HttpGet("generate-report")]
public IActionResult GenerateReport()
{
    // Simulate a long-running task
    Thread.Sleep(10000); // 10 seconds delay

    return Ok("Report generated");
}

This method simulates a 10-second delay (as if it’s generating a report). If a user requests this, they will have to wait 10 seconds before receiving a response. This approach can degrade performance, especially if multiple users request the report at the same time.

To solve this, we can offload the report generation process to a background service, freeing up the API to handle other requests. Here’s where Hosted Services and Task Queues come into play.

The Solution: Background Processing with Hosted Services

What is a Hosted Service?

In .NET Core, a Hosted Service is a class that implements the IHostedService interface and runs background tasks. Hosted services are started when the application starts and stopped when the application shuts down.

There are two key methods in IHostedService:

• StartAsync: Called when the service starts, often used to initialize background tasks.
• StopAsync: Called when the service stops, typically used for cleanup tasks.

Step-by-Step Implementation

Let’s build a background worker using IHostedService and create a task queue to manage long-running tasks.

1. Setting Up the Hosted Service

First, create a class that implements IHostedService. This class will run in the background and process tasks from a queue.

public class BackgroundTaskQueue : IHostedService
{
    private readonly Queue<Func<CancellationToken, Task>> _workItems = new();
    private readonly SemaphoreSlim _signal = new(0);

    public Task StartAsync(CancellationToken cancellationToken)
    {
        // Start the background worker task
        Task.Run(BackgroundProcessing, cancellationToken);
        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }

    private async Task BackgroundProcessing()
    {
        while (true)
        {
            // Wait for a task to be added to the queue
            await _signal.WaitAsync();

            // Dequeue and process the task
            var workItem = _workItems.Dequeue();
            await workItem(CancellationToken.None);
        }
    }

    public void QueueBackgroundWorkItem(Func<CancellationToken, Task> workItem)
    {
        // Add a task to the queue and signal the worker
        _workItems.Enqueue(workItem);
        _signal.Release();
    }
}

Explanation:

• The BackgroundTaskQueue class implements IHostedService, allowing it to run background tasks.
• It uses a Queue<Func<CancellationToken, Task>> to store tasks and a SemaphoreSlim to manage task processing.
• The StartAsync method starts a background worker (BackgroundProcessing) that waits for tasks to be queued.
• When a task is added using QueueBackgroundWorkItem, the task is queued, and the worker is signaled to process it.

---

2. Register the Hosted Service in Startup

Now, we need to register our hosted service in the Startup.cs file to ensure it runs when the application starts.

In .NET 6+, you would register the service in Program.cs:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton<BackgroundTaskQueue>();
builder.Services.AddHostedService(sp => sp.GetRequiredService<BackgroundTaskQueue>());

var app = builder.Build();
app.Run();

This ensures that the BackgroundTaskQueue starts automatically when your application starts.

3. Using the Background Task Queue in a Controller

Now, let’s modify our API to use the task queue instead of running long tasks directly in the request.

[Route("api/[controller]")]
[ApiController]
public class ReportController : ControllerBase
{
    private readonly BackgroundTaskQueue _taskQueue;

    public ReportController(BackgroundTaskQueue taskQueue)
    {
        _taskQueue = taskQueue;
    }

    [HttpGet("generate-report")]
    public IActionResult GenerateReport()
    {
        // Queue the report generation task
        _taskQueue.QueueBackgroundWorkItem(async token =>
        {
            await Task.Delay(10000); // Simulating long-running task
            Console.WriteLine("Report generated in background.");
        });

        // Return an immediate response to the client
        return Ok("Report generation started. You will be notified when it's done.");
    }
}

Explanation:

• The GenerateReport method now queues the task to generate the report in the background.
• Instead of waiting for the task to finish, the API returns an immediate response, telling the client that the process has started.
• The actual report generation happens asynchronously in the background worker.

4. Handling Task Completion and Notifications

In real-world scenarios, you may want to notify the user when the background task is completed, either through email, push notification, or updating the database.

For simplicity, you can log or print a message upon task completion:

_taskQueue.QueueBackgroundWorkItem(async token =>
{
    await Task.Delay(10000); // Simulating long-running task
    Console.WriteLine("Report generation completed.");
    // Add code to send notification or update status in DB here.
});

You can enhance this by storing task completion statuses in a database and polling the client for updates.

Advantages of Using Hosted Services and Task Queues

1. Non-blocking API: API endpoints no longer block the client while waiting for long tasks to complete.
2. Scalability: The API can handle more requests, as long-running tasks are handled in the background.
3. Better User Experience: Users receive an immediate response instead of waiting for the entire process to complete.
4. Resource Efficiency: By managing tasks in a queue, system resources are used more efficiently, ensuring high performance under heavy load.

Additional Tips for Real-World Implementation to handle Long-Running Background Tasks

• Task Status Management: To track task progress and completion, you can store task statuses in a database and provide API endpoints for clients to query task status.
• Notifications: You can send email or push notifications when tasks are completed to inform users.
• Concurrency Control: Limit the number of concurrent tasks to prevent server overload, especially in high-traffic environments.
• Use Asynchronous Programming: To learn more about Asynchronous programming and Multi-threading in .NET, you can check this blog.

Conclusion

In this blog, we explored how to handle long-running background tasks in .NET Core using hosted services and task queues. By offloading resource-intensive tasks to a background worker, you can ensure that your API remains responsive and scalable. This approach not only improves user experience but also enhances the overall performance of your web application.

With hosted services, you can handle complex business logic and tasks like file processing, sending notifications, or even integrating third-party services—all without blocking API requests. This is a powerful pattern for building efficient, scalable applications in .NET Core.