Configure and Validate Custom Polly Resilience Strategies

Introduction

Polly is a .NET library for resilience and transient fault handling. To aid in creating robust .NET applications with Polly, Microsoft offers the Microsoft.Extensions.Http.Resilience package, tailored to improve the resilience of the HttpClient class. This package enables the addition of various resilience mechanisms to an HttpClient through chained calls on the IHttpClientBuilder type.

In this article, we will concentrate exclusively on adding a custom resilience strategy for retrying HTTP calls made through the HttpClient class and, more specifically, on how to ensure that this custom strategy behaves as expected in faulting scenarios.

Polly.Demo Application

The demo source code for the Polly retry strategy is available at: https://github.com/praveenprabharavindran/polly.demo. This code demonstrates how Polly could be integrated into a .NET application to handle retries and timeouts for HTTP requests. The code uses the ResilientConsumerController to make calls to the ChaoticService. The ChaoticService responds with the current server time. However, it's intentionally engineered to include a random delay significant enough to activate a Polly retry on the ResilientConsumer side.

Key Components

• RetryConfig: A configuration class holding retry-related settings, such as the maximum number of retry attempts and the delay between retries.
• HttpRetryStrategyFactory: A factory class that utilizes RetryConfig to construct a ResiliencePipeline featuring retry and timeout policies. This class ensures a uniform approach to creating retry strategies from a given configuration.
  C#

  //HttpRetryStrategyFactory.cs
  
       public ResiliencePipeline<HttpResponseMessage> Create()
      {
          var timeoutStrategyOptions = MapToTimeoutStrategyOptions(_config);
          var retryStrategyOptions = MapToHttpRetryStrategyOptions(_config);
          var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
              .AddRetry(retryStrategyOptions)
              .AddTimeout(timeoutStrategyOptions)
              .Build();
          return pipeline;
      }

• ResilientConsumerController: An API controller that makes HTTP requests using an HttpClient instance. It's resilient to transient failures due to the attached Polly policies.

HttpClient Setup: In Program.cs, the HttpClient is configured with the AddResilienceHandler extension method. This resilience handler is setup using a lambda expression, which retrieves an instance of HttpRetryStrategyFactory from the service provider. The retry strategy, created using the HttpRetryStrategyFactory, is then added to the pipelineBuilder argument. The provided code snippet illustrates how to attach the resilience pipeline to the HttpClient named ChaoticService.

//Program.cs 

builder.Services.AddHttpClient("ChaoticService")
    .AddResilienceHandler("RetryStrategy", (pipelineBuilder, context) =>
    {
        var strategyFactory = 
            context.ServiceProvider.GetRequiredService<IHttpRetryStrategyFactory>();
        pipelineBuilder.AddPipeline(strategyFactory.Create());
    });

Using the Application

Configuration: Set up the RetryConfig in your application's configuration file (e.g., appsettings.json). Define the maximum number of retry attempts, the retry delay, and the timeout for each attempt.

Executing: Start by running the Polly.Demo.Api project. This opens a browser to: https://localhost:7018/ResilientConsumer. The ResilientConsumer endpoint then reaches out to the ChaoticService API to fetch the current time.

The ChaoticService is designed to introduce a delay in random calls. Specifically, there is a 50% chance that any response will be delayed by 3 seconds. This intermittent delay is significant enough to trigger a Polly retry mechanism on the ResilientConsumer side.

Logging: The application logs both retry and timeout events, providing insights into its resilience behavior and aiding in troubleshooting.

In summary, the ResilientConsumer will almost always receive a response, although this might result from multiple retries. If you reload the ResilientConsumer a few times, you will notice that sometimes, it takes longer to reload. In those cases, check the console. You should see log entries such as "Execution timed out after 2 seconds" indicating a timeout.

How the Polly Retry Strategy Works

The Polly retry strategy merges a per-attempt timeout policy with a retry policy. If an attempt to the Chaotic Service fails, then the retry policy activates and retries the call.

Retry Policy: This triggers a specified number of retry attempts with a delay between each, when an HTTP request fails. It is beneficial for handling transient errors, such as temporary network failures.

Timeout Policy: This sets a time limit for each HTTP request. If a request doesn't complete within this timeframe, it is aborted. This prevents the application from waiting indefinitely for a response.

Ensuring Robust Configuration: Integration Testing of HttpRetryStrategy

For validating the configuration, we use a test named HttpRetryStrategyShould.RetryWhenTimeoutExpires. This test is designed to validate the behavior of the HttpRetryStrategy in scenarios where a timeout occurs during an HTTP request. It's important to emphasize that our focus is not on testing the components from the Polly library. Rather, our objective is to verify that the policies have been accurately configured to construct the resilience strategy.

Test Overview

The test is structured into three key sections: Arrange, Act, and Assert.

1. Arrange

   In this section, we prepare the necessary components for testing the HttpRetryStrategy. The steps include:

   • Setting up the retry configuration, which involves defining the per-attempt timeout, the maximum number of retry attempts, and the delay between retries.
   • Creating an instance of HttpRetryStrategy using the configured settings.
   • Mocking an HttpMessageHandler to simulate delayed responses. The mock's SendAsync method is configured to delay responses for longer than the per-attempt timeout, ensuring that each request times out and triggers a retry.
   • Initializing an HttpClient with the mocked HttpMessageHandler.

2. Act

   Here, we execute the test scenario using the ExecuteAsync method on the HttpRetryStrategy. The method is invoked with a lambda expression to perform an HttpClient.GetAsync call to a dummy URL as demonstrated below:

   C#

   await retryStrategy
       .ExecuteAsync(async token => await httpClient.GetAsync
                    ("http://dummy.url", token), CancellationToken.None)
       .ConfigureAwait(false)
   

   The ExecuteAsync method is expected to throw a TimeoutRejectedException due to the induced timeouts.

3. Assert

   The assertion phase involves verifying that the SendAsync method on the mockHttpMessageHandler was called the expected number of times (three, in this case). This count includes the initial attempt and the subsequent retries, confirming that the retry logic is functioning correctly.

Key Points

• The test ensures that the retry mechanism activates when a per-attempt timeout is exceeded.
• By mocking the response delay to be longer than the configured timeout, we can accurately test the retry behavior.
• Verifying the number of calls to SendAsync confirms the number of retry attempts made by the strategy.

Testing Resilience Strategies with AddResilienceHandler

Our previous test, while beneficial, did not entirely mimic the scenario where resilience strategies are incorporated using the AddResilienceHandler extension method. Even though the results of our prior test are similar, if this difference matters to you and your specific use case requires testing a strategy attached via the AddResilienceHandler extension method, it remains possible, though it may require additional effort.

The additional effort arises because there is no standalone HttpClientBuilder class in the .NET standard libraries. Instead, the IHttpClientBuilder interface is accessible through extension methods like AddHttpClient() on IServiceCollection. This setup is integral to configuring named and typed HttpClient instances within the ASP.NET Core framework, leveraging its dependency injection capabilities.

Given this context, to simulate a similar setup outside of the conventional ASP.NET Core dependency injection framework, one would need to instantiate ServiceCollection directly. This approach allows us to use AddHttpClient(), thereby obtaining an IHttpClientBuilder instance. Once you have this, you can customize the HttpClient configuration by applying the AddResilienceHandler extension method.

Test Overview

In the accompanying code, you will find another test case named HttpRetryStrategyShould.RetryWhenTimeoutExpiresWithServiceCollection. This test is almost similar to the previous one, except for the following changes:

• Arrange

  • Service Collection Configuration: The test sets up a ServiceCollection and adds necessary services and configurations:
    • It registers the retryConfig as a singleton.
    • It adds a singleton for IHttpRetryStrategyFactory, the factory class for creating instances of the retry strategy.
  • HttpClient Setup: The HttpClient is configured with the retry strategy in a way similar to how to we configured it in the program.cs. The resilience handler (AddResilienceHandler) is applied to the client, using the retry strategy produced by IHttpRetryStrategyFactory:
  C#

  // Test case: HttpRetryStrategyShould.RetryWhenTimeoutExpiresWithServiceCollection
  
  .
  .
  .
  
  services.AddHttpClient(mockHttpClientName)
          .ConfigurePrimaryHttpMessageHandler(() => mockHttpMessageHandler.Object)
          .AddResilienceHandler("RetryStrategy", (pipelineBuilder, context) =>
          {
                 var strategyFactory =
                 context.ServiceProvider.GetRequiredService<IHttpRetryStrategyFactory>();
                 pipelineBuilder.AddPipeline(strategyFactory.Create());
          });
  

• Act

  Here, we make a call to HttpClient.GetAsync() directly, i.e., without passing it as an argument to the retryStrategies ExecuteAsync() method.

  C#

  await Assert.ThrowsAsync<TimeoutRejectedException>(() => httpClient.GetAsync(dummyUrl));
  

• Assert

  The assert section is same as our previous test.

Key Points

• This test closely mirrors our prior one. In this instance, we utilized the .NET's ServicesCollection class to obtain an instance of IHttpClientBuilder, to which we then attached the Polly RetryStrategy using the AddResilienceHandler method.

Conclusion

In this article, we explored the integration of custom Polly resilience strategies into .NET applications and examined two different methods to validate policies for managing HTTP call failures.

History

• 1^st February, 2024: Initial version