Error Handling (Retry Design Pattern)

• Download source - 20.8 KB

Introduction

Retry mechanism is simple, well known and widely used in places where a call is possible to fail but more than one are more probable to succeed.

The advent of cloud computing and the incremental inclination to build applications and solutions based on the cloud and the need for designing robust and stable solution considering the instability of any network have put more emphasis on this retry mechanism.

Hence, there are a few related mechanisms that are documented in the shape of design patterns, I am talking mainly about the “Retry” and “Circuit Breaker” design patterns.

Using the Code

The code shows the seed building block only, although it is covered by tests and working, yet there is some wide room for enhancements. I will be working on that for my real production but this is some sample for other to reuse or build upon.

The code is used in the tests and also in the program.cs.

Just play with the parameters.

Try:

breakerToRecover = TimeSpan.FromSeconds(5);

then try:

breakerToRecover = TimeSpan.FromSeconds(2);

Also change these numbers and see different behaviors:

private const int numberOfFailures = 5;
private const int numberOfRetry = 5;
private const int numberOfCircuitTrials = 5;

The Code

The design is to support DI, Fluent interface, hence let's start with a simple interface that can be extended to support more functionalities.

public interface IRetryBlock
   {
       /// <summary>
       /// Ons the specified condition.
       /// </summary>
       /// <typeparam name="ExceptionType">The type of the exception type.</typeparam>
       /// <param name="condition">The condition.</param>
       /// <returns>RetryBlock.</returns>
       IRetryBlock On<ExceptionType>(Func<Exception, bool> condition = null);
 
       /// <summary>
       /// Retries the specified retry count.
       /// </summary>
       /// <param name="retryCount">The retry count.</param>
       /// <param name="retryInterval">The retry interval.</param>
       /// <returns>RetryBlock.</returns>
       IRetryBlock Retry(int retryCount, TimeSpan retryInterval);
 
       /// <summary>
       /// Retries the asynchronous.
       /// </summary>
       /// <param name="retryCount">The retry count.</param>
       /// <param name="retryInterval">The retry interval.</param>
       /// <returns>Task&lt;RetryBlock&gt;.</returns>
       Task<IRetryBlock> RetryAsync(int retryCount, TimeSpan retryInterval);
   }

Then the implementation:

public abstract class RetryBlock : IRetryBlock
  {
      /// <summary>
      /// The _to do
      /// </summary>
      private Action _toDo;
 
      /// <summary>
      /// The _transient exceptions
      /// </summary>
      private readonly ConcurrentDictionary<Type, Func<Exception, bool>> _transientExceptions;
 
      /// <summary>
      /// The default condition
      /// </summary>
      private readonly Func<Exception, bool> defaultCondition = null;
 
      /// <summary>
      /// Initializes a new instance of the <see cref="DefaultRetryBlock"/> class.
      /// </summary>
      /// <param name="toDo">To do.</param>
      internal RetryBlock(Action toDo)
      {
          _toDo += toDo;
          _transientExceptions = new ConcurrentDictionary<Type, Func<Exception, bool>>();
      }
 
      /// <summary>
      /// Determine if the exception is transient.
      /// In some cases this may be as simple as checking the exception type, in other
      /// cases it may be necessary to inspect other properties of the exception.
      /// </summary>
      /// <param name="ex">The ex.</param>
      /// <returns><c>true</c> if the specified ex is transient; 
      /// otherwise, <c>false</c>.</returns>
      protected virtual bool IsTransient(Exception ex)
      {
          if (ex is OperationTransientException) return true;
 
          if (_transientExceptions.ContainsKey(ex.GetType()))
          {
              Func<Exception, bool> exceptionCondition;
              _transientExceptions.TryGetValue(ex.GetType(), out exceptionCondition);
              return exceptionCondition.Invoke(ex);
          }
          return false;
      }
 
      /// <summary>
      /// Ons the specified condition.
      /// </summary>
      /// <typeparam name="ExceptionType">The type of the exception type.</typeparam>
      /// <param name="condition">The condition.</param>
      /// <returns>RetryBlock.</returns>
      public virtual IRetryBlock On<ExceptionType>(Func<Exception, bool> condition = null)
      {
          _transientExceptions.TryAdd(typeof(ExceptionType), condition ?? defaultCondition);
          return this;
      }
 
      /// <summary>
      /// Retries the execution multiple times as retry count, with intervals between  them.
      /// </summary>
      /// <param name="retryCount">The retry count.</param>
      /// <param name="retryInterval">The retry interval.</param>
      /// <returns>RetryBlock.</returns>
      public virtual IRetryBlock Retry(int retryCount, TimeSpan retryInterval)
      {
          var currentRetry = 0;
          while (true)
          {
              try
              {
                  this.Execute();
                  break;
              }
              catch (Exception ex)
              {
                  currentRetry++;
                  // Check if the exception thrown was a transient exception
                  // based on the logic in the error detection strategy.
                  // Determine whether to retry the operation, as well as how
                  // long to wait, based on the retry strategy.
                  if (currentRetry > retryCount || !IsTransient(ex))
                  {
                      // If this is not a transient error
                      // or we should not retry re-throw the exception.
                      throw;
                  }
              }
 
              // Wait to retry the operation.
              // Consider calculating an exponential delay here and
              // using a strategy best suited for the operation and fault.
              Console.WriteLine($"Current Retry: {currentRetry}");
              Thread.Sleep(retryInterval);
          }
          return this;
      }
 
      private IRetryBlock Execute()
      {
          if (_toDo == null) throw new NullReferenceException("_toDo");
          _toDo?.Invoke();
          return this;
      }
 
      public virtual Task<IRetryBlock> RetryAsync(int retryCount, 
      TimeSpan retryInterval) => Task.Run(() => { return Retry(retryCount, retryInterval); });
 
      private Task<IRetryBlock> ExecuteAsync() => Task.Run(() => { return Execute(); });
  }

Helper class is just to create an instance of the (IRetryBlock), the configure method is for you to define an instance of another implementation.

public partial class Retry
  {
      private static Func<Action, IRetryBlock> _builder;
 
      /// <summary>
      /// Handles the specified to do.
      /// </summary>
      /// <param name="toDo">To do.</param>
      /// <returns>DesignPatterns.RetryBlock.</returns>
      public static IRetryBlock Handle(Action toDo) => _builder == null ? 
      new DefaultRetryBlock(toDo) : _builder.Invoke(toDo);
 
      public static void Configure(Func<Action, IRetryBlock> builder)
      {
          _builder = builder;
      }
  }

Points of Interest

There are some unit tests to cover some scenarios, I will add more in the complete version.

Circuit breaker is another design pattern that is also well known and used in the similar scenarios where you use the "Retry", yet in my example, I used them together and tried multiple scenarios and showed how it acts.

Let’s see these examples:

private const int numberOfFailures = 5; 
private const int numberOfRetry = 5; 
private const int numberOfCircuitTrials = 5;

This will cause results like:

private const int numberOfFailures = 4;
private const int numberOfRetry = 5;
private const int numberOfCircuitTrials = 5;

private const int numberOfFailures = 5
private const int numberOfRetry = 5;
private const int numberOfCircuitTrials = 4;

and the like.

It all depends on how many failures are occurring before the first successful call.

Retry mechanism will take care of errors if the expected exception occurs less times than retry counts.

Circuit breaker will disconnect the system informing that system is unavailable if the number of failures reached the number configured in the circuit.

There is the timeout configuration if it is too small for the circuit to recover, it will succeed..

References

1. Circuit Breaker Pattern
2. Retry Pattern
3. Coderview