Transient Exception Handling

Introduction

I am in the process of reading this very book: Cloud Design Patterns which discusses a great many patterns for the cloud (geared primarily at Azure), and there is talk of a “Circuit Breaker” and “Retry Pattern”.

The “Circuit Breaker” pattern is a new one to me, but the “Retry Pattern” is certainly something I have seen many times before. The basic idea is this:

You want to run a bit of code, but some Exception could occur, so you have some sort of retry policy in place, say one that retries every 10 seconds, that might work, if the Exception is a transient one caused by the network say.

Now I don’t know about you but I have hand coded this sort of thing a lot and never spent too much time (apart from this attempt I did when using Rx which does actually work very well:

• http://stackoverflow.com/questions/20189166/rx-back-off-and-retry

The thing is, this is a recurring issue, so surely there is help out there for this issue. Turns out there is.

I have seen a few libraries that help out in this area, here are a few that I have found.

Palmer

This project is hosted at GutHub: https://github.com/mitchdenny/palmer/.

And here are a few examples (taken from Github).

Retry For Some Time

Retry for 15 seconds:

Retry .On<WebException>()
      .For(TimeSpan.FromSeconds(15))
      .With(context => { // Code that might periodically fail due to connectivity issues. }); 

Retry Forever

Keep Retrying:

Retry .On<WebException>()
      .Indefinitely()
      .With(context => { // Code that might throw a web exception. }); 

Multiple Exceptions

You can also deal with multiple exceptions:

Retry .On<WebException>()
      .For(5)
      .AndOn<SqlException>()
      .For(5)  
      .With(context => { // Code that might throw a web exception, or a sql exception. }); 

For more examples, check out the Palmer github link posted above.

Polly

There is another nice library, which you can use which is also hosted at GitHub, it is called “Polly”: https://github.com/michael-wolfenden/Polly

So let's see some examples, again these are taken directly from the GitHub readme.

Step 1: Specify the Type of Exceptions You Want the Policy to Handle

// Single exception type 
Policy .Handle<DivideByZeroException>() 

// Single exception type with condition 
Policy .Handle<SqlException>(ex => ex.Number == 1205) 

// Multiple exception types 
Policy .Handle<DivideByZeroException>() 
       .Or<ArgumentException>() 

// Multiple exception types with condition
 Policy .Handle<SqlException>(ex => ex.Number == 1205) 
        .Or<ArgumentException>(ex => x.ParamName == "example") 

Step 2: Specify How the Policy Should Handle Those Exceptions

Retry

// Retry once 
Policy .Handle<DivideByZeroException>() .Retry() 

// Retry multiple times 
Policy .Handle<DivideByZeroException>() .Retry(3) 

// Retry multiple times, calling an action on each retry 
// with the current exception and retry count 
Policy .Handle<DivideByZeroException>() 
       .Retry(3, (exception, retryCount) => { // do something }); 

// Retry multiple times, calling an action on each retry 
// with the current exception, retry count and context 
// provided to Execute() 
Policy .Handle<DivideByZeroException>() 
       .Retry(3, (exception, retryCount, context) => { // do something }); 

Retry Forever

// Retry forever 
Policy .Handle<DivideByZeroException>() 
       .RetryForever() 

// Retry forever, calling an action on each retry with the 
// current exception 
Policy .Handle<DivideByZeroException>() 
       .RetryForever(exception => { // do something }); 

// Retry forever, calling an action on each retry with the 
// current exception and context provided to Execute() 
Policy .Handle<DivideByZeroException>() 
       .RetryForever((exception, context) => { // do something }); 

Retry and Wait

// Retry, waiting a specified duration between each retry 
Policy .Handle<DivideByZeroException>() 
       .WaitAndRetry(new[] 
       { TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(3) }); 

// Retry, waiting a specified duration between each retry, 
// calling an action on each retry with the current exception 
// and duration 
Policy .Handle<DivideByZeroException>() 
       .WaitAndRetry(new[] { 1.Seconds(), 2.Seconds(), 3.Seconds() }, 
                    (exception, timeSpan) => { // do something }); 

// Retry, waiting a specified duration between each retry, 
// calling an action on each retry with the current exception, 
// duration and context provided to Execute() 
Policy .Handle<DivideByZeroException>() 
       .WaitAndRetry(new[] { 1.Seconds(), 2.Seconds(), 3.Seconds() }, 
                     (exception, timeSpan, context) => { // do something }); 

// Retry a specified number of times, using a function to 
// calculate the duration to wait between retries based on 
// the current retry attempt (allows for exponential backoff) 
// In this case will wait for 
// 1 ^ 2 = 2 seconds then 
// 2 ^ 2 = 4 seconds then 
// 3 ^ 2 = 8 seconds then 
// 4 ^ 2 = 16 seconds then 
// 5 ^ 2 = 32 seconds 
Policy .Handle<DivideByZeroException>() 
       .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) ); 

// Retry a specified number of times, using a function to 
// calculate the duration to wait between retries based on 
// the current retry attempt, calling an action on each retry 
// with the current exception, duration and context provided 
// to Execute() 
Policy .Handle<DivideByZeroException>() 
       .WaitAndRetry( 5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), 
                 (exception, timeSpan, context) => { // do something } ) 

Step 3: Execute the Policy

// Execute an action var policy = Policy .Handle<DivideByZeroException>() .Retry(); 
policy.Execute(() => DoSomething()); 

// Execute an action passing arbitrary context data 
var policy = Policy .Handle<DivideByZeroException>() 
                    .Retry(3, (exception, retryCount, context) => 
                    { var methodThatRaisedException = context["methodName"]; 

Log(exception, methodThatRaisedException); }); 

policy.Execute( () => DoSomething(), new Dictionary<string, object>() 
{{ "methodName", "some method" }} ); 

// Execute a function returning a result 
var policy = Policy .Handle<DivideByZeroException>() .Retry(); 
var result = policy.Execute(() => DoSomething()); 

// Execute a function returning a result passing arbitrary context data 
var policy = Policy .Handle<DivideByZeroException>() 
                    .Retry(3, (exception, retryCount, context) => 
                    { object methodThatRaisedException = context["methodName"]; 

Log(exception, methodThatRaisedException) }); 

var result = policy.Execute( () => DoSomething(), 
new Dictionary<string, object>() 
{{ "methodName", "some method" }} ); 

// You can of course chain it all together 
Policy .Handle<SqlException>(ex => ex.Number == 1205)
       .Or<ArgumentException>(ex => ex.ParamName == "example")
       .Retry()
       .Execute(() => DoSomething()); 

Transient Application Block

There is also the Microsoft Patterns & Practices offering “Transient Application Block”. Now P&P pattern can be, well a bit over the top to be frank. This one is not so bad though, it's mainly geared toward working with Azure, but like a lot of stuff in Azure can be used outside of Azure.

There is a nice CodeProject article on how to use this application block outside of Azure which you can read right here:

• Transient Fault Handling Application Block Simplified

Here is a small example of how to use this block taken from the article noted here...

static void Main(string[] args) 
{ 
    try 
    { 
        // Step 1 
        var retryStrategy = new Incremental
        (RETRY_COUNT, TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(2)); 

        // Step 2 
        var retryPolicy = new RetryPolicy<CustomTransientErrorDetectionStrategy>(retryStrategy); 

        // Step 3 
        retryPolicy.ExecuteAction(NavigateTo); 
    } 
    catch (Exception e) 
    { 
        Console.WriteLine(e.Message); 
    } 
} 

// This is the code that may experience transient errors 
static private void NavigateTo() 
{ 
    Console.WriteLine(DateTime.Now); 
    WebClient wc = new WebClient(); 
    wc.DownloadString("<a href="file:///c://temp.txt">c:\\temp.txt</a>"); 
} 

...where the above code makes use of this custom ITransientErrorDetectionStrategy implementation:

internal class CustomTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy 
{ 
    public bool IsTransient(Exception ex) 
    { 
        if (ex is WebException) 
            return true; 

        return false; 
    } 
} 

Conclusion

Anyway there you go, I know I have done nothing more than paste a few links to other peoples' work here, but there may be some people that did not know about these very useful libraries and you may think aha that is the one for me. So happy fault handling.

Until next time!