Click here to Skip to main content
15,867,568 members
Articles / Programming Languages / C#

FETCH! Retrieve data from a remote web server with one line of code (with MVC reCAPTCHA example)

Rate me:
Please Sign up or sign in to vote.
4.92/5 (36 votes)
20 Aug 2015CPOL3 min read 49.4K   71   27
Often an application needs data from an HTTP server, such as web services, or to scrape content. .NET provides HttpWebRequest, but it requires a lot of code to perform trivial operations. This class simplifies the task dramatically by abstracting the HTTP protocol, streams, and exceptions.
Source (RedCell.Net on GitHub)
Binary Component (RedCell.Net.dll)
Compiled Help (RedCell.Net.chm)

Introduction

This is one of those classes I've had in my toolbox for a many years, and I still use it regularly. It was inspired by the FreeBSD fetch command, which is more intuitive than curl. It compiles against version 2.0 of the .NET Framework, so it can be used in almost any desktop or web application. Its only dependency is System.dll.

Background

Class diagramSometimes you just want to get the result of an HTTP request and use it. Maybe you want to download a whole webpage to scrape its content for some data, or maybe your're dealing with a web service that just returns a number, string, some XML, or some JSON. Fetch was create for these cases. It features:

  • Make simple GET or POST requests with a single line of code.
  • Include credentials for authentication.
  • Configurable retry and timeout.
  • Modifyable headers.
  • Get the result returned in the type of your choice.

Using the code

To use the Fetch class in your code, either add Fetch.cs to your project, or reference RedCell.Net.dll in your project.

Simple Use (static)

There are static Get and Post methods for simple requests. These come in two varieties, generic and binary. The binary methods return byte[]. Here are some examples of static usage:

C#
using RedCell.Net;

string url = "http://example.com/api";

// Retrieve some binary data.
byte[] image = Fetch.Get(url);

// Retrieve a string.
string slogan = Fetch.Get<string>(url);

// Post some data and expect an int back.
var formData = new Dictionary<string,string> {
    { "firstname", "Yvan" },
    { "lastname", "Rodrigues" }
};
int age = Fetch.Post<int>(url, formData);

The static methods will return null if an error occurs. Some default values can be set for static invocation:

C#
Fetch.DefaultRetries = 5;         // Number of times to retry.
Fetch.DefaultTimeout = 60000;     // Time to wait for a response (ms).
Fetch.DefaultRetrySleep = 10000;  // Pause between retries (ms).

Instance Use

Instantiation of the class is required to access additional features, and for lower-level access to options.

C#
using System.Net;  
using RedCell.Net;

string url = "http://example.com/api";

// Create a fetcher.
var f = new Fetch(url);

// Set some options.
f.Retries = 1;
f.Timeout = 10;

// Include authentication credentials
f.Credential = new NetworkCredential("yvan", "happy-fun-password");

// Make the request.
// The result will be stored as byte[] in the ResponseData property.
f.Get(); 

// Exceptions are caught, so we have to check for success.
if(!f.Success)
{
    // If the fetch failed, examine f.Response to find out why.
}

float response = f.ConvertResponse<float>();

Applied Example: Recaptcha MVC

Recaptcha

Today I wanted to add the new reCAPTCHA to a web form in an MVC web application. There are tons of libraries to do this on Nuget and GitHub, but I really wanted to cut-and-paste something dead simple. Here's the simple approach, using Fetch. In this example, my form uses the same view for showing and handling the form, like:

HTML
@if (Request.HttpMethod == "POST")
{
    <p>Thanks!</p>
}
@else
{
    <form> 20 questions </form>
}
I am using Json.NET (Newtonsoft.Json), which is available through Nuget.

Three steps

1. Sign up and get your keys.

2. Add to your web form view before the submit button:

HTML
<script src="https://www.google.com/recaptcha/api.js"></script>
<div class="g-recaptcha" data-sitekey="put-your-site-token-here" ></div>
3. Add to your controller:
C#
protected bool VerifyRecaptcha()
{
    const string key = "your-secret-key";
    const string url = "https://www.google.com/recaptcha/api/siteverify";
    string recaptcha = Request.Form["g-recaptcha-response"];
    string ip = Request.ServerVariables["REMOTE_HOST"];
    var values = new Dictionary<string, string>
    {
        { "secret", key },
        { "response", recaptcha },
        { "remoteip", ip }
    };
    string response = Fetch.Post<string>(url, values);
    if (response == null) return false; // You may wish to do a more sophisticated handler.
    var data = JObject.Parse(response);
    return data.Value<bool>("success");
}

The g-recaptcha-response field from the form is POSTed to the web service along with the private key and the IP address. We use Fetch.Post with string as the type argument, knowing we are expecting a JSON object. Json.NET is then used to parse the response and check for success.

A nice approach is to create a base class for your controllers e.g. RecaptchaController that contains this class, and have your other classes subclass it.

4. Finally, invoke it from your controller:

C#
public ActionResult MoreInformation()
{
    if (Request.HttpMethod == "POST" && VerifyRecaptcha())
        SendConfirmation("Thanks, we'll be in touch);

    return View();
}

Points of Interest

Fetch is a simple wrapper for HttpWebRequest and HttpWebResponse. A basic request with these is normally about 20 lines of code. The main reason is that getting the data content into or out of these classes requires handling of Streams. Internally, Fetch creates a MemoryStream, copies the web stream, and turns it into byte[].

One of the nice features is the ability to use the generic methods to get strongly typed data. This is performed internally by the ConvertResponse<T> method. The tricky part is that arbitrary types cannot be cast to T (unless the where constraint is used); however object can be cast to a generic type, so values are boxed before casting.

History

  • August 13, 2015 · v1.0.0 · Original article
  • August 20, 2015 · v1.0.1 · Bug fix

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Engineer Robotic Assistance Devices / AITX
Canada Canada
Yvan Rodrigues has 30 years of experience in information systems and software development for the industry. He is Senior Concept Designer at Robotic Assistance Devices

He is a Certified Technician (C.Tech.), a professional designation granted by the Institute of Engineering Technology of Ontario (IETO).

Yvan draws on experience as owner of Red Cell Innovation Inc., Mabel's Labels Inc. as Manager of Systems and Development, the University of Waterloo as Information Systems Manager, and OTTO Motors as Senior Systems Engineer and Senior Concept Designer.

Yvan is currently focused on design of embedded systems.

Comments and Discussions

 
QuestionWhat is relation between reCAPTCHA and fetching data Pin
Tridip Bhattacharjee25-Apr-17 22:41
professionalTridip Bhattacharjee25-Apr-17 22:41 
AnswerRe: What is relation between reCAPTCHA and fetching data Pin
Yvan Rodrigues12-Jan-18 9:21
professionalYvan Rodrigues12-Jan-18 9:21 
GeneralMy vote of 5 Pin
D V L11-Sep-15 23:50
professionalD V L11-Sep-15 23:50 
QuestionVery useful, but have you thought about adding an async version? Pin
Jon Smith29-Aug-15 22:06
Jon Smith29-Aug-15 22:06 
QuestionThe help file does not display properly. Pin
Scott4427-Aug-15 6:29
Scott4427-Aug-15 6:29 
AnswerRe: The help file does not display properly. Pin
Yvan Rodrigues27-Aug-15 7:46
professionalYvan Rodrigues27-Aug-15 7:46 
GeneralRe: The help file does not display properly. Pin
DSAlCoda10-Sep-15 4:43
DSAlCoda10-Sep-15 4:43 
QuestionGreat job Pin
Thompson2121-Aug-15 0:31
Thompson2121-Aug-15 0:31 
QuestionCommand llne Pin
dandy7220-Aug-15 7:04
dandy7220-Aug-15 7:04 
AnswerRe: Command llne Pin
Yvan Rodrigues22-Aug-15 8:16
professionalYvan Rodrigues22-Aug-15 8:16 
GeneralRe: Command llne Pin
dandy7222-Aug-15 10:47
dandy7222-Aug-15 10:47 
GeneralVery nice! Pin
Ravi Bhavnani20-Aug-15 4:41
professionalRavi Bhavnani20-Aug-15 4:41 
AnswerRe: Very nice! Pin
Yvan Rodrigues22-Aug-15 8:18
professionalYvan Rodrigues22-Aug-15 8:18 
SuggestionI think there is a '=" where there should be a '+=' ? Pin
orjan rmt18-Aug-15 5:10
orjan rmt18-Aug-15 5:10 
AnswerRe: I think there is a '=" where there should be a '+=' ? Pin
Yvan Rodrigues20-Aug-15 2:40
professionalYvan Rodrigues20-Aug-15 2:40 
QuestionMy vote of 5 Pin
Moshe Ventura16-Aug-15 0:57
Moshe Ventura16-Aug-15 0:57 
AnswerRe: My vote of 5 Pin
Yvan Rodrigues20-Aug-15 3:01
professionalYvan Rodrigues20-Aug-15 3:01 
GeneralLike it Pin
Moses Machua14-Aug-15 23:25
Moses Machua14-Aug-15 23:25 
AnswerRe: Like it Pin
Yvan Rodrigues15-Aug-15 9:24
professionalYvan Rodrigues15-Aug-15 9:24 
GeneralSlick Pin
ArchAngel12314-Aug-15 10:36
ArchAngel12314-Aug-15 10:36 
AnswerRe: Slick Pin
Yvan Rodrigues15-Aug-15 9:25
professionalYvan Rodrigues15-Aug-15 9:25 
GeneralMy vote of 5 Pin
Gustav Brock13-Aug-15 22:39
professionalGustav Brock13-Aug-15 22:39 
AnswerRe: My vote of 5 Pin
Yvan Rodrigues15-Aug-15 9:25
professionalYvan Rodrigues15-Aug-15 9:25 
GeneralMy vote of 5 Pin
Oshtri Deka13-Aug-15 11:36
professionalOshtri Deka13-Aug-15 11:36 
AnswerRe: My vote of 5 Pin
Yvan Rodrigues15-Aug-15 9:26
professionalYvan Rodrigues15-Aug-15 9:26 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.