How Can I Write An Http Aggregator In C#

Question

1.00/5 (1 vote)

See more:

for e.g i want to store website url address in array Website URLs will be stored in the hard coded array.

For example A[1] = “www.msn.com”;
A[2]= www.gmail.com; .. .. Etc..

This program is always running and waiting for certain condition ( in this case a time) to occur,

for example when the time is 6.00 AM the program hits all the websites listed in the array and fetch the HTTP content ( the final HTML output) in a output.html file.

For example if www.msn.com returns the following HTTP content
<html> <body> This is Msn.com </body> </html>

and www.hotmail.com returns the following HTTP content
<html> <body> This is hotmail.com </body> </html>

If these are the only two website ( in the array) than the final output in form of output.html would be :
<html> <body> This is Msn.com </body>
</html> <html> <body> This is hotmail.com </body> </html>

In a first phase it will be just a simple C# application.

What would be configurable via app.config or equivalent ? ·

The time ( 6.00 AM) in hour : minute : AM/PM format, which shows when it should generate that file. The file name of the generated file · The folder where it should save the file.

Condition : I will NOT re-start the application. The changes to the app.config are immediate.

Posted 19-May-14 23:21pm

Member 10820750

Add a Solution

Comments

José Amílcar Casimiro 20-May-14 5:52am

You have your requirements, what are your difficulties?

Member 10820750 20-May-14 10:12am

My Aim Is to code a program that on a particular time for eg 6:00 Am fetches the entire list of websites from an hard coded array like a[0]="www.msm.com"; a[1]="www.fb.com"; ...... and so on and then stores the HTML output of particluar website in a particular file like for eg msm.html,fb.html........ and so on and then finally after this procedure ends combine each website outputed html file into single html file for eg output.html

Need For Coding :

1) This program is always running and waiting for certain condition ( in this case a time) to occur, for example when the time is 6.00 AM the program hits all the websites listed in the array and fetch the HTTP content ( the final HTML output) in a output.html file.

2)What would be configurable via app.config or equivalent ?
· i) The time ( 6.00 AM) in hour : minute : AM/PM format, which shows when it should generate that file.
ii) The file name of the generated file ·
iii) The folder where it should save the file.

Condition : I do NOT re-start the application. The changes to the app.config are immediate.

3)Use timer based thread or timer control.

One timer control ( times out at every 10 seconds ) reads the app.config file at every 10 seconds.

In app.config all the settings are stored including the settings of 2nd timer control.

2nd timer control has the start time and when to expire. And his settings are stored in app.config.

10 second timer control readjusts the 2nd timer control.

Upon 2nd timer control timeout it does the main work ( http aggregation ).

Stephen Hewison 20-May-14 6:07am

You haven't actually described your problem.

Member 10820750 20-May-14 10:12am

My Aim Is to code a program that on a particular time for eg 6:00 Am fetches the entire list of websites from an hard coded array like a[0]="www.msm.com"; a[1]="www.fb.com"; ...... and so on and then stores the HTML output of particluar website in a particular file like for eg msm.html,fb.html........ and so on and then finally after this procedure ends combine each website outputed html file into single html file for eg output.html

Need For Coding :

1) This program is always running and waiting for certain condition ( in this case a time) to occur, for example when the time is 6.00 AM the program hits all the websites listed in the array and fetch the HTTP content ( the final HTML output) in a output.html file.

2)What would be configurable via app.config or equivalent ?
· i) The time ( 6.00 AM) in hour : minute : AM/PM format, which shows when it should generate that file.
ii) The file name of the generated file ·
iii) The folder where it should save the file.

Condition : I do NOT re-start the application. The changes to the app.config are immediate.

3)Use timer based thread or timer control.

One timer control ( times out at every 10 seconds ) reads the app.config file at every 10 seconds.

In app.config all the settings are stored including the settings of 2nd timer control.

2nd timer control has the start time and when to expire. And his settings are stored in app.config.

10 second timer control readjusts the 2nd timer control.

Upon 2nd timer control timeout it does the main work ( http aggregation ).

Member 10820750 20-May-14 8:13am

Well Sir the difficulties are that :

1) Use timer based thread or timer control.

One timer control ( times out at every 10 seconds ) reads the app.config file at every 10 seconds.So how can i do that? any code examples would help me a lot.

In app.config all the settings are stored including the settings of 2nd timer control.

2nd timer control has the start time and when to expire. And its settings are stored in app.config.

1st 10 second timer control readjusts the 2nd timer control.

Upon 2nd timer control timeout it does the main work ( http aggregation ).

This is a typical console application and not a web based application ..

so any example codes or ideas???

Ziee-M 20-May-14 9:43am

I don't see the need to read app.config. app.config contain connection string to databases and other configuration stuff.
Help us understand your issue a little better.

You need to develope an app that connects to diffrent websites, retrive html code, and do some stuff?

Start by explaining the need, then how you see it done using code.

Thanks.

Member 10820750 20-May-14 10:12am

My Aim Is to code a program that on a particular time for eg 6:00 Am fetches the entire list of websites from an hard coded array like a[0]="www.msm.com"; a[1]="www.fb.com"; ...... and so on and then stores the HTML output of particluar website in a particular file like for eg msm.html,fb.html........ and so on and then finally after this procedure ends combine each website outputed html file into single html file for eg output.html

Need For Coding :

1) This program is always running and waiting for certain condition ( in this case a time) to occur, for example when the time is 6.00 AM the program hits all the websites listed in the array and fetch the HTTP content ( the final HTML output) in a output.html file.

2)What would be configurable via app.config or equivalent ?
· i) The time ( 6.00 AM) in hour : minute : AM/PM format, which shows when it should generate that file.
ii) The file name of the generated file ·
iii) The folder where it should save the file.

Condition : I do NOT re-start the application. The changes to the app.config are immediate.

3)Use timer based thread or timer control.

One timer control ( times out at every 10 seconds ) reads the app.config file at every 10 seconds.

In app.config all the settings are stored including the settings of 2nd timer control.

2nd timer control has the start time and when to expire. And his settings are stored in app.config.

10 second timer control readjusts the 2nd timer control.

Upon 2nd timer control timeout it does the main work ( http aggregation ).

Member 10820750 20-May-14 10:12am

My Aim Is to code a program that on a particular time for eg 6:00 Am fetches the entire list of websites from an hard coded array like a[0]="www.msm.com"; a[1]="www.fb.com"; ...... and so on and then stores the HTML output of particluar website in a particular file like for eg msm.html,fb.html........ and so on and then finally after this procedure ends combine each website outputed html file into single html file for eg output.html

Need For Coding :

1) This program is always running and waiting for certain condition ( in this case a time) to occur, for example when the time is 6.00 AM the program hits all the websites listed in the array and fetch the HTTP content ( the final HTML output) in a output.html file.

2)What would be configurable via app.config or equivalent ?
· i) The time ( 6.00 AM) in hour : minute : AM/PM format, which shows when it should generate that file.
ii) The file name of the generated file ·
iii) The folder where it should save the file.

Condition : I do NOT re-start the application. The changes to the app.config are immediate.

3)Use timer based thread or timer control.

One timer control ( times out at every 10 seconds ) reads the app.config file at every 10 seconds.

In app.config all the settings are stored including the settings of 2nd timer control.

2nd timer control has the start time and when to expire. And his settings are stored in app.config.

10 second timer control readjusts the 2nd timer control.

Upon 2nd timer control timeout it does the main work ( http aggregation ).

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Ziee-M · Answer 1 · 2014-05-20T04:44:00

Hi, its much clearer now.

Here is globaly how you can achive your goal :

I) Running an app at a specific time.
-the best solution : use windows tasks. windows allow you to run a specific software at a specific time (daily weekly or monthly).
=> So you will no longer have to worry about your app running at a specifc time each day.

-For the file location, you will probably specife your location in the code.
specifiying the location in the app.config is possible (i guess), but i don't now how to do it.

II) General Idea to capture webPages html code.
I am working a lot with this kind of apps. you have 2 solutions. either use WebBrowser or httpwebRequets.
the easiest is the later. httpwebRequest allow you te get the html code of a page synchronously. its perfect to start with this technique.

Here is a sample to get an url html code:

C#

public string CapturePage(string url)
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            request.Accept = "*/*";

            request.AllowAutoRedirect = true;
            request.Timeout = 60000;
            request.UserAgent = "http_requester/0.1";
            request.Method = "GET";
            request.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();

            StreamReader sr = new StreamReader(response.GetResponseStream());
            ///copier les donnees du stream dans une variable
            string sourceCode = sr.ReadToEnd();            
            
            sr.Close();
            response.Close();

            return sourceCode;

        }

I don't think you will have to use threads for now. In the other hand, its much better to develop the app as console application, so it automatically close when the task is finished.

III) timers:
Timers don't have anything to do with threads. the idea is to instanciate one or many timers.
once the timer reach its cooldown time, an event is raised.

here is a link to understand the logic: (MSDN)
http://msdn.microsoft.com/fr-fr/library/system.timers.timer(v=vs.110).aspx[^]

Pretty Easy :) no thread needed

Don't hesitate to ask more questions.

Hope it helps.

How Can I Write An Http Aggregator In C#

1 solution

Solution 1

Add your solution here

Preview 0

Existing Members

...or Join us