First of all,
Web scraping is something irrelevant to ASP.NET, "include files" or other server-side technology:
http://en.wikipedia.org/wiki/Web_scraping[
^].
All what matters is HTTP request you can make and HTTP response. All details of server-side behavior are hidden from the client side. You can achieve all the Web server user can get, perhaps a little more, but no less.
Please see my comment to the question: you did not describe your problem. So, I can advice just one thing: instead of
System.Net.WebClient use more general and hence powerful class
System.Net.HttpWebRequest,
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx[
^],
http://msdn.microsoft.com/en-us/library/system.net.webrequest.aspx[
^].
Please see my past answers:
get specific data from web page[
^],
How to get the data from another site[
^].
—SA
Submit an article or tip