You can always do it using
System.Net.HttpWebRequest. Using this class one can mimic everything the human user could do using a Web browser (including submitting a form using POST, Ajax requests, etc.) in every detail, and something the human user can not do. You can do whatever Web crawlers do as well, see
http://en.wikipedia.org/wiki/Web_crawler[
^].
See MSDN
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx[
^] for further detail.
A legal warning made by wizardzz is valid of course, but it is more related to what you're going to do with collected data. If you only do what a regular user can do and don't break protection systems, you can be more or less sure. However, some legal practices in US and some other countries are absolutely crazy, so in certain situations you indeed can have legal problems on the most innocent actions from your side.
In real life, however, Web crawling is a common practice. Google and other search engines are still alive as you can see…
—SA
Submit an article or tip