|
Hi,
I need to crawl a website of below url format and store the content locally in a csv file
now, I only need to crawl URL of below format. I should be able to do a loop with query string ranging from 0 to 1000000(or a max value present).
http://www.mysite.com.au/products/products.asp?p=101
the only way i think will be to do a loop from 0 to 1000000 but again there are some id that doesnt exists and redirects to main page which i need to exclude from crawl list i.e it redirects to http://www.mysite.com.au/products page.
below is the code so far i have coded. please assist me how can i achieve this.
public static void CrawlSite()
{
Console.WriteLine("Beginning crawl.");
CrawlPage("http://www.mysite.com.au/products/products.asp?p="); Console.WriteLine("Finished crawl.");
}
private static void CrawlPage(string url)
{
for(int i=0; i<=100000; i++)
{
Console.WriteLine("Crawling " + url+i.tostring());
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.UserAgent = "blah!";
WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string htmlText = reader.ReadToEnd();
using (StreamWriter sw = File.AppendText(@"c:\logs\data.txt"))
{
sw.WriteLine(htmlText);
sw.Close();
}
}
}
|
|
|
|
|
Look into Regex, that would be the way to go.
|
|
|
|
|
I am trying to set an integer variable to null if no vaue but it's not accepting? how can I set the variable to null if the value in the database is null?
This is what I am trying:
if (sql_reader["company_id"] != null)
company_id = Convert.ToInt32(sql_reader["company_id"]);
else
company_id = null;
and this is what I am getting:
Cannot convert null to 'int' because it is a non-nullable value type
I also tried DBNull.Value but getting:
Cannot implicitly convert type 'System.DBNull' to 'int'
|
|
|
|
|
use nullable types
ex:
int? something = null;
if(something.HasValue)
So all you have to do is declare your company_id as int? instead of int
|
|
|
|
|
You need to make company_id a nullable type by making its declaration
int? company_id = null;
You check if it is null as follows:
if(company_id.HasValue)
System.Console.WriteLine(company_id.Value);
else
System.Console.WriteLine("company_id is null");
Graham
Librarians rule, Ook!
|
|
|
|
|
Hello
This is an article which shows you how to set processor affinity on a per-process basis ...[^]
I'm wondering if:
a. Processor Affinity
b. Memory Allocation Limit+Tracking
can be configured on per-process and per-thread basis
I understand from Perfmon:
a. Processor counter: %C1/2/3.. Time, %Processor Time, Interrupts/sec ...etc
b. Memory counter: Available MBytes, Committed Bytes, Page Faults/sec ...etc
However, we're writing a quite demanding (although not realtime) parallel processing analytic application. We want to say, for example, route slow calc requests to designated threads where processor affinity can be set so these threads are assigned to more processor (we have 8 CPU's) and higher memory limits.
Can this be done with .NET 3.5?
Thanks
dev
|
|
|
|
|
Hi,
IMO most situations do not warrant you to worry about processor affinity; it seems to be a common misconception that the user/programmer, by statically setting some values, could do a better job than the Windows scheduler does.
when you set processor affinity for processes or threads, you are limiting the freedom Windows has. Normally it would try and balance the load for all processors, and try and keep a thread/process on the same processor if there isn't a reason to move it (to achieve better cache efficiency, as level-one caches are local to each processor).
You can't limit the memory usage by processors or threads; memory is tracked at the process level.
Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that. [The QA section does it automatically now, I hope we soon get it on regular forums as well]
|
|
|
|
|
|
Do not create any columns in the datagridview. when you set the datasource, it automatically adds the columns from the table.
I am assuming that there is no trouble till the point of "data.Fill"
|
|
|
|
|
Bizcrack wrote: lost the allignment by copy+paste
you should use PRE tags, not CODE tags, for showing multi-line code!
Luc Pattyn [Forum Guidelines] [Why QA sucks] [My Articles]
I only read code that is properly formatted, adding PRE tags is the easiest way to obtain that. [The QA section does it automatically now, I hope we soon get it on regular forums as well]
|
|
|
|
|
I tried removing the columns, but that does not seem to work.
|
|
|
|
|
Any help? I didn't get the solution yet.
|
|
|
|
|
Hi,
The default scrollbar of windows seems to be to narrow for the ListView i'm developing for the application, this is why i have created a custom ScrollBar. The question is how do i attach the scrollbar to the listview that when scrolled the listview will scroll.
Please advice.
Thanks
|
|
|
|
|
|
I am Manually calling a context menu from a listview based on if it is on an item or not. When i right-click, the function validates if an item is present or not, then shows the correct contextMenu.
The problem that i am running into is that the context menu is showing quiet a bit off of the correct coordinates, (top-left of the mouse, off by an inch or so)
Here is an exert of that code.
private void iconView_MouseUp(object sender, MouseEventArgs e)
{
if (e.Button == MouseButtons.Right)
{
ListViewHitTestInfo hitTestInfo = iconView.HitTest(e.Location);
string filePath, fName;
if (hitTestInfo.Item != null)
{
fName = hitTestInfo.Item.Text;
filePath = getNodePath();
filePath = filePath + @"\" + fName;
ShellMenu.ShellContextMenu ctxMnu = new ShellMenu.ShellContextMenu();
FileInfo[] arrFI = new FileInfo[1];
arrFI[0] = new FileInfo(filePath);
ctxMnu.ShowContextMenu(arrFI, this.PointToScreen(e.Location));
}
else
{
iconContext.Visible = true;
iconContext.Show(this, e.Location);
}
}
}
}
Here is the kicker, i am using a fellow coders ShellContextCode, and it is doing the same thing, so i am pretty sure something is not happening on an application or system level.
-Edit-
Just had a brainstorm, maybe the location being reported is relative to the app, then the input is relative to the screen. I am going to test this while waiting for an experts view.
|
|
|
|
|
kodysumter wrote: ctxMnu.ShowContextMenu(arrFI, this.PointToScreen(e.Location));
I wonder why you are doing it on screen coordinates?
i think you should simply do:
ctxMnu.Show(iconView, e.X, e.Y);
this will show the context menu where you right clicked. do a hit test to confirm whether you should or shouldn't show the contextmenu.
|
|
|
|
|
Ive used all variations.
ctxMnu.Show(iconView, e.X, e.Y);
ctxMnu.Show(iconView, e.Location);
ctxMnu.Show(iconView, this.PointToScreen(e.X, e.Y));
ctxMnu.Show(iconView, this.PointToScreen(e.Location));
This is happening with BOTH menus that are popping up, (the context menu, and the Shell Context.
|
|
|
|
|
Usually, the best method that worked for me so far is:
1. Let windows handle how to select an item. This is automatically done when mousedown is fired.
2. check for any item being selected. if it is selected, show the menu otherwise not.
this is a code snippet which is working.
private void listView1_MouseUp(object sender, MouseEventArgs e)
{
if (listView1.SelectedItems.Count > 0)
{
contextMenuStrip1.Show(listView1, e.X, e.Y);
}
}
|
|
|
|
|
That would work, if an item was deselected before. Problem is that an item will remain highlighted and selected if i right click white space within a listview control. And i cannot/will not disable the selected item because it is needed for some data being showed within the context menu.
|
|
|
|
|
Could you clarify?
kodysumter wrote: item will remain highlighted and selected if i right click white space within a listview control
This can be sorted in MouseDown event. Do a hit test to check if there is an item under the mouse. If not, deselect items in the listview. At Mouseup event, since no items are selected, menu won't be displayed.
kodysumter wrote: And i cannot/will not disable the selected item because it is needed for some data being showed within the context menu.
I do not understand this part. Do you want to show the menu when there is no item selected? or the mouse is right clicked at some place where there is no item? In that case too, you should deselect the listview at mousedown and then show a different contextmenu at mouseup. because, ultimately, there is no item selected when the mouse is clicked elsewhere.
|
|
|
|
|
hey guys...i have a combobox in my form and i fill it with one column from my db...everything is fine till here...and i open another form if the user wants to add another data different than the db that are in combobox...and in new form user enter new data to a textbox and i write that data to a txt file...and i want to add that data to combobox also..that is why i wrote that function
public void DosyaOku(object sender, EventArgs e)
{
StreamReader read = new StreamReader("Firmalar.txt");
string kelime = read.ReadLine();
while (kelime != null)
{
cmbFirma.Items.Add(kelime);
kelime = read.ReadLine();
}
read.Close();
}
but whenever i try to call it..the program gives error and says combobox has a datasource already
so how can i add new data to combobox ??
thanks for your help
|
|
|
|
|
Try setting the datasource equal to null.
|
|
|
|
|
where should i make it null ????
|
|
|
|
|
Hello
I have the following piece of code (RSSToolkit). The RSS feed that I try to parse has multiple link elements and each such element has "rel" as attribute but with a different value. I'm interested in the link element with "absolute" as "rel" value.
[XmlElementAttribute("link")]
[XmlAttributeAttribute(AttributeName = "rel")]
public string LinkAlternate
{
get
{
return _link_alternate;
}
set
{
_link_alternate = value;
}
}
Thanks for your time
|
|
|
|
|
Hi. I'm having a problem with focusing from textbox to a button. Is there a way in doing this? I tried coding it but it doesn't work. Here's my code so far:
The textbox:
private void txtBoxLogIn_TextChanged(object sender, EventArgs e)
{
txtBoxLogIn.KeyPress += new KeyPressEventHandler(txtBoxLogIn_KeyPress);
}
and the eventhandler:
private void txtBoxLogIn_KeyPress(object sender, KeyPressEventArgs e)
{
if (e.KeyChar == (char)Keys.Enter)
{
txtBoxPassword.Focus();
}
else if (e.KeyChar == (char)Keys.Enter && txtBoxPassword.Focused == true)
{
btnProceedLogIn.Focus();
}
}
I'll be using only one eventhandler in transferring focus to other controls.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
