|
Howdy!!
I've been working on a VB.net program to:
1. Rip data from a **Huge** Text file.
2. Encode that data to an XML URL
3. Pull the data from the XML from the website
4. Enter that data into an Access Database
Access 2002 to create reports from data.
Is there anyone out there that would be willing to help me optomize my program and database for performance and help me with error handling???
Right now it takes over a half hour to itterate through the .txt file (roughly 10,000 lines) and do its thing. My 'error handling' is me clicking OK on tons of errors... lol
Email me @ geoff at azpmc dot com if you'd like to help out a r00kie!
   ![Rose | [Rose]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/rose.gif)
|
|
|
|
|
Oh yeah, this post was mine^^^^^^^
I just got arond to creating a log-in.
|
|
|
|
|
First question - why are you using XML ? Because it's coming through a web service ? If the files are huge, then wrapping them in XML tags won't do anything good. Secondly, how are you parsing the XML document ? You should use an XMLReader to provide a forward only reader, instead of XMLDocument, which will give you the DOM, at huge cost on a document of this size.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
The original text file is just a bunch of addresses. That part never sees XML.
Part one of the program takes the street number, st. name, city, state, zip and puts into a URL, calls the url as the source of the XML document. It then uses XMLtextReader to do this:
(the XML file is pretty small.... really small...why does this program chug so bad????)
Private Sub readXML(ByVal newURL As String, ByVal stnumber As Integer)
Dim anewURL As String = newURL
Dim reader As XmlTextReader = New XmlTextReader(anewURL)
Dim output As String
Dim name As String
Dim contents As String
Dim fName As String
Dim lName As String
Dim stName As String
Dim phoneNumber As String
Dim phone2 As String
Dim phone3 As String
Dim phone4 As String
Dim city As String
Dim state As String
Dim zip As Integer
Dim yesNo As String
Dim countYesNo As String
Dim contentInteger As Integer
reader.WhitespaceHandling = WhitespaceHandling.None
While reader.Read()
reader.MoveToContent()
If reader.NodeType = Xml.XmlNodeType.Element Then
name = reader.Name
contents = reader.ReadString()
reader.MoveToElement()
reader.MoveToFirstAttribute()
If name = "NAME" Then
reader.MoveToFirstAttribute()
contents = reader.ReadString()
fName = contents
End If
If name = "ADDRESS" Then
reader.MoveToFirstAttribute()
contents = reader.ReadString()
stnumber = contents
End If
Select Case name
Case "COUNT"
countYesNo = contents
Case "STATUS"
yesNo = contents
Case "LAST"
lName = contents
Case "STREET"
stName = contents
Case "CITY"
city = contents
Case "STATE"
state = contents
Case "ZIP"
zip = contents
Case "PHONE"
phoneNumber = contents
End Select
End If
End While
If yesNo = "OK" Then
addPhoneNumber(fName, lName, stnumber, stName, phoneNumber, city, state, zip)
End If
reader.Close()
End Sub
-- modified at 0:52 Friday 21st October, 2005
|
|
|
|
|
Does anyone out there think that there's a way to get this function to run faster?
|
|
|
|
|
vb_n00b wrote: I've been working on a VB.net program to:
1. Rip data from a **Huge** Text file.
2. Encode that data to an XML URL
3. Pull the data from the XML from the website
4. Enter that data into an Access Database
What format is the text file in? If it’s a comma separated file (or you can change the file format to a CSV) then adding this data to the access db is easy.
I have always wanted to try this, but never had a need for it.
Create a record set based on the CSV file, close the connection open a new connection to the access db. Attach the new connection to the record set and update it to the access db. Done, no extra data parsing needed, again never tried anything like this. If it works let me know…
Hope that helps.
DEBUGGING : Removing the needles from the haystack.
|
|
|
|
|
The text file contains a giant list of addresses and a bunch of other usless junk.
They look like this:
10/21/2005 9:12:34AM2331 WEST RIVER RDSearch Report
West Dennis, MA 02661
10/21/2005 9:12:34AM2331 WEST RIVER RDSearch Report
West Dennis, MA 02661
10/21/2005 9:12:34AM2331 WEST RIVER RDSearch Report
West Dennis, MA 02661
10/21/2005 9:12:34AM2331 WEST RIVER RDSearch Report
West Dennis, MA 02661
etc...
I have to parse the data from that to a URL something like this:
www.lookuptheaddress.com&stnum=2331&stName=West+river+rd&city=west+dennis...
That URL produces an XML page that I need to pull Name and Phone Number from and then slap into a database for later use...
i think what is bogging it down is the XML data parsing... Im going to re-work it using the XMLDocument methods and see if that helps any...
Thanks for the help and I welcome ANY input on this porject!!
_thanks again!
|
|
|
|
|
Desert_Dude wrote: That URL produces an XML page that I need to pull Name and Phone Number from and then slap into a database for later use...
Why can’t you add the information to the database as your creating the XML document (that’s provided the XML Document is needed for something else)? Just fire off a simple insert into the database, you have all of the data already right there.
Desert_Dude wrote: i think what is bogging it down is the XML data parsing... Im going to re-work it using the XMLDocument methods and see if that helps any...
Test it; see how long each piece of the pie takes. Start with cleaning up the flat file, then creating the XML document and finally adding the data.
DEBUGGING : Removing the needles from the haystack.
|
|
|
|
|
sfdougl wrote: Desert_Dude wrote:
That URL produces an XML page that I need to pull Name and Phone Number from and then slap into a database for later use...
Why can’t you add the information to the database as your creating the XML document (that’s provided the XML Document is needed for something else)? Just fire off a simple insert into the database, you have all of the data already right there.
The Name and Number come from the XML... it's a lookup service that im using on the internet.
sfdougl wrote: Desert_Dude wrote:
i think what is bogging it down is the XML data parsing... Im going to re-work it using the XMLDocument methods and see if that helps any...
Test it; see how long each piece of the pie takes. Start with cleaning up the flat file, then creating the XML document and finally adding the data.
What would be a good method for testing and timing operations besides guestimating?
Thanks!
|
|
|
|
|
Desert_Dude wrote: What would be a good method for testing and timing operations besides guestimating?
Well the simplest way is to output to the debugger, say output the start time of a sub and the end time when it finishes. I don’t know how you have your code laid out.
If you’re doing an internet look up while parsing the data that maybe the cause of the speed issue. Remember what your doing here pushing a request, waiting for a response then getting the response.
DEBUGGING : Removing the needles from the haystack.
|
|
|
|
|
Yeah,
My suspicion was that the XML lookup was causing the delay. I'll debug using a more seat-of-the-pants kind of way cuz Im such a n00b @ VB. It may make my brain fry to have to learn another thing like using the debugger... :/
When I get this thing up and running smoothly, I'll post up my code for anyone to look at and maybe use...
I'm pretty proud of myself for getting it to work at all.
I'll keep ya posted.
|
|
|
|
|
Desert_Dude wrote: My suspicion was that the XML lookup was causing the delay
Most likely the cause.
Desert_Dude wrote: debug using a more seat-of-the-pants kind of way cuz Im such a n00b @ VB. It may make my brain fry to have to learn another thing like using the debugger... :/
debug.print <string text=""> is your freind.
Desert_Dude wrote: I'm pretty proud of myself for getting it to work at all
Okay.
DEBUGGING : Removing the needles from the haystack.
|
|
|
|
|
How can I move a dataset to a specific record?
I have a textbox that you can type in a records "ID" and when they hit F2 I need the dataset (or even just the primary datatable) to move to that record.
Thanks
|
|
|
|
|
Do you mean the datatable ( which I believe you can search with SQL to get the row you want ), or the grid that displays the data ?
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
I have a tabbed form and all the fields from the SQL database are displayed in various textboxes,comboboxes,checkboxes,etc... (no datagrids). I have a datatable,dataadapter,currency manager,and datview for each of the 6 tables in sql. The "lead table" is the one that if they type in the ID number then I need to move to the record that matches or closest match. I can do it using the filter method, but that will make the current view have only 1 record and I can not move next or previous. Thanks
|
|
|
|
|
please help
how to develop application for multiple user
in vb.net
eg: adding records deleting & updating to table.
help me by providing source code
my mail id is u_rock100@rediffmail.com
thank in advanced to all vb.net experts.
|
|
|
|
|
You want to have multiple users on a database based app ? The obvious way is to store usernames and passwords in your database, and give each user an ID and then use that ID through the database to record what belongs to whom.
If you want users to be able to do different things, then you need to add some code for permissions as well. It's best not to use multiple logins on the database, as it causes .NET connection pooling to not work as well as it otherwise would.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
I have two things (maybe one)that I am trying to work on, and the second one is escaping me HELP!!
Here are my questions:
1. Write a standard windows application that minimizes to the system tray area. Double-click to bring it back, right click for menu. E.g outlook 2003, the volume control...
2. same application or seperate. Give me the same information and similar layout as you get when you right click My Computer and select Properties. Only the General Tab!!! Don't worry about anything else.
Kyanide
|
|
|
|
|
Looks like homework to me. Review your course notes, and then let us know what you come up with. If you get stuck on something specific, we'll be glad to help out.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
|
how to use datagrid as a data entry and display form in visual basic.net windows forms? please help me
|
|
|
|
|
That's basically what the datagrid is for, so google 'datagrid', or even search this site, for a ton of info.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
can someone help me to design a control to call up the current rate of convertion between any two world currencies.this should also be able to display as a graph,the history of the flactuations of the selected rate. i need the procedure(codes)
thank you for helping me
|
|
|
|
|
rexford nyarko wrote: i need the procedure(codes)
I don't see anyone writing the code for you.
Basically, I assume you're going to use this control online ? If not, where will the info come from ? Getting the data is the only sticking point, the rest is pretty trivial, any tutorial on user controls will help with that.
Christian Graus - Microsoft MVP - C++
|
|
|
|
|
I used Visual studio .NET 2003 to create the sql database, and now I need to know how to import it into frontpage 2003 so I can use it with my webpage that I have created.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
