|
Hi,
I have csv file which contains 1 million records and with column 160. I have to insert(update the csv file) more 60 columns in the same file so the total column finally will generated 220.
60 columns and there data will get update like -
for first Row I have to select values from column 2,3,4 and generate the url and execute the url so that I will get xml data. Then I have to parse that data and fill 60 columns for that perticular row.
Like same I have to do for 2 row and 3row....upto 1 million. I have to process this file within 3 hrs.
How I can improve the performance of the reading and writing to csv file.
Regards,
sjs
modified 19-Dec-13 8:59am.
|
|
|
|
|
Sionce we know nothing of your code, it's prretty much impossible to tell you how to improve the performance.
|
|
|
|
|
Here, on CP, Sebastien Lorion's very popular 2011 article, and code, "A Fast CSV Reader," immediately comes to mind: [^]. I think you'll find a good strategy for optimizing access to your file in that article.
But, given: "I have to select values from column 2,3,4 and generate the url and execute the url so that I will get xml data:" does this mean you are writing XML into your CSV file ?
... edit ... you might also examine the open-source file library, FileHelpers: [^]. I have not used this library.
If you seek to aid everyone that suffers in the galaxy, you will only weaken yourself … and weaken them. It is the internal struggles, when fought and won on their own, that yield the strongest rewards… If you care for others, then dispense with pity and sacrifice and recognize the value in letting them fight their own battles." Darth Traya
modified 19-Dec-13 10:16am.
|
|
|
|
|
1 million records isn't so much. The real advantage is you can spawn multiple threads each with a different start point all writing to a different output file then have another app combine the outputs. But if the file was only a few GB I would just load the entire thing into ram in one quick blit and get it over with.
|
|
|
|
|
|
sjs4u wrote: I have to process this file within 3 hrs.
Sorry, but that's not how it works. You can't take an arbitrary proces and demand it's being done within a certain time-frame 
CSV-files aren't meant to be read or manipulated "fast"; use a database if speed is important.
sjs4u wrote: How I can improve the performance of the reading and writing to csv file.
Divide the workload over multiple PC's - then again, we don't know if that's even possible; depends on the structure of the file.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
|
|
|
|
|
I suspect the problem is not reading/writing the CSV but the manipulation of the records you have to do.
Try breaking the operation into blocks. Read the file, store into a database, process the records and update the database, write out the results. Then identify the slowest operation and work on that.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
sjs4u wrote: 1 million records and ...and execute the url so
That is your bottle neck.
If you need to make 1 million entirely different requests that is going to be a problem. If in fact the requests are duplicates then you can cache the results the first time and then use the cached data after that.
Other than that the url request can be put into a thread, and within reason, you can be waiting on a number of those at one time before proceeding.
|
|
|
|
|
Hi,
have this
public class RecipsMap : ClassMap<Data.Database.Recipe>
{
public RecipsMap()
{
this.Table("Recipe");
this.Id(x => x.Id).GeneratedBy.Native().Column("Id").UnsavedValue(0);
this.Map(x => x.Name).Not.Nullable().Length(50).Unique();
this.Map(x => x.Description).Nullable();
this.Map(x => x.Span);
this.HasMany<Data.Database.Temperature>(x => x.Temperatures).Table("Temperature").Inverse().Cascade.AllDeleteOrphan();
this.HasMany<Data.Database.LampControl>(x => x.LampControls).Table("LampControl").Inverse().Cascade.AllDeleteOrphan();
}
}
with one-to-many relationship.
And in my Repository class i define 2 methods for clear table. Full clear and remove one record by name.
public void Delete()
{
var metadata = this._sessionFactory.GetClassMetadata(typeof(T)) as NHibernate.Persister.Entity.AbstractEntityPersister;
string table = metadata.TableName;
using (var session = this._sessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
string deleteAll = string.Format("DELETE FROM \"{0}\"", table);
session.CreateQuery(deleteAll).ExecuteUpdate();
transaction.Commit();
}
}
this.OnOnDataChange(this.Get());
}
public void Delete(string name)
{
using (var session = this._sessionFactory.OpenSession())
{
using (var transaction = session.BeginTransaction())
{
T recipe = session.QueryOver<T>().Where(Restrictions.Eq("Name", name)).SingleOrDefault();
session.Delete(recipe);
transaction.Commit();
}
}
this.OnOnDataChange(this.Get());
}
But,trouble is that data are not deleting cascadely in first case "delete all", but everything is deleting while deleting separate object in second case. If selecting by name as here - all right, if delete all - only recipe table is removing and all other (Temperatures and LampControl tables in relationship) stay at there places. Who dealed with this plz give me advice - how to remove all data from table?
Thanks.
|
|
|
|
|
LIttle_Cat wrote: Who dealed with this plz give me advice - how to remove all data from table?
I have not dealt with Hibernate; but then again, if it doesn't support cascading delete's, you can always delete them yourself. Either by executing an extra delete-statement, or by defining a trigger on the database-tables.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
|
|
|
|
|
Thanks for Your attention.
A little bit I understand the proccess but don't know how to solve the problem. I have table with field Id(external key), other two tables references to tjis table. While delete NHibernate kills one child and meets external key which is also must be deleted but can't be deleted because it is one another child. So the question is: how to do it by NHibernate without isage of separate removing? And why this problem does not raising while removing separate strings in parent table, only with cascade removing? The same is going with SQLite database
|
|
|
|
|
Hi,
I have RDLC report with a viewer on the form. It's working great on my development laptop.
but when I deployed the application to another machine (using InstallShield) I am getting the following error.
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
Microsoft.Reporting.WinForms.LocalProcessingException: An error occurred during local report processing. ---> System.ApplicationException: The report definition for report 'C:\Documents and Settings\USER\Desktop\invoice.rdlc' has not been specified ---> System.IO.FileNotFoundException: Could not find file 'C:\Documents and Settings\USER\Desktop\invoice.rdlc'.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at Microsoft.ReportingServices.StandalonePreviewStore.GetReportDefinition(ReportID reportId)
at Microsoft.Reporting.PreviewStore.GetCompiledReport(CatalogItemContextBase context, Boolean rebuild, Byte[]& reportDefinition, ControlSnapshot& snapshot)
--- End of inner exception stack trace ---
at Microsoft.Reporting.PreviewStore.GetCompiledReport(CatalogItemContextBase context, Boolean rebuild, Byte[]& reportDefinition, ControlSnapshot& snapshot)
at Microsoft.Reporting.LocalService.GetCompiledReport(CatalogItemContextBase itemContext, Boolean rebuild, ControlSnapshot& snapshot)
at Microsoft.Reporting.WinForms.LocalReport.EnsureExecutionSession()
--- End of inner exception stack trace ---
at Microsoft.Reporting.WinForms.LocalReport.EnsureExecutionSession()
at Microsoft.Reporting.WinForms.LocalReport.SetParameters(IEnumerable`1 parameters)
at cleaning_company_software.frmMain.print_invoice2(Int32 invoice_id) in C:\Users\Jassim\Documents\Visual Studio 2010\Projects\WipePro\WipePro\main_form.cs:line 840
at cleaning_company_software.frmMain.barButtonItem7_ItemClick(Object sender, ItemClickEventArgs e) in C:\Users\Jassim\Documents\Visual Studio 2010\Projects\WipePro\WipePro\main_form.cs:line 2596
at DevExpress.XtraBars.BarItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarBaseButtonItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarItemLink.OnLinkClick()
at DevExpress.XtraBars.BarItemLink.OnLinkAction(BarLinkAction action, Object actionArgs)
at DevExpress.XtraBars.BarButtonItemLink.OnLinkAction(BarLinkAction action, Object actionArgs)
at DevExpress.XtraBars.BarItemLink.OnLinkActionCore(BarLinkAction action, Object actionArgs)
at DevExpress.XtraBars.ViewInfo.BarSelectionInfo.ClickLink(BarItemLink link)
at DevExpress.XtraBars.ViewInfo.BarSelectionInfo.UnPressLink(BarItemLink link)
at DevExpress.XtraBars.Controls.CustomLinksControl.OnMouseUp(MouseEventArgs e)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/WINDOWS/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
WipePro
Assembly Version: 1.0.0.0
Win32 Version: 1.00
CodeBase: file:///C:/Program%20Files/WipePro/WipePro.exe
----------------------------------------
DevExpress.Utils.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.Utils.v11.1.DLL
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
DevExpress.Data.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.Data.v11.1.DLL
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
DevExpress.XtraEditors.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.XtraEditors.v11.1.DLL
----------------------------------------
DevExpress.XtraGrid.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.XtraGrid.v11.1.DLL
----------------------------------------
DevExpress.XtraNavBar.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.XtraNavBar.v11.1.DLL
----------------------------------------
DevExpress.XtraBars.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.XtraBars.v11.1.DLL
----------------------------------------
DevExpress.Printing.v11.1.Core
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.Printing.v11.1.Core.DLL
----------------------------------------
Microsoft.ReportViewer.WinForms
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1
CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/Microsoft.ReportViewer.WinForms/10.0.0.0__b03f5f7f11d50a3a/Microsoft.ReportViewer.WinForms.dll
----------------------------------------
DevExpress.XtraPrinting.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.XtraPrinting.v11.1.DLL
----------------------------------------
System.Data
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/System.Data/v4.0_4.0.0.0__b77a5c561934e089/System.Data.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.ReportViewer.Common
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1
CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/Microsoft.ReportViewer.Common/10.0.0.0__b03f5f7f11d50a3a/Microsoft.ReportViewer.Common.dll
----------------------------------------
DevExpress.XtraTreeList.v11.1
Assembly Version: 11.1.4.0
Win32 Version: 11.1.4.0
CodeBase: file:///C:/Program%20Files/WipePro/DevExpress.XtraTreeList.v11.1.DLL
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
MySql.Data
Assembly Version: 6.6.4.0
Win32 Version: 6.6.4.0
CodeBase: file:///C:/Program%20Files/WipePro/MySql.Data.DLL
----------------------------------------
System.Transactions
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/System.Transactions/v4.0_4.0.0.0__b77a5c561934e089/System.Transactions.dll
----------------------------------------
System.EnterpriseServices
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/System.EnterpriseServices/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.EnterpriseServices.dll
----------------------------------------
System.Numerics
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Numerics/v4.0_4.0.0.0__b77a5c561934e089/System.Numerics.dll
----------------------------------------
PasswordHash.NET
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Program%20Files/WipePro/PasswordHash.NET.DLL
----------------------------------------
************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.
For example:
<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>
When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
Technology News @ www.JassimRahma.com
|
|
|
|
|
Did you even read the exception message? Is telling you exactly what the problem is. This should give you a large clue as to what you should be looking at in your code.
|
|
|
|
|
How to draw a half pixel line in c#
Please help me
Thanks in Advance
|
|
|
|
|
While mathematically, and abstractly, you could model some fraction of a pixel, when it comes to "reality," a pixel is a fundamental atomic unit of a hardware display, or print/render, device which may be dependent (as in screen resolution/bit-depth) on software settings.
So, I am wondering what your goal is here. Are you wanting to model in floating point before you, ultimately, generate ints to affect some display or printed output ?
Please clarify.
If you seek to aid everyone that suffers in the galaxy, you will only weaken yourself … and weaken them. It is the internal struggles, when fought and won on their own, that yield the strongest rewards… If you care for others, then dispense with pity and sacrifice and recognize the value in letting them fight their own battles." Darth Traya
|
|
|
|
|
I thought you knew that the Pixel is made of different numbers of the three types of Colorons and an Alpha particle?
Clearly, half a pixel would be no Alpha particles (which is good from a Health and Safety point of view), no Redulons, 127 Greons and 255 Bluons.
It's also not visible as a result...Dark Pictures! 
|
|
|
|
|
The speed of your light is a constant
If you seek to aid everyone that suffers in the galaxy, you will only weaken yourself … and weaken them. It is the internal struggles, when fought and won on their own, that yield the strongest rewards… If you care for others, then dispense with pity and sacrifice and recognize the value in letting them fight their own battles." Darth Traya
|
|
|
|
|
I take it what you are wanting to do is reduce an image size by 1/2?
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
|
|
|
|
|
I want to draw half pixel size line
|
|
|
|
|
The smallest line you can draw is one pixel wide; there is nothing smaller.
Veni, vidi, abiit domum
|
|
|
|
|
The concept of half a pixel is like the concept of half an atom - the idea itself does not make sense.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
|
|
|
|
|
I want to deserialize a dictionary with datacontractserializer, the xml schema is the following :
<General xmlns="http://schemas.datacontract.org/2004/07/MyProject" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<Mydictionary xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<a:KeyValueOfunsignedByteMyclass>
<a:Key>1</a:Key>
<a:Value>
....
</a:Value>
</a:KeyValueOfunsignedByteMyclass>
<a:KeyValueOfunsignedByteMyclass>
<a:Key>2</a:Key>
<a:Value>
....
</a:Value>
</a:KeyValueOfunsignedByteMyclass>
</Mydictionary>
But on deserializing, I get this error :
"An item with the same key has already been added."
(referencing KeyValueOfunsignedByteMyclass)
Can someone help me to resolve this problem please?
Thanks
|
|
|
|
|
I think you could get more helpful responses here if you posted the same code you posted on the same question on StackOverFlow: [^] ... where you show your code for the 'xml_todata method which does the deserilization.
And, if you do that post that code, which doesn't compile as shown, people here are going to ask you the same questions the SO folks asked you, so: please post code that will compile and in which you show us the declaration of the variable 'deserializ.
It also might be very useful if you show how you applied the DataContractAttribute and DataMemberAttribute Attributes to the Dictionary you serialized.
If you seek to aid everyone that suffers in the galaxy, you will only weaken yourself … and weaken them. It is the internal struggles, when fought and won on their own, that yield the strongest rewards… If you care for others, then dispense with pity and sacrifice and recognize the value in letting them fight their own battles." Darth Traya
|
|
|
|
|
I am using AviFile api to make a screen capture video. I want to capture it as small videos of 2 minutes length instead of creating one single video at a stretch. When the video starts getting recorded, a pop-up box to chose the compression option is displayed by default when the avisaveoptions function is called. This creates a problem since I have to choose the option every 2 mins. I would like to avoid this by setting the default value to 'Microsoft Video 1'. How can I do this?
thanks in advance
|
|
|
|
|
Hello
I use this code
AviManager aviManager =new AviManager(@"new.avi", false);
//add a new video stream and one frame to the new file
VideoStream aviStream =aviManager.AddVideoStream(true, 2, bitmap);//(dlg,Rate, bmp);
how I can choice compression method without show dialog, I want microsoft video 1 compression method.
thanks.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
