Faster data processing.

Question

1.00/5 (2 votes)

See more:

Hi,

currently i am working on telcom project where i have to process few hundered cdr files in a minitue( cdr is a text file which contains details\
related with a call in comma seperated format.each line represent each call.most of the file contains up to 8k lines).we have to process each line(formating,rating etc etc) and insert those values in a table. we were using
c# for for the front end and sql for back end processing.everything working fine for a few hundered cdr files per minitue..but as per the current requiremnt we have to process a few thousand files at a time.n our current process fails
terribly.
so is there any alternative for more faster data processing.i heard about big data(hadoop).but cdrs are perfectly structed data.which is not the ideal case for big data applications.please suggest any other alternatives..

What I have tried:

we were inserting each data on row by row basis.i tried to change it to bulk insert but it didnt work because the table to which we are inserting data is used by other services too(such as reporting service),so we cannot lock table for for than 100ms

Posted 16-May-16 20:47pm

sreeragns

Updated 11-May-17 13:52pm

Add a Solution

Comments

Mehdi Gholam 17-May-16 3:01am

First, measure what your bottle neck is by timing every step.

Patrice T 18-May-16 17:52pm

You will need to show code that process your data.

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

CPallini · Accepted Answer · 2016-05-16T21:21:00

In my opinion you have first to find out where your program is failing, that is where is the bottleneck. It could be, for instance, either in the CDR format parsing or in the data insertion inside the database (or both).
Once you located the bottleneck you could try to improve the performance, (e.g. optimizing the parsing algorithm).