Click here to Skip to main content
15,892,059 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
C#
I have a topology running with parallelism as (1,8,1)(spout,logic bolt, write bolt) with number of ackers set as 12( 12 are available slots in my cluster). The max spout pending is 200 and timeout.secs is 200. I have to process 14 lac inputs.

My cluster consist of 1 nimbus & 3 supervisors ( dual core , 4 GB each) SO currently it takes 44 hours to give the output for 14 lac inputs.

My application is running separately as an application on another server. I did this to separate the application from the storm jar because I suspected that application was reserving some memory. I ran 50 instances of the application so it can process 50 tuples at a time But even this didnt help.

with this configuration, the topology processes around 6000/6500 tuples in 10 mins Messages are not failing anywhere but overall latency 16000 ms. If i try increasing the paralelism the rate of topology reduces(4000/ 3000 in 10 mins).

The behaviour is not constanst so kindly help me to do the math.


What I have tried:

I have mentioned everything in my question
Posted
Updated 27-Feb-16 8:20am

1 solution

Problem can be anywhere. You have to define/search where is the bottle neck.

Your network can be in downgraded mode because of bad wiring or bad switch.
Computers can be slowed down because of lack of memory.
Your programs can be artificially complicated or not optimized.

It can be anything and you are the only one that can do tests to see where is the problem.
Solving this problem is a specialized job because the reason can be pretty complicated.
 
Share this answer
 
Comments
rehabrish 28-Feb-16 1:55am    
the pings are proper in no time without any lag so i dont think there is problem in the network
could you suggest something eith respect to the configuration or storm parameters

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900