I have a topology running with parallelism as (1,8,1)(spout,logic bolt, write bolt) with number of ackers set as 12( 12 are available slots in my cluster). The max spout pending is 200 and timeout.secs is 200. I have to process 14 lac inputs. My cluster consist of 1 nimbus & 3 supervisors ( dual core , 4 GB each) SO currently it takes 44 hours to give the output for 14 lac inputs. My application is running separately as an application on another server. I did this to separate the application from the storm jar because I suspected that application was reserving some memory. I ran 50 instances of the application so it can process 50 tuples at a time But even this didnt help. with this configuration, the topology processes around 6000/6500 tuples in 10 mins Messages are not failing anywhere but overall latency 16000 ms. If i try increasing the paralelism the rate of topology reduces(4000/ 3000 in 10 mins). The behaviour is not constanst so kindly help me to do the math.
var
This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)