A Data Processing Design Pattern for Intermittent Input Data

For processing continuous data input, RAM and CPU utilization has to be optimized. If there are multiple threads collecting and submitting data for processing, then you have two options from there. One is to create equal amount of input threads for processing data or store the input data in memory and process it one by one. Creating large number of threads chokes up the CPU and holding everything in memory exhausts the RAM. We need a balanced solution.