Unless you buy a solution to do this, expect this project to be massive. I have done something similar and it took a lot of man hours to get exactly what we wanted out of it.
I suggest looking at systems that already exist, like:
http://www.endace.com/[
^]
Submit an article or tip