Generating 1:1  ipfix from 10g pipeline - Getting the data -  Part 1

Goal : generate lossles ipfix flow's from distributed pipe to monitor application or network performance, identify bootlenecks and generate alerts if possible. 
 Why this way ? : It was expensive to do it with proprietary solutions. Plus we needed to have a flexible, open source option to work on. The closest solution cost was $1M 
 Challenges ; 
 
 We are going to do it using cpu, so lot's of flows require lots of processing power, 
 short flows create large ipfix messages than theirselves, any dns flow is is mostly 100 bytes long but a ipfix message for that flow costs us 1000 bytes 
 many packets traversing the kernel would create losses so we needed to bypass that 
 we need to create custom dashboards, or create alerts. 
 deduplication 
 
 Plan : 
 
 get mirrors from devices 
 produce ipfix from mirrors 
 send ipfix data to elastiflow 
 design custom dashboards 
 design monitor and alerters 
 
 What happened along the way ; 
 
 Obviously there will be lot's of storage requirement and cpu requirements how ever the current write rate wasn't going to be deadly so we decided to go with using only one server with many ssd drives. We lost the array drives because of a failure in array controller 
 We were going to aggregate mirrors from fabric using a Mellanox switch by utilizing it's bridge functions, the asic couldn't manage packet replication this step failed, we bought additional cards and skipped aggregation layer. 
 The intel x810-CAQxx cards failed to go in Zero Copy mode, we needed to wait for 2 months for a new firmware from intel 
 The tool that we used created problems with intel cards firmware and kernel module. We had to reflash them twice till we found a working state 
 The tool had a bug with combining mirrors so we waited for a bugfix. 
 
 Current status 
 We have a working setup, using
this version of intel module
 
 With this version of firmware 
 
 
 You can find some performance outputs as shown below 
 Server's status 
 
 Current traffic rate 
 
 Current io rate 
 
 Disk config 
 
 What's Next : 
 Well, we started to deep dive into traffic and analyze, create widgets and all the necessary stuff to have a Management dashboard. We saw some interesting stuff too which will need a lot of troubleshooting and investigation.