Network

Delay and loss tests using netem on a healthy network

Purpose of this document

We needed a lab environment that can provide a "break on purpose network" to developers to test their code on it. You can add;

to a connection on the fly. To do it we need "netem" modules help.

NETEM

Since kernel 2.6 netem is included in iproute2 package, basically all modern distro's has it now. You can find a documentation on Netem

i am going to explain the required steps here ;

Extending l2 networks using wireguard & gretap

At some cases, you might need to extend a vlan over l3 links, and for some reason while;

Then here is your solution, with a simple dnat entry you can extend your vlan.

In my case, customer was in need to backup their vm's using VMWare's replication to their DR center and whenever they needed, they do like to use the DR as Active data center. And changing configs on a disaster situation was the least they want to deal with.

Of course this is only a half of a complete DR solution may less, but restoring a services directly from DR was the requirement as i've been told.

Requirements:

Some caveats ;

install wireguard

To install wireguard vpn on ubuntu 18.04

#sudo apt update
#sudo apt-get install libmnl-dev libelf-dev linux-headers-$(uname -r) build-essential pkg-config
#sudo apt install wireguard 

After installation you'll need to restart operating system and check for installed module, output should be as shown below.

#lsmod| grep wireguard
udp_tunnel	16384 1 wireguard
ip6_udp_tunnel 16384 1 wireguard

generete keys

Wireguard need public and private keys to operate, there is a tool called wg which can generate them as shown below

#wg genkey | sudo tee /etc/wireguard/privatekey | wg pubkey | sudo tee /etc/wireguard/publickey

You can find the keys at /etc/wireguard

create wireguard0.conf

You'll need to create a file called wireguard0.conf at /etc/wireguard

Contents of the file should be like this ;

For initiator:

[interface]
private_key= the_private_key_you_generated
address= 10.10.10.1/24 #ip address of the wireguard0 

[peer]
PublicKey = #he public key that you generated on the responder host
AllowedIPs = 10.10.10.0/24 #and the other networks that you'd like route through
EndPoint = internet_address:port 
PersistenKeepAlive = 15 #seconds

For responder:

[interface]
private_key= the_private_key_you_generated
address= 10.10.10.2/24 #ip address of the wireguard0 
listenPort = 4900

[peer]
PublicKey = #he public key that you generated on the initiator host
AllowedIPs = 10.10.10.0/24 #nd the other networks that you'd like route through
PersistenKeepAlive = 15 #seconds

initiate tunnel

to start the tunnel

wg-quick up wireguard0

to check the status

wg show wireguard0
interface: wireguard0
  public key: +/1R3JqLKlszbaGUSBtckoxNOMuSvLYKUCl03ShoFw8=
  private key: (hidden)
  listening port: 4900

peer: 2f/RmbuvKtR/L2ZFlQBHsVGkTXkA6d1pJO1ay5EjwSQ=
  endpoint: 172.21.23.111:49792
  allowed ips: 10.10.10.0/24, 192.168.5.0/24
  latest handshake: 1 minute, 31 seconds ago
  transfer: 19.10 MiB received, 11.95 MiB sent

install bridge-utils

To install bridge utils

#apt install bridge-utils

enable br_netfilter

to enable

#modprobe br_netfilter

to keep loading on boot

#sudo sh -c 'echo "br_netfilter" > /etc/modules-load.d/br_netfilter.conf'
#cat net.bridge.bridge-nf-call-ip6tables = 1 >> /etc/sysctl.d/bridge.conf

enable routing

To enable routing on the fly

sysctl -w net.ipv4.ip_forward=1

to make it permanent, add lines below to /etc/sysctl.conf ;

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

fix mss on exit interface

When you are going to use ip tunnel for underlay and gretap for overlay, there will be some serious mss/mtu size problems to fix that, we need the br_netfilter module that we installed before and a special chain to limit the mss size to the max mtu of interface

#iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

We specially use -I to put it at the top of the forward chain.

configure gretap interface

For the initiator:

For responder

configure bridge

Now you will add the interfaces to bridge

#brctl addbr br0

add interfaces to bridge br0

#brctl addif br0 ens4
#brctl addif br0 gretap0

Bring everything up

#wp-quick up wireguard0
#ip link set up dev br0
#ip link set up dev gretap0
#ip link et up dev ens4

tests

do pings i've checked that dhcp is working,

what needs to be done are ;

Generating 1:1 ipfix from 10g pipeline - Getting the data - Part 1

Goal : generate lossles ipfix flow's from distributed pipe to monitor application or network performance, identify bootlenecks and generate alerts if possible.

Why this way ? : It was expensive to do it with proprietary solutions. Plus we needed to have a flexible, open source option to work on. The closest solution cost was $1M

Challenges ;

Plan :

What happened along the way;

Current status

We have a working setup, using this version of intel module

With this version of firmware


You can find some performance outputs as shown below

Server's status

Current traffic rate

Current io rate

Disk config

What's Next:

Well, we started to deep dive into traffic and analyze, create widgets and all the necessary stuff to have a Management dashboard. We saw some interesting stuff too which will need a lot of troubleshooting and investigation.