lvs-dpdk

This project has ported LVS FULLNAT/DR/NAT and SNAT-GATEWAY to OpenFastPath(base on odp-dpdk).

NAT is only available on single core while FULLNAT,DR and SNAT-GATEWAY support multi-cores, because in lvs-dpdk each core has one local session table and depend on flow director.

LVS-FULLNAT origin source code is at https://github.com/alibaba/LVS

LVS-SNAT gateway origin source code is at https://github.com/jlijian3/lvs-snat

I had forked OpenFastPath project and added support for flow director and ofp_vs example, see https://github.com/lvsgate/ofp/tree/master/example/ofp_vs

I had forked odp-dpdk project and added support for flow director, see https://github.com/lvsgate/odp-dpdk.git

Please note that this project is experimental, it only had limited testing and never runned in production environment.

Prerequisites

Intel x86 CPU
NIC which support flow director, if you want to run on multi-cores
lvs-dpdk has been compiled and tested on Centos 7.2 with 3.10 kernel
5G memory at least

Performance

CPU E5-2682 v4 @ 2.50GHz
NIC intel 82599 10G
fullnat 800w pps if use 5 cores (in 400w pps and out 400w pps)

Build all in one step

git clone https://github.com/lvsgate/lvs-dpdk.git
cd lvs-dpdk
./build_all.sh
cd ofp_vs

If build all successfully Goto step 5, otherwise build one by one.

Build steps one by one

1. Fetch and compile DPDK

git clone git://dpdk.org/dpdk ./<dpdk-dir>
cd <dpdk-dir>
git checkout -b 17.02 tags/v17.02
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
cd <dpdk-dir>/x86_64-native-linuxapp-gcc
sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_PCAP=).*,\1y,' .config
cd ..
make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS="-fPIC"

#Add env value
export RTE_SDK=<dpdk-dir>
	export RTE_TARGET=x86_64-native-linuxapp-gcc

2. Fetch and compile odp-dpdk

git clone  https://github.com/lvsgate/odp-dpdk.git <odp-dir>
cd <odp-dir>
./bootstrap
#SHARED library
./configure --enable-shared --with-platform=linux-dpdk --enable-helper-linux --with-sdk-install-path=<dpdk-dir>/x86_64-native-linuxapp-gcc --prefix=<INSTALL ODP-DPDK TO THIS DIR>
#If build only static library, dpdk driver register function defined with __attribute__(construction) won't be called.

make
make install

3. Fetch and compile ofp

yum install libnl3 libnl3-cli libnl3-devel #or download rpm from https://www.infradead.org/~tgr/libnl/
git clone https://github.com/lvsgate/ofp.git <ofp-dir>
cd <ofp-dir>
./bootstrap
./configure --disable-libconfig --with-odp-lib=odp-dpdk --with-odp=<ODP-DPDK INSTALLATION DIR> --enable-shared=no --enable-sp=yes --disable-mtrie CPPFLAGS=-I<ODP-DPDK INSTALLATION DIR>/include/odp/arch/x86_64-linux/
make

4. Fetch and compiled lvs-dpdk tools

git clone https://github.com/lvsgate/lvs-dpdk.git
cd lvs-dpdk/tools/keepalived
sh configure --prefix=/usr --sysconfdir=/etc/ CPPFLAGS=-I/usr/include/libnl3/ LDFLAGS=-L/usr/lib64/
make
make install
cd lvs-dpdk/tools/ipvsadm
make
make install

5. Prepare DPDK for running lvs-dpdk

echo 8192 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 8192 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
mkdir /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
modprobe uio
insmod <dpdk-dir>/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
cd <dpdk-dir>
./usertools/dpdk-devbind.py --status
./usertools/dpdk-devbind.py --bind=igb_uio <pci-id-1>
./usertools/dpdk-devbind.py --bind=igb_uio <pci-id-2>

6. Run ofp_vs

modprobe ip_vs  #add this line to /etc/rc.local, because ipvsadm and keepalived depend on it.
cd <ofp-dir>/examples/ofp_vs
./ofp_vs -i 0,1 -c 0x00f0 -p 1 -f ofp.conf # -i <port1>,<port2>  
                                 # -c <worker core mask> 
                                 # -o <required for snat-gw, outer port to wan, snat-gw fdir rule will be add to this port>
                                 # -p <required for fullnat, inner port to lan, fullnat fdir rule  will be add to this port>
                                 # -f <config file include default command which you can change in ofp cli>
#If worker core count > 0, -o is required for snat-gw, -p is required for fullnat.

7. Connect to ofp cli or edit ofp.conf to configure network

telnet localhost 2345
type in "?" or "help"
>>> ?
# fp0 equal to port number 0 in dpdk
>>> ifconfig fp0 <ip_addr>/<net_mask> 
>>> ifconfig fp1 <ip_addr>/<net_mask> 
#default gw don't work if enable mtries routing, may be ofp's bug.
>>> route add 0.0.0.0/0 gw <next hop> dev fp0
>>> route add <ip_addr>/<net_mask> gw <next hop> dev fp1
#Add vs ip to lo if you want to ping it
>>> ifconfig lo0 <vip1/32>
>>> ifconfig lo1 <vip2/32>

8. Configure network on your switch/router

#The ip on the port is used as nexthop of vs/snat ip or local address.
#H3C switch example.
#add route for virutal server address or SNAT source address
>>> ip route-static <vs.ip.x.0> 255.255.255.0 <nexthop ip on the interface fp0> 
#add route for local address of FULLNAT
>>> ip route-static <local.addr.x.0> 255.255.255.0 <nexthop ip on the interface fp1>

9. Start and configure SNAT-GATEWAY

./ofp_vs -i 0,1 -c 0x00f0 -o 0 -f ofp.conf  #-o specify the port to wan
telnet localhost 2345
>>> snat enable
>>> snat add from 10.1.0.0/16 to 0.0.0.0/0 out_dev fp0 source 192.168.50.253 - 192.168.50.253 algo sd
>>> snat add from 10.1.0.10/32 to 0.0.0.0/0 out_dev fp0 source 192.168.50.100 - 192.168.50.103 algo sdfn
>>> snat del from 10.1.0.10/32 to 0.0.0.0/0 out_dev fp0
>>> snat show

10. Start and configure FULLNAT virtual server

./ofp_vs -i 0,1 -c 0x00f0 -p 1 -f ofp.conf  #-p specify the port to lan

#ipvsadm and keepalived will comunicate with ofp_vs process but not the kernel module.
#Create FULLNAT virtual server, local address count must be greater than worker count
ipvsadm  -A  -t <vip:vport> -s rr
ipvsadm  -a  -t <vip:vport> -r <rsip1:rsport> -b
ipvsadm  -a  -t <vip:vport> -r <rsip2:rsport> -b
ipvsadm  -P  -t <vip:vport> -z <local_addr1>
ipvsadm  -P  -t <vip:vport> -z <local_addr2>
ipvsadm  -P  -t <vip:vport> -z <local_addr3>
ipvsadm  -P  -t <vip:vport> -z <local_addr4>
...
...
ipvsadm  -P  -t <vip:vport> -z <local_addr254>
ipvsadm -ln
ipvadm -G

11. Try to visit vs now

curl <vip:vport>

12. More details about ofp and odp-dpdk

http://www.openfastpath.org/
http://opendataplane.org/
https://github.com/OpenFastPath/ofp
https://github.com/lvsgate/ofp
https://github.com/lvsgate/odp-dpdk

lvsgate/lvs-dpdk

lvsgate

Reviews

Repository Details