Network performance testing tools
by Wenwei Weng
As my day job is to develop routing platforms to route packets, from time to time I need to measure the performance of router in terms of throughput, i.e. pump traffic into UUT ingress interface, and get traffic out from egress interface. In my view, there are three classes of testing tools:
- Dedicated testing equipment like IXIA, Spirent. These are expensive and accurate equipments. It requires dedicated hardware, training.
- Open source tools like Pktgen-DPDK, [TRex](https://github.com/cisco-system-traffic-generator): These tools are wokring with NIC directly, and can create high rate relative precise traffiic.
- Open source tools like iperf, netperf, nuttcp. These are free tools, easy to install, easy to use. It generates traffic using socket. So timing wise, it is much less accurate. However it still gives roughly the performance number.
In this post, I will go throught netperf, iperf, nuttcp.
Test bed set up
I uses three Cisco UCS boxes below.
some details:
- The middle box "iotg-epc-ucs3" runs Fedora 24, and is used as UUT.
- The box iotg-epc-ucs2 runs ubuntu 16.04LTS, is used as server
- The box weng-ucs1 runs ubuntu 16.04LTS, is used as client.
[weng@iotg-epc-ucs3 ~]$ ifconfig enp11s0f0
enp11s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.10.2 netmask 255.255.255.0 broadcast 10.10.10.255
inet6 fe80::c204:be95:d906:ba13 prefixlen 64 scopeid 0x20<link>
ether 00:10:18:a7:82:28 txqueuelen 1000 (Ethernet)
RX packets 2189444 bytes 153377208 (146.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23132269 bytes 22045460531 (20.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[weng@iotg-epc-ucs3 ~]$ ifconfig enp11s0f1
enp11s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 20.20.20.2 netmask 255.255.255.0 broadcast 20.20.20.255
inet6 fe80::ee39:1018:b17e:3f76 prefixlen 64 scopeid 0x20<link>
ether 00:10:18:a7:82:2a txqueuelen 1000 (Ethernet)
RX packets 23176127 bytes 22049032366 (20.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2200466 bytes 154280229 (147.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[weng@iotg-epc-ucs3 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 enp12s0f0
default gateway 0.0.0.0 UG 101 0 0 enp12s0f1
default gateway 0.0.0.0 UG 102 0 0 ens1f0
10.10.10.0 0.0.0.0 255.255.255.0 U 100 0 0 enp11s0f0
20.20.20.0 0.0.0.0 255.255.255.0 U 100 0 0 enp11s0f1
30.30.30.0 0.0.0.0 255.255.255.0 U 100 0 0 enp12s0f0
40.40.40.0 0.0.0.0 255.255.255.0 U 100 0 0 enp12s0f1
172.0.0.0 gateway 255.0.0.0 UG 1 0 0 ens1f0
172.27.161.0 0.0.0.0 255.255.255.128 U 100 0 0 ens1f0
[weng@iotg-epc-ucs3 ~]$ cat /proc/sys/net/ipv4/ip_forward # make sure IP forwad is turned on
1
[weng@iotg-epc-ucs3 ~]$ # make sure firewall is disable as it might block the testing traffic
[weng@iotg-epc-ucs3 ~]$ systemctl status firewalld
\u25cf firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Thu 2016-09-08 11:15:25 PDT; 1 day 10h ago
Process: 1306 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, s
Main PID: 1306 (code=exited, status=0/SUCCESS)
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavaila
[weng@iotg-epc-ucs3 ~]$ epc@iotg-epc-ucs2:~$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:10:18:c9:54:d8
inet addr:10.10.10.1 Bcast:10.10.10.255 Mask:255.255.255.0
inet6 addr: fe80::210:18ff:fec9:54d8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:23133246 errors:1 dropped:0 overruns:0 frame:1
TX packets:2191846 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22045747179 (22.0 GB) TX bytes:153829920 (153.8 MB)
epc@iotg-epc-ucs2:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.10.10.2 0.0.0.0 UG 0 0 0 eth0
10.10.10.0 * 255.255.255.0 U 1 0 0 eth0
20.20.20.0 10.10.10.2 255.255.255.0 UG 0 0 0 eth0
30.30.30.0 * 255.255.255.0 U 1 0 0 eth1
40.40.40.0 30.30.30.2 255.255.255.0 UG 0 0 0 eth1
171.0.0.0 * 255.0.0.0 U 0 0 0 eth4
172.0.0.0 172.27.161.1 255.0.0.0 UG 0 0 0 eth4
172.27.161.0 * 255.255.255.128 U 1 0 0 eth4
173.0.0.0 * 255.0.0.0 U 0 0 0 eth4
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
epc@iotg-epc-ucs2:~$ which netperf
/usr/local/bin/netperf
epc@iotg-epc-ucs2:~$ which iperf3
/usr/bin/iperf3
epc@iotg-epc-ucs2:~$ which nuttcp
/usr/bin/nuttcp
epc@iotg-epc-ucs2:~$ weng@weng-ucs1:~$ ifconfig eth5
eth5 Link encap:Ethernet HWaddr cc:ef:48:b4:4b:f9
inet addr:20.20.20.1 Bcast:20.20.20.255 Mask:255.255.255.0
inet6 addr: fe80::ceef:48ff:feb4:4bf9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1442040 errors:0 dropped:0 overruns:0 frame:0
TX packets:16883772 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:95197117 (95.1 MB) TX bytes:12520409066 (12.5 GB)
Memory:b3220000-b323ffff
weng@weng-ucs1:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 40.40.40.2 0.0.0.0 UG 100 0 0 eth1
default 172.27.161.1 0.0.0.0 UG 101 0 0 eth4
10.10.10.0 20.20.20.2 255.255.255.0 UG 0 0 0 eth5
20.20.20.0 * 255.255.255.0 U 100 0 0 eth5
30.30.30.0 40.40.40.2 255.255.255.0 UG 0 0 0 eth1
40.40.40.0 * 255.255.255.0 U 100 0 0 eth1
link-local * 255.255.0.0 U 1000 0 0 eth5
172.27.161.0 * 255.255.255.128 U 100 0 0 eth4
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
weng@weng-ucs1:~$ which iperf3
/usr/bin/iperf3
weng@weng-ucs1:~$ which netperf
/usr/local/bin/netperf
weng@weng-ucs1:~$ which nuttcp
/usr/bin/nuttcp
weng@weng-ucs1:~$ ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
64 bytes from 10.10.10.1: icmp_seq=1 ttl=63 time=0.374 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=63 time=0.365 ms
64 bytes from 10.10.10.1: icmp_seq=3 ttl=63 time=0.360 ms
64 bytes from 10.10.10.1: icmp_seq=4 ttl=63 time=0.391 ms
64 bytes from 10.10.10.1: icmp_seq=5 ttl=63 time=0.397 ms
64 bytes from 10.10.10.1: icmp_seq=6 ttl=63 time=0.368 ms
^C
--- 10.10.10.1 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5000ms
rtt min/avg/max/mdev = 0.360/0.375/0.397/0.028 ms
weng@weng-ucs1:~$ netperf
netperf is my preferred tool, it allows the specify the packet size.
Start server, which listens on port 5000 , using local ipv4 address 10.10.10.1.
epc@iotg-epc-ucs2:~$ sudo netserver -L 10.10.10.1 -4 -p 5000 -D
Starting netserver with host '10.10.10.1' port '5000' and family AF_INETStart client to measure throughput.
weng@weng-ucs1:~$ sudo netperf -H 10.10.10.1 -p 5000 -t TCP_STREAM -- -m 64
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.10.1 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 64 10.03 940.66
weng@weng-ucs1:~$ sudo netperf -H 10.10.10.1 -p 5000 -t TCP_STREAM -- -m 32
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.10.1 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 32 10.00 767.18
weng@weng-ucs1:~$ iperf3 (doc see https://iperf.fr/iperf-doc.php.)
Start iperf3 server, listen on port 6000.
epc@iotg-epc-ucs2:~$ sudo iperf3 -s -p 6000 -B 10.10.10.1
Server listening on 6000
-----------------------------------------------------------
Accepted connection from 20.20.20.1, port 33256
[ 5] local 10.10.10.1 port 6000 connected to 20.20.20.1 port 33258
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 83.4 MBytes 699 Mbits/sec
[ 5] 1.00-2.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 2.00-3.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 3.00-4.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 4.00-5.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 5.00-6.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 6.00-7.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 7.00-8.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 8.00-9.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 9.00-10.00 sec 87.1 MBytes 731 Mbits/sec
[ 5] 10.00-10.04 sec 3.30 MBytes 730 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 874 MBytes 730 Mbits/sec 0 sender
[ 5] 0.00-10.04 sec 870 MBytes 727 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 6000
-----------------------------------------------------------
Accepted connection from 20.20.20.1, port 33260
[ 5] local 10.10.10.1 port 6000 connected to 20.20.20.1 port 33262
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 96.1 MBytes 806 Mbits/sec
[ 5] 1.00-2.00 sec 101 MBytes 847 Mbits/sec
[ 5] 2.00-3.00 sec 101 MBytes 847 Mbits/sec
[ 5] 3.00-4.00 sec 101 MBytes 847 Mbits/sec
[ 5] 4.00-5.00 sec 101 MBytes 847 Mbits/sec
[ 5] 5.00-6.00 sec 101 MBytes 847 Mbits/sec
[ 5] 6.00-7.00 sec 101 MBytes 847 Mbits/sec
[ 5] 7.00-8.00 sec 101 MBytes 847 Mbits/sec
[ 5] 8.00-9.00 sec 101 MBytes 847 Mbits/sec
[ 5] 9.00-10.00 sec 101 MBytes 847 Mbits/sec
[ 5] 10.00-10.04 sec 3.71 MBytes 846 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 1012 MBytes 846 Mbits/sec 71 sender
[ 5] 0.00-10.04 sec 1009 MBytes 843 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 6000
-----------------------------------------------------------Start iperf3 client.
weng@weng-ucs1:~$ sudo iperf3 -p 6000 -c 10.10.10.1 -M 256
[sudo] password for weng:
Connecting to host 10.10.10.1, port 6000
[ 4] local 20.20.20.1 port 33258 connected to 10.10.10.1 port 6000
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 89.6 MBytes 752 Mbits/sec 0 327 KBytes
[ 4] 1.00-2.00 sec 87.4 MBytes 733 Mbits/sec 0 327 KBytes
[ 4] 2.00-3.00 sec 86.2 MBytes 723 Mbits/sec 0 342 KBytes
[ 4] 3.00-4.00 sec 87.5 MBytes 734 Mbits/sec 0 342 KBytes
[ 4] 4.00-5.00 sec 87.7 MBytes 735 Mbits/sec 0 342 KBytes
[ 4] 5.00-6.00 sec 86.5 MBytes 725 Mbits/sec 0 342 KBytes
[ 4] 6.00-7.00 sec 87.5 MBytes 734 Mbits/sec 0 342 KBytes
[ 4] 7.00-8.00 sec 86.4 MBytes 725 Mbits/sec 0 342 KBytes
[ 4] 8.00-9.00 sec 87.5 MBytes 734 Mbits/sec 0 342 KBytes
[ 4] 9.00-10.00 sec 87.4 MBytes 733 Mbits/sec 0 342 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 874 MBytes 733 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 870 MBytes 730 Mbits/sec receiver
iperf Done.
weng@weng-ucs1:~$ sudo iperf3 -p 6000 -c 10.10.10.1 -M 512
Connecting to host 10.10.10.1, port 6000
[ 4] local 20.20.20.1 port 33262 connected to 10.10.10.1 port 6000
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 102 MBytes 856 Mbits/sec 71 314 KBytes
[ 4] 1.00-2.00 sec 101 MBytes 850 Mbits/sec 0 328 KBytes
[ 4] 2.00-3.00 sec 101 MBytes 850 Mbits/sec 0 328 KBytes
[ 4] 3.00-4.00 sec 101 MBytes 850 Mbits/sec 0 371 KBytes
[ 4] 4.00-5.00 sec 101 MBytes 850 Mbits/sec 0 380 KBytes
[ 4] 5.00-6.00 sec 100 MBytes 840 Mbits/sec 0 384 KBytes
[ 4] 6.00-7.00 sec 101 MBytes 850 Mbits/sec 0 387 KBytes
[ 4] 7.00-8.00 sec 101 MBytes 850 Mbits/sec 0 387 KBytes
[ 4] 8.00-9.00 sec 101 MBytes 851 Mbits/sec 0 387 KBytes
[ 4] 9.00-10.00 sec 100 MBytes 841 Mbits/sec 0 387 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1012 MBytes 849 Mbits/sec 71 sender
[ 4] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec receiver
iperf Done.
weng@weng-ucs1:~$ nuttcp
epc@iotg-epc-ucs2:~$ sudo nuttcp -S -P 7000weng@weng-ucs1:~$ sudo nuttcp -i1 -P 7000 -M 512 10.10.10.1
100.5000 MB / 1.00 sec = 843.0492 Mbps 0 retrans
101.0000 MB / 1.00 sec = 847.2316 Mbps 0 retrans
101.0000 MB / 1.00 sec = 847.2528 Mbps 0 retrans
100.8750 MB / 1.00 sec = 846.2169 Mbps 0 retrans
100.9375 MB / 1.00 sec = 846.7175 Mbps 0 retrans
101.0000 MB / 1.00 sec = 847.2367 Mbps 0 retrans
101.0625 MB / 1.00 sec = 847.7805 Mbps 0 retrans
101.0000 MB / 1.00 sec = 847.2647 Mbps 0 retrans
101.0000 MB / 1.00 sec = 847.2367 Mbps 0 retrans
101.0625 MB / 1.00 sec = 847.7686 Mbps 0 retrans
1012.1250 MB / 10.03 sec = 846.7138 Mbps 3 %TX 30 %RX 0 retrans 0.40 msRTT
weng@weng-ucs1:~$ sudo nuttcp -i1 -P 7000 -M 1024 10.10.10.1
108.6875 MB / 1.00 sec = 911.7295 Mbps 0 retrans
109.4375 MB / 1.00 sec = 918.0044 Mbps 0 retrans
109.3125 MB / 1.00 sec = 917.0045 Mbps 0 retrans
109.5000 MB / 1.00 sec = 918.5416 Mbps 0 retrans
109.4375 MB / 1.00 sec = 918.0384 Mbps 0 retrans
109.4375 MB / 1.00 sec = 918.0338 Mbps 0 retrans
109.4375 MB / 1.00 sec = 918.0228 Mbps 0 retrans
109.5000 MB / 1.00 sec = 918.5581 Mbps 0 retrans
109.4375 MB / 1.00 sec = 918.0209 Mbps 0 retrans
109.5000 MB / 1.00 sec = 918.5425 Mbps 0 retrans
1096.0625 MB / 10.02 sec = 917.3818 Mbps 5 %TX 19 %RX 0 retrans 0.46 msRTT
weng@weng-ucs1:~$ Reference:
- netperf: http://www.netperf.org/svn/netperf2/tags/netperf-2.7.0/doc/netperf.html
- iperf: https://iperf.fr/iperf-doc.php
- nuttcp: http://manpages.ubuntu.com/manpages/trusty/man8/nuttcp.8.html
Subscribe via RSS