As my day job is to develop routing platforms to route packets, from time to time I need to measure the performance of router in terms of throughput, i.e. pump traffic into UUT ingress interface, and get traffic out from egress interface. In my view, there are three classes of testing tools:

  1. Dedicated testing equipment like IXIA, Spirent. These are expensive and accurate equipments. It requires dedicated hardware, training.
  2. Open source tools like Pktgen-DPDK, [TRex](https://github.com/cisco-system-traffic-generator): These tools are wokring with NIC directly, and can create high rate relative precise traffiic.
  3. Open source tools like iperf, netperf, nuttcp. These are free tools, easy to install, easy to use. It generates traffic using socket. So timing wise, it is much less accurate. However it still gives roughly the performance number.

Linux network performance tools

In this post, I will go throught netperf, iperf, nuttcp.

Test bed set up

I uses three Cisco UCS boxes below.

net performance test bed

some details:

  1. The middle box "iotg-epc-ucs3" runs Fedora 24, and is used as UUT.
  2. [weng@iotg-epc-ucs3 ~]$   ifconfig enp11s0f0
    enp11s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 10.10.10.2  netmask 255.255.255.0  broadcast 10.10.10.255
            inet6 fe80::c204:be95:d906:ba13  prefixlen 64  scopeid 0x20<link>
            ether 00:10:18:a7:82:28  txqueuelen 1000  (Ethernet)
            RX packets 2189444  bytes 153377208 (146.2 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 23132269  bytes 22045460531 (20.5 GiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    [weng@iotg-epc-ucs3 ~]$   ifconfig enp11s0f1
    enp11s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 20.20.20.2  netmask 255.255.255.0  broadcast 20.20.20.255
            inet6 fe80::ee39:1018:b17e:3f76  prefixlen 64  scopeid 0x20<link>
            ether 00:10:18:a7:82:2a  txqueuelen 1000  (Ethernet)
            RX packets 23176127  bytes 22049032366 (20.5 GiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 2200466  bytes 154280229 (147.1 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    [weng@iotg-epc-ucs3 ~]$ route
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    default         gateway         0.0.0.0         UG    100    0        0 enp12s0f0
    default         gateway         0.0.0.0         UG    101    0        0 enp12s0f1
    default         gateway         0.0.0.0         UG    102    0        0 ens1f0
    10.10.10.0      0.0.0.0         255.255.255.0   U     100    0        0 enp11s0f0
    20.20.20.0      0.0.0.0         255.255.255.0   U     100    0        0 enp11s0f1
    30.30.30.0      0.0.0.0         255.255.255.0   U     100    0        0 enp12s0f0
    40.40.40.0      0.0.0.0         255.255.255.0   U     100    0        0 enp12s0f1
    172.0.0.0       gateway         255.0.0.0       UG    1      0        0 ens1f0
    172.27.161.0    0.0.0.0         255.255.255.128 U     100    0        0 ens1f0
    [weng@iotg-epc-ucs3 ~]$ cat /proc/sys/net/ipv4/ip_forward # make sure IP forwad is turned on
    1
    [weng@iotg-epc-ucs3 ~]$ # make sure firewall is disable as it might block the testing traffic
    [weng@iotg-epc-ucs3 ~]$ systemctl status firewalld
    \u25cf firewalld.service - firewalld - dynamic firewall daemon
       Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
       Active: inactive (dead) since Thu 2016-09-08 11:15:25 PDT; 1 day 10h ago
      Process: 1306 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, s
     Main PID: 1306 (code=exited, status=0/SUCCESS)
    
    Warning: Journal has been rotated since unit was started. Log output is incomplete or unavaila
    [weng@iotg-epc-ucs3 ~]$ 
  3. The box iotg-epc-ucs2 runs ubuntu 16.04LTS, is used as server
  4. epc@iotg-epc-ucs2:~$ ifconfig eth0
    eth0      Link encap:Ethernet  HWaddr 00:10:18:c9:54:d8  
              inet addr:10.10.10.1  Bcast:10.10.10.255  Mask:255.255.255.0
              inet6 addr: fe80::210:18ff:fec9:54d8/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:23133246 errors:1 dropped:0 overruns:0 frame:1
              TX packets:2191846 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:22045747179 (22.0 GB)  TX bytes:153829920 (153.8 MB)
    
    epc@iotg-epc-ucs2:~$ route
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    default         10.10.10.2      0.0.0.0         UG    0      0        0 eth0
    10.10.10.0      *               255.255.255.0   U     1      0        0 eth0
    20.20.20.0      10.10.10.2      255.255.255.0   UG    0      0        0 eth0
    30.30.30.0      *               255.255.255.0   U     1      0        0 eth1
    40.40.40.0      30.30.30.2      255.255.255.0   UG    0      0        0 eth1
    171.0.0.0       *               255.0.0.0       U     0      0        0 eth4
    172.0.0.0       172.27.161.1    255.0.0.0       UG    0      0        0 eth4
    172.27.161.0    *               255.255.255.128 U     1      0        0 eth4
    173.0.0.0       *               255.0.0.0       U     0      0        0 eth4
    192.168.122.0   *               255.255.255.0   U     0      0        0 virbr0
    epc@iotg-epc-ucs2:~$ which netperf
    /usr/local/bin/netperf
    epc@iotg-epc-ucs2:~$ which iperf3
    /usr/bin/iperf3
    epc@iotg-epc-ucs2:~$ which nuttcp
    /usr/bin/nuttcp
    epc@iotg-epc-ucs2:~$ 
  5. The box weng-ucs1 runs ubuntu 16.04LTS, is used as client.
  6. weng@weng-ucs1:~$ ifconfig eth5
    eth5      Link encap:Ethernet  HWaddr cc:ef:48:b4:4b:f9  
              inet addr:20.20.20.1  Bcast:20.20.20.255  Mask:255.255.255.0
              inet6 addr: fe80::ceef:48ff:feb4:4bf9/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:1442040 errors:0 dropped:0 overruns:0 frame:0
              TX packets:16883772 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:95197117 (95.1 MB)  TX bytes:12520409066 (12.5 GB)
              Memory:b3220000-b323ffff 
    
    weng@weng-ucs1:~$ route
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    default         40.40.40.2      0.0.0.0         UG    100    0        0 eth1
    default         172.27.161.1    0.0.0.0         UG    101    0        0 eth4
    10.10.10.0      20.20.20.2      255.255.255.0   UG    0      0        0 eth5
    20.20.20.0      *               255.255.255.0   U     100    0        0 eth5
    30.30.30.0      40.40.40.2      255.255.255.0   UG    0      0        0 eth1
    40.40.40.0      *               255.255.255.0   U     100    0        0 eth1
    link-local      *               255.255.0.0     U     1000   0        0 eth5
    172.27.161.0    *               255.255.255.128 U     100    0        0 eth4
    192.168.122.0   *               255.255.255.0   U     0      0        0 virbr0
    weng@weng-ucs1:~$ which iperf3
    /usr/bin/iperf3
    weng@weng-ucs1:~$ which netperf
    /usr/local/bin/netperf
    weng@weng-ucs1:~$ which nuttcp
    /usr/bin/nuttcp
    weng@weng-ucs1:~$ ping 10.10.10.1
    PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data.
    64 bytes from 10.10.10.1: icmp_seq=1 ttl=63 time=0.374 ms
    64 bytes from 10.10.10.1: icmp_seq=2 ttl=63 time=0.365 ms
    64 bytes from 10.10.10.1: icmp_seq=3 ttl=63 time=0.360 ms
    64 bytes from 10.10.10.1: icmp_seq=4 ttl=63 time=0.391 ms
    64 bytes from 10.10.10.1: icmp_seq=5 ttl=63 time=0.397 ms
    64 bytes from 10.10.10.1: icmp_seq=6 ttl=63 time=0.368 ms
    ^C
    --- 10.10.10.1 ping statistics ---
    6 packets transmitted, 6 received, 0% packet loss, time 5000ms
    rtt min/avg/max/mdev = 0.360/0.375/0.397/0.028 ms
    weng@weng-ucs1:~$ 

netperf

netperf is my preferred tool, it allows the specify the packet size.

Start server, which listens on port 5000 , using local ipv4 address 10.10.10.1.

epc@iotg-epc-ucs2:~$ sudo netserver -L 10.10.10.1 -4 -p 5000 -D
Starting netserver with host '10.10.10.1' port '5000' and family AF_INET

Start client to measure throughput.

weng@weng-ucs1:~$ sudo netperf -H 10.10.10.1 -p 5000 -t TCP_STREAM -- -m 64
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.10.1 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384     64    10.03     940.66   
weng@weng-ucs1:~$ sudo netperf -H 10.10.10.1 -p 5000 -t TCP_STREAM -- -m 32
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.10.1 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384     32    10.00     767.18   
weng@weng-ucs1:~$ 

iperf3 (doc see https://iperf.fr/iperf-doc.php.)

Start iperf3 server, listen on port 6000.

epc@iotg-epc-ucs2:~$ sudo iperf3 -s -p 6000 -B 10.10.10.1 
Server listening on 6000
-----------------------------------------------------------
Accepted connection from 20.20.20.1, port 33256
[  5] local 10.10.10.1 port 6000 connected to 20.20.20.1 port 33258
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  83.4 MBytes   699 Mbits/sec                  
[  5]   1.00-2.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   2.00-3.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   3.00-4.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   4.00-5.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   5.00-6.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   6.00-7.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   7.00-8.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   8.00-9.00   sec  87.1 MBytes   731 Mbits/sec                  
[  5]   9.00-10.00  sec  87.1 MBytes   731 Mbits/sec                  
[  5]  10.00-10.04  sec  3.30 MBytes   730 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec   874 MBytes   730 Mbits/sec    0             sender
[  5]   0.00-10.04  sec   870 MBytes   727 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 6000
-----------------------------------------------------------
Accepted connection from 20.20.20.1, port 33260
[  5] local 10.10.10.1 port 6000 connected to 20.20.20.1 port 33262
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  96.1 MBytes   806 Mbits/sec                  
[  5]   1.00-2.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   2.00-3.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   3.00-4.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   4.00-5.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   5.00-6.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   6.00-7.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   7.00-8.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   8.00-9.00   sec   101 MBytes   847 Mbits/sec                  
[  5]   9.00-10.00  sec   101 MBytes   847 Mbits/sec                  
[  5]  10.00-10.04  sec  3.71 MBytes   846 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.04  sec  1012 MBytes   846 Mbits/sec   71             sender
[  5]   0.00-10.04  sec  1009 MBytes   843 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 6000
-----------------------------------------------------------

Start iperf3 client.

weng@weng-ucs1:~$  sudo iperf3 -p 6000 -c 10.10.10.1 -M 256
[sudo] password for weng: 
Connecting to host 10.10.10.1, port 6000
[  4] local 20.20.20.1 port 33258 connected to 10.10.10.1 port 6000
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  89.6 MBytes   752 Mbits/sec    0    327 KBytes       
[  4]   1.00-2.00   sec  87.4 MBytes   733 Mbits/sec    0    327 KBytes       
[  4]   2.00-3.00   sec  86.2 MBytes   723 Mbits/sec    0    342 KBytes       
[  4]   3.00-4.00   sec  87.5 MBytes   734 Mbits/sec    0    342 KBytes       
[  4]   4.00-5.00   sec  87.7 MBytes   735 Mbits/sec    0    342 KBytes       
[  4]   5.00-6.00   sec  86.5 MBytes   725 Mbits/sec    0    342 KBytes       
[  4]   6.00-7.00   sec  87.5 MBytes   734 Mbits/sec    0    342 KBytes       
[  4]   7.00-8.00   sec  86.4 MBytes   725 Mbits/sec    0    342 KBytes       
[  4]   8.00-9.00   sec  87.5 MBytes   734 Mbits/sec    0    342 KBytes       
[  4]   9.00-10.00  sec  87.4 MBytes   733 Mbits/sec    0    342 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   874 MBytes   733 Mbits/sec    0             sender
[  4]   0.00-10.00  sec   870 MBytes   730 Mbits/sec                  receiver

iperf Done.
weng@weng-ucs1:~$  sudo iperf3 -p 6000 -c 10.10.10.1 -M 512
Connecting to host 10.10.10.1, port 6000
[  4] local 20.20.20.1 port 33262 connected to 10.10.10.1 port 6000
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   102 MBytes   856 Mbits/sec   71    314 KBytes       
[  4]   1.00-2.00   sec   101 MBytes   850 Mbits/sec    0    328 KBytes       
[  4]   2.00-3.00   sec   101 MBytes   850 Mbits/sec    0    328 KBytes       
[  4]   3.00-4.00   sec   101 MBytes   850 Mbits/sec    0    371 KBytes       
[  4]   4.00-5.00   sec   101 MBytes   850 Mbits/sec    0    380 KBytes       
[  4]   5.00-6.00   sec   100 MBytes   840 Mbits/sec    0    384 KBytes       
[  4]   6.00-7.00   sec   101 MBytes   850 Mbits/sec    0    387 KBytes       
[  4]   7.00-8.00   sec   101 MBytes   850 Mbits/sec    0    387 KBytes       
[  4]   8.00-9.00   sec   101 MBytes   851 Mbits/sec    0    387 KBytes       
[  4]   9.00-10.00  sec   100 MBytes   841 Mbits/sec    0    387 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1012 MBytes   849 Mbits/sec   71             sender
[  4]   0.00-10.00  sec  1009 MBytes   846 Mbits/sec                  receiver

iperf Done.
weng@weng-ucs1:~$ 

nuttcp

epc@iotg-epc-ucs2:~$  sudo nuttcp -S  -P 7000
weng@weng-ucs1:~$ sudo nuttcp -i1 -P 7000 -M 512 10.10.10.1
  100.5000 MB /   1.00 sec =  843.0492 Mbps     0 retrans
  101.0000 MB /   1.00 sec =  847.2316 Mbps     0 retrans
  101.0000 MB /   1.00 sec =  847.2528 Mbps     0 retrans
  100.8750 MB /   1.00 sec =  846.2169 Mbps     0 retrans
  100.9375 MB /   1.00 sec =  846.7175 Mbps     0 retrans
  101.0000 MB /   1.00 sec =  847.2367 Mbps     0 retrans
  101.0625 MB /   1.00 sec =  847.7805 Mbps     0 retrans
  101.0000 MB /   1.00 sec =  847.2647 Mbps     0 retrans
  101.0000 MB /   1.00 sec =  847.2367 Mbps     0 retrans
  101.0625 MB /   1.00 sec =  847.7686 Mbps     0 retrans

 1012.1250 MB /  10.03 sec =  846.7138 Mbps 3 %TX 30 %RX 0 retrans 0.40 msRTT
weng@weng-ucs1:~$ sudo nuttcp -i1 -P 7000 -M 1024 10.10.10.1
  108.6875 MB /   1.00 sec =  911.7295 Mbps     0 retrans
  109.4375 MB /   1.00 sec =  918.0044 Mbps     0 retrans
  109.3125 MB /   1.00 sec =  917.0045 Mbps     0 retrans
  109.5000 MB /   1.00 sec =  918.5416 Mbps     0 retrans
  109.4375 MB /   1.00 sec =  918.0384 Mbps     0 retrans
  109.4375 MB /   1.00 sec =  918.0338 Mbps     0 retrans
  109.4375 MB /   1.00 sec =  918.0228 Mbps     0 retrans
  109.5000 MB /   1.00 sec =  918.5581 Mbps     0 retrans
  109.4375 MB /   1.00 sec =  918.0209 Mbps     0 retrans
  109.5000 MB /   1.00 sec =  918.5425 Mbps     0 retrans

 1096.0625 MB /  10.02 sec =  917.3818 Mbps 5 %TX 19 %RX 0 retrans 0.46 msRTT
weng@weng-ucs1:~$ 

Reference:

  • netperf: http://www.netperf.org/svn/netperf2/tags/netperf-2.7.0/doc/netperf.html
  • iperf: https://iperf.fr/iperf-doc.php
  • nuttcp: http://manpages.ubuntu.com/manpages/trusty/man8/nuttcp.8.html