As a result of setting up my site-to-site VPN see this it is much easier to transfer things from one site to the other.
The questions that came to mind were:
What is the bandwidth of my VPN ?
What is the limiting factor ? My ISPs or the Edge Router ?
To answer to the above I used the iperf3 a tool that is already available on the EdgeRouter.
Site 1 that has an 500Mbps/25Mbps link will be used as the client.
Site 2 that has an 120Mbps/6Mbps link will be used as server.
I do this because I do not want to be limited by the 6Mbps uplink from Site 2. In this setup Site 1 pushes data onto the Site 2 so the limiting links will be the 25Mbps upload from Site 1 and the 120Mbps download on Site 2.
Note that Site 2 iperf3 must be executed first the Site 1 iperf3 command will start sending data.
Bellow the output from Site 2:
ubnt@ubnt:~$ iperf3 -s
Accepted connection from 10.99.99.1, port 44388
[ 5] local 192.168.9.1 port 5201 connected to 10.99.99.1 port 44389
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.18 sec 1.00 MBytes 7.10 Mbits/sec
[ 5] 1.18-2.22 sec 1.00 MBytes 8.06 Mbits/sec
[ 5] 2.22-3.13 sec 896 KBytes 8.05 Mbits/sec
[ 5] 3.13-4.20 sec 1.00 MBytes 7.88 Mbits/sec
[ 5] 4.20-5.15 sec 896 KBytes 7.69 Mbits/sec
[ 5] 5.15-6.25 sec 1.00 MBytes 7.63 Mbits/sec
[ 5] 6.25-7.19 sec 1.00 MBytes 8.95 Mbits/sec
[ 5] 7.19-8.23 sec 1.00 MBytes 8.10 Mbits/sec
[ 5] 8.23-9.23 sec 1.00 MBytes 8.37 Mbits/sec
[ 5] 9.23-10.20 sec 1.00 MBytes 8.64 Mbits/sec
[ 5] 10.20-10.20 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.20 sec 9.75 MBytes 8.02 Mbits/sec 9 sender
[ 5] 0.00-10.20 sec 9.75 MBytes 8.02 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
Bellow the output from Site 1:
ubnt@ubnt:~$ iperf3 -c 192.168.9.1
Connecting to host 192.168.9.1, port 5201
[ 4] local 10.99.99.1 port 44389 connected to 192.168.9.1 port 5201
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-1.03 sec 1.00 MBytes 8.18 Mbits/sec 5
[ 4] 1.03-2.08 sec 1.00 MBytes 7.99 Mbits/sec 0
[ 4] 2.08-3.07 sec 1.00 MBytes 8.41 Mbits/sec 0
[ 4] 3.07-4.03 sec 896 KBytes 7.70 Mbits/sec 0
[ 4] 4.03-5.01 sec 896 KBytes 7.50 Mbits/sec 4
[ 4] 5.01-6.01 sec 896 KBytes 7.31 Mbits/sec 0
[ 4] 6.01-7.03 sec 1.12 MBytes 9.28 Mbits/sec 0
[ 4] 7.03-8.04 sec 1.00 MBytes 8.27 Mbits/sec 0
[ 4] 8.04-9.07 sec 1.00 MBytes 8.14 Mbits/sec 0
[ 4] 9.07-10.03 sec 1.00 MBytes 8.79 Mbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.03 sec 9.75 MBytes 8.16 Mbits/sec 9 sender
[ 4] 0.00-10.03 sec 9.75 MBytes 8.16 Mbits/sec receiver
iperf Done.
So the result is a bit over 8Mbps/8Mbps symmetric link.
In conclusion:
1. There is no limit due to the ISPs. 8Mbps is bellow both limiting links, the 25Mbps upload from Site 1 and the 120Mbps download on Site 2.
2. This is quite low bandwith, consistent with the fact that there is no hardware acceleration for OpenVPN site-to-site in EdgeRouter.
3. Strange that I did not notice a 100% CPU utilization on any of the routers, the highest peak went to 70% on the client site.
4. There are probably ways to improve this link.