Title: Performance Issues with Gigabit Encryption
We have a problem with OpenVPN that we are hoping someone can provide some advice for….
We have been lab testing OpenVPN for potential use for gigabit encryption and are experiencing very poor levels of performance. We were wondering if anyone else has experienced this and come up with a set of configuration settings that fix the problem. We have checked the FAQ and searched the mailing list archives and we see a number of people experiencing similar issues but in most cases it appears the people reporting these problems were maxing our their processors which our testing indicates we are not.
Onto the problem……..
OpenVPN version is the latest available for download from the OpenVPN site. Compiled on the hardware it is being run on and configured in accordance with the sample bridge configuration provided on the site.
The layout of our test scenario is as follows:
A is connected to B via Cat 6 x-over cable. Interfaces on both ends are hard set to gigabit.
B is connected to C via Cat 6 x-over cable. Interfaces on both ends are hard set to gigabit.
C is connected to D via Cat 6 x-over cable. Interfaces on both ends are hard set to gigabit.
Box B and C control the OpenVPN link (site to site bridged mode configuration) using SSL encryption. Both of these boxes are RHEL 4 ES running a 2.6 SMP kernel. The boxes have been stripped down from a OS perspective to only that necessary to perform the VPN functions and administration of the boxes so there is nothing else sucking up resources on them.
Hardware specs of Box B and C are Dual Zeon 2Ghz hyperthreaded processors with 2Gb of RAM and 2 x 1Gigabit Broadcom network interface cards, fast SCSI raid hard disks.
Box A simulates a workstation at one end of the link accessing box D a servers at the other end of the link. Traffic between Box A and Box B is not encrypted, traffic is encrypted by Box B and decrypted by Box C then passed in the clear again to the server at Box D to simulate clients at one site accessing a server at another site over a dedicated dark fibre.
When we measure the unencrypted throughput speed across the bridge between Box B and Box C (using IPERF to generate the traffic and measure the overall throughput achieved), we get a maximum throughput of 940Mbits/s which is what you would reasonably expect from a 1Gigabit link. When we activate OpenVPN on the bridged link we get a maximum throughput of 230Mbits/s with averages being around 160Mbits/s. We have also used other test tools to determine if the tool we were using was accurately reporting the throughput or not and these tools confirmed our original throughput findings.
Performance monitoring of Box B and C during the testing indicates the first processor is only 40% utilised and the second processor is not being utilised at all on each box. Memory is only 30% utilised and there is no paging taking place on either box. We have used ‘tcpdump’ to monitor the interfaces on the link to see if there were any ICMP messages indicating fragmentation problems but this test also came back negative. We have checked the logs to see if there is anything in them that would aid in diagnosing the problem but there are no error messages at all in the logs.
We have conducted the tests between Boxes A and D and also between Boxes B and C (just to make sure it wasn’t a problem with the end client or server) the results in performance are the same.
We have tried it with hyperthreading turned on and off in the BIOS however this produced no difference in the results.
I realise encryption has an overhead but it should be in the vicinity of 20-30%, not 70% so we obviously have a problem somewhere.
Now to our questions…….
Q1. What is the maximum sustained speed anyone has been able to achieve with SSL encryption on a Gigabit link?
Q2. Does anone have any suggestions of things to try.
Q3. Does someone have a configuration they have used for a bridged site to site VPN (that achieved speeds faster than 230Mbits/s on a gigabit link) that we could use to replace our entire existing configuration with from scratch for the scenario mentioned above.
Thanks in advance,