OpenVPN Cryptographic Layer

This technical overview of OpenVPN's cryptographic layer assumes a prior understanding of modern cryptographic concepts. For additional discussion on OpenVPN security, refer to Hardening OpenVPN Security.

OpenVPN has two authentication modes:

In static key mode, a pre-shared key is generated and shared between both OpenVPN peers before the tunnel is started. This static key contains four independent keys: HMAC send, HMAC receive, encrypt, and decrypt. By default, in static key mode, both hosts will use the same HMAC key and the same encrypt/decrypt key. However, using the direction parameter to --secret, it is possible to use all four keys independently.

In SSL/TLS mode, an SSL session is established with bidirectional authentication (i.e., each side of the connection must present its own certificate). If the SSL/TLS authentication succeeds, encryption/decryption and HMAC key source material is then randomly generated by OpenSSL's RAND_bytes function and exchanged over the SSL/TLS connection. Both sides of the connection contribute random source material. This mode never uses any key bidirectionally, so each peer has a distinct send HMAC, receive HMAC, packet encrypt, and packet decrypt key. If --key-method 2 is used, the actual keys are generated from the random source material using the TLS PRF function. If --key-method 1 is used, the keys are generated directly from the OpenSSL RAND_bytes function. --key-method 2 was introduced with OpenVPN 1.5.0 and will be made the default in OpenVPN 2.0.

During SSL/TLS rekeying, a transition window parameter permits overlap between old and new key usage, so there is no time pressure or latency bottleneck during renegotiations.

Because SSL/TLS is designed to operate over a reliable transport, OpenVPN provides a reliable transport layer on top of UDP (see diagram below).

Once each peer has its set of keys, the tunnel forwarding operation commences.

The encrypted packet is formatted as follows:

The HMAC and explicit IV are outside of the encrypted envelope.

The per-packet IV is randomized using a nonce-based PRNG initially seeded from the OpenSSL RAND_bytes function.

The OpenSSL EVP interface provides HMAC, encryption, and decryption functions and allows the user to select an arbitrary cipher, key size, and message digest for HMAC. BlowFish is the default cipher, and SHA1 is the default message digest. The OpenSSL EVP interface handles padding to an even multiple of block size using PKCS#5 padding. CBC-mode cipher usage is encouraged but not required.

One notable security improvement that OpenVPN provides over vanilla TLS is that it allows the user to use a pre-shared passphrase (or static key) in conjunction with the --tls-auth directive to generate an HMAC key to authenticate the packets that are themselves part of the TLS handshake sequence. This protects against buffer overflows in the OpenSSL TLS implementation because an attacker cannot even initiate a TLS handshake without being able to generate packets with the correct HMAC signature.

OpenVPN multiplexes the SSL/TLS session used for authentication and key exchange with the actual encrypted tunnel data stream. OpenVPN provides the SSL/TLS connection with a reliable transport layer (as it is designed to operate over). After being encrypted and signed with an HMAC, the actual IP packets are tunneled over UDP without any reliability layer. So if --proto udp is used, no IP packets are tunneled over a reliable transport, eliminating the problem of reliability-layer collisions — Of course, if you are tunneling a TCP session over OpenVPN running in UDP mode, the TCP protocol itself will provide the reliability layer.

SSL/TLS -> Reliability Layer -> \
           --tls-auth HMAC       \
                                  \ 
                                   > Multiplexer ----> UDP
                                  /                    Transport
IP        Encrypt and HMAC       /
Tunnel -> using OpenSSL EVP --> /
Packets   interface.

This model has the benefit that SSL/TLS sees a reliable transport layer while the IP packet forwarder sees an unreliable transport layer -- exactly what both components want to see. The reliability and authentication layers are completely independent of one another, i.e., the sequence number is embedded inside the HMAC-signed envelope and is not used for authentication purposes.