OpenVPN cryptographic layer

This is a technical overview of OpenVPN’s cryptographic layer, and assumes a prior understanding of modern cryptographic concepts. For additional discussion on OpenVPN security, see this FAQ item.

OpenVPN has two authentication modes:

  • Static Key — Use a pre-shared static key
  • TLS — Use SSL/TLS + certificates for authentication and key exchange

In static key mode, a pre-shared key is generated and shared between both OpenVPN peers before the tunnel is started. This static key contains 4 independent keys: HMAC send, HMAC receive, encrypt, and decrypt. By default in static key mode, both hosts will use the same HMAC key and the same encrypt/decrypt key. However, using the direction parameter to –secret, it is possible to use all 4 keys independently.

In SSL/TLS mode, an SSL session is established with bidirectional authentication (i.e. each side of the connection must present its own certificate). If the SSL/TLS authentication succeeds, encryption/decryption and HMAC key source material is then randomly generated by OpenSSL’s RAND_bytes function and exchanged over the SSL/TLS connection. Both sides of the connection contribute random source material. This mode never uses any key bidirectionally, so each peer has a distinct send HMAC, receive HMAC, packet encrypt, and packet decrypt key. If –key-method 2 is used, the actual keys are generated from the random source material using the TLS PRF function. If –key-method 1 is used, the keys are generated directly from the OpenSSL RAND_bytes function. –key-method 2 was introduced with OpenVPN 1.5.0 and will be made the default in OpenVPN 2.0.

During SSL/TLS rekeying, there is a transition-window parameter that permits overlap between old and new key usage, so there is no time pressure or latency bottleneck during SSL/TLS renegotiations.

Because SSL/TLS is designed to operate over a reliable transport, OpenVPN provides a reliable transport layer on top of UDP (see diagram below).

Once each peer has its set of keys, the tunnel forwarding operation commences.

The encrypted packet is formatted as follows:

  • HMAC(explicit IV, encrypted envelope)
  • Explicit IV
  • Encrypted Envelope

The plaintext of the encrypted envelope is formatted as follows:

  • 64 bit sequence number
  • payload data, i.e. IP packet or Ethernet frame

The HMAC and explicit IV are outside of the encrypted envelope.

The per-packet IV is randomized using a nonce-based PRNG that is initially seeded from the OpenSSL RAND_bytes function.

HMAC, encryption, and decryption functions are provided by the OpenSSL EVP interface and allows the user to select an arbitrary cipher, key size, and message digest for HMAC. BlowFish is the default cipher and SHA1 is the default message digest. The OpenSSL EVP interface handles padding to an even multiple of block size using PKCS#5 padding. CBC-mode cipher usage is encouraged but not required.

One notable security improvement that OpenVPN provides over vanilla TLS is that it gives the user the opportunity to use a pre-shared passphrase (or static key) in conjunction with the –tls-auth directive to generate an HMAC key to authenticate the packets that are themselves part of the TLS handshake sequence. This protects against buffer overflows in the OpenSSL TLS implementation, because an attacker cannot even initiate a TLS handshake without being able to generate packets with the currect HMAC signature.

OpenVPN multiplexes the SSL/TLS session used for authentication and key exchange with the actual encrypted tunnel data stream. OpenVPN provides the SSL/TLS connection with a reliable transport layer (as it is designed to operate over). The actual IP packets, after being encrypted and signed with an HMAC, are tunnelled over UDP without any reliability layer. So if –proto udpis used, no IP packets are tunneled over a reliable transport, eliminating the problem of reliability-layer collisions — Of course, if you are tunneling a TCP session over OpenVPN running in UDP mode, the TCP protocol itself will provide the reliability layer.

SSL/TLS -> Reliability Layer -> \
           --tls-auth HMAC       \
                                  \ 
                                   > Multiplexer ----> UDP
                                  /                    Transport
IP        Encrypt and HMAC       /
Tunnel -> using OpenSSL EVP --> /
Packets   interface.

This model has the benefit that SSL/TLS sees a reliable transport layer while the IP packet forwarder sees an unreliable transport layer — exactly what both components want to see. The reliability and authentication layers are completely independent of one another, i.e. the sequence number is embedded inside the HMAC-signed envelope and is not used for authentication purposes.