No flaws found in OpenVPN software. Our response to the CVE-2019-14899 vulnerability report.

Setting up an OpenVPN Access Server cluster

Request More Information

By submitting this form, you agree to our Terms of Use and acknowledge our Privacy Statement

Introduction

A feature newly introduced in Access Server since the official release 2.7.4 is the ability to create a cluster of Access Servers for the purpose of high availability and increased load capacity. Such a cluster is an answer to the requirement that our customers have expressed for a high-availability solution and it also provides the ability to spread the load across multiple servers. There are still improvements coming to the product, like a new licensing system that lets you share a license across multiple nodes, and we are of course open to hearing feedback from our customers (use the support ticket system, please). Access Server 2.7.4 is a first phase implementation of a clustering feature.

A single OpenVPN Access Server contains everything it needs to offer its services to connecting VPN clients. All the user credentials, certificates, services, and access rules, can all be present on one Access Server. This is also a single point of failure. To resolve this we used to rely in the past on our LAN-model UCARP/VRRP based failover system to run a standby server that can take over if the primary fails. However, that model is limited to running on a local network that supports VRRP/UCARP traffic to pass through. Some networks don’t allow this traffic. Amazon AWS for example doesn’t, so this model doesn’t work there. The cluster feature works on Amazon AWS and many other platforms as it doesn’t rely on this VRRP/UCARP type traffic.

In the cluster feature, we separate the storage of certificates and credentials from the OpenVPN service daemons. Our cluster solution allows you to use a MySQL type database system (MySQL, MariaDB, Amazon RDS, for example) to store all the configuration of the Access Server. These database systems can be single servers or clusters with high-availability. Amazon RDS for example can offer a fault-tolerant solution that automatically switches over to a backup server if the primary server fails, and is the recommended solutions at this time. Multiple Access Servers in clustering mode can attach to such a central database and offer VPN services. A DNS-based round-robin system can ensure that VPN clients will have a single address to connect to, like vpn.example.com, and this resolves at random to any of the nodes in your cluster setup. A VPN will try these different servers in semi-random order. If an OpenVPN Access Server server fails, it will lead to a temporary connection failure for the connected client, until the client automatically tries to reconnect, and then ends up connecting to one of the other servers in the cluster. The failed server could then either be repaired, or it can be removed from the DNS records and a new Access Server could be setup, attached to the cluster, and then added to the round-robin DNS record to replace the failed node.

Development status

As mentioned this is a first phase implementation. It is primarily aimed at Amazon AWS, where with tiered instances on Amazon AWS adding a new node can even be automated, and it can then be automatically licensed. At the moment, a requirement is that each node has its own license, and tiered instances make this occur automatically. However, we recognize that a better licensing system where licenses can be shared across nodes in a cluster would be much more useful. We are therefore already taking steps to prepare for a new, more flexible licensing system that allows having one license shared across multiple nodes. This is expected to be released in Q1 or Q2 of 2020.

We will also be continuing to improve the clustering feature to make deployment and maintenance easier.

Some important notes

Most of our customers will currently be using the default SQLite3 database files to store all the configuration of a single OpenVPN Access Server installation. When you install the Access Server from scratch, it will still default to SQLite3, even with the new versions of Access Server with clustering support built-in. You can upgrade an existing Access Server installation with the new version and it will continue to function as before, and retain all your settings. The cluster function is an on/off toggle that is not enabled by default, and with the cluster function switched off, it will just be a normal standalone Access Server as with previous versions. But of course now with an updated Admin UI interface with all of the familiar functions still present. Only when you go into the Admin UI and enable the cluster function, do the new functions and settings for clustering mode come into play.

The cluster function requires that you migrate your settings to a MySQL type database such as MySQL, MariaDB, or Amazon RDS. In theory, as long as it is a MySQL 5.6 or higher compatible system, it should work. Amazon Aurora available as an engine for an Amazon RDS database is such a compatible system. With previous Access Server releases, the conversion from SQLite3 local storage to a MySQL type system was already possible, and back then it had to be done on the command line manually. We now have a conversion tool built right into the Admin UI to handle that part. You do still need to have some database client support software installed for the connection to be made, and you do need to still set up the new database system yourself first. In our guide we are going to be assuming a situation where Amazon RDS is used, so this means we’re doing this cluster setup on Amazon AWS, but a MySQL or MariaDB setup can also be used for the database storage, and that means it’s not tied to Amazon AWS. To eliminate a single point of failure we do advise using a fault-tolerant setup for the database system, whatever solution you choose to use.

The conversion process from local SQLite3 to MySQL type database system currently has few safety checks in place. What this means is that if you convert an Access Server configuration to MySQL type database system, and then repeat that same act again from another Access Server, you will likely wipe the original settings, and the latest converted settings will be the only valid settings. So please do not repeat a database conversion to the same target MySQL database system. We will improve this behavior in future releases.

Before you begin

If you intend to upgrade an existing Access Server installation to the new cluster-ready version, then backup your settings first. You can use the backup commands on the command line found here to make a complete backup of the settings on your Access Server installation safely without having to stop operations on your server. If you’ve never made a backup before, now would be a good time to remind you that it would be a good idea to set up an automated backup plan. The information stored in the Access Server is unique and cannot be replaced, unless you wish to reinstall all your existing clients. Backups can prevent that scenario.

Setting up Amazon RDS

It is important to note here that you are not required to use Amazon RDS, but it is recommended, since it is fault-tolerant. Another MySQL or MariaDB system also works, and this also means you are not tied to Amazon AWS. We just assume this is the easiest setup for our customers and serves as a general guide on how to set up an OpenVPN Access Server cluster. The term cluster is also used in database systems that are fault-tolerant. Please do not confuse a database cluster setup with an Access Server cluster setup, they are two different things. You need both a cluster database setup and an Access Server cluster setup with multiple nodes to be fully fault-tolerant.

  • Log on to the Amazon AWS console.
  • Under Services look under the Database header for the RDS option (Managed Relational Database Service).
  • Click Create database and select the engine of choice: Amazon Aurora, MySQL, or MariaDB.
  • Default options are usually fine here, click the Next button to continue.
  • Specify instance size and fault-tolerance settings – this depends entirely on how heavily you intend to use the system.
  • Specify the DB instance identifier, the Master username, and the Master password (twice).
  • Take note of the username and password as you will need them later, and click the Next button to continue.
  • Select which VPC and subnet this should be launched on. You can choose if it should be publicly reachable or not.
  • It is not necessary to create a default database here. Review the other settings to set them as you prefer, defaults are usually fine.
  • Complete the launch and then find your new Amazon RDS database in the Instances or Clusters overview.
  • Wait until the status shows that it is Available before attempting to use it.
  • Find the Cluster endpoint if you are using a cluster, or just the instance’s Endpoint when it’s just an instance.
  • Take note of the endpoint name, you will need it together with the Master username and the Master password later.

This should result in an Amazon RDS instance or cluster that is still empty right now, but is capable of storing database information. A logical next step is to install one OpenVPN Access Server from scratch, or take an existing Access Server setup, and updating it to the latest official OpenVPN Access Server release available on our software packages download page. Then the required MySQL client software can be installed, and a connection to this RDS system can be setup and tested. You are then ready to convert the database from local SQLite3 storage to the Amazon RDS database, and the cluster function can then be enabled, and additional Access Server nodes added.

Please keep in mind that Amazon RDS databases are protected by security groups. These are an Amazon specific security system that functions like a firewall. It is therefore necessary to adjust the security group settings so that your Access Server nodes that you intend to connect to this Amazon RDS database can actually reach this database.
Our example RDS database has these 3 important items that we will need later:

  • Endpoint: astest-cluster.cluster-cw1zos4nytdr.us-west-1.rds.amazonaws.com
  • Master username: adminuser
  • Master password: Fw4MjHs5KPjScCkz7Yxyp5YKNh
    (this is just some made-up example)

Initial setup of first Access Server cluster node

At this moment, version 2.7.4 which contains the clustering functionality, is available on our website as software package but we also offer images for, ESXi, HyperV, Microsoft Azure, Google Cloud Platform, and DigitalOcean. Amazon AWS version 2.7.4 is on its way and is expected to go live in August 2019. If you want to use it on AWS now, you can deploy version 2.6.1 and update it to 2.7.4 with the software package on our website.

Our images are based on Ubuntu18 x64. You may also choose to set up your own Linux system from scratch. The available installation options are listed here:

When you choose to use an existing Access Server that you are already using now, and that already has users and settings configured, you can upgrade it to the latest version with the instructions below, and the licenses and users and settings will all be retained. As mentioned earlier we do advise making a backup first. The new version of Access Server with version number 2.7.4 comes with a new optional cluster feature added that does not necessarily have to be used. But in this guide of course we are going to explain how to enable and use that feature.

If the image you have launched is not version 2.7.4, or you install our software on your own self-installed OS, then in order to upgrade the Access Server to this new release, you will need to download the Access Server package from the page above and place it somewhere on the intended server host. You can do this via a roundabout way by using your desktop computer to download the installation package from our website, and then uploading it using a tool such as SCP or WinSCP. But an easier method is to use wget, which is a tool designed to retrieve files directly from the Internet and save it directly on the file system of the Linux operating system where you are upgrading the OpenVPN Access Server program.

You can right-click the download link and select “Copy Link Address" or “Copy target" or such. The exact wording depends on the browser used. The goal is having the link to the installation package in your copy/paste buffer. Next go to the Linux server where you want to install the OpenVPN Access Server program and use wget to download the installation package file directly to the server.

Type wget followed by the pasted URL:

wget <paste copied url>

For example for Ubuntu 18 x64 installation package:

wget https://openvpn.net/downloads/openvpn-as-latest-ubuntu18.amd_64.deb

Note: if you get a certificate mismatch warning we suggest you use the –no-check-certificate flag to force the download.

Optional step for advanced users: You can compare the downloaded file with the SHA256sum hash mentioned on the software package overview page. Use command line “sha256sum openvpn-as-x.x.x-Ubuntu18.amd64.deb" to generate the hash, and compare it to what is listed on the site. If they match you can be certain that you have the right file and it has downloaded correctly.

Now that the installation package file is downloaded to your system you can perform the upgrade with the following command:

Upgrade installation package on Debian/Ubuntu system:

dpkg -i openvpn-as-latest-Ubuntu18.amd64.deb

Upgrade installation package on RedHat/CentOS/Fedora system:

rpm -Uvh openvpn-as-latest-CentOS7.x86_64.rpm

The upgrade process should then commence and finish. Afterwards you should reboot:

reboot

You should now have an Access Server that runs our latest release version, and you should be able to log in to the new Admin UI and see the new functions in the menu for clustering.

Firewall configuration – ports to open

These are the ports that need to be open. Most of the cloud images and appliances we offer have these ports open already by default. Amazon AWS is an exception, port 945 isn’t open by default there, so this may need to be opened in the security group settings.

These ports mentioned below assume standard configuration. If you know you have changed your ports then please adjust as necessary.

  • TCP 22 – for SSH access
  • TCP 443 – for web interface access, and OpenVPN TCP connections
  • TCP 943 – for web interface access
  • TCP 945 – for cluster control channel  <- this port is new and might not be open on existing installations yet, so be sure to open it.
  • UDP 1194 – for OpenVPN UDP connections

Configure and test database server connection

From the steps in the section about setting up Amazon RDS we will take the endpoint name, the Master username, and the Master password, and use them to test if this connection works. When access Server can make a connection to the database, you can continue with the steps to setup a new cluster. To reiterate, these are the example settings we are using in this guide:

  • Endpoint: astest-cluster.cluster-cw1zos4nytdr.us-west-1.rds.amazonaws.com
  • Master username: adminuser
  • Master password: Fw4MjHs5KPjScCkz7Yxyp5YKNh
    (this is just a made-up

As usual for all our command line tools documentation we are assuming you are logged on as root user and are in the /usr/local/openvpn_as/scripts/ folder. These instructions are for Ubuntu/Debian but by replacing apt-get with yum you should be able to achieve the correct results on CentOS and Red Hat.

Install the required software:

apt-get update
apt-get install mysql-client libmysqlclient-dev

Next, connect to the RDS instance with MySQL command line tool:

mysql -h astest-cluster.cluster-cw1zos4nytdr.us-west-1.rds.amazonaws.com -u adminuser -p

When asked for a password, provide the Master password.
You should be seeing a message like this, and you can exit with the exit command:

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 13
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> exit
Bye

This system is now able to make a successful connection to the Amazon RDS database. You can now proceed to the next step.

Setup a new cluster

Log on to the Admin UI of the Access Server, and go to Configuration and then go to Cluster. You will see a page where you can select to Setup a New Cluster. For our initial Access Server node, this is what we’ll choose. For any future nodes you want to add to an existing cluster setup, you can choose Join existing cluster instead.

Note: there is a check in Access Server that tries to verify that the required MySQL library is installed in the operating system. On Ubuntu systems this is called libmyselfclient-dev and can be installed with the following command:

apt-get install mysql-client libmysqlclient-dev

If you see that this check fails with the message “AS has detected that MySQL Client is not installed on this machine" and doesn’t let you continue, but you’re sure the required libraries are installed, you can override this check by adding this line into the file as.conf and restarting the Access Server and trying again. To do this follow the steps below:

Open as.conf in nano text editor:

nano /usr/local/openvpn_as/etc/as.conf

At the bottom add this:

OVERRIDE_LIBMYSQLCLIENT_ASSERT=1

Press ctrl+x and press enter to save settings and exit the file.

Then restart the Access Server service:

service openvpnas restart

And try again.

If your database backend is on MySQL already, then you can just click Save to confirm that you want to convert your current Access Server node to a cluster node. It will then restart and you’re ready to start using your cluster setup. However, if your database backend is the default SQLite3, and not yet on the MySQL type database system, then you will see additional fields where you can enter the following information to do the conversion to MySQL type database. In the following steps we will assume you have yet to convert to a MySQL type database.

During this step of setting up a new cluster, any existing user certificates and settings will be converted and placed into the cluster configuration. This conversion can only be done once, so it’s not possible to repeat this to combine multiple different Access Servers into one cluster. The first Access Server you use to setup a cluster will be the master data set and any Access Server nodes that you add to the cluster later on will use that master data set, and changes made afterward to user configurations in a cluster will apply for all the nodes in the cluster.

  • MySQL Username: enter the Master username here.
  • MySQL Password: enter the Master password here.
  • Hostname or IP: enter the endpoint name here.
  • MySQL port: the default is port 3306.

Press the Save button to convert the local SQLite3 databases to the new MySQL type databases. The Access Server will now take a short while to convert things. If your user base is very large, it may take some time for this to complete fully. Once it is done, a restart of the Access Server will be done automatically. Once ready you will be back at the Admin UI login prompt, and you’re ready to start using your cluster setup.

Round robin DNS

We advise that this cluster setup is used in combination with a round robin DNS record. This is basically a DNS name like vpn.yourcompany.com that contains multiple A records. Each A record points to one of the servers in your cluster. When a client connects, it will resolve to one of those nodes, and connect to it. If that node becomes unavailable for whatever reason, it will try the next node automatically. We have made adjustments in the server and the client software to accommodate automatic switching to the next available server node. If a node becomes defunct, you can remove it from the DNS record. We advise that you keep the TTL on such records low, so that it picks up on changes to the DNS records more quickly.

The web interface of the Access Server has an option in the cluster section to have new nodes that join the cluster automatically configure themselves to provide client connection profiles with that cluster-wide round robin DNS name, and also an option to set a single round-robin DNS name on all nodes currently in the cluster setup in one go. It is necessary for each individual node to know about this round robin DNS name, because each node individually is capable of generating client connection profiles, and these contain the connection endpoint address that a VPN client will need to know to start a connection. If it points to only a particular node (with a non round robin hostname or IP address) in the cluster, then when that node goes down, clients will not be able to establish a connection with the other nodes, as they will try to connect to this particular node (with non Round Robin hostname or IP address) only. This is why round robin DNS is recommended for this type of setup.

Adding more nodes to the cluster

You can set up a new Access Server and install the latest Access Server build on it, and log on the Admin UI. Then you go to Cluster and you select Join existing cluster. You can then enter the database connection details and join the cluster. It is important to note that if this is an Access Server with existing users and configuration, then this node that you are adding will be wiped clean (but backups are made automatically just in case) and will get its user certificates and other information from the cluster instead.

Share