One of the features in OpenVPN Access Server is to create a cluster of Access Servers for the purpose of high availability and increased load capacity. Such a cluster is an answer to the requirement that our customers have expressed for a high-availability solution and it also provides the ability to spread the load across multiple servers. For this mode it is recommended to use the subscription mode to license your cluster nodes. There are still improvements coming to the clustering mode.
A single OpenVPN Access Server contains everything it needs to offer its services to connecting VPN clients. All the user credentials, certificates, services, and access rules, can all be present on one Access Server. This is also a single point of failure. To resolve this we used to rely in the past on our LAN-model UCARP/VRRP based failover system to run a standby server that can take over if the primary fails. However, that model is limited to running on a local network that supports VRRP/UCARP traffic to pass through. Some networks don’t allow this traffic. Amazon AWS for example doesn’t, so this model doesn’t work there. The cluster feature works on Amazon AWS and many other platforms as it doesn’t rely on this VRRP/UCARP type traffic.
In the cluster feature, we separate the storage of certificates and credentials from the OpenVPN service daemons. Our cluster solution allows you to use a MySQL type database system (MySQL, MariaDB, Amazon RDS, for example) to store all the configuration of the Access Server. These database systems can be single servers or clusters with high-availability. Amazon RDS for example can offer a fault-tolerant solution that automatically switches over to a backup server if the primary server fails, and is the recommended solutions at this time. Multiple Access Servers in clustering mode can attach to such a central database and offer VPN services. A DNS-based round-robin system can ensure that VPN clients will have a single address to connect to, like vpn.example.com, and this resolves at random to any of the nodes in your cluster setup. A VPN will try these different servers in semi-random order. If an OpenVPN Access Server server fails, it will lead to a temporary connection failure for the connected client, until the client automatically tries to reconnect, and then ends up connecting to one of the other servers in the cluster. The failed server could then either be repaired, or it can be removed from the DNS records and a new Access Server could be setup, attached to the cluster, and then added to the round-robin DNS record to replace the failed node.
This is a first phase implementation. It is primarily aimed at Amazon AWS, where with tiered instances on Amazon AWS adding a new node can even be automated, and it can then be automatically licensed. A new, more flexible licensing system that allows having one license shared across multiple nodes is now available for our customers on the Access Server portal. We will be continuing to improve the clustering feature to make deployment and maintenance easier.
Some important notes
Most of our customers will currently be using the default SQLite3 database files to store all the configuration of a single OpenVPN Access Server installation. When you install the Access Server from scratch, it will still default to SQLite3, even with the new versions of Access Server with clustering support built-in. You can upgrade an existing Access Server installation with the latest version and it will continue to function as before, and retain all your settings. The cluster function is an on/off toggle that is not enabled by default, and with the cluster function switched off, it will just be a normal standalone Access Server as with previous versions. But of course now with an updated Admin UI interface with all of the familiar functions still present. Only when you go into the Admin UI and enable the cluster function, do the new functions and settings for clustering mode come into play.
The cluster function requires that you migrate your settings to a MySQL type database such as MySQL, MariaDB, or Amazon RDS. In theory, as long as it is a MySQL 5.6 or higher compatible system, it should work. Amazon Aurora available as an engine for an Amazon RDS database is such a compatible system. With previous Access Server releases, the conversion from SQLite3 local storage to a MySQL type system was already possible, and back then it had to be done on the command line manually. We now have a conversion tool built right into the Admin UI to handle that part. You do still need to have some database client support software installed for the connection to be made, and you do need to still set up the new database system yourself first. In our guide we are going to be assuming a situation where Amazon RDS is used, so this means we’re doing this cluster setup on Amazon AWS, but a MySQL or MariaDB setup can also be used for the database storage, and that means it’s not tied to Amazon AWS. To eliminate a single point of failure we do advise using a fault-tolerant setup for the database system, whatever solution you choose to use.
The conversion process from local SQLite3 to MySQL type database system currently has few safety checks in place. What this means is that if you convert an Access Server configuration to MySQL type database system, and then repeat that same act again from another Access Server, you will likely wipe the original settings, and the latest converted settings will be the only valid settings. So please do not repeat a database conversion to the same target MySQL database system. We will improve this behavior in future releases.
Before you begin
If you intend to upgrade an existing Access Server installation to the new cluster-ready version, then backup your settings first. You can use the backup commands on the command line found here to make a complete backup of the settings on your Access Server installation safely without having to stop operations on your server. If you’ve never made a backup before, now would be a good time to remind you that it would be a good idea to set up an automated backup plan. The information stored in the Access Server is unique and cannot be replaced, unless you wish to reinstall all your existing clients. Backups can prevent that scenario.
Setting up Amazon RDS
It is important to note here that you are not required to use Amazon RDS, but it is recommended, since it is fault-tolerant. Another MySQL or MariaDB system also works, and this also means you are not tied to Amazon AWS. We just assume this is the easiest setup for our customers and serves as a general guide on how to set up an OpenVPN Access Server cluster. The term cluster is also used in database systems that are fault-tolerant. Please do not confuse a database cluster setup with an Access Server cluster setup, they are two different things. You need both a cluster database setup and an Access Server cluster setup with multiple nodes to be fully fault-tolerant.
- Log on to the Amazon AWS console.
- Under Services look under the Database header for the RDS option (Managed Relational Database Service).
- Click Create database and select the engine of choice: Amazon Aurora, MySQL, or MariaDB.
- Default options are usually fine here, click the Next button to continue.
- Specify instance size and fault-tolerance settings – this depends entirely on how heavily you intend to use the system.
- Specify the DB instance identifier, the Master username, and the Master password (twice).
- Take note of the username and password as you will need them later, and click the Next button to continue.
- Select which VPC and subnet this should be launched on. You can choose if it should be publicly reachable or not.
- It is not necessary to create a default database here. Review the other settings to set them as you prefer, defaults are usually fine.
- Complete the launch and then find your new Amazon RDS database in the Instances or Clusters overview.
- Wait until the status shows that it is Available before attempting to use it.
- Find the Cluster endpoint if you are using a cluster, or just the instance’s Endpoint when it’s just an instance.
- Take note of the endpoint name, you will need it together with the Master username and the Master password later.
This should result in an Amazon RDS instance or cluster that is still empty right now, but is capable of storing database information. A logical next step is to install one OpenVPN Access Server from scratch, or take an existing Access Server setup, and updating it to the latest official OpenVPN Access Server release available on our software packages download page. Then the required MySQL client software can be installed, and a connection to this RDS system can be setup and tested. You are then ready to convert the database from local SQLite3 storage to the Amazon RDS database, and the cluster function can then be enabled, and additional Access Server nodes added.
Please keep in mind that Amazon RDS databases are protected by security groups. These are an Amazon specific security system that functions like a firewall. It is therefore necessary to adjust the security group settings so that your Access Server nodes that you intend to connect to this Amazon RDS database can actually reach this database.
Our example RDS database has these 3 important items that we will need later:
- Endpoint: astest-cluster.cluster-cw1zos4nytdr.us-west-1.rds.amazonaws.com
- Master username: adminuser
- Master password: Fw4MjHs5KPjScCkz7Yxyp5YKNh
(this is just some made-up example)
Alternatively, set up your own MySQL server
Amazon RDS has the advantage it has options to automatically backup the databases regularly and to very easily implement an automatic failover, so that the database server is also high-availability. We therefore recommend that Amazon AWS RDS is used, especially when using OpenVPN Access Server on Amazon AWS. But it is possible to use your own MySQL server too. And it is possible to add high-availability there as well, if you put in the time and effort to do this properly.
Install the MySQL server software on your server:
apt update apt install mysql-server
During installation you may be asked to setup a root password for MySQL server. In some situations you won’t be asked and then the root account can access the MySQL server from localhost without password. You can then refine by setting a password on the root account as per documentation of the MySQL server. By default, MySQL server only runs on localhost and won’t let users from outside log in. However if you run an OpenVPN Access Server cluster, you will need to allow other machines to log in to this MySQL server from remote addresses. To set up an account that can log in remotely, and to have MySQL listen on all network interfaces so it can be reached, you can reference the simple instructions below. Experienced users may note that you can lock things down further in terms of security.
Login to the MySQL server:
mysql -u root -p
Grant privileges to the ‘root’ user account from ‘any’ network address:
grant all privileges on *.* to 'root'@'%' identified by 'PASSWORDGOESHERE';
Obviously, replace PASSWORDGOESHERE with a secure password that you will use for OpenVPN Access Server to access this MySQL server over the network. Please also note that granting access from the network may be different depending on your MySQL server type and version. Refer to the documentation of your MySQL type server if you have problems with the above command, to learn how to do it properly on your particular version of MySQL type server.
Now edit your MySQL server listening address:
Locate the line that starts with bind-address (or add it if it doesn’t exist) and change it to this:
bind-address = 0.0.0.0
Press ctrl+x, followed by y, to exit and save the file.
Now restart the MySQL server service for the changes to take effect:
service mysql restart
You should now have a MySQL server up and running that is ready to set up your Access Server cluster with. As mentioned, there are ways to do security better by for example using a non-root account instead, and after initial cluster creation, you can remove access rights and have them set to only enough rights to read, insert, and change records in existing databases.
Initial setup of first Access Server cluster node
The OpenVPN Access Server since version 2.7.4 supports the clustering capability. So if you have anything older that you wish to enable clustering on, you should upgrade to the latest version first. All our offerings from the software repository, our software appliances, and our cloud images, support clustering. For upgrading, it is recommended to upgrade the cluster nodes to the same version, preferably the latest.
Our images are based on Ubuntu18 x64. You may also choose to set up your own Linux system from scratch. The available installation options are listed here:
And if you have an existing Access Server that you want to use, we recommend upgrading it to the latest version first before starting to set up your cluster:
Firewall configuration – ports to open
These are the ports that need to be open. Most of the cloud images and appliances we offer have these ports open already by default. Amazon AWS is an exception, port 945 isn’t open by default there, so this may need to be opened in the security group settings.
These ports mentioned below assume standard configuration. If you know you have changed your ports then please adjust as necessary.
- TCP 22 – for SSH access
- TCP 443 – for web interface access, and OpenVPN TCP connections
- TCP 943 – for web interface access
- TCP 945 – for cluster control channel <- this port is new and might not be open on existing installations yet, so be sure to open it.
- UDP 1194 – for OpenVPN UDP connections
Configure and test database server connection
From the steps in the section about setting up Amazon RDS we will take the endpoint name, the Master username, and the Master password, and use them to test if this connection works. When access Server can make a connection to the database, you can continue with the steps to setup a new cluster. To reiterate, these are the example settings we are using in this guide:
- Endpoint: astest-cluster.cluster-cw1zos4nytdr.us-west-1.rds.amazonaws.com
- Master username: adminuser
- Master password: Fw4MjHs5KPjScCkz7Yxyp5YKNh
(this is just a made-up
As usual for all our command line tools documentation we are assuming you are logged on as root user and are in the /usr/local/openvpn_as/scripts/ folder. These instructions are for Ubuntu/Debian but by replacing apt with yum you should be able to achieve the correct results on CentOS and Red Hat.
Install the required software:
apt update apt install mysql-client
Next, connect to the RDS instance with MySQL command line tool:
mysql -h astest-cluster.cluster-cw1zos4nytdr.us-west-1.rds.amazonaws.com -u adminuser -p
When asked for a password, provide the Master password.
You should be seeing a message like this, and you can exit with the exit command:
Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 13 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> exit Bye
This system is now able to make a successful connection to the Amazon RDS database. You can now proceed to the next step.
Setup a new cluster
Log on to the Admin UI of the Access Server, and go to Configuration and then go to Cluster. You will see a page where you can select to Setup a New Cluster. For our initial Access Server node, this is what we’ll choose. For any future nodes you want to add to an existing cluster setup, you can choose Join existing cluster instead.
If your database backend is on MySQL already, then you can just click Save to confirm that you want to convert your current Access Server node to a cluster node. It will then restart and you’re ready to start using your cluster setup. However, if your database backend is the default SQLite3, and not yet on the MySQL type database system, then you will see additional fields where you can enter the following information to do the conversion to MySQL type database. In the following steps we will assume you have yet to convert to a MySQL type database.
During this step of setting up a new cluster, any existing user certificates and settings will be converted and placed into the cluster configuration. This conversion can only be done once, so it’s not possible to repeat this to combine multiple different Access Servers into one cluster. The first Access Server you use to setup a cluster will be the master data set and any Access Server nodes that you add to the cluster later on will use that master data set, and changes made afterward to user configurations in a cluster will apply for all the nodes in the cluster.
- MySQL Username: enter the Master username here.
- MySQL Password: enter the Master password here.
- Hostname or IP: enter the endpoint name here.
- MySQL port: the default is port 3306.
Press the Save button to convert the local SQLite3 databases to the new MySQL type databases. The Access Server will now take a short while to convert things. If your user base is very large, it may take some time for this to complete fully. Once it is done, a restart of the Access Server will be done automatically. Once ready you will be back at the Admin UI login prompt, and you’re ready to start using your cluster setup.
Round robin DNS
We advise that this cluster setup is used in combination with a round robin DNS record. This is basically a DNS name like vpn.yourcompany.com that contains multiple A records. Each A record points to one of the servers in your cluster. When a client connects, it will resolve to one of those nodes, and connect to it. If that node becomes unavailable for whatever reason, it will try the next node automatically. We have made adjustments in the server and the client software to accommodate automatic switching to the next available server node. If a node becomes defunct, you can remove it from the DNS record. We advise that you keep the TTL on such records low, so that it picks up on changes to the DNS records more quickly.
The web interface of the Access Server has an option in the cluster section to have new nodes that join the cluster automatically configure themselves to provide client connection profiles with that cluster-wide round robin DNS name, and also an option to set a single round-robin DNS name on all nodes currently in the cluster setup in one go. It is necessary for each individual node to know about this round robin DNS name, because each node individually is capable of generating client connection profiles, and these contain the connection endpoint address that a VPN client will need to know to start a connection. If it points to only a particular node (with a non round robin hostname or IP address) in the cluster, then when that node goes down, clients will not be able to establish a connection with the other nodes, as they will try to connect to this particular node (with non Round Robin hostname or IP address) only. This is why round robin DNS is recommended for this type of setup.
Automatic failover and geolocation routing
If you set up a cluster with a round robin DNS record, then automatically the VPN clients will try each server IP address in this record in a semi-random order until it is able to connect. That at least ensures that if a node goes down, the VPN client can reconnect to another node and get connected again. But if you want it to be smarter and also do a healthcheck so that when a node goes down, it is automatically removed from the DNS record, and added again when it comes back online, then you need something smarter to manage your DNS. Those smarter DNS solutions often also support geolocation routing, which tries to give you a server IP of a server close to you. This requires some additional setup.
If you wish to do geolocation based DNS resolution, and automatic failover in that as well, you would have to use some service to intelligently manage your DNS records, like Amazon Route 53, which is capable of doing healthchecks on servers, and based on that, adjust DNS records to remove dead servers and bring them back automatically when they come back up. You can also configure geolocation routing, by which the DNS server tries to determine what area you are in and based on that give you an IP of a server that is in your region. This detection may not always work and in that case a default record can be specified to ensure that at least some server is reached.
See these documents for more information:
If you do not wish to use Amazon AWS, there may be other service providers that do a similar thing. This is not a feature that only Amazon AWS has, but it is also not a standard feature in DNS. Access Server itself does not manage your DNS for you. The DNS system is where the resolution to and selection of the IP address occurs that the VPN clients will be connecting to, and it is here where you can implement such intelligent selection of which server to connect to.
Adding more nodes to the cluster
You can set up a new Access Server and install the latest Access Server build on it, and log on the Admin UI. Then you go to Cluster and you select Join existing cluster. You can then enter the database connection details and join the cluster. It is important to note that if this is an Access Server with existing users and configuration, then this node that you are adding will be wiped clean (but backups are made automatically just in case) and will get its user certificates and other information from the cluster instead.