Tutorial: Test and Troubleshoot Access Server Cluster Connectivity

Abstract

Learn how to test and troubleshoot connectivity in an Access Server cluster — covers verifying client connections, testing inter-node communication, and executing remote API calls on cluster members using the Web API and sacli.

Overview

This tutorial shows you how to test and troubleshoot connectivity in an Access Server cluster.

You'll verify:

Connectivity from an administrator's device to each cluster node.
Web service availability on each cluster node.
Inter-node connectivity and inter-node communication.
Cluster database and cluster node status.
Web API access from a remote client.
Local and remote API calls between cluster nodes.

These tests can help identify networking, firewall, Web API, and cluster configuration issues that could affect cluster operation. To learn more about the clustering feature, refer to Access Server Clustering.

Prerequisites

Access Server 3.1.0 or newer (for using the Web API examples in this tutorial).
A configured Access Server cluster.
Root access to the console of each cluster node.
A configured MySQL database for cluster storage.
Network access between the cluster nodes.
(Optional) A remote machine for running API calls.

Refer to Cluster Setup if you haven't configured the cluster yet or need to verify its configuration.

Important

The cluster API specification can change between Access Server releases. Verify the API documentation for your installed version before running the Web API examples.

Example environment

The examples in this tutorial use the following two-node cluster:

Setting	Example value
Round-robin DNS hostname	`rrdns-nodes.example.com`
First cluster node	`Node1`
First node hostname	`node1.example.com`
First node IP address	`198.51.100.5`
Second cluster node	`Node2`
Second node hostname	`node2.example.com`
Second node IP address	`198.51.100.6`
Default web service port	TCP `943`

Replace these values with the hostnames, IP addresses, ports, and cluster node names for your environment.

Note

In our documentation, we use example IPv4 addresses and subnets reserved for documentation, such as 192.0.2.0/24, 198.51.100.0/24, and 203.0.113.0/24.

Ensure you replace them with valid IPv4 addresses and subnets for your network(s).

Placeholder values

Placeholder	Description
`<NODE_NAME>`	Cluster node name
`<ACCESS_SERVER_IP>`	Cluster node IP address
`<ACCESS_SERVER_DOMAIN>`	Cluster node hostname
`<RRDNS_DOMAIN>`	Round-robin DNS hostname
`<ACCESS_SERVER_PORT>`	Access Server web service port
`<AUTH_TOKEN>`	Web API authentication token

All command-line examples assume you've connected to an Access Server console with root privileges unless stated otherwise.

Step 1: Test administrator connectivity to each node

First, confirm that your computer can reach the Admin Web UI on each cluster node. These tests validate administrative web connectivity and are separate from VPN client connectivity.

Access Server uses TCP port 943 for web services by default. Replace the port in the examples if your deployment uses a different value.

Test connectivity from a browser

Open the Admin Web UI using the round-robin DNS hostname:
```
https://rrdns-nodes.example.com:943/admin/login
```

Open the Admin Web UI using the hostname of each cluster node:

https://node1.example.com:943/admin/login

https://node2.example.com:943/admin/login

Open the Admin Web UI using the IP address of each cluster node:

https://198.51.100.5:943/admin/login

https://198.51.100.6:943/admin/login

Confirm that the Admin Web UI loads for each address.
If a page doesn't load, open your browser's developer tools and select the Network tab. Retry the request, then review the request headers, response, status code, and connection errors.

Tip

When contacting OpenVPN Support, include screenshots or recordings showing the failed request and the open developer tools.

Test connectivity with curl

Run the following commands from your computer.

Test the round-robin DNS hostname:

curl -vvk https://rrdns-nodes.example.com:943/admin/login

Test each cluster node hostname:

curl -vvk https://node1.example.com:943/admin/login

curl -vvk https://node2.example.com:943/admin/login

Test each cluster node IP address:
```
curl -vvk https://198.51.100.5:943/admin/login
```
```
curl -vvk https://198.51.100.6:943/admin/login
```
- A successful connection confirms that your computer can reach the web service on the specified cluster node. The response may still return an HTTP authentication or login response, which is expected.

Step 2: Verify the web service on each cluster node

Confirm that each cluster node is configured to use the expected Admin Web UI port and is listening for connections on that port.

Access Server includes sacli, its command-line administration utility. For an introduction to the tool and its syntax, refer to Tutorial: An Intro to the sacli Command-Line Utility.

Connect to the console and get root privileges.

Verify the Admin Web UI port configured:

sacli ConfigQuery | grep admin_ui.https.port

Example output:
```
"admin_ui.https.port": "943"
```

Verify that Access Server is listening on the configured port:

netstat -tulpn | grep 943

Example output:

tcp        0      0 0.0.0.0:943             0.0.0.0:*               LISTEN      699419/python3

If netstat isn't installed, install the net-tools package:
- Ubuntu/Debian:
```
apt install net-tools
```
- RHEL:
```
dnf install net-tools
```

Verify that iptables/nftables rules exist for that Admin Web UI port.

If the host uses iptables, run:

iptables-save -c | grep '943'

Example output:

[6:336] -A INPUT -p tcp -m state --state NEW -m tcp --dport 943 -j AS0_WEBACCEPT

If the host uses nftables, run:

nft -a list chain ip as_filter INPUT | grep 943

Example output:

counter packets 91448 bytes 4634926 ct state new tcp dport 943 jump AS_WEBACCEPT # handle 63

Repeat these checks on every cluster node.

Step 3: Test inter-node connectivity and communication

Access Server 3.0.0 and newer use the Admin Web UI port for cluster communication. The default is TCP 943. Earlier releases used TCP 945.

Verify that traffic to the configured web service port is allowed through any host firewall, cloud security group, network firewall, or router between the cluster nodes.

Verify the Admin Web UI port configured:

sacli ConfigQuery | grep admin_ui.https.port

Example output:
```
"admin_ui.https.port": "943"
```

Run the following on either cluster node to verify that the api_endpoint has the Admin Web UI port:

sacli --method GET --url 'api/cluster/nodes/list' --value '{}' apicall

Example output:

{
  "nodes": {
    "Node1": {
      "api_endpoint": "https://198.51.100.5:943",
      "certificate_fingerprint": "50:05:D6:08:98:81:16:A1:BE:CC:CD:61:3D:DC:AD:95:A0:8F:1D:72:32:A5:19:2E:99:D1:0D:DD:76:10:98:32",
      "name": "Node1",
      "restart_pending": false
    },
    "Node2": {
      "api_endpoint": "https://198.51.100.6:943",
      "certificate_fingerprint": "50:05:D6:08:98:81:16:A1:BE:CC:CD:61:3D:DC:AD:95:A0:8F:1D:72:32:A5:19:2E:99:D1:0D:DD:76:10:98:32",
      "name": "Node2",
      "restart_pending": false
    }
  }
}

Confirm that each api_endpoint contains the correct IP address or hostname and web service port for that cluster node.

Test TCP connectivity between the nodes:

From Node1, test connectivity to Node2:
```
nc -z -v 198.51.100.6 943
```

Example output:

Connection to 198.51.100.6 943 port [tcp/*] succeeded!

From Node2, test connectivity to Node1:
```
nc -z -v 198.51.100.5 943
```

Example output:

Connection to 198.51.100.5 943 port [tcp/*] succeeded!

Ensure both tests succeed for reliable inter-node communication.

If an API endpoint contains an incorrect address or port, update it with the following remote API call:

sacli --method POST --url 'api/cluster/<NODE_NAME>/set' --value '{"api_endpoint": "https://<ACCESS_SERVER_IP>:<ACCESS_SERVER_PORT>"}' apicall

Restart the Access Server services:

sacli --method POST --url 'api/server/restart?restartMode=warm' --value '' apicall

Repeat the node list and TCP connectivity tests after the restart.

Step 4: Verify cluster node and ClusterDB status

Use sacli to confirm that all cluster nodes are registered and reachable.

View the configured cluster nodes:

sacli ClusterQuery

Example output:

{
  "Node1": {
    "api_endpoint": "https://198.51.100.5:943",
    "certificate_fingerprint": "58:DF:12:B8:D1:A7:D4:F2:77:23:9F:5E:D6:69:B2:55:10:5F:6B:BF:6E:6D:CC:E9:1B:92:93:30:28:9B:C0:A6",
    "name": "Node1"
  },
  "Node2": {
    "api_endpoint": "https://198.51.100.6:943",
    "certificate_fingerprint": "58:DF:12:B8:D1:A7:D4:F2:77:23:9F:5E:D6:69:B2:55:10:5F:6B:BF:6E:6D:CC:E9:1B:92:93:30:28:9B:C0:A6",
    "name": "Node2"
  }
}

Confirm that every expected cluster member appears with the correct API endpoint.

View the runtime status of the cluster nodes:

sacli ClusterNodesStatus

Example output:

{
  "Node1": {
    "activation_type": "Not Activated",
    "api_endpoint": "https://198.51.100.5:943",
    "as_build": "e22fe316",
    "as_version": "3.1.0 (e22fe316)",
    "auth": "local",
    "certificate_fingerprint": "58:DF:12:B8:D1:A7:D4:F2:77:23:9F:5E:D6:69:B2:55:10:5F:6B:BF:6E:6D:CC:E9:1B:92:93:30:28:9B:C0:A6",
    "cipher": "AES-256-CBC",
    "current_users": 0,
    "license": "2",
    "name": "Node1",
    "osi_layer": "3 (routing/NAT)",
    "restart_pending": false,
    "server_name": "198.51.100.5",
    "status": "reachable",
    "user_data": {
      "n_clients": 0,
      "ovpn_dco_available": true,
      "ovpn_dco_ver": "ovpn-dco version 0.2.20251017-2+noble"
    },
    "vlis": "all interfaces",
    "vpn_port": "tcp/443, udp/1194",
    "vserv_priv_access": "NAT"
  }
}

Confirm that each cluster node reports:
- The expected API endpoint.
- The expected Access Server version.
- "status": "reachable".
- No unexpected pending restart.
- The expected VPN ports and routing mode.
Tip
A node that reports as unreachable may have an incorrect API endpoint, a blocked port, an unavailable web service, or a certificate-related communication problem.

Step 5: Test remote Web API calls to each cluster node

Use this procedure to confirm that a remote client can authenticate to the Web API and call API endpoints on an individual cluster node.

Generate an authentication token for an admin user using cURL (ensure the account has administrative privileges):

curl -X 'POST' 'https://198.51.100.5:943/api/auth/login/userpassword' -H 'accept: */*' -H 'Content-Type: application/json' -d '{"request_admin": true, "username": "<ADMIN_USERNAME>","password": "<ADMIN_PASSWORD>"}' -k

Example output:

{
  "auth_token": "<AUTH_TOKEN>",
  "expires_after": "2025-10-23T21:00:23.000000Z",
  "renewable_until": "2025-10-24T00:50:23.000000Z",
  "user_properties": {
    "user_type": "admin",
    "username": "openvpn"
  }
}

Important

Treat the authentication token like a password. Don't store it in an insecure location or include it in screenshots, logs, or support requests.

Use the token to call the following endpoints on each cluster node:

curl -k -v "https://<ACCESS_SERVER_IP>:943/api/config-profile/list" -H "x-openvpn-as-authtoken: <AUTH_TOKEN>"

curl -k -v "https://<ACCESS_SERVER_IP>:943/api/server/config" -H "x-openvpn-as-authtoken: <AUTH_TOKEN>"

curl -k -v "https://<ACCESS_SERVER_IP>:943/api/license/info" -H "x-openvpn-as-authtoken: <AUTH_TOKEN>"

curl -k -v "https://<ACCESS_SERVER_IP>:943/api/helper/status-overview" -H "x-openvpn-as-authtoken: <AUTH_TOKEN>"

Repeat the calls using the IP address or hostname of each cluster node.
- Successful responses confirm that the remote client can reach and authenticate to the Web API on the tested node.

Step 6: Test local API calls on a cluster node

Use sacli apicall to call API endpoints locally on a cluster node.

Run the following commands to call API endpoints directly from a node:

sacli --method GET --url "api/config-profile/list" apicall

sacli --method GET --url 'api/server/config' apicall

sacli --method GET --url 'api/license/info' apicall

sacli --method POST --url 'api/helper/status-overview' apicall

Successful responses confirm that the local Access Server service can process the API calls.

Use ClusterRPC to route a remote API call through the cluster communication layer, even when targeting the node from which you run the command:

For example, from Node1, run:

sacli --node_name "Node1" --method GET --url '/api/server/info' ClusterRPC

sacli --node_name "Node1" --method GET --url '/api/server/config' ClusterRPC

sacli --node_name "Node1" --method GET --url '/api/license/info' ClusterRPC

sacli --node_name "Node1" --method POST --url '/api/helper/status-overview' ClusterRPC

These tests help distinguish a local Web API failure from a ClusterRPC or inter-node communication failure.

Step 7: Test remote API calls between cluster nodes

Use ClusterRPC to execute API calls on one cluster node from another cluster node.

Run the following commands from Node1 to execute API calls on Node2:

sacli --node_name "Node2" --method GET --url '/api/server/info' ClusterRPC

sacli --node_name "Node2" --method GET --url '/api/server/config' ClusterRPC

sacli --node_name "Node2" --method GET --url '/api/license/info' ClusterRPC

sacli --node_name "Node2" --method POST --url '/api/helper/status-overview' ClusterRPC

Repeat the test in the opposite direction by running the commands on Node2 and setting --node_name to Node1.
- Successful calls in both directions confirm that:
  - Each node can reach the other node's API endpoint.
  - Inter-node TLS communication succeeds.
  - ClusterRPC can route and process remote API calls.
  - The cluster communication path required for coordinated operations is functioning.

Step 8: Test VPN client connectivity

The previous steps verify administrative, Web API, and inter-node connectivity. They don’t confirm that VPN clients can connect successfully through the cluster.

To test VPN client connectivity:

Import a valid connection profile into OpenVPN Connect or another supported VPN client.
Connect using the round-robin DNS hostname.
Confirm that the VPN connection succeeds.
In the Admin Web UI, select Status → Active Connections.
Identify the cluster node handling the connection.
Disconnect and reconnect the VPN client several times.
Confirm that the client can connect successfully when directed to either cluster node.

These tests help verify that DNS distribution, VPN listener ports, certificates, authentication, and routing are working across the cluster.

Troubleshooting

Enable additional HTTP client debugging

Access Server includes debug flags that can provide more information about local and remote API calls.

To enable HTTP client debugging:

Connect to the console of the cluster node and get root privileges.
Add the following line to /usr/local/openvpn_as/etc/as.conf:
```
DEBUG_HTTP_CLIENT=1
```
You can append it with this command:
```
echo "DEBUG_HTTP_CLIENT=1" >> /usr/local/openvpn_as/etc/as.conf
```
Restart Access Server:
```
systemctl restart openvpnas
```
Repeat the apicall and ClusterRPC tests from Steps 6 and 7.
Review the command output and Access Server logs for additional connection details.
Important
Debug flags can generate large amounts of log data. Remove the debug flag after collecting the required information, then restart Access Server again.

To remove the flag:

sed -i '/^DEBUG_HTTP_CLIENT=1$/d' /usr/local/openvpn_as/etc/as.conf
systemctl restart openvpnas

Common symptoms

The Admin Web UI is unavailable on one node

Verify:

The configured Admin Web UI port.
That Access Server is listening on the port.
The host firewall rules.
Any cloud security groups or external firewalls.
DNS resolution for the node hostname.

The nodes are reachable from a client but not from each other

Verify:

Bidirectional TCP connectivity between the node IP addresses.
The api_endpoint value for each cluster node.
Internal firewall and router rules.
Whether the nodes are using addresses that are reachable from each other.

Local apicall succeeds but ClusterRPC fails

This generally indicates a problem in the cluster communication path rather than with the local Web API. Review:

The target node name.
The target node’s API endpoint.
Inter-node TCP connectivity.
Certificate fingerprints.
HTTP client debug output.

A cluster node reports "status": "unreachable"

Verify that:

The node is running.
Access Server services are active.
The API endpoint is correct.
The web service port is open.
The other cluster nodes can reach it.

In this section:

Tutorial: Test and Troubleshoot Access Server Cluster Connectivity

Overview

Prerequisites

Important

Example environment

Note

Placeholder values

Step 1: Test administrator connectivity to each node

Test connectivity from a browser

Tip

Test connectivity with curl

Step 2: Verify the web service on each cluster node

Step 3: Test inter-node connectivity and communication

Step 4: Verify cluster node and ClusterDB status

Tip

Step 5: Test remote Web API calls to each cluster node

Important

Step 6: Test local API calls on a cluster node

Step 7: Test remote API calls between cluster nodes

Step 8: Test VPN client connectivity

Troubleshooting

Enable additional HTTP client debugging

Important

Common symptoms

The Admin Web UI is unavailable on one node

The nodes are reachable from a client but not from each other

Local apicall succeeds but ClusterRPC fails

A cluster node reports "status": "unreachable"

Search results