Re: [Openvpn-users] Intermittently lost connections - affecting IP Phones

[Leonard Isham <leonard.isham@xxxxxxxxx>]

On 2/8/06, Cameron Gocke <livedrive777@xxxxxxxxx> wrote:
> On 2/7/06, James Yonan <jim@xxxxxxxxx> wrote:
> > Cameron Gocke wrote:
> > > Over the last couple of months I've had numerous reports of
> > > intermittent loss of connectivity.  This doesn't last for more than a
> > > few seconds, but it results in the users' dropping their IP phone
> > > connection.  At first I started troubleshooting the Inactivity timeout
> > > messages, but have all but eliminated those and the issue continues.
> > > (Aside from which I would assume that the IP phone would be sending
> > > traffic over the VPN and therefore keep it from timing out.)
> > >
> > > I am getting these messages:
> > > Wed Feb 01 15:21:58 2006 clientname/71.99.3.226:1142 TLS: tls_process:
> > > killed expiring key
> > >
> > > Almost exactly every hour on the client side, but am not losing
> > > connection nearly that often.
> > >
> > That's nothing to worry about -- it's the normal rekeying process.
> > > I've done some preliminary testing, and I don't see any evidence that
> > > their internet connectivity is dropping at any point.
> > >
> > > Does anyone have any suggestions for me?
> > >
> > Have you tried calibrating the keepalive parameters?  Use a larger
> > timeout (the second parameter) to decrease the number of client restarts
> > due to short-term network outages.
>
> Currently I've changed my keepalive settings to the following:
> keepalive 10 600
> I've definitely noticed a decreased number of client restarts, and
> given the information I've been able to gleen from othe posts I'm
> doubting they will ever disappear entirely.
>
> >
> > How do you characterize the network conditions at the time of the dropouts:
> >
> > (1) How many clients are connected to the OpenVPN server?
> I have about 80-90 concurrent sessions over this instance of OpenVPN
> server, I have three other instances running that have only 3 or 4
> connections on them currently.
>
> > (2) What percentage of the CPU is taken by the OpenVPN server?
> I've never seen the CPU spike over 3 or 4 % really
>
> > (3) Does the server log file show anything interesting at this point in
> > time?
>   At times I have seen the (WSAECONNRESET) (code=10054) messages at
> approx the same time as their disconnect, but difficult to say exactly
> and the message doesn't indicate the session information, so I don't
> know for sure that this is related.
>   Yesterday a person was disconencted and I could find no server log
> errors, but on his client log he had two replay-window backtrack
> occurred messages.  I haven't typically seen many of those so again
> don't neccesarily feel like I have a smoking gun there.
>   I was getting the messages on the server related to the client
> resets, and that is when I changed the keepalive settings.
>   The only other messages I've gotten are when I switched the client
> to connect over TCP instead of UDP.  After doing that with three
> clients only one of them has experienced a disconnect and that was
> where I got the message: MULTI: packet dropped due to output
> saturation (multi_process_incoming_tun).  On the whole the connection
> over TCP has been a good be more stable so far, but obviously slower
> and supposedly with more voice stuttering over the IP phone.

There might be clues here.

1. UDP has no packet delivery guarantee.

2. TCP will retransmit dropped packets and will dynamically adjust
packet rate to keep the connection up.

3. VoIP uses UDP to eliminate stuttering produced by TCP's normal
response to a dropped packet.  TCP will normally request, unless SACK
is used, a retransmission of all packets after a lost packet which can
cause stuttering.

4. As I understand it VoIP drops out of order packets, but if you
tunnel over TCP then TCP will retransmit lost and subsequent packets
causing stuttering.

6. With UDP you can't correlate (WSAECONNRESET) (code=10054) messages
too all of your reported VoIP drops which may indicate lost or delayed
packets, but not a lost VPN connection.

7. Your one TCP drop was correlated to a "MULTI: packet dropped due to
output saturation (multi_process_incoming_tun)" message.

Conclusion: You appear to be experiencing intermittent lost or delayed
packets.  Encryption and decryption do add to the delay (latency) of
the packets.

You could use a packet sniffer to see if you experience packet loss or
latency issues with your TCP VPNs.  It is much easier to find with TCP
due to tracking the sequence number.

If both your endpoints support SACK
(http://www.faqs.org/rfcs/rfc2018.html) this might help minimize or
eliminate stuttering.

HTH

> > (4) How close to saturation are the network pipes over which the OpenVPN
> > traffic is flowing?
> Well, the traffic flowing into OpenVPN from the internet shouldn't be
> saturating out i-net pipes.  We monitor it pretty well, and don't
> typically go over 50% utilization for any period of time.
>
> > (5) How many seconds do the dropouts last for?
> I'd say just 5-10 seconds at which point connectivity is restored.
> The VPN client; however, never registers that it is actually
> disconnected, traffic simply stops flowing.
>
> > (6) Are you running OpenVPN over TCP or UDP?
> UDP for everyone but a handful of people that I've moved over to TCP
> in the hopes of stabalizing their phone connections.
>
> >
> > James
> >
> >
>
> Thanks for the insights here.  Every bit of info I get feels like it
> gives me a new direction to look into.
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> _______________________________________________
> Openvpn-users mailing list
> Openvpn-users@xxxxxxxxxxxxxxxxxxxxx
> https://lists.sourceforge.net/lists/listinfo/openvpn-users
>


--
Leonard Isham, CISSP
Ostendo non ostento.