Best practices for managing TCP ephemeral port ranges and TCP time wait on NetBackup servers

Problem

There are two aspects to this topic.

First, how to ensure that as many TCP ports as possible are available for application use.  The number of ports is finite.  When the range is exhausted applications must fail, although many - NetBackup included - will retry briefly in hopes one does become available quickly.

Second, how to ensure that recently used TCP ports can be safely reused by other processes.  Otherwise applications may receive the wrong data, that was intended for a previous process, and either fail or behave in an unexpected fashion.

The goal is to allow NetBackup to perform as many operations as possible, as quickly as possible, without conflict.

This is mostly a challenge of scale.  Computing hosts have become bigger, faster, and more interconnected (memory, CPU, network interfaces, I/O devices, etc.), but the limitations of TCP port availability have remained largely unchanged for decades.

NetBackup has made design and implementation changes over the years to minimize the number of ports needed for concurrent operation.  However, certain new features and functionality have also expanded the number of ports potentially in concurrent use.

NetBackup 4.5 minimized ephemeral port use by deprecating legacy call-back.
NetBackup 7.1 minimized service port use by allowing many legacy services to accept connections via PBX.
NetBackup 8.1 significantly increased ephemeral port use when introducing secure communication connections.
NetBackup 8.2 roughly halved the ephemeral port use added by secure communication connections.
NetBackup 8.2 removed the need for some connections and shortened the lifespan of others.

Note: NetBackup 8.2+ also no longer limits the startup of backup jobs to 1 job per second.  This will allow jobs to go active more quickly, and will increase the concurrent connection and port usage in the seconds and minutes immediately following when a backup window opens.  Sites that spread their backup and duplication windows throughout the night/weekend/day are unlikely to see any effect, other than jobs going from Queued to Active more quickly (until the concurrent per policy/client/storage limits are reached).  Sites that have many or all backup windows configured to open simultaneously may find that their platforms and NetBackup now have the capacity to saturate the ephemeral port window, which can lead to connection and job failures.  Further, each TCP end-point also consumes non-swapable kernel memory for the TCP Send and Receive queues.  The increased number of concurrent connections will reduce the amount of memory available for other processing and may slow the performance of applications.  The impact should be almost exclusive to the master server, but a media server supporting many hundreds of concurrent jobs might also be affected to some extent.

Caution: Starting a thousand or more restore job streams simultaneously also has the potential to exhaust the ephemeral port range on a NetBackup 8.1+ master server if the media servers are also at NetBackup 8.1+.

Background

Operating systems have a fixed number of TCP ports, a subset of which are in the ephemeral port range.  When a port range is exhausted, applications are unable to obtain a port for inter process connections and will fail.

Each TCP connection, inter-host or intra-host, is from a source TCP port number to a destination TCP port number.  TCP requires that the 4-tuple of source IP address, source port number, destination IP address, and destination port number be unique for each TCP connection on the host.  If they are not unique, data within a TCP packets intended for one application may be delivered to a different application.  The latter application will likely be confused by the unexpected data, react in an unexpected way, and most likely fail.

Each service must be listening on a well-known TCP port.  Each client processes must have prior knowledge of that port number in order to connect.  Hence services cannot easily be relocated to a different port number without impact the configuration of all connecting clients.

When processes connect to a service, they bind a port from the ephemeral port range to use as the source port.  Hence many processes on many different hosts can connect to the well-known service port on a single host, but each connection will have a unique pair of source IP address and source TCP port.

When a TCP connection is completed, TCP will typically place the 4-tuple into TCP Time Wait state.  This allows retransmitted TCP packets, already in-flight to arrive and be discarded before the source port number is allowed to be reused by another application.

Note:  Multiple rapid connection attempts, either Intentional [denial-of-service (DOS) attack] or unintentional [NetBackup repeated connection retry attempts due to asymmetrical routes, etc], may similarly consume available ports on the host can lead to a similar situation.

Error Message

When the ephemeral port range is exhausted, NetBackup processes may log that they are unable to bind a socket because of either EADDRINUSE (98 or 125) or WSAEADDRINUSE (10048).  E.g.

<8> ... bind() failed ERRNO=98 ...
<16> getsockbound: bind() failed, Address already in use (125)
ERROR V-14-2-15261 INET bind failed: Address already in use

The nbpxyhelper debug log (NetBackup 8.1 and newer) may show socket pair creation failed with VN_STATUS_SOCKET_FAILED (10).  The process utilizing the vnetd proxy process reports EC_socket_pair_failed (7638).  E.g.

[Debug] NB 51216 nbpxyhelper 486 [ProxyConnection::makeSocketPair] Error: Failed to create VNET socket pair. Status: 10

<16> dump_proxy_info: statusmsg: Failed to create a local socket pair., nbu status = 7638

If the amount of memory on the host is insufficient to support the connection load, then bind requests will fail with ENOMEM (12) or WSAENOBUFS (10055).  E.g.

<8> ... bind() failed ERRNO=10055 ...

Solution

First review the concurrent connections at the time of a failure and determine if an intentional or unintentional denial-of-service attack is the the reason for the port range exhaustion.  If so, eliminating the cause of the undesired connections will be the best solution.

If all connections were relevant, and simply due to the job load on the host, then the following steps can used to determine the configuration and usage of TCP ports in the ephemeral range on the host, and whether any adjustments to the configuration should be made.

1. If running NetBackup 8.0 or prior, and the NetBackup Client Port Window will override the operating system ephemeral port range.  This is how to display the current value, and shows the default value.  

cd <install>/netbackup/bin
nbgetconfig CLIENT_PORT_WINDOW
CLIENT_PORT_WINDOW = 0 0

If set to a value other than the default, restore the default value, and then monitor the resulting behavior before considering any of the steps that follow.

2. Determine the current lower and upper bound for the ephemeral port range.

3. Identify the highest numbered TCP port used for listening by the services on the host.  

For NetBackup, 13787 is the highest port number.  If non-NetBackup services are present on the host, the netstat command can be used to identify TCP ports used for listening and determine if they are above or below those used by NetBackup services.  In some cases, the listening process ID or process name will also be displayed, or it can be determined using other commands.

Linux:        
    netstat -n -a -p -t | grep LISTEN

Solaris 11:    
    netstat -n -a -u -P tcp | grep LISTEN
    ps -ef
    
AIX, HP-UX, Solaris 10:
    netstat -n -a | grep LISTEN
    lsof
    pfiles

Windows:
    netstat -n -a -o | FINDSTR LISTEN
    tasklist /v

4. Determine if there truly is a need to make additional TCP ports available on the host.

When ports are not available, the NetBackup debug logs will show a failure of the operating system bind() API.  (On NetBackup 8.1+, it may also be shown as a socket pair creation failures in the nbpxyhelper debug log.)  If bind failures are not occurring, there is no direct value in adjusting the TCP port ranges.  

Please note that on busy master servers there may be value in proactively monitoring the number of TCP ports in use to determine if the ephemeral port range is approaching saturation.  This can be done by collecting the 'netstat -n -a' output during the time frame when the highest number of jobs are being queued and/or going active.  The collection should occur at 5-10 second intervals to accommodate capturing brief spikes in activity followed by rapid TCP timer expiration.  Inspecting the output from each interval to determine the number of concurrent connections and TCP port numbers in use.

5. If bind failures are present, or netstat shows [near] port range saturation, then the following options should be considered.

1. Reconfigure the start windows for backup and duplication jobs, or the restore commands, to spread out the initiation of jobs to a comparatively longer span of time.  The initiation of each job results in a burst of short-lived connections between the media server and the master server.  Spreading out the jobs by a few seconds or minutes (or longer) gives TCP a chance to timeout recently used ports and safely reuse them for new connections.
    
2. Decrease the lower bound of the ephemeral port range to be just higher than the largest port number used by any [NetBackup] service on the host. 
   
   Similarly, increase the top of the ephemeral port range to the highest value permitted by the operating system vendor, if not already at the upper limit.  The TCP defined upper limit is 65535.
   
   The 'no', ndd', 'sysctl', and 'netsh' commands shown above all have options to modify the current settings.

Note: It is possible to decrease the lower bound of the ephemeral port range to a value below the highest port number observed in step #2 above.  But if a service is stopped, then another processes may randomly bind the well-known service port as a source port before the service can be restarted.  The service will not be able to restart until the conflicting process has released the port or been terminated.  The commands noted in step #3 above (used without the LISTEN filter) can be used to identify the conflicting process.

3. In rare instances it may be observed that half or more of the connections in the 'netstat -n -a' output are in TCP TIME_WAIT state.

TCP Time Wait is a normal state.  It indicates that a connection has been closed, but the operating system is waiting a bit to ensure that the port numbers are not reused before any retransmitted TCP packets that are already in-flight can be received and discarded.

Note: Connections in TCP TIME_WAIT are not a concern if the ephemeral port range is not saturated, and any TCP memory associate with the connections is not starving other processing on the host.

If-and-only-if the number of connections in Time Wait is directly affecting operations, review the operating system tunables that affect the duration of TCP Time Wait state.

Note: Some operating system default to more than 60 seconds.  In recent years, with faster networks and lower 2MSL (2 x maximum segment lifetime), some operating system vendors have begun to recommend a value of 60 seconds, and in some instances even 30 seconds.  Please consult your operating system vendor before changing these values significantly from the default values, especially Linux.  Never use a Time Wait value of less than 10 seconds, that is the default retransmission timeout on Windows, and the default is even longer on UNIX.

Following any changes, review the NetBackup debug logs on the host following high work-load time frames.  The sudden appearance of protocol or handshake failures is a key indication that retransmitted packet data is being delivered to a unsuspecting process following port reuse.  If present, any changes to Time Wait tuning should then be reverted.

4. In very rare instance it may be observed that most of the connections in the 'netstat -n -a' output are in TCP FIN_WAIT or CLOSE_WAIT state.

This indicates that the process at the FIN_WAIT end of the connection is finished with the connection, but the process at the CLOSE_WAIT end is not yet done.  If many hundreds of connections are in this state, and specific connections (identified by their unique 4-tuple) persist in this state for more than 30-60 seconds across several netstat collections, then the debug logs for the process on the CLOSE_WAIT end should be reviewed to determine how best to remove the unexpected delay.

Note: Adjusting the TCP tunables at the operating system level will not decrease or otherwise affect CLOSE_WAIT. 

Note: Adjusting any TCP tunables that affect FIN_WAIT may introduce problems similar to those associated with TIME_WAIT.  Especially since the remote process has not finished with it's end of the connection so initial packets (not just retransmitted packets) may be sent many seconds or minutes later.  See the discussions on TCP Time Wait above.