A NDMP backup fails immediately with status 99 "ndmp_data_connect failed, status = 23"

Problem

A NDMP backup immediately fails with status 99 even when the NDMP communication port 10000 is permitted to pass network traffic via a either an active Firewall or Anti-Virus Software.

Error Message

The logs example capture that the filer is unable to complete a NDMP data connection back via port 3499 to a designated Media Server. In the following example the NDMP filer was unable to complete a data connection back to media server which had a port range from 1025 to 5000.

NetBackup Job detailed status shows: 

Cause

The NDMP Filer is unable to connect to the desired port range, this will be due to either a firewall or other port blocking mechanism which is preventing the flow of traffic from the inbound connection from filer to media server from this port range.

An inbound NDMP filer requires data connection that selects a random port number in response to operational requests, this data connection that can either be blocked or is idle an connection that has eventually terminated under some of the following conditions.

Inbound NDMP data connections are initiated back Media Server upon request, unless defined on the NDMP filer then by definition it will select a random TCP/IP port number that will be assigned for NDMP data connections.

Therefore just allowing TCP/IP port 10000 communication through a firewall or anti-virus is insufficient. A prerequisite would be to allow the designated Media Server and NDMP filer to have defined and extensive port range that is allows for the inbound random port selections.

Solution

The port requirements to backup and restore an NDMP server are as follows:

TCP port 10000 must be open from the media server (DMA) to the NDMP filer (tape or disk) for all types of NDMP operations; local, remote, and 3-way. And the NetBackup SERVER_PORT_WINDOW must be open inbound from the filer to the media server for remote NDMP. It must also be open for efficient catalog file (TIR data) movement during local or 3-way NDMP.

In more recent releases some NDMP Vendors have started to offer the capability to assign a designated range of TCP/IP ports which can be used to control the port range for the NDMP data connection in response to NDMP_DATA_LISTEN and NDMP_MOVER_LISTEN operations.

Please Consult the appropriate Vendor Documentation on how to configure and define the required ports ranges to NDMP_DATA_LISTEN and NDMP_MOVER_LISTEN operations. 

Also consider that an active Firewall Rules that have an active policy, that restrictions in place to either block or limit which random ports can used during a data connection.

Do also take into account that a Third Party Security Program such as Anti-Virus, could be intercepting and holding or preventing the random connecting port beyond the expected time to live internal and reaching it final destination.