Unable to add NetBackup AIR replication target after successful upgrade to version 8.1.x.

Problem

Unable to add NetBackup AIR replication target after successful upgrade to version 8.1.x.

Error Message

Consider the following scenarios:

1. Targeted AIR has been running fine prior to upgrade to version 8.1.x. After successful upgrade, AIR jobs start failing with the following error: 

Dec 1, 2018 2:21:09 PM - Critical bpdm (pid=1111) Storage Server Error: (Storage server: PureDisk:prodmsdp.domain.local) CALaunchAIRReplicate: Failed to complete launchAIRReplicate webservice (Could not setup replication: get Remote SPA ( drmsdp.domain.local ) webservice failed, could not determine whether target is PDDE or PDDO (broken pipe) ) V-454-61
Dec 1, 2018 2:21:10 PM - Error bpdm (pid=1111) <async> copy image failed: error 2060017: system call failed
Dec 1, 2018 2:21:10 PM - Error bpdm (pid=1111) copy failed: error 174
Dec 1, 2018 2:21:10 PM - Error bpdm (pid=1111) <async> cancel failed: error 2060001: one or more invalid arguments
Dec 1, 2018 2:21:10 PM - Error bpdm (pid=1111) copy cancel failed: error 174

2. Targeted AIR has not been configured prior to upgrade, or replication target has been removed post upgrade. In this scenario, user is unable to add the replication target (back) to the source MSDP storage server. The error can be seen in the spad nbrmms session log:

13:26:56 INFO [2222] [139869969426176]: WSRequestExt: submitting &request=7&login=root&passwd=*&action=getSpaID13:26:56 INFO [2222] [139869969426176]: sessionStartAgent: Server is Version 11.0200.0018.07, Protocol Version 6.6.1.1.1
13:28:26 ERR [2222] [139869969426176]: 25056: SSL initialization failed: could not connect to server[SSL error = 5, errno = 110]
13:28:26 ERR [2222] [139869969426176]: 25099: Could not send binary message: sent -1 instead of 48 bytes. Cause: broken pipe13:28:26 ERR [2222] [139869969426176]: 25099: CRSessionStart: connect to server failed!
13:28:26 ERR [2222] [139869969426176]: 25099:  Connection failed broken pipe
13:28:26 ERR [2222] [139869969426176]: 25099: CAGetSpaID failed for URL: ; reason: broken pipe
13:28:26 ERR [2222] [139869969426176]: -1: spad request failed:

TIPS: Spad nbrmms session log can be found within the source MSDP storage server in the following location:

Windows: <MSDP path>\log\spad\<source media server>\nbrmms.exe\spad\

UNIX/Linux: <MSDP path>/log/spad/<source media server>/nbrmms/spad/

Cause

1. NetBackup 8.1.x introduces a new enhancement whereby compatible hosts (version 8.1.x onwards) must communicate with each other in secure mode. For a preconfigured targeted AIR, the trust between source and target master servers needs to be updated, else AIR jobs will stop working.

2. The source MSDP storage server must know the CA of the target NetBackup domain and vice versa, otherwise the replication will fail due to unauthorized connection.

3. If firewall is in use, certain ports must be open bidirectionally. Even after opening the required ports, there have been instances where the firewall still inadvertently denies NetBackup replication traffic. In this case, one way to find out is by watching for packet denials within the firewall itself. 

Solution

1. Follow this link for the instruction on updating master servers' trust. The steps vary depending on whether the source and target master servers are running the same version or mixed.

2. Follow this link to deploy target domain's CA and the certificate on the source MSDP storage server.

3. Open the following TCP ports bidirectionally:

• Between source and target master servers: Port 1556.
• Between source MSDP and target MSDP storage servers: Port 1556, 10082 and 10102.

NOTE: If symptom persists, again, it may be necessary to watch the firewall for packet denials and rectify accordingly.