NetBackup 8.1 and above, master server loses connectivity to the media servers for no apparent reason

Problem

NetBackup 8.1 and above, master server loses connectivity to the media servers for no apparent reason. The media servers can be any supported version of NetBackup, the ability to communicate to the master is possible (via ssh, telnet, ping), but when running NetBackup jobs or using the NetBackup commands to test connectivity will result in failures.

Error Message

While testing communications between servers using bptestbpcd command from the master server, or a media server, directed to a client or media server:

<16>bptestbpcd main: Function ConnectToBPCD(exbackupaps1) failed: 8506
<16>bptestbpcd main: The certificate has expired.
<16>bptestbpcd main: Unable to perform peer host name validation. Curl error has occurred for peer name: <media_server>, self name: <master>: 8506
        [PROXY] Encountered error (VALIDATE_PEER_HOST_PROTOCOL_RUNNING) while processing(ValidatePeerHostProtocol).: 1
The certificate has expired.

The directory path to the bptestbpcd command is as follows...

UNIX/Linux: /usr/openv/netbackup/bin/admincmd/
Windows: install_path\NetBackup\bin\admincmd\

Cause

In this case the Tomcat certificate has expired. By design, the expiration date should auto-extend up to 180 days in advance of the expiration date. In some cases the certificate does not auto-extended due to permission issues on certain key files or previous intervention where NetBackup Secure Communications were rebuilt.

There are two methods to confirm that the Tomcat certificate has expired:

1. With logging:
   • Set the ' ENABLE_NBCURL_VERBOSE = 1 ' configuration parameter
   • Run command: nbcertcmd -ping
   • View the details in the nbcert log.
     UNIX/Linux: /usr/openv/netbackup/logs/nbcert/
     Windows: install_path\NetBackup\logs\nbcert\
   • Run command: nbcertcmd -listAllCertificates
   • The expiration of the Tomcat certificate *may* be different, but if either results show an expired certificate for Tomcat, the certificate will need to be updated.
2. With command line:
   • Note: Adjust paths as necessary...
   • UNIX/Linux:
     /usr/openv/java/jre/bin/keytool -list -v -keystore /usr/openv/var/global/vxss/tomcatcreds/nbwebservice.jks < /usr/openv/var/global/jkskey | grep -e until -e Owner
/usr/openv/java/jre/bin/keytool -list -v -keystore /usr/openv/var/global/wsl/credentials/nbwebservice.jks < /usr/openv/var/global/jkskey | grep -e until -e Owner
   • Windows:
     "C:\Program Files\Veritas\NetBackup\jre\bin\keytool.exe" -list -v -keystore "C:\Program Files\Veritas\NetBackup\var\global\vxss\tomcatcreds\nbwebservice.jks" < "C:\Program Files\Veritas\NetBackup\var\global\jkskey" | findstr "until Owner"
"C:\Program Files\Veritas\NetBackup\jre\bin\keytool.exe" -list -v -keystore "C:\Program Files\Veritas\NetBackup\var\global\wsl\credentials\nbwebservice.jks" < "C:\Program Files\Veritas\NetBackup\var\global\jkskey" | findstr "until Owner"

The jkskey file contains the password needed to open the nbwebservice.jks file using the keytool command. The command strings will open the file and feed it the password in one step. Each .jks file contains 3 certificates. The middle Certificate is the TOMCAT certificate

Example Output:

Owner: O=vx, OU=root@jdhv2.community.veritas.com, CN=nbatd
Valid from: Tue Sep 04 11:46:55 CDT 2018 until: Mon Aug 30 13:01:55 CDT 2038
Owner: O=vx, OU=TOMCAT@jdhv2.community.veritas.com,CN=jdhv2        <- TOMCAT Identity String
Valid from: Tue Sep 04 11:47:59 CDT 2018 until: Wed Sep 04 13:02:59 CDT 2019        <- TOMCAT Cert Expiration Date
Owner: O=vx, OU=root@jdhv2.community.veritas.com, CN=broker
Valid from: Tue Sep 04 11:46:55 CDT 2018 until: Mon Aug 30 13:01:55 CDT 2038

In this case the TOMCAT certificate expires Wed Sep 04 13:02:59 CDT 2019. If the date displayed is prior to the current date than the certificate has expired and must be renewed.

Solution

The following commands on the Master server will create a valid certificate.

UNIX/Linux Steps:

1. /usr/openv/netbackup/bin/admincmd/nbcertconfig -t
   Note: The -t and -f options will be needed for NetBackup versions 8.1.1 and higher.
   /usr/openv/netbackup/bin/admincmd/nbcertconfig -t -f
2. /usr/openv/wmc/bin/install/configureCerts
3. /usr/openv/wmc/bin/install/setupWmc
4. /usr/openv/netbackup/bin/nbwmc stop
5. /usr/openv/netbackup/bin/nbwmc start
6. /usr/openv/netbackup/bin/nbcertcmd -getCACertificate
7. usr/openv/netbackup/bin/nbcertcmd -getCertificate -force

Windows Steps:

1. On the Master, run services.msc and locate NetBackup Web Management Console service (nbwmc)
2. Identify the account used to start the nbwmc service
3. Locate /  Acquire the password for this account
4. Open an Administrator CMD prompt on the Master
5. Create the following Environment Variable for the CMD window by running: set WEBSVC_PASSWORD=<passwordHere>
6. Run: install_path\NetBackup\bin\admincmd\nbcertconfig -t
   Note: The -t and -f options will be needed for NetBackup versions 8.1.1 and higher.
   install_path\NetBackup\bin\admincmd\nbcertconfig -t -f
   • If this fails, it is likely to be due to an incorrect password.
     To verify the password is correct, use the following command to spawn a new CMD prompt window running as the account in question: runas /user:<user> cmd.exe
     Example local account: runas /user:nbwebsvc cmd.exe
     Example domain account: runas /user:COMPANY\nbwebsvc cmd.exe
   • If the new CMD window opens successfully, it means the credentials were correct and the new window can simply be closed.
   • If the new CMD window fails to open, examine the on-screen language to identify why.
7. CD into install_path\NetBackup\wmc\bin\install
8. Run: configureCerts.bat
9. Run: setupWmc
10. Restart the nbwmc service
11. Run: nbcertcmd -getCACertificate
12. Run: nbcertcmd -getCertificate -force

Note: For clustered environments, please ensure to use the '-cluster' switch with 'nbcertcmd' and/or refer to the commands reference guide for further details.

While it is preferred to have NetBackup auto-renew the certificates, the commands can also be run manually in advance of expiration. As the nbwmc process is cycled there may be a disruption to normal activities such as backups.