Unable to deploy certificate after install/upgrade of NetBackup Master 8.1 [Windows Master] EXIT STATUS 8625: Server is unavailable to process the request. Please try later.

Problem

Unable to deploy certificate after install/upgrade to NetBackup Master 8.1 [Windows Master]; nbcertcmd; EXIT STATUS 8625: Server is unavailable to process the request. Please try later.

Error Message

nbcertcmd -getCertificate -force
nbcertcmd: The -getCertificate operation failed for server [MASTER HOSTNAME].
EXIT STATUS 8625: Server is unavailable to process the request. Please try later.

ERROR: /usr/openv/wmc/bin/install/configureCerts failed. Aborting...
ERROR: NetBackup Web Service configuration failed

Cause

Anti-Virus and/or Firewall, Kaspersky in one customer's case. At the time of the install/upgrade the antivirus blocked the creation of the CRL (certificate revocation list) and other web service components.

Troubleshooting

• Key logs from the master server:
  • nbcert (nbcertcmd commands)
  • catalina.log (web server)
  • nbwebservice (OID 466 and 484)
  • nbatd (OID 18)

nbatd unified logs, OID 18:

20/04/2018 08:22:15.899 [debugmsgs] (msgtransport.cpp:287) recv timeout not found,Default <60>
20/04/2018 08:22:15.899 [debugmsgs] (e:\production\vxat\v6.3.63.0\win-x64\vxat\server\authentication\mavericks\authbroker\tpcauthserver\client_handler.cpp:990) Received PK_AUTH_START_CRL
20/04/2018 08:22:15.899 [debugmsgs] (e:\production\vxat\v6.3.63.0\win-x64\vxat\server\authentication\mavericks\authbroker\registry\atregistry.cpp:610) AtRegistry<LOCK>::OpenSection failed. sub_section = NBU_HOSTS, fCreate 0, Errno = -1
20/04/2018 08:22:15.899 [debugmsgs] (e:\production\vxat\v6.3.63.0\win-x64\vxat\server\authentication\mavericks\authbroker\registry\atregistry.cpp:610) AtRegistry<LOCK>::OpenSection failed. sub_section = NBU_HOSTS@[MASTER HOSTNAME], fCreate 0, Errno = -1
20/04/2018 08:22:15.899 [debugmsgs] (e:\production\vxat\v6.3.63.0\win-x64\vxat\server\authentication\mavericks\authbroker\registry\atregistry.cpp:610) AtRegistry<LOCK>::OpenSection failed. sub_section = NBU_HOSTS, fCreate 0, Errno = -1
20/04/2018 08:22:15.899 [debugmsgs] (e:\production\vxat\v6.3.63.0\win-x64\vxat\server\authentication\mavericks\authbroker\registry\atregistry.cpp:610) AtRegistry<LOCK>::OpenSection failed. sub_section = NBU_HOSTS@[MASTER HOSTNAME], fCreate 0, Errno = -1
20/04/2018 08:22:15.899 [debugmsgs] (e:\production\vxat\v6.3.63.0\win-x64\vxat\server\authentication\mavericks\authbroker\tpcauthserver\client_handler.cpp:1008) Could not create context for Generate CRL
20/04/2018 08:22:15.899 [debugmsgs] (msgtransport.cpp:253) send timeout not found,Default <60>

nbwebservice unified logs, OID 466:

20/04/2018 08:22:15.899 [com.netbackup.security.common.NBCertRenew] EAT_LOG:(sslconn.c,716)freeing SSL <1dc1f270>
20/04/2018 08:22:15.899 [Error] EAT_LOG:(at_utils.c,144) ERROR STACK REPORT BEGIN
20/04/2018 08:22:15.899 [Error] EAT_LOG:(at_utils.c,148) Frame :0
20/04/2018 08:22:15.899 [Error] EAT_LOG:(at_utils.c,158) File: at_client_api.c:4099
20/04/2018 08:22:15.899 [Error] EAT_LOG:(at_utils.c,161) Error data: vrtsAtGenerateCrl
20/04/2018 08:22:15.899 [Error] EAT_LOG:(at_utils.c,171) ERROR STACK REPORT END
20/04/2018 08:22:15.899 [Error] Domain Name: NBU_HOSTS, Domain Type: vx
20/04/2018 08:22:15.899 [Error] Broker Name: localhost!1556!nbatd, Port: 1556
20/04/2018 08:22:15.899 [Error] Error occurred while trying to get CRL with 6084 and message Unexpected VxAT Error
20/04/2018 08:22:15.899 [com.netbackup.security.certificate.audit.CRLGenerationAuditor] CRL Operation do not need auditing
20/04/2018 08:22:15.899 [Error] CRL generation failed on host : [MASTER HOSTNAME]

Solution

1. Uninstall the Anti-Virus, disabling was not enough, it still left NetBackup processes in quarantine and continued to block the generation of CRL
2. Reboot master server
3. Run the following commands, in order:
   • First run the following command, WEBSVC_PASSWORD sets the password that 'nbcertconfig' uses with '-u'
     • set WEBSVC_PASSWORD=**** [password of webuser]
       • Note: The webuser password can be found in the bp.conf file, or registry 'HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Config\WEBSVC_USER'.
   • install_path\NetBackup\bin\admincmd\nbcertconfig -u -i
   • install_path\NetBackup\bin\admincmd\nbcertconfig -m
   • install_path\NetBackup\bin\admincmd\nbcertconfig -t
     • Note: 'configureEnv' is an optional command, unless install_path\NetBackup\wmc\bin\setenv.bat is missing, corrupt, or has false information inside. If you need configureEnv, -nbHostName should match the first SERVER, and you may need information about the environment [https://www.veritas.com/support/en_US/article.100041568].
   • install_path\NetBackup\wmc\bin\install\configureEnv -nbInstallDir "install_path" -nbHostName <host_name> -isClustered 0/1
   • install_path\NetBackup\wmc\bin\install\configureWmc
   • install_path\NetBackup\wmc\bin\install\configureCerts
   • install_path\NetBackup\wmc\bin\install\setupWmc
4. All commands most come back successful
5. Restart NetBackup services [bpdown/bpup]

Note: If you run "configureWmc" command after the HotFix of article.100044601 was installed, security.war or netbackup.war is replaced with the original file. Please refer to the following article for details and execute the steps described in the Solution section.

Tomcat and nbwebsvc certificates are not renewed after HotFix in article.100044601 was applied.