NetBackup 8.1 master server installation or upgrade returns "WARNING: Failed to deploy AT certificate for web user."

Problem

During an upgrade or installation of 8.1, a warning is posted stating that the AT certificate for web user failed to deploy:

Deploying AT certificate for web user.

WARNING: Failed to deploy AT certificate for web user.
         You must separately run /usr/openv/netbackup/bin/admincmd/nbcertconfig -u to
         resolve the issue.
Deploying AT certificate for web services.
Generating AT certificate for tomcat.

Error Message

The error "WARNING: Failed to deploy AT certificate for web user." can occur for a number of reasons.  For this particular issue, the IPV4 loopback address was not in the /etc/hosts file, and IPV6 was not enabled.

This can be confirmed reviewing the nbcert log file.  The error stack returned when running nbcertconfig -u will be as follows:

09:14:53.742 [19427] <2> nbcertconfig: /usr/openv/netbackup/bin/admincmd/nbcertconfig -u
/cut//
09:14:56.052 [19427] <2> nbcertconfig: Generate Web User Certificate ...
//cut//
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_client_transport.c,469)Failed (pbxConnect) to connect to localhost 1556
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_client_transport.c,281)pbx connect failed. rc = -1, host = localhost, PBXPort = 1556, PBX_ServiceID = nbatd
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_client_proxy.c,278)Unable to open a connection to localhost!1556!nbatd
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,144)     ERROR STACK REPORT BEGIN
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,148)     Frame :0
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,158)     File: ../at_client_transport.c:282
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,161)     Error data: PBX connect failed
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,148)     Frame :1
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,158)     File: ../at_client_protocol.c:4519
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,161)     Error data: Failed to connect to the server
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,148)     Frame :2
09:14:58.186 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,158)     File: ../at_client_protocol.c:1564
09:14:58.187 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,161)     Error data: Failed to gets brokers public key
09:14:58.187 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,148)     Frame :3
09:14:58.187 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,158)     File: ../at_client_api.c:884
09:14:58.187 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,161)     Error data: vrtsAtSetupTrust
09:14:58.187 [19427] <16> nbcertconfig: EAT_LOG:(../at_utils.c,171)     ERROR STACK REPORT END
09:14:58.187 [19427] <16> setupTrustAuthorizedUser: vrtsAtSetupTrust() failed, error =24582
//cut//
09:15:18.224 [19427] <16> setupTrustAuthorizedUser: vrtsAtSetupTrust() failed, error =24582
09:15:18.225 [19427] <16> authenticateUser: setupTrustAuthorizedUser() failed, error =116
09:15:18.225 [19427] <16> generateNbWebSvcUserCert: authenticateUser() failed, error = 116
09:15:18.225 [19427] <16> nbcertconfig: EXIT STATUS 116 AT configuration for web service user failed

Review of the pbx log will show that no connection attempt was made at the time nbcertconfig -u was exeucted.  The log file may show EWOULDBLOCK errors for local host connections at other times:

12/8/2017 09:14:49.120  PBX_Manager::handle_input, peer fd = 79, From 127.0.0.1, Port 35711
12/8/2017 09:14:49.120  PBX_Client_Proxy::parse_line, line = ack=1  From 127.0.0.1, Port 35711
12/8/2017 09:14:49.120  PBX_Client_Proxy::handle_input read returned: EWOULDBLOCK, will retry

Cause

Lack of the IPV4 localhost entry in the /etc/hosts file

Solution

There are two possible solutions based on whether or not IPV6 should be used for localhost connections.

Solution:  Use if  IPV4 only should be used for localhost connections

1.  Modify the /etc/hosts file to add the IPV4 localhost  line to the /etc/hosts file:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

2. Verify the permissions on the /etc/hosts file, make sure all users have the ability to read:

ls -al /etc/hosts
-rw-r--r--  2 root root 431 Dec 12  2014 /etc/hosts

3.  Recycle the operating system network services.  Please review the appropriate operating system documentation.

4.  Run:   /usr/openv/netbackup/bin/bpclntcmd -clear_host_cache

5.  Recycle NetBackup

6.  Execute the following in the order specified:

/usr/openv/netbackup/bin/admincmd/nbcertconfig -u
/usr/openv/netbackup/bin/admincmd/nbcertconfig -m
/usr/openv/netbackup/bin/admincmd/nbcertconfig -t -f
/usr/openv/wmc/bin/install/configureEnv -platform <platform_value> -nbHostName <host_name>
/usr/openv/wmc/bin/install/configureWmc
/usr/openv/wmc/bin/install/configureCerts
/usr/openv/wmc/bin/install/setupWmc
/usr/openv/netbackup/bin/nbwmc start
/usr/openv/netbackup/bin/nbcertcmd -getCACertificate
/usr/openv/netbackup/bin/nbcertcmd -getCertificate -force
/usr/openv/netbackup/bin/admincmd/bpnbaz -ProvisionCert <master_server_name>

Solution:  Use if IPV4 and IPV6 localhost connections are authorized.

A. Complete steps  1 - 4 in the previous section.

B.  Enable NetBackup for IPv6: 

echo IP_ADDRESS_FAMILY=AF_UNSPEC|nbsetconfig

C.   Restart NetBackup services.

D.   Execute the following commands:

/usr/openv/netbackup/bin/admincmd/nbcertconfig -u
/usr/openv/netbackup/bin/admincmd/nbcertconfig -m
/usr/openv/netbackup/bin/admincmd/nbcertconfig -t -f
/usr/openv/wmc/bin/install/configureEnv -platform <platform_value> -nbHostName <host_name>
/usr/openv/wmc/bin/install/configureWmc
/usr/openv/wmc/bin/install/configureCerts
/usr/openv/wmc/bin/install/setupWmc
/usr/openv/netbackup/bin/nbwmc start
/usr/openv/netbackup/bin/nbcertcmd -getCACertificate
/usr/openv/netbackup/bin/nbcertcmd -getCertificate -force
/usr/openv/netbackup/bin/admincmd/bpnbaz -ProvisionCert <master_server_name>

NOTE: If you run "configureWmc" command after the HotFix of article.100044601 was installed, security.war or netbackup.war is replaced with the original file. Please refer to the following article for details and execute the steps described in the Solution section.

Tomcat and nbwebsvc certificates are not renewed after HotFix in article.100044601 was applied.