NetBackup web management console service is not starting after the NetBackup master server upgrade from 8.2 to 9.1.0.1

Problem

• NetBackup web management console service not starting after the NetBackup upgrade from 8.2 to 9.1.0.1. 
• The upgrade of NetBackup was successful but unable to start the NetBackup web management console service post NBU upgrade to 9.1.0.1

Error Message

<Install log snip>

01-17-2022,07:35:40 : Temporarily setting ENABLE_NBCURL_VERBOSE to 1.
01-17-2022,07:35:40 : Running command: "E:\Program Files\Veritas\NetBackup\bin\\bptestnetconn.exe" -H nbmaster1 -wnbwmc/netbackup -T 8 -e 43 -v.
01-17-2022,07:35:40 : Waiting for command: bptestnetconn.exe
01-17-2022,08:09:34 : Reading pipe broken.
01-17-2022,08:09:34 : The process terminated.
01-17-2022,08:09:34 : Command produced the following output (will display up to 8192 characters):
01-17-2022,08:09:34 : -------------------------------------------------------------------------->
01-17-2022,08:09:34 : adding hostname = nbmaster1
01-17-2022,08:09:34 : Checking responsiveness of 'nbwmc/netbackup' webapp...
01-17-2022,08:09:34 : nbwmc/netbackup web service test for host: nbmaster1 : 2033436 ms [FAIL]
01-17-2022,08:09:34 : --------------------------------------------------------------------------<
01-17-2022,08:09:34 : Command returned status 26.
+ 01-17-2022,08:09:34 : , ERROR: NetBackup Web management console service is not responsive.
+ 01-17-2022,08:09:34 : Not attempting to deploy certificates as the web service did not start.
01-17-2022,08:09:34 : Attempting to delete install_token...
01-17-2022,08:09:34 : Successfully deleted install_token.
+ 01-17-2022,08:09:34 : WARNING: NetBackup web services could not start on this host. This will cause failure of certain NetBackup operations. For details refer to: http://www.veritas.com/docs/000115774
01-17-2022,08:09:34 : CustomAction Deferred_DeployServerCertificate returned actual error code 1603 but will be translated to success due to continue marking

Post NBU upgrade to 9.1.0.1, catalina log is complaining about being Unable to acquire JDBC Connection.Catalina log (catalina.YYYY-MM-DD.log) is showing below error.

Catalina log location at master server

• Unix: /usr/openv/wmc/webserver/logs
• Windows: install_path\NetBackup\wmc\webserver\logs

java.sql.SQLException: JZ00L: Login failed. Examine the SQLWarnings chained to this exception for the reason(s).

Attempt to reset the NBDB password with command "nbdb_admin -dba <new_password>" to sync web.conf failed with the below error.

The nbdb log location at master server

• Unix: /usr/openv/netbackup/logs/nbdb
• Windows: install_path\NetBackup\logs\nbdb

10:59:38.112 [9508.8396] <16> updateWebConf: Error setting password for user NBWEBSVC : 77. Please delete the file : E:\Program Files\Veritas\NetBackup\var\global\wsl\config\web.conf, and try to re-run this command
10:59:38.112 [9508.8396] <2> updateWebConf: updateWebConf exiting with status = 77
10:59:38.113 [9508.8396] <4> secure_db: Exiting with rc = 77
10:59:38.113 [9508.8396] <16> nbdb_admin: Change of dba password for NBDB failed.
10:59:38.113 [9508.8396] <16> nbdb_admin: One or more pwd modifications failed, rc is: 77
10:59:38.117 [9508.8396] <4> nbdb_admin: Exiting with rc = 77
 

Deleting the file E:\Program Files\Veritas\NetBackup\var\global\wsl\config\web.conf did not help in this case.

Cause

After reverting the NBU to the previous version we've noticed that the command nbdb_admin -dba <new_password> still fails with the same error: Change of dba password for NBDB failed. Exiting with rc = 77

Importing the certificates with the 8.2 drpkg file failing too with the below error.

The nbcert log location at master server

• Unix: /usr/openv/netbackup/logs/nbcert
• Windows: install_path\NetBackup\logs\nbcert

14:42:19.292 [7804.10064] <2> nbhostidentity: nbhostidentity -import  -infile D:\Catalog_DR\NBU_Catalog_ctime_FULL.drpkg
.
.
14:42:45.419 [7804.10064] <2> IdentityUtil::executeJavaHostIdentityApp: Executing command: ""C:Program FilesJavajdk1.8.0_261bin" -classpath "E:\Program 
Files\Veritas\NetBackup\web\nbu_common.jar;E:\Program Files\Veritas\NetBackup\web\nb_drpkg.jar;E:\Program Files\Veritas\NetBackup\web\commons-io-2.0.1.jar" 
com.netbackup.nbdrpkg.NBDRPackage -testpassphrase -infile "NBU_Catalog_ctime_FULL.drpkg""
14:42:45.419 [7804.10064] <2> SystemCmd::Execute: Executing cmd (""C:Program FilesJavajdk1.8.0_261bin" -classpath "E:\Program 
Files\Veritas\NetBackup\web\nbu_common.jar;E:\Program Files\Veritas\NetBackup\web\nb_drpkg.jar;E:\Program Files\Veritas\NetBackup\web\commons-io-2.0.1.jar" 
com.netbackup.nbdrpkg.NBDRPackage -testpassphrase -infile "NBU_Catalog_ctime_FULL.drpkg"" )
14:42:45.419 [7804.10064] <2> SystemCmd::Execute: Executing redirected cmd (""C:Program FilesJavajdk1.8.0_261bin" -classpath "E:\Program 
Files\Veritas\NetBackup\web\nbu_common.jar;E:\Program Files\Veritas\NetBackup\web\nb_drpkg.jar;E:\Program Files\Veritas\NetBackup\web\commons-io-2.0.1.jar" 
com.netbackup.nbdrpkg.NBDRPackage -testpassphrase -infile "NBU_Catalog_ctime_FULL.drpkg""  <NUL 
>C:\Users\sshaikha\AppData\Local\Temp\2\Syste-07804637246565417064000000000-a10064 
2>C:\Users\sshaikha\AppData\Local\Temp\2\Syste-07804637246565418064000000001-a10064 )
14:42:45.441 [7804.10064] <2> SystemCmd::Execute: Command status: 1
14:42:45.442 [7804.10064] <16> IdentityUtil::executeJavaHostIdentityApp: [STDERR]: '"C:Program FilesJavajdk1.8.0_261bin"' is not recognized as an internal or 
external command,
14:42:45.442 [7804.10064] <16> IdentityUtil::executeJavaHostIdentityApp: [STDERR]: operable program or batch file.
 

In an ideal scenario - "C:Program FilesJavajdk1.8.0_261bin" would point to the java binary, which appears to be incorrect here.

Reviewing the available NBSU (in versions 8.2 and 9.1.0.1) points to the JAVA_PATH variable incorrectly pointing to the one we see in the logs. 
It could be some stale entry from the OS side or an incorrect registry variable from the NetBackup side.

----------
JAVA_PATH = C:Program FilesJavajdk1.8.0_261bin
----------

The above incorrect JAVA_PATH from the OS breaks the NetBackup operations such as nbdb_admin command to reset the password, nbhostidentity command, and so on.
 

Solution

1. Capture the output of the below command from the master server

----------
reg query HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Paths
----------

2. If JAVA_PATH is explicitly defined, we need to fix the entry. If it is not defined (the default behavior), it could be coming up from the OS PATH variable incorrectly containing JAVA_PATH% in it

----------
C:\Windows\System32\WindowsPowerShell\v1.0\JAVA_HOME%\bin
----------

3. Update the OS - PATH variable by removing this incorrect entry mentioned above in point 2. Replace the above-mentioned entry with the below (keeping other entries as it is)

----------
C:\Windows\System32\WindowsPowerShell\v1.0\
----------

4. Open a new command prompt and execute the below command to check whether the JAVA_PATH variable is reporting correctly. If yes, try resetting the NBDB password (nbdb_admin -dba nbusql)

----------
E:\Program Files\Veritas\netbackup\bin\goodies\listpaths.exe | findstr JAVA_PATH
----------

If the variable still reports an incorrect entry, perform the below steps (steps 5-7)

5. Open regedit, and navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Veritas\NetBackup\CurrentVersion\Paths.Right-click and select New String Value 

    ----------
    Value Name: JAVA_PATH
    Type: REG_SZ
    Value data: C:\\Program Files\\Veritas\\NetBackup\\jre\\bin\\java
    ----------

***Make a note of the double slashes.

6. Execute the command and verify whether JAVA_PATH is correctly set.

----------
E:\Program Files\Veritas\netbackup\bin\goodies\listpaths.exe | findstr JAVA_PATH
----------

7. After correctly defining the JAVA_PATH variable, nbdb_admin command to reset the password, nbhostidentity commands are now working fine. 

Once the above commands are successful, the next NBU upgrade attempt to 9.1.0.1 is successful now without any issue/error.