nbcheck reports java_key_store failed during NetBackup upgrade

Description

nbcheck is used to confirm that the Java Keystore files present on a NetBackup master server have valid content prior to performing an upgrade.

This check will compare the SHA1 fingerprints for the Root Broker/Certificate Authority (RB/ca1) and Tomcat (TC/nbwmc) certificate files present on the host to the contents of the Java Keystore files.  If the configuration is not as expected, the check will fail so that the situation can be corrected before the upgrade is performed.

This check runs only on standalone and active-node master server hosts.  The check is applicable to upgrades from NetBackup versions 8.0 or later.

Error Messages

This check performs many steps, any of which might find a problem.  The unique messages for each step are detailed below. All of the messages will include a common footer.  Either 

This test runs when you upgrade the NetBackup master server from 8.0 to 8.1 or later NetBackup version.  Check Java Keystores fingerprints (*.jks and truststoreNBWSS) for tomcatcreds wsl or credentials.

or 

This test checks Java Keystores fingerprints for tomcatcreds or credentials during upgrades of NetBackup primary servers 8.0 and later.

This error indicates that something is wrong with the Root Broker certificate file, and must be resolved before the Java Keystore files can be accurately checked.

not ok java_key_store: Root Broker certificate not available for comparison to Java Keystore.

This error indicates that something is wrong with the Tomcat certificate files, and must be resolved before the Java Keystore files can be accurately checked.

not ok java_key_store: Tomcat certificate not available for comparison to Java Keystore.

Java Keystore files are secured with a password.  These errors indicate that the specified password file is either not present or unable to be accessed for some reason.

not ok java_key_store: JavaKeystore password file is missing or unreadable
   C:\Program Files\Veritas\NetBackup\var\global\jkskey

not ok java_key_store: Smartcard JavaKeystore password file is missing or unreadable
   C:\Program Files\Veritas\NetBackup\var\global\smartcardjkskey

This error indicates that the password file is present, but out of sync with at least one keystore file.  Either a keystore file or the password file has incorrect contents.

not ok java_key_store: Either Keystore file is corrupted or password is incorrect

This error indicates that the keytool command either failed to execute or was not able to extract information from the indicated keystore file.  There is a problem either with the keytool program or the keystore file is corrupt.

not ok java_key_store: Keytool command failed for file [C:\Program Files\Veritas\NetBackup\var\global\vxss\tomcatcreds\nbwebservice.jks]

This error indicates the keystore file could be read, but does not contain any SHA1 fingerprints.  There is a problem with the specific keystore file.

not ok java_key_store: Missing fingerprints for file [C:\Program Files\Veritas\NetBackup\var\global\wsl\credentials\nbwebservice.jks]

These errors indicate that a keystore file is missing a SHA1 fingerprint for either the Root Broker (ca1/smartcard) or TC (nbwmc) services.

not ok java_key_store: Expecting a parse-able fingerprint for ca1, missing. Regenerate web service certificates

not ok java_key_store: Expecting a parse-able fingerprint for nbwmc, missing. Regenerate web service certificates

These errors indicate that the Root Broker (CA) and Tomcat (TC) certificate and Java Keystore files are present and have readable contents, but the SHA1 fingerprints in the keystore files do not match those in the certificate files.

not ok java_key_store: [<hexadecimal_string>] fingerprint does not match any CA/tomcat certificate files.
  At least one Java Keystore CA fingerprint does not match the RB certificate fingerprint.

not ok java_key_store: [<hexadecimal_string>] fingerprint does not match any CA/tomcat certificate files.
At least one Java Keystore nbwmc fingerprint does not match the Tomcat certificate fingerprint.

Note: For this specific situation (fingprint does not match any CA certificate), please see the Related Article regarding Java Keystore Fingerprint Error During NetBackup Upgrade Pre-check .

Solution

Note: nbcheck only performs the java_key_store check on master servers using a NetBackup Certificate Authority (NBCA).  It does not perform the same check on master servers using an External Certificate Authority (ECA).  This solution should not be used on a host configured for an ECA.

Note: If a problem was reported with the Root Broker certificate file, please engage NetBackup Technical services to ensure the situation is accurately assessed and resolved in the most appropriate way.  This may require reissuing host ID certificates to all NetBackup 8.x hosts.

Note: Verify that the log4j mitigation steps have not been performed, and that the .war files are present before following the rest of this document:
Windows:
cd [install path]NetBackup\wmc\webserver
dir /a /S /b | findstr /r .war$

You should see at least the first 4 of these files at 8.1.2+  (ROOT.war may not exist)

C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps\nbwebservice.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api\nbwss.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api\netbackup.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api\webui.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api_cssc\ROOT.war

Linux/Unix:
find /usr/openv/wmc/webserver -name "*.war"

You should see at least the first 4 of these files at 8.1.2+  (ROOT.war may not exist)

/usr/openv/wmc/webserver/webapps_api/netbackup.war
/usr/openv/wmc/webserver/webapps_api/nbwss.war
/usr/openv/wmc/webserver/webapps_api/webui.war
/usr/openv/wmc/webserver/webapps_api_cssc/ROOT.war
/usr/openv/wmc/webserver/webapps/nbwebservice.war

If these .war files are missing, you must either put them back from a backup or from a system that has not had the log4j mitigation steps performed, or apply the final fix log4j EEB before proceeding below:
https://www.veritas.com/content/support/en_US/article.100052058.html

Otherwise, the following procedure can be used to rebuild the Tomcat certificate and Java Keystore related files.  If desired, NetBackup Technical Services can be engaged to assist with reviewing and correcting the situation.

1. Take a NetBackup catalog backup.
    
2. If this is an active cluster node, freeze the cluster so services do not fail-over to the other node.
    

3. Make sure the nbatd service is running and web services has stopped.

Linux/UNIX:
/usr/openv/netbackup/bin/nbwmc terminate
/usr/openv/netbackup/bin/bpps java | grep nbwmc
/usr/openv/netbackup/bin/bpps java | grep nbatd

Windows:
"<install_path>\netbackup\wmc\bin\nbwmc.exe" -stop -srvname "NetBackup Web Management Console"
"<install_path>\netbackup\bin\bpps" | findstr /i nbwmc
"<install_path>\netbackup\bin\bpps" | findstr /i nbatd

4. If the Tomcat certificate files were detected as missing or corrupt, recreate them.

First make sure the first SERVER in the configuration is correct for the host.  Update the bp.conf file or registry setting if incorrect.

Second, obtain the list of hostnames by which other NetBackup hosts know the master server, so they can be placed into the new certificate.  E.g.  mymaster.com, mymaster, mymaster.backup.com, mymaster-bk

Third, on NetBackup 8.1.1+, use the -f option to force overwrite of existing files.

Linux/UNIX:
/usr/openv/netbackup/bin/nbgetconfig SERVER
/usr/openv/netbackup/bin/admincmd/nbcertconfig -t -user <web_service_user> [-f] [-sub <comma_separated_master_server_hostnames>]

Windows:
"<install_path>\netbackup\bin\nbgetconfig" SERVER
set WEBSVC_PASSWORD=<password_of_user>
"<install_path>\netbackup\bin\admincmd\nbcertconfig.exe" -t -user <web_service_user> [-f] [-sub <comma_separated_master_server_hostnames>]

Note: if the windows password contains any special characters then please refer to the following:

https://www.veritas.com/content/support/en_US/article.100049589.html

5. Setup the environment; to be used by the steps that follow.  

The -nbHostName should match the first SERVER entry.  On Windows, this should also match the -DNB_HOSTNAME value in <install_path>\NetBackup\wmc\bin\nbwmcservice.xml.

The -nbInstallDir should be similar to "C:\Program Files\Veritas", with "\NetBackup" as a sub-directory.

The -platform should be one of: AMD64, hpia64, linuxR_x86, linuxS_x86, rs6000, solaris, solaris_x86.

Linux/UNIX:
cd /usr/openv/wmc/bin/install/
/usr/openv/wmc/bin/install/configureEnv -platform <platform_value> -nbHostName <host_name>

Windows:
cd "<install_path>\netbackup\wmc\bin\install"
"<install_path>\netbackup\wmc\bin\install\configureEnv.bat" -nbInstallDir "<install_path>" -nbHostName <host_name> -isClustered 0/1 

Afterwards the environment setup can be confirmed by inspecting this file:

Linux/UNIX:
more /usr/openv/wmc/bin/setenv

Windows:
more "<install_path>\netbackup\wmc\bin\setenv.bat"

6. Configure web services preparation; sslStore, jkskeys, ports, webrootcert.pem, etc.

Linux/UNIX:
/usr/openv/wmc/bin/install/configureWmc

Windows:
"<install_path>\netbackup\wmc\bin\install\configureWmc.bat"

Note: If you run "configureWmc" command after having installed any HotFix or EEB that updates the security.war or netbackup.war file, the effects of installing the EEB will be reverted.  For an example of the steps necessary to reinstall the HotFix or EEB, see the related article about certificates not renewed after HotFix (100049294).

7. Configure web services; update the Java Keystore files from the certificate files, etc.

Linux/UNIX:
/usr/openv/wmc/bin/install/configureCerts

Windows:
"<install_path>\netbackup\wmc\bin\install\configureCerts.bat"

8. Setup web services; permissions, etc.

Linux/UNIX:
/usr/openv/wmc/bin/install/setupWmc -logFileName <setupWmc_log_file>

Windows:
"<install_path>\netbackup\wmc\bin\install\setupWmc.bat"  -logFileName <setupWmc_log_file>
"<install_path>\netbackup\wmc\bin\install\setupWmc.bat"  -logFileName <setupWmc_log_file>

Note: Run this command twice on Windows to recursively set directory and file permissions.

9. Restart web services, and confirm they are running and responding.

Linux/UNIX:
/usr/openv/netbackup/bin/nbwmc start
/usr/openv/netbackup/bin/bpps java | grep nbwmc
/usr/openv/netbackup/bin/nbcertcmd -ping

Windows:
"<install_path>\netbackup\wmc\bin\nbwmc.exe" -start -srvname "NetBackup Web Management Console"
"<install_path>\netbackup\bin\bpps" | findstr /i nbwmc
"<install_path>\netbackup\bin\nbcertcmd" -ping

10. If this is an active cluster node, unfreeze the cluster.