nbcheck reports tomcat_certificates failure during NetBackup upgrade

Problem

nbcheck is used to confirm that the expected certificate and password files used by the Tomcat service are present on a NetBackup master server, and have valid contents, prior to performing an upgrade.

This check runs only on standalone and active-node master server hosts.  The check is applicable to upgrades from NetBackup versions 8.0 or later.

Error Messages

This check performs many steps, any of which might find a problem.  The unique messages for each step are detailed below. All of the messages will include this common footer.

This test runs when you upgrade the NetBackup master server from 8.0 to 8.1 or later.
Check Tomcat and it's trusted certificates health. See
    https://www.veritas.com/support/en_US/article.100045515
for more information.

This error indicates that Tomcat certificate files are not present in any of the expected directories.

not ok tomcat_certificates: No Tomcat certificate files found.

This error indicates that the files in the expected directories could not be parsed successfully by the vxsslcmd command.

not ok tomcat_certificates: Failed to get Tomcat certificate data.

This error indicates that some of the files contained certificates, but none were Tomcat certificates.

not ok tomcat_certificates: tomcat certificate is missing

This error indicates that a Tomcat certificate was found, but the name of the certificate file does not match with the first SERVER in the NetBackup configuration.

not ok tomcat_certificates: At least one Tomcat certificate file name does not match the first configured SERVER name.
Certificate : <certificate_file_path>

This error indicates that the Tomcat certificate is less than 150 days from expiration and has not auto-renewed.

not ok tomcat_certificates: The Tomcat certificate is not auto-renewing.

This error indicates that the Tomcat certificate, located in the certstore directory, is expired.

not ok tomcat_certificates: Tomcat certificate is expired in certstore directory.

This error indicates that the Tomcat certificate is missing from the certstore directory.

not ok tomcat_certificates: Tomcat certificate is missing from the certstore directory.

This error indicates that the Tomcat certificate, located in the credentials directory, is expired.

not ok tomcat_certificates: Tomcat certificate is expired in credentials directory.

This error indicates that the Tomcat certificate is missing from the credentials directory.

not ok tomcat_certificates: Tomcat certificate is missing from the credentials directory.

This error indicates that the Tomcat certificate, located in the tomcatcreds directory, is expired.

not ok tomcat_certificates: TomcatCert.pem file is expired in tomcatcreds directory.

This error indicates that the Tomcat certificate is missing from the tomcatcreds directory.

not ok tomcat_certificates: TomcatCert.pem file is missing from tomcatcreds directory.

This error indicates that more than one Tomcatcert.pem file exists.

not ok tomcat_certificates: More than one Tomcatcert.pem file exists.

This error indicates that the Tomcat certificate exists in more than 3 files. This error is specific to Windows platforms.

not ok tomcat_certificates: Tomcat certificate exists in more than 3 files.

This error indicates that the Tomcat certificate exists in more than 2 files. This error is specific to Linux/UNIX platforms.

not ok tomcat_certificates: Tomcat certificate exists in more than 2 files.

This error indicates that none of the files in the Tomcat trusted Certificate Authority (CA) tuststore directory could be parsed successfully by the vxsslcmd command.

not ok tomcat_certificates: Failed to get Tomcat trusted certificate data.

This error indicates that a trusted Certificate Authority (CA) certificate was not found in the truststore directory.

not ok tomcat_certificates: Tomcat trusted certificate is missing from the truststore.

This error indicates, on Windows platforms, that there are more trusted CA certificates present than expected.

not ok tomcat_certificates: Tomcat trusted CA certificate exist in more than two file.

This error indicates, on Windows platforms, that the trusted cacert.pem file is missing from the expected directory.

not ok tomcat_certificates: Trusted CA certificate CaCert.pem is missing from tomcatcreds directory.

This error indicates, on Linux/Unix platform, that there exist more trusted CA certificates than expected.

not ok tomcat_certificates: Tomcat trusted CA certificate exist in more than one file.

This error indicates that one of the trusted CA certificates is expired.

not ok tomcat_certificates: One of the tomcat trusted CA certificate is expired.

This error indicates that one or more trusted CA certificate is not yet valid.

not ok Tomcat_certificates: Tomcat trusted CA certificate is yet to start.

This error indicates that the Root Broker (RB) certificate information could not be obtained for comparison to the trusted CA certificate files.

not ok tomcat_certificates: Could not get Root Broker certificate info.

This error indicates that multiple Tomcat trusted certificates are present and have differing SHA1 fingerprints indicating the files unexpectedly have different contents.

not ok tomcat_certificates: Multiple Tomcat certificates present with different fingerprints.

This error indicates that a Tomcat trusted CA certificate has a SHA1 fingerprint that does not match the Root Broker certificate.

not ok tomcat_certificates: Tomcat trusted CA certificate does not match with RB certificate.

This error indicates that the Tomcat private key file is missing from the keystore directory.

not ok tomcat_certificates: Tomcat private key file is missing from keystore.

This error indicates that the Tomcat private key file cannot to be parsed by the vxsslcmd command.

not ok tomcat_certificates: Cannot fetch tomcat private key file information.

This error indicates that the Tomcat private key file has a modulus that does not match with the modulus of the Tomcat certificate.

not ok tomcat_certificates: Tomcat private key modulus does not match with certificate modulus.
Tomcat Certificate : <file_pathname>

Solution

Note: nbcheck only performs the tomcat_certificates check on master servers using a NetBackup Certificate Authority (NBCA).  It does not perform the same check on master servers using an External Certificate Authority (ECA).  This solution should not be used on a host configured for an ECA.

Note: If a problem was reported with the Root Broker certificate file, please engage NetBackup Technical services to ensure the situation is accurately assessed and resolved in the most appropriate way.  This may require reissuing host ID certificates to all NetBackup 8.x hosts.

Note: Verify that the log4j mitigation steps have not been performed, and that the .war files are present before following the rest of this document:
Windows:
cd [install path]NetBackup\wmc\webserver
dir /a /S /b | findstr /r .war$

You should see at least the first 4 of these files at 8.1.2+  (ROOT.war may not exist)

C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps\nbwebservice.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api\nbwss.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api\netbackup.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api\webui.war
C:\Program Files\Veritas\NetBackup\wmc\webserver\webapps_api_cssc\ROOT.war

Linux/Unix:
find /usr/openv/wmc/webserver -name "*.war"

You should see at least the first 4 of these files at 8.1.2+  (ROOT.war may not exist)

/usr/openv/wmc/webserver/webapps_api/netbackup.war
/usr/openv/wmc/webserver/webapps_api/nbwss.war
/usr/openv/wmc/webserver/webapps_api/webui.war
/usr/openv/wmc/webserver/webapps_api_cssc/ROOT.war
/usr/openv/wmc/webserver/webapps/nbwebservice.war

If these .war files are missing, you must either put them back from a backup or from a system that has not had the log4j mitigation steps performed, or apply the final fix log4j EEB before proceeding below:
https://www.veritas.com/support/en_US/article.100052058.html

Otherwise, the following procedure can be used to rebuild the Tomcat certificate related files.  If desired, NetBackup Technical Services can be engaged to assist with reviewing and correcting the situation.

1. Take a NetBackup catalog backup.
    
2. If this is an active cluster node, freeze the cluster so services do not fail-over to the other node.
    
3. Shutdown only the NetBackup web service (nbwmc), and confirm it has stopped.

Linux/UNIX:
/usr/openv/netbackup/bin/nbwmc terminate
/usr/openv/netbackup/bin/bpps java | grep nbwmc

Windows:
"<install_path>\netbackup\wmc\bin\nbwmc.exe" -stop -srvname "NetBackup Web Management Console"
"<install_path>\netbackup\bin\bpps" | findstr /i nbwmc

4. Delete the exiting Tomcat service credential files.

Linux/UNIX:
rm -rf /usr/openv/var/global/vxss/tomcatcreds
rm -f  /usr/openv/var/global/wsl/credentials/truststoreNBWSS

Windows: (not clustered)
rmdir <install_dir>\netbackup\var\global\vxss\tomcatcreds
del <install_dir>\netbackup\var\global\wsl\credentials\truststoreNBWSS

Windows: (active cluster node)
rmdir <shared_dir>\netbackup\var\global\vxss\tomcatcreds
del <shared_dir>\netbackup\var\global\wsl\credentials\truststoreNBWSS

5. Recreate the Tomcat certificate, private key, and trusted Certificate Authority (CA) files.

First make sure the first SERVER in the configuration is correct for the host.  Update the bp.conf file or registry setting if incorrect.

Second, obtain the list of hostnames by which other NetBackup hosts know the master server, so they can be placed into the new certificate.  E.g.  mymaster.com, mymaster, mymaster.backup.com, mymaster-bk

Third, on NetBackup 8.1.1+, use the -f option to force overwrite of existing Tomcat certificate files.

Linux/UNIX:
/usr/openv/netbackup/bin/nbgetconfig SERVER
/usr/openv/netbackup/bin/admincmd/nbcertconfig -t -user <web_service_user> [-f] [-sub <comma_separated_master_server_hostnames>]

Windows:
"<install_dir>\netbackup\bin\nbgetconfig" SERVER
set WEBSVC_PASSWORD=<password_of_user>
"<install_path>\netbackup\bin\admincmd\nbcertconfig.exe" -t -user <web_service_user> [-f] [-sub <comma_separated_master_server_hostnames>]

6. Setup the environment to be used by the steps that follow.  

The <host_name> should match the first SERVER entry.  On Windows, this should also match the -DNB_HOSTNAME value in <install_path>\NetBackup\wmc\bin\nbwmcservice.xml.

The -nbInstallDir should be similar to "C:\Program Files\Veritas", with "\NetBackup" as a sub-directory.

The platform should be one of: AMD64, hpia64, linuxR_x86, linuxS_x86, rs6000, solaris, solaris_x86.

Linux/UNIX:
cd /usr/openv/wmc/bin/install/
/usr/openv/wmc/bin/install/configureEnv -platform <platform_value> -nbHostName <host_name>

Windows:
cd "<install_path>\netbackup\wmc\bin\install"
"<install_path>\netbackup\wmc\bin\install\configureEnv.bat" -nbInstallDir "<install_path>" -nbHostName <host_name> -isClustered 0/1

Afterwards the environment setup can be confirmed by inspecting this file:

Linux/UNIX:
more /usr/openv/wmc/bin/setenv

Windows:
more "<install_path>\netbackup\wmc\bin\setenv.bat"

7. Configure web services preparation; sslStore, jkskeys, ports, webrootcert.pem, etc.

Linux/UNIX:
/usr/openv/wmc/bin/install/configureWmc

Windows:
"<install_path>\netbackup\wmc\bin\install\configureWmc.bat"

NOTE: If you run "configureWmc" command after having installed any HotFix or EEB that updates the security.war or netbackup.war file, the effects of installing the EEB will be reverted.  For an example of the steps necessary to reinstall the HotFix or EEB, see the related article about certificates not renewed after HotFix (100049294).

8. Configure web services; update the Java Keystore files from the certificate files, etc.

Linux/UNIX:
/usr/openv/wmc/bin/install/configureCerts

Windows:
"<install_path>\netbackup\wmc\bin\install\configureCerts.bat"

9. Setup web services; permissions, etc.

Linux/UNIX:
/usr/openv/wmc/bin/install/setupWmc -logFileName <setupWmc_log_file>

Windows:
"<install_path>\netbackup\wmc\bin\install\setupWmc.bat"  -logFileName <setupWmc_log_file>
"<install_path>\netbackup\wmc\bin\install\setupWmc.bat"  -logFileName <setupWmc_log_file>

Note: Run this command twice on Windows to recursively set directory and file permissions.

10. Restart web services, and confirm they are running and responding.

Linux/UNIX:
/usr/openv/netbackup/bin/nbwmc start
/usr/openv/netbackup/bin/bpps java | grep nbwmc
/usr/openv/netbackup/bin/nbcertcmd -ping

Windows:
"<install_path>\netbackup\wmc\bin\nbwmc.exe" -start -srvname "NetBackup Web Management Console"
"<install_path>\netbackup\bin\bpps" | findstr /i nbwmc
"<install_path>\netbackup\bin\nbcertcmd" -ping

11. If this is an active cluster node, unfreeze the cluster.