Java keystore fingerprint error during NetBackup upgrade pre-check

Problem

One of the upgrade pre-check tools reports that there is a problem with the certificate authority (CA) fingerprint in the Java keystore files.

This applies to NetBackup master server upgrades for NetBackup 8.2 - 9.0.

Error Message

From: Veritas™ Services and Operations Readiness Tools (SORT) > NetBackup Family > Installation and Upgrade report

Java keystores fingerprints check
[<hexadecimal_string>] fingerprint does not match any CA/tomcat certificate files. At least one Java Keystore CA fingerprint does not match the RB certificate fingerprint.
      
     This test runs when you upgrade the NetBackup master server from 8.0 to 8.1 or later NetBackup version.
     Check Java Keystores fingerprints (*.jks and truststoreNBWSS) for tomcatcreds wsl or credentials.
     Refer to the following article for more details: Technote 100045513 https://www.veritas.com/docs/100045513.

From: NetBackup upgrade process (nbcheck) report

not ok java_key_store: [<hexadecimal_string>] fingerprint does not match any CA/tomcat certificate files.
     At least one Java Keystore CA fingerprint does not match the RB certificate fingerprint.

In addition the /usr/openv/wmc/webserver/logs/configureCerts.log, cumulative from a prior upgrade, may show an error like this that is relative to one, two, or three of the files indicated here.

Changing the truststore password for the file /usr/openv/var/global/wsl/credentials/truststoreNBWSS
Illegal option:  /usr/openv/var/global/wsl/credentials/truststoreNBWSS
keytool -storepasswd [OPTION]...
...snip...
Changing the truststore password for the file /usr/openv/var/global/wsl/credentials/truststoreMSDP
Illegal option:  /usr/openv/var/global/wsl/credentials/truststoreMSDP
keytool -storepasswd [OPTION]...
...snip...
Changing the truststore password for the file /usr/openv/var/global/wsl/credentials/truststoreResiliencyPlatform
Illegal option:  /usr/openv/var/global/wsl/credentials/truststoreResiliencyPlatform
keytool -storepasswd [OPTION]...

Cause

When initially created, these files contain a placeholder, the then current CA certificate for the master server. Normally the CA certificate of a master server never changes.  But if it does, then the contents of these files may become stale.

There are several causes, both operational and programmatic, that can cause this situation.  But the situation will not be detected until months or years later during the next upgrade.

Solution

Note: This solution should be applied when only the pre-check failures noted above exist, and no others.  The pre-checks noted above rely on baseline information from other files that might not be correct.  If other certificate pre-checks have failed, resolve those issues first, and then see if this check continues to fail.

Once all other check failures have been resolved, confirm the following situation exists.  This solution applies only to this specific situation.

1) Extract the CA fingerprint for the NetBackup Web Management Console (nbwmc).

$ cd /usr/openv/var/global
$ /usr/openv/netbackup/bin/goodies/vxsslcmd x509 -fingerprint -in ./webrootcert.pem -noout 2>/dev/null

SHA1 Fingerprint=<hexadecimal_string>

2) Extract the entries and fingerprints from this file.  Confirm there is only one entry and that the fingerprint matches step 1.

$ /usr/openv/java/jre/bin/keytool -list -keystore ./vxss/tomcatcreds/truststoreNBWSS < ./jkskey

...snip...
Your keystore contains 1 entry
ca1, ...snip...
Certificate fingerprint (SHA1): <hexadecimal_string>

3) Extract the same from these files, if the files exist.  Earlier versions of NetBackup will only have one or two instead of three.

$ /usr/openv/java/jre/bin/keytool -v -list -keystore ./wsl/credentials/truststoreNBWSS < ./jkskey

$ /usr/openv/java/jre/bin/keytool -v -list -keystore ./wsl/credentials/truststoreMSDP < ./jkskey

$ /usr/openv/java/jre/bin/keytool -v -list -keystore ./wsl/credentials/truststoreResiliencyPlatform < ./jkskey

4) Files from step 3 that contain the same fingerprint as steps 1 & 2 are intact, ignore them.

5) Files from step 3 that contain a different fingerprint need to be inspected more closely.

6) If a file contains just one entry, but with a different fingerprint, then overwrite with the file from step 2.

For example:

$ cp -p ./vxss/tomcatcreds/truststoreNBWSS ./wsl/credentials/truststoreNBWSS
$ cp -p ./vxss/tomcatcreds/truststoreNBWSS ./wsl/credentials/truststoreMSDP
$ cp -p ./vxss/tomcatcreds/truststoreNBWSS ./wsl/credentials/truststoreResiliencyPlatform

7) If all files are now correct, rerun the upgrade pre-check tool to confirm the corrections, and then proceed with the upgrade.

8) If a file contains more than one entry, please contact NetBackup Technical Services for guidance.

There are no methods to address this issue by way of a patch or hotfix in the current or previous versions of the software at the present time. However, this issue is currently scheduled to be addressed in the next major revision of the product and in the next version of SORT. Please note that Veritas Technologies LLC reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests.  Veritas’ plans are subject to change, and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.