1. Support
  2. Knowledge base
  3. 100039943

How to identify and correct a corrupted certificate authority (CA) certificate.

Article: 100039943
Last Published: 2021-05-27
Ratings: 6 0
Product(s): NetBackup & Alta Data Protection

Problem

This is "Issue 2: A corrupted certificate authority (CA) certificate” as referenced in the master article for troubleshooting CRL-related problems, 000127887

Default file location:
Windows:    <install_path>\NetBackup\var\webtruststore\cacert.pem
Unix/Linux:  /usr/openv/var/webtruststore/cacert.pem

In the event that the CA certificate is corrupted / unreadable, backups will fail in the following manner:

Media server:
In the case the media server is the entity with the corrupted CA certificate, backups will hang after requesting resources:
Sep 5, 2017 9:42:59 AM - awaiting resource nbmedia2-hcart2-robot-tld-1. Waiting for resources.
          Reason: Media server is currently not connected to master server, Media server: nbmedia2,
          Robot Type(Number): TLD(1), Media ID: N/A, Drive Name: N/A,
          Volume Pool: NetBackup, Storage Unit: nbmedia2-hcart2-robot-tld-1, Drive Scan Host: N/A,
          Disk Pool: N/A, Disk Volume: N/A
//job hung here//


 A review of the media server in the Media and Device Management --> Devices --> Media Servers shows the server as offline, even though all processes are running.
 
User-added image



Client:
In the case the client is the entity with the corrupted CA certificate, backups will fail with status 7625.

Example Job details:
Sep 1, 2017 5:01:36 PM - Error bpbrm (pid=27612) [PROXY] Received status: 7660 with message Unable to read the certificate mapping file.
Sep 5, 2017 11:29:31 AM - Error bpbrm (pid=21674) [PROXY] Received status: 7625 with message A SSL connect failed. Status: 5 Msg: A non-recoverable I/O error occurred. The ssl error queue was empty
Sep 5, 2017 11:29:31 AM - Error bpbrm (pid=21674) bpcd on nbclient1 exited with status 7625: A SSL socket connect failed
Sep 5, 2017 11:29:31 AM - Error bpbrm (pid=21674) [PROXY] Connecting host: nbmaster2..fqdn.com
Sep 5, 2017 11:29:31 AM - Error bpbrm (pid=21674) cannot send mail because BPCD on nbclient1 exited with status 61: the vnetd proxy encountered an error
Sep 5, 2017 11:29:31 AM - Info bpbkar (pid=0) done. status: 7625: A SSL socket connect failed
A SSL socket connect failed.  (7625)
 
For both media servers and clients, the issue can be confirmed by running nbcertcmd -listCACertDetails
nbcertcmd -listCACertDetails
nbcertcmd: The -listCACertDetails operation failed.
EXIT STATUS 13: file read failed


A review of the resulting nbcert log file will show the inability to display CA Certificate, “ PEM_X509_INFO_read_bio failed” and return value//exit status 13.

10:23:59.973 [3436.3556] <2> nbcertcmd: INITIATING: NetBackup 8.1 created: 0
10:23:59.973 [3436.3556] <2> nbcertcmd: nbcertcmd -listCACertDetails
//cut//
10:24:00.098 [3436.3556] <2> nbcertcmd: Displaying trusted CA details...
10:24:00.098 [3436.3556] <2> getCACertPath: CA Cert path is [C:\Program Files\Veritas\NetBackup\var\webtruststore\cacert.pem]
10:24:00.098 [3436.3556] <2> DisplayTrustedCADetails: Access to certificate path is successful
10:24:00.098 [3436.3556] <2> DisplayTrustedCADetails: successfully created BIO
10:24:00.098 [3436.3556] <16> DisplayTrustedCADetails: PEM_X509_INFO_read_bio failed
10:24:00.098 [3436.3556] <16> nbcertcmd: DisplayTrustedCADetails failed. retval =13
10:24:00.098 [3436.3556] <2> nbcertcmd: EXIT STATUS 13: file read failed

Another indication of this issue is the return of bptestbpcd with error 7625:
# bptestbpcd -host nbmedia2
<16>bptestbpcd main: Function ConnectToBPCD (nbmedia2) failed: 7625
<16>bptestbpcd main: A SSL socket connect failed
A SSL socket connect failed

When this occurs, bpclntcmd -pn will show no output if run from the media server or client, nor will any errors be returned.

Use the verbose flag when  this occurs to get further information:   
bpclntcmd -pn -verbose 
 
Reminder:  For information about log file verbose or debug levels, see the parent article, 000127887
 

Solution

To correct this issue, complete the following on the host (client or media server) reporting the error:

1.    Move or remove the file, cacert.pem. The default file location:
  • Windows:    <install_path>\NetBackup\var\webtruststore\cacert.pem
  • Unix/Linux:  /usr/openv/var/webtruststore/cacert.pem

Note:  This removes the cache of all CA certificates on this host from all master servers, not just the one with the corrupt certificate.  

2.    Execute nbcertcmd -getCACertificate:
nbcertcmd -getCACertificate
Authenticity of root certificate cannot be established.
The SHA1 fingerprint of root certificate is [Master CA Certificate fingerprint]
Are you sure you want to continue using this certificate ? (y/n): y
The validation of root certificate fingerprint is successful.
CA certificate stored successfully from server nbmaster2.

Note: If this host is a member of more than one NetBackup domain, check using nbcertcmd -listCertDetails, then use nbcertcmd -getCAcertificate -server <other_master_server> for each of the other domains. 

3.    Verify that nbcertcmd -listCACertDetails now reports that a valid CA certificate exists.
Example:

nbcertcmd -listCACertDetails

      Subject Name : /CN=nbatd/OU=root@nbmaster2.fqdn.com/O=vx
        Start Date : Sep 01 14:40:51 2017 GMT
       Expiry Date : Aug 27 15:55:51 2037 GMT
  SHA1 Fingerprint : [Master CA Certificate fingerprint]

Was this content helpful?

Article Languages