Problem
Corruption of files can occur due to unexpected system events, such as the sudden shutdown or reboot of the system, hardware errors, such as a faulty disks, or software/operating system type errors triggering incorrect writes of data to disk. This article will explain how to identify and correct certificate or Certificate Revocation List (CRL) related issues in NetBackup 8.1 and higher to include the following issues:
-
A corrupted certmapinfo.json file
-
Resulting in status codes/errors include: 13, 5949, 7660
-
-
A corrupted certificate authority (CA) certificate
-
Status codes/errors: hung jobs, offline media servers, 13, 7625, 7660
-
-
A corrupted local host ID-based certificate.
-
Status codes/errors: 25, 7625, 5942
-
-
A corrupt certificate revocation list (CRL)
-
Status codes/errors: 23, 25, 7640, 7654, 9301
-
Error Message
Error codes, status codes and messages that may be experienced include, but are not limited to the following:Error | Message |
13 | File read failed |
23 | Socket read failed. |
25 | Cannot connect on socket. |
61 | The vnetd proxy encountered an error |
5949 | Certificate does not exist. |
5978 | Unable to read the certificate mapping file. |
5942 | Certificate could not be read from the local certificate store |
7624 | SSL accept failed. |
7625 | SSL socket connect failed |
7627 | PEM_X509_INFO_read_bio failed |
7640 | The peer closed the connection |
7660 | The peer host certificate cannot be verified using the Certificate Revocation List. |
9301 | Failed to decode certificate revocation list |
N/A | Backups hung waiting for resources. |
Cause
Best Practices
Certificate, CRL and certmapinfo.json file problems are easiest to identify on the client, or media server reporting the issue.
Log files that should be enabled on the server reporting the issue, when troubleshooting the error codes listed above when they are related to certificate or CRL type issues are:
Log file directory | VERBOSE or Debug Level | OID |
*nbpxyhelper | DebugLevel=4 | 486 |
nbcert | VERBOSE = 5 | |
bpcd | VERBOSE = 5 | |
bprd (master server only) | VERBOSE = 5 |
* Unix/Linux operating systems, this is located in /usr/openv/logs. All other log directories referenced are located in netbackup/logs for both Unix/Linux and Windows.
**Note that the DiagnosticLevel should always be set to 6.
The commands, nbcertcmd, bptestbpcd and bpclntcmd, are useful in troubleshooting certificate and CRL related issues. Common syntax of the commands utilized in troubleshooting are:
bpclntcmd -pn
bpclntcmd -hn <hostname>
bptestbpcd -host [host]
nbcertcmd -getCRL
nbcertcmd -getCertificate
nbcertcmd -getCertificate -force
nbcertcmd -getCertificate -token <reissue_token> -force
nbcertcmd -getCACertificate
nbcertcmd -hostselfcheck
nbcertcmd -listCACertDetails
nbcertcmd -listCertDetails
The commands are all located in:
Windows: <install_path>\Veritas\NetBackup\bin
Unix/Linux: /usr/openv/netbackup/bin
For additional information, see the NetBackup 8.1 Commands Reference Guide.
Solution
Select the following links to be directed identification and solution of each issue:Issue 1: A corrupted certmapinfo.json file
Issue 2: A corrupted certificate authority (CA) certificate
Issue 3: A corrupted local host ID-based certificate
Issue 4 : A corrupt certificate revocation list (CRL)