NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup & Alta Data Protection (10.5)
  1. Introduction
    1.  
      Additional resources on NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX or Linux servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the primary server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Primary server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX primary server and client
      2.  
        Example of host name and service entries on UNIX primary server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host properties to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. Troubleshooting Auto Image Replication
      1.  
        Rules for primary servers used with Auto Image Replication and SLPs
      2. Targeted A.I.R. trusted primary server operation failed in case of external certificate configuration
        1.  
          Add or update trust
        2.  
          Remove trust
      3.  
        About troubleshooting automatic import jobs that SLP components manage
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Troubleshooting error messages in the NetBackup web UI and the NetBackup Administration Console
    29.  
      Extra disk space required for logs and temporary files for the NetBackup Administration Console
    30.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    31.  
      Troubleshooting file-based external certificate issues
    32.  
      Troubleshooting issues with external certificate configuration
    33.  
      Troubleshooting Windows certificate store issues
    34.  
      Troubleshooting backup failures
    35.  
      Troubleshooting backup failure issues with NAT clients or NAT servers
    36.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
    37.  
      Troubleshooting issues with email notifications for Windows systems
    38.  
      Troubleshooting issues with KMS configuration
    39.  
      Troubleshooting issues with initiating the NetBackup CA migration because of large key size
    40.  
      Troubleshooting issues with the non-privileged user (service user) account
    41.  
      Troubleshooting issues with group name format in the auth.conf file
    42.  
      Troubleshooting the VxUpdate add package process
    43.  
      Troubleshooting issues with FIPS mode
    44.  
      Troubleshooting issues with malware scanning
    45.  
      Troubleshooting issues with NetBackup jobs that are enabled for data-in-transit encryption
    46.  
      Troubleshooting issues with Unstructured Data Instant Access
    47.  
      Troubleshooting issues with multifactor authentication
    48.  
      Troubleshooting issues with multi-person authorization
    49.  
      Troubleshooting connections to the NetBackup Scale-Out Relational Database
    50.  
      Troubleshooting issues with private key encryption
    51.  
      Troubleshooting issues with the security configuration risk feature
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
    10. About the NetBackup Smart Diagnosis (nbsmartdiag) utility
      1.  
        Workflow to use the nbsmartdiag utility for NetBackup host communication
    11.  
      About log collection by job ID
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      Recommended backup practices
    3.  
      Requirements and notes for disaster recovery
    4.  
      Disaster recovery packages
    5.  
      About disaster recovery settings
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the primary server disk on Linux
        1.  
          Recovering the primary server when root is intact
        2.  
          Recovering the primary server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the primary server disk for Windows
        1.  
          Recovering the primary server with Windows intact
        2.  
          Recovering the primary server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered primary server after disaster recovery installation
    11.  
      About the DR_PKG_MARKER_FILE environment variable
    12.  
      Restoring the disaster recovery package on Windows
    13.  
      Restoring the disaster recovery package on Linux
    14. Options to recover the NetBackup catalog
      1. Prerequisites for recovering the NetBackup catalog or NetBackup catalog image files
        1.  
          Establishing a connection with NAT media server before catalog recovery
      2.  
        About NetBackup catalog recovery on Windows computers
      3.  
        About NetBackup catalog recovery from disk devices
      4.  
        About NetBackup catalog recovery and symbolic links
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the NetBackup catalog recovery wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
        3.  
          Specifying the NetBackup job ID number after a catalog recovery
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the NetBackup catalog recovery wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup databases
        1.  
          Recovering the NetBackup database from a backup
        2.  
          Recovering the NetBackup database from staging
        3.  
          About processing the NetBackup database in staging
        4.  
          Terminating database connections
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Troubleshooting issues with private key encryption

This topic provides information on how to troubleshoot issues that are specific to private key encryption.

Passphrases are used to encrypt and decrypt the private keys of NetBackup host ID-based certificates. Passphrase keys are used to encrypt and decrypt these passphrases.

The private key of the NetBackup certificate is stored in an encrypted format using AES_256_CBC encryption. The password that is used to encrypt the private keys is stored in file storage and is encrypted using AES_256_GCM encryption.

Private key encryption file paths

Keystore location:

On Windows: Install path\NetBackup\var\vxss\credentials\keystore

Linux: /usr/openv/var/vxss/credentials/keystore

Keystore location for cluster:

/usr/openv/var/global/vxss/credentials/keystore

Nbcert logs:

On Windows: Install path\NetBackup\logs\nbcert

On Linux: /usr/openv/netbackup/logs/nbcert

Passphrase file path: keystorepath + .yekekp

Passphrase key file path: keystorepath + .yekcneekp

certmapinfo.json file path:

On Windows: Install path\NetBackup\var\vxss\certmapinfo.json

On Linux: /usr/openv/var/vxss/certmapinfo.json

Table:

Sr. No.

Issue

Possible reason

Resolution

1

Command: nbcertcmd -listcertdetails

Output:
Private Key Encryption State:
 Encrypted with an unknown 
passphrase

The private key file is tampered.

  1. Clean up the private key file for the server.

  2. Run the following command on all the servers that are associated with the host:

    • nbcertcmd -getCertificate -token reissue_token -server server host name -force

2

For the following problem scenarios, the reason and the resolution are the same:

Command: nbcertcmd -listcertdetails

Output:

Private Key 
Encryption State: Encrypted 
with an unknown passphrase

Command: nbcertcmd -rotatePassphrasekey

The passphrase 
key rotation failed.
EXIT STATUS 1200: Internal 
error

The passphrase file or the passphrase key file is tampered.

  1. Check the last modification date of the passphrase file.

  2. Clean up the keystore folder including the hidden files.

  3. Run the following command on all the servers that are associated with the host:

    • nbcertcmd -getCertificate -token reissue_token -server server host name -force

3

While you perform catalog restore after the fresh NetBackup installation, you can see both the newly-created private keys from the fresh installation and the restored ones.

Command:

ls -la

total 20 drwx------ 2 nbsvcusr nbsvcusr 171 Jun 19 19:38

drwx------ 3 nbsvcusr nbsvcusr 133 Jun 19 19:25 ..

-rw------- 1 nbsvcusr nbsvcusr 1858 Jun 19 19:38

015b91f5-74b5-44fb- 865f-6d65827cdb30-key.pem

-rw------- 1 nbsvcusr nbsvcusr 1858 Jun 19 19:38

015b91f5-74b5-44fb-865f- 6d65827cdb3r-key.pem

Restoring the catalog reintegrates the existing private keys and passphrase files into the keystore. The keystore then includes both the newly-created private keys from the fresh installation and the restored ones.

  • Clear the private key files that do not have entry in the certmapinfo.json file.

Location of the certmapinfo.json file on Unix: /usr/openv/var/vxss/certmapinfo.json

4

The following notification is seen on the NetBackup web UI:

Reissuing the host certificates during private key encryption failed for the following hosts: host1

Reissue of the certificate is attempted during the private key encryption operation.

  • Run the following command:

    nbcert -listCertDetails -json

    The subsequent restart of the services may encrypt all the private keys and the output of this command shows all the keys in the Encrypted state.

If all the keys are not encrypted, run one of the following commands for the private keys with state other than Encrypted:

  • nbcertcmd -reissuecertificates -server server

  • nbcertcmd -getCertificate -token reissue_token -server server host name -force

5

The attempt to rotate the passphrase failed, the private key files and the passphrase file could not be restored.

Command: [root@example keystore]

nbcertcmd -rotatepassphrase

This operation performs the rotation of passphrase that encrypts the private key of the host ID-based certificates.

It is strongly recommended that you stop the NetBackup services before you perform this operation. Ensure that you restart the services after the operation is performed.

Are you sure you want to proceed with this operation? (y/n) y

The passphrase 
rotation failed.
EXIT STATUS 9141: Keystore 
is in inconsistent state.

Command:
ls -la
total 20
drwx------ 2 nbsvcusr 
nbsvcusr  176 Jul 16 11:55 .
drwx------ 3 nbsvcusr 
nbsvcusr  133 Jul  4 22:24 ..
-rw------- 1 nbsvcusr 
nbsvcusr 1858 Jul 16 11:51 
5176ec69-d3cb-44d7-a229-
799555b7bd7e-key.pem
-rw------- 1 nbsvcusr 
nbsvcusr 1858 Jul 16 11:54
 5176ec69-d3cb-44d7-a229-
799555b7bd7e-key.pem_bkup
-rw------- 1 nbsvcusr 
nbsvcusr 1858 Jul 16 11:51 
PrivKeyFile-2048.pem
-rw-r--r-- 1 nbsvcusr 
nbsvcusr 1072 Jul 16 11:51
 .yekcneekp
-rw-r--r-- 1 nbsvcusr 
nbsvcusr 271 Jul 16 11:52
 .yekekp

The restore operation failed because of the absence of backup files or an issue with the file rewrite process.

  • Check if the backup files are present(files that have the suffix '_bkup') in the same keystore folder.

  • Perform following:

    • Verify the status using

      nbcertcmd -listcertdetails

    • If all the primary servers are showing the private key encryption status as Encrypted, clean up the backup files manually and retry the rotation operation.

  • If the issue still persists, check the following:

    • If some of the primary servers show a private key and the encryption status is 'encrypted with unknown passphrase', restore the passphrase file and the corresponding private key files.

    • Again, check the status using

      nbcertcmd -listcertdetails. Verify if the correct encryption status is shown for the remaining private keys. If it does, retry the rotation operation.

  • If the issue still persists, check the following:

    • If backup files are not present and the command

      nbcertcmd -

      listcertdetails

      shows the incorrect encryption status, clean up the keystore.

    • Run

      nbcertcmd -getCertificate

      with the reissueToken option for all servers.