NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (10.3.0.1, 10.3)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX or Linux servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the primary server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Primary server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX primary server and client
      2.  
        Example of host name and service entries on UNIX primary server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host properties to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. Troubleshooting Auto Image Replication
      1.  
        Rules for primary servers used with Auto Image Replication and SLPs
      2. Targeted A.I.R. trusted primary server operation failed in case of external certificate configuration
        1.  
          Add or update trust
        2.  
          Remove trust
      3.  
        About troubleshooting automatic import jobs that SLP components manage
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Troubleshooting error messages in the NetBackup Administration Console
    29.  
      Extra disk space required for logs and temporary files for the NetBackup Administration Console
    30.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    31.  
      Troubleshooting file-based external certificate issues
    32.  
      Troubleshooting Windows certificate store issues
    33.  
      Troubleshooting backup failures
    34.  
      Troubleshooting backup failure issues with NAT clients or NAT servers
    35.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
    36.  
      Issues with email notifications for Windows systems
    37.  
      Issues with KMS configuration
    38.  
      Issues with initiating the NetBackup CA migration because of large key size
    39.  
      Issues with the non-privileged user (service user) account
    40.  
      Issues with group name format in the auth.conf file
    41.  
      Troubleshooting the VxUpdate add package process
    42.  
      Issues with FIPS mode
    43.  
      Issues with malware scanning
    44.  
      Issues with NetBackup jobs that are enabled for data-in-transit encryption
    45.  
      Issues with Unstructured Data Instant Access
    46.  
      Troubleshooting issues with multi-factor authentication
    47.  
      Troubleshooting issues with multi-person authorization
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
    10. About the NetBackup Smart Diagnosis (nbsmartdiag) utility
      1.  
        Workflow to use the nbsmartdiag utility for NetBackup host communication
    11.  
      About log collection by job ID
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the primary server disk for UNIX and Linux
        1.  
          Recovering the primary server when root is intact
        2.  
          Recovering the primary server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the primary server disk for Windows
        1.  
          Recovering the primary server with Windows intact
        2.  
          Recovering the primary server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered primary server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About the catalog backup process
      2.  
        Prerequisites for recovering the NetBackup catalog or NetBackup catalog image files
      3.  
        About NetBackup catalog recovery on Windows computers
      4.  
        About NetBackup catalog recovery from disk devices
      5.  
        About NetBackup catalog recovery and symbolic links
      6. About NetBackup catalog recovery
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      7.  
        NetBackup disaster recovery email example
      8. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the NetBackup catalog recovery wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      9.  
        Establishing a connection with NAT media server before catalog recovery
      10. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the NetBackup catalog recovery wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      11. About recovering the NetBackup databases
        1.  
          Recovering the NetBackup database from a backup
        2.  
          Recovering the NetBackup database from staging
        3.  
          About processing the NetBackup database in staging
        4.  
          Terminating database connections
      12.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      13.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      14.  
        Recovering the NetBackup catalog without the disaster recovery file
      15.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      16.  
        Restoring files from a NetBackup online catalog backup
      17.  
        Unfreezing the NetBackup online catalog recovery media
      18.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Recovering the NetBackup catalog image files using the NetBackup catalog recovery wizard

This procedure describes how to recover the NetBackup catalog image files by using the NetBackup catalog recovery wizard.

Warning:

Do not run any client backups before you recover the NetBackup catalog.

To recover the catalog image files using the NetBackup catalog recovery wizard

  1. Run the nbgetconfig command and save the output. This output can be used after the catalog recovery to recover the host-specific information that is overwritten during the catalog recovery.

    For example:

    ./nbgetconfig > sample.txt

  2. Review the prerequisites before starting the catalog recovery.

    See Prerequisites for recovering the NetBackup catalog or NetBackup catalog image files.

  3. If NetBackup is not running, start all of the NetBackup services by entering the following:

    • On UNIX and Linux:

      /usr/openv/netbackup/bin/bp.start_all

    • On Windows:

      install_path\NetBackup\bin\bpup

  4. If the catalog backup and the recovery devices are not available, perform the following steps:
    • Configure the necessary recovery device in NetBackup.

    • Make available to NetBackup the media that contains the catalog backup: Inventory the robot or the disk pool, add the media for standalone drives, configure the storage server and disk pool, or so on.

    • Create symbolic links to match those in the original environment.

      See About NetBackup catalog recovery and symbolic links.

    For tape storage or BasicDisk storage, see the NetBackup Administrator's Guide, Volume I. For disk storage types, see the guide that describes the option. See the following website for NetBackup documentation:

  5. Open the NetBackup web UI.
  6. At the top, click Settings > NetBackup catalog recovery.
  7. Specify where the disaster recovery file is stored. You can browse to select the file or enter the full pathname to the disaster recovery file.

    In most cases, you specify the most recent disaster recovery information file available. If the most recent catalog backup is an incremental backup, use the disaster recovery file from the incremental backup. (There is no need to first restore the full backup and then follow with the incremental backup.)

    If some form of corruption has occurred, you may want to restore to an earlier state of the catalog.

    Click Next to continue.

  8. NetBackup searches for the media that are required to recover the catalog. It then informs you of the progress and locates the necessary backup ID of the disaster recovery image. If the media is not located, NetBackup indicates which media is needed to update the database.

    If necessary, follow the wizard instructions to insert the media that is indicated and run an inventory to update the NetBackup database. The information that is displayed on this panel depends on whether the recovery is from a full backup or an incremental backup.

    When the required media sources are all found, click Next.

  9. Select Recover only NetBackup catalog image and configuration files.

    Select a Job priority if wanted and then click Next to initiate the recovery.

  10. NetBackup displays the recovery progress.

    Your action depends on the outcome of the recovery, as follows:

    Not successful

    Consult the log file messages for an indication of the problem. Click Cancel, fix the problem, and then run the wizard again.

    Successful

    Click Next to continue to the final wizard panel.

  11. After the recovery completes, click Sign Out.

    Each image file is restored to the proper image directory and the configuration files are restored.

  12. If you want to recover the image header information without recovering the entire NetBackup database, perform the following steps:
    • Step a - Back up the target database. Run the following command.

      nbdb_backup -online directory

      Make sure that you do not specify the staging folder as the output directory. (The staging folder contains the schema data and configuration data for the NetBackup database from the catalog backup. Image .f and configuration files are recovered to their final destinations.)

    • Step b - Recover the NetBackup database from the staging directory.

      nbdb_restore -recover -staging
    • Step c - Export the image header data that you want to import from the backup.

      For example, the following command exports export all image header data. The data is exported to the netbackup/db.export directory.

      cat_export -all
    • Step d- Recover the NetBackup database with the following command.

      nbdb_restore -recover directory

      Make sure that you specify the same directory as in step a.

    • Step e- Run the cat_import command to import the image header data that you extracted in step c.

      cat_import -all -replace_destination -delete_source

      The command does the following:

      • Imports all of the image header data in the netbackup/db.export directory.

      • Replaces any image header data that was exported that already exists in the target database.

      • Removes the image header data that resides in the netbackup/db.export directory.

    • Step f- If you recovered the catalog from a disk device, you may have to fix the disk media ID references. Run the following command:

      nbcatsync -sync_dr_file DR file path -dryrun

      Replace DR file path with the path to the catalog DR file.

    • Step g - If the result of the dry run is satisfactory, run the following command:

      nbcatsync -sync_dr_file DR file path
  13. Before you continue, be aware of the following points:

    • If you recovered the catalog from removable media, NetBackup freezes the catalog media.

    • Before you restart NetBackup, Veritas recommends that you freeze the media that contains the backups more recent than the date of the catalog from which you recovered.

    • NetBackup does not run scheduled backup jobs until you stop and then restart NetBackup.

      You can submit backup jobs manually before you stop and restart NetBackup. However, if you do not freeze the media that contains the backups more recent than the date of the catalog from which you recovered, NetBackup may overwrite that media.

    • Because this operation is a partial recovery, you must recover the database portion of the catalog.

  14. Recover the host settings that you backed up in step 1. Run the following command.
    ./nbsetconfig sample.txt
  15. Stop and restart the NetBackup services on the primary server, as follows:

    • On UNIX and Linux:

      /usr/openv/netbackup/bin/bp.kill_all
      /usr/openv/netbackup/bin/bp.start_all
    • On Windows:

      install_path\NetBackup\bin\bpdown
      install_path\NetBackup\bin\bpup
  16. After the services are restarted, run the following command:

    On a non-clustered setup:

    Windows:

    install_path\netbackup\bin\nbcertcmd -renewcertificate

    UNIX:

    /usr/openv/netbackup/bin/nbcertcmd -renewcertificate

    On a clustered setup:

    Windows:

    install_path\netbackup\bin\nbcertcmd -renewcertificate -cluster

    UNIX:

    /usr/openv/netbackup/bin/nbcertcmd -renewcertificate -cluster
  17. If the catalog recovery is part of a server recovery procedure, complete the remaining steps in the appropriate recovery procedure.

    Recovery can include the following:

    • Importing the backups from the backup media into the catalog.

    • Write protecting the media.

    • Ejecting the media and setting it aside.

    • Freezing the media.