NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (10.3.0.1, 10.3)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX or Linux servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the primary server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Primary server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX primary server and client
      2.  
        Example of host name and service entries on UNIX primary server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host properties to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. Troubleshooting Auto Image Replication
      1.  
        Rules for primary servers used with Auto Image Replication and SLPs
      2. Targeted A.I.R. trusted primary server operation failed in case of external certificate configuration
        1.  
          Add or update trust
        2.  
          Remove trust
      3.  
        About troubleshooting automatic import jobs that SLP components manage
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Troubleshooting error messages in the NetBackup Administration Console
    29.  
      Extra disk space required for logs and temporary files for the NetBackup Administration Console
    30.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    31.  
      Troubleshooting file-based external certificate issues
    32.  
      Troubleshooting Windows certificate store issues
    33.  
      Troubleshooting backup failures
    34.  
      Troubleshooting backup failure issues with NAT clients or NAT servers
    35.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
    36.  
      Issues with email notifications for Windows systems
    37.  
      Issues with KMS configuration
    38.  
      Issues with initiating the NetBackup CA migration because of large key size
    39.  
      Issues with the non-privileged user (service user) account
    40.  
      Issues with group name format in the auth.conf file
    41.  
      Troubleshooting the VxUpdate add package process
    42.  
      Issues with FIPS mode
    43.  
      Issues with malware scanning
    44.  
      Issues with NetBackup jobs that are enabled for data-in-transit encryption
    45.  
      Issues with Unstructured Data Instant Access
    46.  
      Troubleshooting issues with multi-factor authentication
    47.  
      Troubleshooting issues with multi-person authorization
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
    10. About the NetBackup Smart Diagnosis (nbsmartdiag) utility
      1.  
        Workflow to use the nbsmartdiag utility for NetBackup host communication
    11.  
      About log collection by job ID
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the primary server disk for UNIX and Linux
        1.  
          Recovering the primary server when root is intact
        2.  
          Recovering the primary server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the primary server disk for Windows
        1.  
          Recovering the primary server with Windows intact
        2.  
          Recovering the primary server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered primary server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About the catalog backup process
      2.  
        Prerequisites for recovering the NetBackup catalog or NetBackup catalog image files
      3.  
        About NetBackup catalog recovery on Windows computers
      4.  
        About NetBackup catalog recovery from disk devices
      5.  
        About NetBackup catalog recovery and symbolic links
      6. About NetBackup catalog recovery
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      7.  
        NetBackup disaster recovery email example
      8. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the NetBackup catalog recovery wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      9.  
        Establishing a connection with NAT media server before catalog recovery
      10. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the NetBackup catalog recovery wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      11. About recovering the NetBackup databases
        1.  
          Recovering the NetBackup database from a backup
        2.  
          Recovering the NetBackup database from staging
        3.  
          About processing the NetBackup database in staging
        4.  
          Terminating database connections
      12.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      13.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      14.  
        Recovering the NetBackup catalog without the disaster recovery file
      15.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      16.  
        Restoring files from a NetBackup online catalog backup
      17.  
        Unfreezing the NetBackup online catalog recovery media
      18.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Issues with malware scanning

Table: Issues related to malware scanning

Error/Issue

Description

Workaround

Failed to get response from NetBackup malware utility.

(Applicable on scan host RHEL 8.x and NFS version 4.x) When scanning large size backup (~ 200 million files), following error is displayed on the Web UI for failed job:

Failed to get response from NetBackup 
malware utility.

While scan is in progress on scan host, NFS mount points are not accessible from scan host. Scan job remains in progress and timeout after two days. NFS exports on storage server are accessible.

Ensure that you use NFS version 3 for mounting IA mounts on scan host over NFS by setting the following configuration in /etc/nfsmount.conf file on scan host:

# grep Defaultvers /etc/nfsmount.conf Defaultvers=3

Failed to connect to the scan host.

SSH connection to scan host from media server failed.

Verify the following scan host credentials:

  • RSA (SHA256) key

  • User name

  • Password

Refer to NetBackup Web UI Administrator's Guide for the scan host configuration.

Failed to determine the scan host OS.

Error can be due to unsupported scan host.

For a complete list of supported platforms for the scan host, refer to the Software Compatibility list document.

Failed to copy NetBackup malware utility to the scan host.

  • Not enough space is available on the scan host.

  • SSH user does not have access to the required directories on the scan host.

  • On a Windows scan host, check for space availability in C:\ folder.

  • On a Linux scan host, check for space availability in /tmp folder.

Failed to get the scan host credentials.

Media server is not able to fetch the credentials to access scan host from the Primary.

Check that credentials for scan host are specified.

Time-out has occurred during the scan.

Default scan operation time out is two days. Time to scan may vary depending on the factors sch as workload type, network bandwidth, backup size.

Scan time-out is configurable and can be changed by setting the MALWARE_SCAN_OPERATION_TIMEOUT configuration key.

  • Minimum value: 1 hour

  • Maximum value: 30 days

Failed to get response from NetBackup malware utility.

Mismatch between nbmalwareutil binary and the ScanManager

Contact NetBackup support.

Failed to launch the scanner.

Malware scanner-specific failure message.

Refer to nbmalwarescanner logs on the media server.

Failed to mount the backup image.

IA share is not accessible from the scan host.

Check IA configuration on storage server. Verify on activity monitor that IA job is successful.

Failed to unmount the backup image.

IA share is busy or not accessible.

Refer to nbmalwarescanner logs on the media server.

Failed to run a scan.

Generic failure during the scan of a backup image.

Refer to nbmalwarescanner logs on the media server.

Instant access mount created but not deleted by malware scan.

Generic failure during the scan of a backup image.

  • Verify if any scan is in progress.

  • If no scan is in progress, then obtain the list of such instant access mounts with ID's of the instant access mount created using the GET IA API from the following directory:

    /netbackup/recovery/workloads/{workload}/instant-access-mounts

  • Using the DELETE API, delete the instant access mount:

    /netbackup/recovery/workloads/{workload}/instant-access-mounts/{mounId}

  • Failed to an open file.

  • Unable to create a directory.

  • Failed to generate the result file.

  • Failed to open output file.

  • Unable to create directory for result file.

  • Failed to open the result file.

  • Unable to create mount destination directory.

  • Unable to create directory for a log file.

  • Not enough space is available on the scan host.

  • SSH user does not have access to the required directories on the scan host.

  • On a Windows scan host, check space is availability in C:\

  • On a Linux scan host check space is availability in /tmp

All mount drives are exhausted.

Only five backup images can be mounted at the same time on windows scan host.

  • Ensure that scan host is not part of multiple NetBackup domains.

  • Check if there are any Stale mounts on the scan host by running net use.

  • Following drive letters are used for mounting the IA shares on the windows scan host. Ensure that they are not in use. L:\ M:\ N:\ O:\ P:\

Either the Windows Defender is not installed or the environment variable is not set.

Microsoft Windows Defender is not installed on the scan host or not configured properly.

Ensure that Microsoft Windows Defender is installed on scan host.

Refer NetBackup Web UI Administrator's Guide for the scan host configuration.

Either Symantec protection engine is not installed or the environment variable is not set.

Symantec Protection Engine is not installed on the scan host or not configured properly.

Ensure that Symantec Protection Engine is installed on scan host.

Refer NetBackup Web UI Administrator's Guide for the scan host configuration.

Failed to perform malware scan of the backup image.

Generic error for Scan failure.

Contact NetBackup support.

Net bios name can be at most 15 chars long.

Storage server host name cannot be more than 15 characters for the SMB share support.

If Windows Server 2016 is used to set up Active Directory domain, then it does not allow a connection to a storage server with host name of length more than 15 characters.

Ensure the character limit.

Failed to run a scan.

Generic failure during scanning backup image.

Check for the following errors:

  • Refer to nbmalwarescanner logs on the media server.

  • Check for space on media server storage.

  • Check for NFS service failure on media server.

Too many infected files in the selected time range.

Review the nbmalwarescanner to view the infected files list for the backup images in the selected date range.

Update the date range or recovery files and folders selection to reduce the number of infected files. Retry the operation. You can also perform one of the following:

  • Select the Allow recovery of files impacted by malware option which can be used to recover selective clean files.

  • Skip that backup image from recovery.

Large number of infected files.

There are too many infected files in the selected scan result. If the scan result has infected files greater than 5000, the following message is displayed:

Large number of infected files. To view the complete list of infected files, export the list.

Export the infected file list in .csv format and download it to view it.

There are many infected files in the selected scan result or the infected file paths are long to be captured in the database. Following error message is displayed:

Large number of infected files.

This result cannot be exported or viewed.

As the results cannot be exported or viewed, review the scan logs to view a detailed list of the infected files for the selected scan result.

Scan operation is divided into parts.

For large size backup, scan operation is divided into parts.

For example, if total number of files in the backup are 1,000,000, the scan operation will be divided into two parts of 500,000 files each.

Each part would be created and scanned separately. Each part can be assigned with different scan host. The Malware detection UI displays only single entry for backup.

Each divided part details can be obtained by using the REST API.

The NB_MALWARE_SCANNER_PATH environment variable is missing.

When performing a malware scan operation with the NetBackup Malware Scanner installed on the scan host, it fails with the following error message:

Missing environment variable NB_MALWARE_SCANNER_PATH

Ensure that NetBackup Malware Scanner is installed. Note the install location.

Login on the scan host as user using the same user credentials that were provided during scan host configuration on the primary server. Add the following lines to ~/.bashrc:

export NB_MALWARE_SCANNER_PATH=<installLocation>/savapi-sdk-linux64/bin

export PATH=$PATH:$NB_MALWARE_SCANNER_PATH

Issue related to NAS-Data-Protection

When upgrading NetBackup from previous version to NetBackup version 10.2.1 with the following options selected, the No images match the search criteria message is displayed:

Options

Fields

Search by: Backup images

Policy type: NAS-Data-Protection

Copies: Copy2

Malware scan status: Not scanned (Default)

Search by: Assets by policy type

Policy type: NAS-Data-Protection

Copies: Copy2

Scanner host pool: Select the required scanner host pool.

Malware scan status: Not scanned (Default)

Workaround

To view the images that are backed up, ensure that you select the Malware scan status option as All to scan the NAS-Data-Protection backup images created on earlier version of NetBackup media server.