NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (10.3.0.1, 10.3)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX or Linux servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the primary server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Primary server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX primary server and client
      2.  
        Example of host name and service entries on UNIX primary server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host properties to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. Troubleshooting Auto Image Replication
      1.  
        Rules for primary servers used with Auto Image Replication and SLPs
      2. Targeted A.I.R. trusted primary server operation failed in case of external certificate configuration
        1.  
          Add or update trust
        2.  
          Remove trust
      3.  
        About troubleshooting automatic import jobs that SLP components manage
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Troubleshooting error messages in the NetBackup Administration Console
    29.  
      Extra disk space required for logs and temporary files for the NetBackup Administration Console
    30.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    31.  
      Troubleshooting file-based external certificate issues
    32.  
      Troubleshooting Windows certificate store issues
    33.  
      Troubleshooting backup failures
    34.  
      Troubleshooting backup failure issues with NAT clients or NAT servers
    35.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
    36.  
      Issues with email notifications for Windows systems
    37.  
      Issues with KMS configuration
    38.  
      Issues with initiating the NetBackup CA migration because of large key size
    39.  
      Issues with the non-privileged user (service user) account
    40.  
      Issues with group name format in the auth.conf file
    41.  
      Troubleshooting the VxUpdate add package process
    42.  
      Issues with FIPS mode
    43.  
      Issues with malware scanning
    44.  
      Issues with NetBackup jobs that are enabled for data-in-transit encryption
    45.  
      Issues with Unstructured Data Instant Access
    46.  
      Troubleshooting issues with multi-factor authentication
    47.  
      Troubleshooting issues with multi-person authorization
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
    10. About the NetBackup Smart Diagnosis (nbsmartdiag) utility
      1.  
        Workflow to use the nbsmartdiag utility for NetBackup host communication
    11.  
      About log collection by job ID
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the primary server disk for UNIX and Linux
        1.  
          Recovering the primary server when root is intact
        2.  
          Recovering the primary server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the primary server disk for Windows
        1.  
          Recovering the primary server with Windows intact
        2.  
          Recovering the primary server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered primary server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About the catalog backup process
      2.  
        Prerequisites for recovering the NetBackup catalog or NetBackup catalog image files
      3.  
        About NetBackup catalog recovery on Windows computers
      4.  
        About NetBackup catalog recovery from disk devices
      5.  
        About NetBackup catalog recovery and symbolic links
      6. About NetBackup catalog recovery
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      7.  
        NetBackup disaster recovery email example
      8. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the NetBackup catalog recovery wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      9.  
        Establishing a connection with NAT media server before catalog recovery
      10. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the NetBackup catalog recovery wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      11. About recovering the NetBackup databases
        1.  
          Recovering the NetBackup database from a backup
        2.  
          Recovering the NetBackup database from staging
        3.  
          About processing the NetBackup database in staging
        4.  
          Terminating database connections
      12.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      13.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      14.  
        Recovering the NetBackup catalog without the disaster recovery file
      15.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      16.  
        Restoring files from a NetBackup online catalog backup
      17.  
        Unfreezing the NetBackup online catalog recovery media
      18.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Issues with the non-privileged user (service user) account

This topic provides troubleshooting information about the issues specific to the non-privileged, non-root, or service user.

Most of the primary server services can be run as non-privileged user, which is highly recommended. This new user is called service user.

For more information on the service user, see the NetBackup Security and Encryption Guide.

The nbcertcmd command option logs

The nbcertcmd command options internally run under the service user context. You can find the logs of the nbcertcmd command options in the SERVICE_USER.xxxxxx_xxxxx.log file.

Table: Troubleshooting service user issues

Sr. No.

Issue

Possible reason

Resolution

1

During NetBackup installation or upgrade on UNIX platform, unable to specify the service user even after three prompts.

Possible reasons are as follows:

  • Reason 1 - The service user does not exist locally, in LDAP, or in NIS.

  • Reason 2 - nbwebsvc is used as a service user.

  • Reason 3 - nbwebgrp is not a secondary group of the service user.

Resolutions are as follows:

  • Resolution 1 - Run the following command:

    id service_user

    The ID command must be successful.

  • Resolution 2 - Run the nbgetconfig command to check the NetBackup configuration file (bp.conf) for the WEBSVC_USER entry.

    The service user cannot be same as the value that is set for the WEBSVC_USER configuration option.

  • Resolution 3 - Run the nbgetconfig command to check the NetBackup configuration file (bp.conf) for the WEBSVC_USER entry.

    Run the following command:

    id service_user

    In the command output, ensure that gid is not equal to the gid of the WEBSVC_GROUP option value and groups have the value WEBSVC_GROUP.

2

During NetBackup installation on an inactive cluster node on UNIX platform, one of the following errors occurs:

  • Service user name on active node does not match with service user name entered on inactive node.

  • SERVICE_USER_ID '10021' retrieved from active node does not match with the user ID '1002' of local user 'nonroot'.

The service user name and the user ID do not match.

Ensure that the service user name and the user ID match on all cluster nodes and the same is provided during NetBackup installation on active and inactive nodes.

3

During NetBackup upgrade of an inactive cluster node on UNIX platform, the following error occurs:

Failed to retrieve the 'SERVICE_USER' or 'SERVICE_USER_ID' entries from the configuration file on the server 'cluster_virtual_name'. You must provide the same 'SERVICE_USER' (daemon user name) that is configured on the active node.

The bpgetconfig command could not retrieve the service user and the ID from active node.

Provide the service user as that of the active node and ensure that the service user has the same user ID on all cluster nodes.

4

During NetBackup installation or upgrade on UNIX platform, the following error occurs:

The user serviceuser cannot be set as the owner of files in /usr/openv.

This may be because of the issues while changing the ownership of the installation directory.

Fix the errors specified in installation trace under the following heading:

Fix below errors and then retry

5

NetBackup host communication does not work when external CA is configured with Windows Certificate Store and services run in a Local Service account context.

NetBackup services do not have access to the private key. Usually, the error in this case can be seen in the nbpxyhelper logs:

The Windows API CryptAcquireCertificatePrivateKey fails with error 0x80090016: Keyset does not exist.

Check private key permissions as follows:

Right-click the certificate. Go to All Tasks > Manage Private Keys.

All NetBackup services should have permissions to read the private key.

Run the following command to set permissions:

nbcertcmd -setWinCertPrivKeyPermissions

Run the following command to validate the configuration:

nbcertcmd -ecaHealthCheck

6

The setconfig command fails with the following error:

Failed to open /usr/openv/netbackup/bp.conf.d53: Permission denied (13)

Ownership of /usr/openv/netbackup is changed to the root user.

Other possible reason may be that the language pack is installed using rpm.

Run the following command to fix the ownership issues:

/usr/openv/netbackup/bin/goodies/ update_install_folder_perms

7

  • Create or update operation fails for catalog backup policy.

  • Catalog backup fails.

  • Catalog recovery fails.

Service user account may not have access to the disaster recovery (DR) path specified in policy.

Review status code 9201 and 9202.

Refer to the NetBackup Status Codes Guide.

Refer to the NetBackup Security and Encryption Guide for giving access permissions to the service user account.

8

Disaster recovery fails.

The NBHostIdentity -import command fails.

Ensure the following:

  • The service user exists on the system prior to disaster recovery (DR).

  • The service user has access to the DR package.

9

Any of the following commands fail with error: Ensure that the service user account [service_user_name] has access permissions on the specified paths and their contents.

  • nbdb_admin

  • nbdb_move

  • nbdb_backup

  • nbdb_restore

  • nbdb_unload

  • create_nbdb

  • cat_export

  • cat_import

Path:

For UNIX - Install_Path/db/bin

For Windows - Install_Path\netbackup\bin

Service user account may not have access permissions on specified paths and their contents.

Refer to the NetBackup Security and Encryption Guide for giving access permissions to the service user account.

10

Adding VMware server operation fails

500 system error

Ensure that the temp directory (/tmp) is accessible to the service user account

11

Issue in bpjava-test-login workflow

File ownership is shown as 'root'

Change the ownership of the file to the service user account.

12

nbcertcmd operations fail.

Lack of permissions

Check if the certmapinfo.json file is created and owned by the service user.

13

nbcertcmd or bpnbaz fails with error code 123.

The private key file (PrivKeyFile-2048.pem), public key file (PubKeyFile-2048.pem), or access control list (ACL) update failed.

Ensure that NetBackup SIDs are configured and both public and private keys are present in AT_DATA_DIR.

14

nbserviceusercmd -changeUser operation failed with authorization failure, when NBAC is configured.

The new service user is not part of the NBAC security admin group.

Add the new service user in the NBAC security admin group. Run the following command:

vssaz addazgrpmember --azgrpname \"Security Administrators\" --prplinfo prplinfo

15

After NetBackup 9.1 installation and upgrade, NetBackup Administration Console login fails for root user, if NetBackup access control (NBAC) or Enhanced Auditing (EA) is enabled.

The user certificate directory is changed.

If NBAC or EA is enabled in your environment, you must run the bpnbat -login command after NetBackup upgrade.

16

The nbcertcmd -enrollCertificate command fails as external CA (ECA) health check fails.

An error occurs while accessing the files at the following path:

certificates/private key/passphrase file/crl

The nbcertcmd -enrollCertificate command runs under the service user context, however the service user does not have access to the associated files.

Provide the required access to the service user.

It is recommended that you run the following command to verify the access rights before running the enrollCertificate command again:

nbcertcmd -ecaHealthCheck -serviceUser user_name

17

Before upgrade or change user, the service user is deleted.

The service user may be deleted because of certain user actions.

Do the following:

Reconfigure the user to restore the service user. Refer to the article.

Run the following commands::

  • useradd -c 'NetBackup Services account' -d /usr/openv/ nbsvcusr -u old uid

  • usermod -a -G nbwebgrp nbsvcusr

18

During backup or restore, operation error is encountered.

The media server version is earlier than the client version.

Upgrade the media server or use an alternate media server with the version that is later or same as the client version.