Search <book_title>...

NetBackup™ Troubleshooting Guide

Last Published: 2023-10-23

Product(s): NetBackup (10.3.0.1, 10.3)

Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service

The NetBackup Messaging Broker service is not running

Do the following:

Ensure that the service is configured and started on the primary server. To configure the service, run the configureMQ command.
Refer to the NetBackup Commands Reference Guide.

The NetBackup Messaging Broker service is not able to start

Reasons:

Ports that are configured for the service is in use by some other process.
The configuration file is corrupted.

Do the following:

Check the configureMQ command logs for failure.
Check the nbmqbroker service logs for failure.
Run the configureMQ command.
Refer to the NetBackup Commands Reference Guide.

The NetBackup Messaging Broker service is not connected to the NAT client

Reasons:

The port configured for the service is not available for use.
Connection fails with some SSL exception.
The nbmqbroker service is not restarted after the configureWebServerCerts command is run on the primary server.

Do the following:

Ensure that the port configured for the nbmqbroker service is available for use and accessible by NetBackup hosts.
Check the connectivity between the primary server and the NAT client using the nbcertcmd -ping command.
- If the command is not successfully executed, refer to the troubleshooting section for the NetBackup web service.
- If the command is successfully executed, run configureMQ command to configure the nbmqbroker service.
Restart the nbmqbroker service.

Subscriber or publisher is not able to connect to the NetBackup Messaging Broker service

Reasons:

The JSON web token (JWT) for the NAT client cannot be refreshed.
The security certificate of the NAT client is revoked.
The NetBackup Web Management Console (or nbwmc) service is not running.

Do the following:

Refer to the subscriber troubleshooting steps.
If the client's security certificate is revoked, reissue the certificate.
Start the nbwmc service.

The NetBackup Messaging Broker service is not able to start after disaster recovery

Reasons:

The disaster recovery package is lost.
The configureMQ command is not run after the disaster recovery (DR) installation.

Do the following:

Run the configureMQ or configureMQ -defaultPorts command.
Refer to the NetBackup Commands Reference Guide.

The NetBackup Messaging Broker service fails to start on Windows if the 8dot3 short file name setting is disabled on the volume where NetBackup is installed

To check if the installation root folder has the 8dot3 file name setting enabled, run the following command from your folder:

>dir /x

Example: The 'Program Files' directory has the 8dot3 file name setting enabled, therefore the short name 'PROGRA~1' is generated.

But it differs for the 'not8 Dot3' directory.

C:\>dir /x

The volume in drive C has no label.

The Volume Serial Number is FE21-2F8E

Directory of C:\

-5.6.3

12/06/2019  02:24 PM    <DIR>   	            not8 Dot3

12/02/2019  06:35 AM    <DIR>   PROGRA~1     Program Files

12/02/2019  10:44 AM    <DIR>   PROGRA~2     Program Files (x86)

Do the following to resolve the issue:

Enable 8dot3 name file setting for the NetBackup installation root folder using the fsutil command.
Refer to the following article: Fsutil 8dot3name
If the problem persists, contact Technical Support.

The NetBackup Messaging Broker service behaves incorrectly after restoring the disaster recovery package in case of external CA setup

Consider the following scenario:

NetBackup is configured to use only external CA-singed certificates at the time of catalog backup. Therefore, the disaster recovery package that was created during catalog backup contains the required external certificates. If the host identity is recovered using such disaster recovery package after NetBackup installation, the nbmqbroker service may behave incorrectly because of the NetBackup CA-signed certificates that were issued during installation.

To resolve the issue

Verify if your the NetBackup environment uses only external CA-signed certificates. Run the following command:
nbcertcmd -getSecConfig -caUsage
Check the certificates that the nbmqbroker service uses. Run the following command:
On Unix: cat /usr/openv/var/global/mqbroker/mqbroker.config | grep ssl_options
On windows: type "NetBackup_Install_path\var\global\mqbroker\mqbroker.config" | findstr "ssl_options"
If only external CA-signed certificates are used in your environment, the command shows the path with externalcacreds entry.
If the command shows the path with nbcacreds entry, NetBackup CA-signed certificates are used.
For example:
{ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/nbcacreds/ca.pem"}, {ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/nbcacreds/ca.pem"},
You need to remove the NetBackup certificates so that the nbmqbroker service works appropriately.
Run the following command to remove the NetBackup certificates:
configureWebServerCerts -removeNBCert
Restart the NetBackup Web Management Console (nbwmc) service and the nbqmbroker service to reflect the changes.
Check the certificates that the nbmqbroker service uses. Run the following command:
On Unix: cat /usr/openv/var/global/mqbroker/mqbroker.config | grep ssl_options
On windows: type "NetBackup_Install_path\var\global\mqbroker\mqbroker.config" | findstr "ssl_options"
Expected output for external certificate only mode:
{ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/externalcacreds/ca.pem"}, {ssl_options, [{cacertfile, "/usr/openv/var/global/mqbroker/certstore/externalcacreds/ca.pem"},
See Restoring disaster recovery package on UNIX.
See Restoring disaster recovery package on Windows.

New nbmqbroker service-specific notifications are not displayed in the NetBackup web UI on Linux

The nbmqbroker service logs show the following errors:

escript: exception error: undefined function rabbitmqctl_escript:main/1

in function  escript:run/2 (escript.erl, line 758)

in call from escript:start/1 (escript.erl, line 277)

in call from init:start_em/1

in call from init:do_boot/3

Root cause:

Certain configuration changes on the primary server may result into inconsistency in nbmqbroker service configuration. To resolve the issue, you need to reconfigure the nbmqbroker service.

To reconfigure the nbmqbroker service

Stop the nbmqbroker service by running the following command:
/usr/openv/mqbroker/bin/nbmqbroker stop
Run the following command to configure the nbmqbroker environment:
/usr/openv/mqbroker/bin/install/configureMQEnv
Run the following command to configure the nbmqbroker service:
/usr/openv/mqbroker/bin/install/configureMQ
Start the nbmqbroker service by running any of the following commands:
- /usr/openv/mqbroker/bin/nbmqbroker start
- bp.start_all command
For more information on the commands, refer to the NetBackup Commands Reference Guide.

The NetBackup Messaging Broker service does not start on IPv6-only primary server

Reasons:

The primary sever name is possibly being resolved to both IPv4 and IPv6 addresses although only IPv6 address is used.

Run the following command to check if the output contains an IPv4 address: nslookup primary_server_name

Sample output:

# nslookup primary-server.com

Server: 2600:100:f0a1:9000::a

Address: 2600:100:f0a1:9000::a#53

Non-authoritative answer:

Name: primary-server.com

Address: 10.200.100.60

Name: primary-server.com

Address: 2600:100:f0a1:9014::335

Expected output:

# nslookup primary-server.com

Server: 2600:100:f0a1:9000::a

Address: 2600:100:f0a1:9000::a#53

Non-authoritative answer:

Name: primary-server.com

Address: 2600:100:f0a1:9014::335

Do the following:

Fix all the configurations to create an appropriate IPv6-only setup.
If the issue still persists, do the following configuration changes to start the nbmqbroker service.
With this configuration, the nbmqbroker service always attempts to first use the IPv6 address for name resolution.

To change the configurations

Do the following to create the required file.
Use an appropriate text editor (vi on Linux or Notepad on Windows) and create a file called erl_inetrc in the given directory:
On Linux, create the erl_inetrc file in the following directory:
/usr/openv/var/global/mqbroker/erl_inetrc
Run the following command:
cat > /usr/openv/var/global/mqbroker/erl_inetrc
On Windows, create the erl_inetrc file in the following directory: NetBackup_Install_path\var\global\mqbroker\
Add the following line in the erl_inetrc file:
{inet6,true}.
Note that the trailing dot (.) is mandatory.
On UNIX, run the following command to check the permissions of the /usr/openv/mqbroker/bin/setmqenv file:
ls -l /usr/openv/mqbroker/bin/setmqenv
The output is as follows:
```
-rwxr-x---. 1 nbwebsvc nbwebgrp 3869 date
 /usr/openv/mqbroker/bin/setmqenv
```
Do the following:
On Linux:
Add the following lines in the /usr/openv/var/global/mqbroker/advanced_setmqenv file:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/usr/openv/var/global/mqbroker/erl_inetrc' -proto_dist inet6_tcp"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp"
On Windows:
Add the following lines in the NetBackup_Install_path\var\global\mqbroker\advanced_setmqenv file:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-kernel inetrc 'E:/NetBackup/var/global/mqbroker/erl_inetrc' -proto_dist inet6_tcp
RABBITMQ_CTL_ERL_ARGS=-proto_dist inet6_tcp
Ensure that the file permissions are not changed after the update.
Start the nbmqbroker service.