NetBackup Messaging Broker service fails to start after upgrade.

Problem

NetBackup Messaging Broker service fails to start after an upgrade or reboot.

Error Message

Starting nbmqbroker...
NetBackup Messaging Broker service configuration failed. For more details, refer to /usr/openv/mqbroker/logs/configureMQ.log.

The ConfigureMQ_<date>.log shows the message below and it keeps repeating.

Sat Dec 16 10:47:19 2023 Performing cluster check
Sat Dec 16 10:47:19 2023 skipIsActiveCheck (isCluster) value is : 1
Sat Dec 16 10:47:19 2023 skipIsActiveCheck (isActiveNode) value is : 1
Sat Dec 16 10:47:19 2023 Setting fips_mode to false in MQ broker configuration
Sat Dec 16 10:47:19 2023 copy /usr/openv/var/global/mqbroker/mqbroker.config to /usr/openv/var/global/mqbroker/mqbroker.config.orig
Sat Dec 16 10:47:19 2023 copy /usr/openv/var/global/mqbroker/mqbroker.config.new to /usr/openv/var/global/mqbroker/mqbroker.config
Sat Dec 16 10:47:19 2023 Setting fips_mode to false in erlang crypto file
Sat Dec 16 10:47:19 2023 Failed to configure nbmqbroker.
 

The nbmqbroker_<date>.log' shows the message below and it keeps repeating.

12:16:23 15:34:35 - Generating cookie using vxsslcmd.
12:16:23 15:34:35 - Starting nbmqbroker. Using ERLANG_HOME=/usr/openv/mqbroker/erlang.
nohup: failed to run command '/usr/openv/mqbroker/rabbitmq-server/sbin/rabbitmq-server': No such file or directory
Registering service nbmqbroker
Successfully registered service nbmqbroker for monitoring.
12:16:23 15:34:35 - Registered the nbmqbroker daemon to be monitored by nbsvcmon
12:16:23 15:34:35 - checking service start status count: 0
nohup: failed to run command '/usr/openv/mqbroker/rabbitmq-server/sbin/rabbitmqctl': No such file or directory
12:16:23 15:34:36 - checking service start status count: 46
nohup: failed to run command '/usr/openv/mqbroker/rabbitmq-server/sbin/rabbitmqctl': No such file or directory

Cause

During a NetBackup upgrade, if an error occurs during MQBroker configuration, the MQBroker directories for rabbitmq-server and erlang may not be created causing MQBroker Services startup to fail.

ls -l /usr/openv/mqbroker/
drwxr-xr-x. 4 nbwebsvc nbwebgrp     4096 Jun 12  2023 bin
-rw-r--r--. 1 nbwebsvc nbwebgrp 25470969 Dec 10  2022 erlang_OTP.tar.gz
drwxrwx---. 2 nbwebsvc nbwebgrp     4096 Dec 18 00:00 logs
drwxr-xr-x. 2 nbwebsvc nbwebgrp     4096 Jun 12  2023 openssl_libs
-rw-r--r--. 1 nbwebsvc nbwebgrp 17188514 Dec 10  2022 rabbitmq-server.tar.gz
drwxr-xr-x. 2 nbwebsvc nbwebgrp     4096 Jun 12  2023 templates

Solution

Follow the steps below to correct the MQbroker installation for the version of NetBackup affected.

Before proceeding with the repair procedures below, note that the NetBackup installer uses different methods to deploy the binaries for MQBroker depending on the version of NetBackup.

In the releases of NetBackup 10.2.0.1 and older, the MQBroker binaries are extracted from tar files contained with the NetBackup install RPM files.

- erlang_OTP.tar.gz
- rabbitmq-server.tar.gz 

In the releases of NetBackup 10.3 and newer, the tar file method has been deprecated in favor of native RPM installation and the tar files are no longer used for installation of MQBroker.

NOTE:
Be certain to use the correct repair method below for the installed version of NetBackup.
Using the repair steps for 10.2.0.1 and older on a 10.3 or newer installation will result in a corrupted MQBroker installation with mixed versions of binaries. 
If the version mismatch is present, apply the correct repair steps for the version of NetBackup noted below.

NOTE: 
Some NetBackup Appliance upgrades to 5.3 or higher do not remove the previous installation's MQBroker files for erlang_OTP.tar.gz and rabbitmq-server.tar.gz from the MQBroker directory.

This has been known to cause issues with MQBroker upgrades. Follow the steps before for repairing MQBroker on NetBackup 10.3 to correct the issue. 

NetBackup versions 10.2.0.1 and older using tar file deployment:

From an elevated command prompt:

• Stop nbwmc and mqbroker services.

/usr/openv/netbackup/bin/nbwmc terminate
/usr/openv/mqbroker/bin/nbmqbroker stop

• Verify that the services are stopped

# /usr/openv/netbackup/bin/bpps | egrep -i "nbwmc|mqbroker" 

• Create rabbitmq-server and erlang folder in /usr/openv/mqbroker/ location.
  
  # mkdir /usr/openv/mqbroker/erlang /usr/openv/mqbroker/rabbitmq-server
   
• Extract the rabbitmq-server.tar.gz file from /usr/openv/mqbroker into /usr/openv/mqbroker/rabbitmq-server/
  
  # tar xzvf /usr/openv/mqbroker/rabbitmq-server.tar.gz -C /usr/openv/mqbroker/rabbitmq-server/
   
• Extract the erlang_OTP.tar.gz file from /usr/openv/mqbroker into /usr/openv/mqbroker/erlang/

# tar xzvf /usr/openv/mqbroker/erlang_OTP.tar.gz -C /usr/openv/mqbroker/erlang/

• Run 'install_bp' to reconfigure mqbroker 
    
  # /usr/openv/netbackup/bin/install_bp
      -- A license may be required to complete this step.

• Start nbwmc and mqbroker services. 

/usr/openv/netbackup/bin/nbwmc -start
/usr/openv/mqbroker/bin/nbmqbroker start

• Verify that the services are started

/usr/openv/netbackup/bin/bpps | egrep -i "nbwmc|mqbroker"

NetBackup versions from 10.3 and above using RPM deployment:
 

From an elevated command prompt:

• Stop nbwmc and mqbroker services.

# /usr/openv/netbackup/bin/nbwmc terminate
# /usr/openv/mqbroker/bin/nbmqbroker stop

• Verify that the services are stopped

# bpps | egrep -i "nbwmc|mqbroker" command.

• Rename mqbroker directory
  
    # mv /usr/openv/mqbroker /usr/openv/mqbroker.mixedversions
   
• Reinstall the MQbroker files using the rpm bundle -> VRTSnbmqbrkr.rpm 
  
  NBAPPLIANCE - The NetBackup install rpm files are located in /inst/linuxR_x86/anb/.
  BYO Linux   - Locate the NetBackup install bundle and cd into /<path_to_install>/linuxR_x86/anb/.
   
  Example from NetBackup Appliance:  
  # rpm -ihv /inst/linuxR_x86/anb/VRTSnbmqbrkr.rpm --replacepkgs --replacefiles
   
• Run 'install_bp' to reconfigure mqbroker 
    
  # /usr/openv/netbackup/bin/install_bp
      -- A license may be required to complete this step.
   
• Check that MQBroker is listening

  # netstat -panot | egrep 13781
     tcp6       0      0 :::13781                :::*                    LISTEN      15779/beam.smp       off (0.00/0/0)

• Review the configureMQ.log to correct start up

Example output from NetBackup Appliance:

nba5250:/ # tail -n 10 /usr/openv/mqbroker/logs/configureMQ.log

Tue Mar 26 14:21:36 2024 skipIsActiveCheck (isCluster) value is : 1
Tue Mar 26 14:21:36 2024 skipIsActiveCheck (isActiveNode) value is : 1
Tue Mar 26 14:21:36 2024 Setting fips_mode to false in MQ broker configuration
Tue Mar 26 14:21:36 2024 copy /usr/openv/var/global/mqbroker/mqbroker.config to /usr/openv/var/global/mqbroker/mqbroker.config.orig
Tue Mar 26 14:21:36 2024 copy /usr/openv/var/global/mqbroker/mqbroker.config.new to /usr/openv/var/global/mqbroker/mqbroker.config
Tue Mar 26 14:21:36 2024 Setting fips_mode to false in erlang crypto file
Tue Mar 26 14:21:36 2024 copy /usr/openv/mqbroker/erlang/lib/crypto-5.1.4/ebin/crypto.app to /usr/openv/mqbroker/erlang/lib/crypto-5.1.4/ebin/crypto.app.orig
Tue Mar 26 14:21:36 2024 move /usr/openv/mqbroker/erlang/lib/crypto-5.1.4/ebin/crypto.app.new to /usr/openv/mqbroker/erlang/lib/crypto-5.1.4/ebin/crypto.app
Tue Mar 26 14:21:36 2024 Successfully configured nbmqbroker.

About erlang_OTP.tar.gz and rabbitmq-server.tar.gz bundles

The tarballs for erlang and rabbitmq-server can be found in the following RPM and NetBackup version.

NetBackup 8.2 to 10.1

VRTSnetbp.rpm
/usr/openv/mqbroker/erlang_OTP.tar.gz
/usr/openv/mqbroker/rabbitmq-server.tar.gz

NetBackup 10.1.1 to 10.2.0.1

VRTSnbmqbrkr.rpm
/usr/openv/mqbroker/erlang_OTP.tar.gz
/usr/openv/mqbroker/rabbitmq-server.tar.gz

NetBackup 10.3 to current

The tarball method to install has been deprecated. 
The erlang and rabbitmq-server binaries are installed through the VRTSnbmqbrkr.rpm deployment.