Unable to start mqbroker - (epmd running but beam.smp not starting)

Problem

Unable to start mqbroker on the primary server. Checking bpps command output shows epmd running but beam.smp does not show up in the output.

Error Message

Below generic message is recorded while trying to start mqbroker

Starting NetBackup Messaging Queue Broker could take a couple of minutes ...................................................
NetBackup Message Queue Broker not started.

The erl crash dump shows the message below

Slogan: init terminating in do_boot ({error,{crypto,{{case_clause,{_}},[{_},{_},{_},{_},{_}]}}})

The erl crash dump gets generated under the mqbroker home path

# ls -ltr /usr/openv/var/global/mqbroker/mqhome/
total 1024
-rw-r-----. 1 nbwebsvc nbwebgrp 1047542 Aug  7 19:42 erl_crash.dump

The 'nbmqbroker_<date>.log' shows the below message and it keeps repeating with the count value increasing sequentially

08:07:23 19:35:58 - Starting NetBackup Messaging Queue Broker could take a couple of minutes.
Will ping nbmqbroker@primary-server. This only checks if the OS process is running and registered with epmd. Timeout: 60000 ms.
Error:
Failed to connect and authenticate to nbmqbroker@primary-server in 60000 ms
08:07:23 19:35:59 - Generating cookie using vxsslcmd.
08:07:23 19:35:59 - Starting nbmqbroker. Using ERLANG_HOME=/usr/openv/mqbroker/erlang.
08:07:23 19:36:03 - Registered the nbmqbroker daemon to be monitored by nbsvcmon
08:07:23 19:36:03 - checking service start status count: 0
Error: unable to perform an operation on node 'nbmqbroker@primary-server'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node nbmqbroker@primary-server
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: [nbmqbroker@primary-server]

nbmqbroker@primary-server:
  * connected to epmd (port 13778) on primary-server
  * epmd reports: node 'nbmqbroker' not running at all
                  no other nodes on primary-server
  * suggestion: start the node

Current node details:
 * node name: 'rabbitmqcli-59-nbmqbroker@primary-server'
 * effective user's home directory: /usr/openv/var/global/mqbroker/mqhome
 * Erlang cookie hash: mZsLSLySmfUuUiIWk4+hyw==

08:07:23 19:36:05 - checking service start status count: 1

Cause

The 'crypto.app' file is corrupt. In this case, the file itself was 0 bytes in size.

-rw-r--r--. 1 nbwebsvc nbwebgrp   0 Aug  7 19:50 /usr/openv/mqbroker/erlang/lib/crypto-5.1.2/ebin/crypto.app.orig
-rw-r--r--. 1 nbwebsvc nbwebgrp   0 Aug  7 19:50 /usr/openv/mqbroker/erlang/lib/crypto-5.1.2/ebin/crypto.app
 

Solution

1. Stop nbwmc and mqbroker 
   # /usr/openv/netbackup/bin/nbwmc terminate
   # /usr/openv/mqbroker/bin/nbmqbroker stop
    
2. Verify that the services are stopped (#bpps | egrep -i "nbwmc|mqbroker")
    
3. Extract the 'crypto.app' file
   
   NetBackup versions 10.2.0.1 and older

• Extract the 'crypto.app' file from erlang_OTP.tar.gz (/usr/openv/mqbroker/erlang_OTP.tar.gz) 
  Copy the extracted file to - '/usr/openv/mqbroker/erlang/lib/crypto-5.1.2/ebin/'
  
  NetBackup versions from 10.3 and above
  • Note: In the releases of NetBackup 10.3 and newer, the erlang_OTP.tar.gz is no longer available and the crypto.app file is part of the VRTSnbmqbrkr.rpm package that gets installed during the rpm installation.
     

• Extract the crypto.app file from the VRTSnbmqbrkr.rpm file using the below command. (this requires access to the NetBackup installer)
  The VRTSnbmqbrkr.rpm is located at '{path_to_install}/linuxR_x86/anb/'

  rpm2cpio VRTSnbmqbrkr.rpm | cpio -idv ./usr/openv/mqbroker/erlang/lib/crypto-{version}/ebin/crypto.app

  Example:
  Below is an example from NetBackup 10.4.
  rpm2cpio VRTSnbmqbrkr.rpm | cpio -idv ./usr/openv/mqbroker/erlang/lib/crypto-5.1.4/ebin/crypto.app
   

4. Set the file ownership as nbwebsvc:nbwebgrp (easy option is to run the 'setupMQBroker' script and it will take care of the permission/ownership).
   Note: The 'crypto.app.orig' file will automatically get updated when the service is started.
    
5. Start NetBackup services. 
   # /usr/openv/netbackup/bin/nbwmc -start
   # /usr/openv/mqbroker/bin/nbmqbroker start

If any other files are affected or there is a need to reinstall mqbroker, it can be done following the steps outlined in the below article
https://www.veritas.com/support/en_US/article.100061842