How to troubleshoot Call Home on the NetBackup appliance

Description

Veritas has not received any Call Home data from your appliance <hostname> (serial#) for more than 24 hours since DD MMM YYYY HH:MM:SS GMT

For Call Home to work correctly, the appliance must access the following servers via the Internet either directly, or through a proxy server. AutoSupport, a mechanism that monitors the appliance proactively, uses the Call Home data to analyze and resolve any issues that the appliance may encounter. All communications are initiated by the appliance.

Call Home uses HTTPS (secure and encrypted protocol) with port number 443 for all communication with the following Veritas AutoSupport servers:
         api.appliance.veritas.com
         receiver.appliance.veritas.com
         sort.veritas.com

After an appliance is provisioned in the Veritas AutoSupport database with the mac address of appliance eth0 interface, upon first communication from appliance to AutoSupport servers the appliance will write a /usr/openv/runtime_data/callhome_secret containing a password hash.


In some customer environments, internet access is governed by a proxy server. The appliance administrator in such an environment may need to complete the Call Home Proxy configuration via NetBackup Appliance Shell Menu or web UI with some or all of the following proxy server details such as:
- proxy server hostname or IP address
- proxy server port number
- proxy server username and password

To add a Call Home proxy server from the NetBackup Appliance Shell Menu:

1. Log on to the NetBackup Appliance Shell Menu.
2. To enable proxy settings, run the Main > Settings > Alerts > CallHome Proxy Enable command.
3. To add a proxy server, run the Main > Settings > Alerts > CallHome Proxy Add command.

   • You are prompted to enter the name of the proxy server. The proxy server name is the TCP/IP address or the fully qualified domain name of the proxy server. Be careful not to add a trailing space at the end of the value.

   • After you have entered a name for the proxy server, you are prompted to enter the port number for the proxy server.

   • Further, you are required to answer the following:

     Do you want to set credentials for proxy server? (yes/no)

   • On answering yes, you are prompted to enter a user name for the proxy server.

   • After you have entered the user name, you are prompted to enter a password for the user. On entering the required information, the following message is displayed:

     Successfully set proxy server
4. To disable proxy settings, run the Main > Settings > Alerts > CallHome Proxy Disable command.

Further, you can also use the NetBackup Appliance Shell Menu to enable or disable proxy server tunneling for your appliance. To do so, run the Main > Settings > CallHome Proxy EnableTunnel and Main > Settings > Alerts > CallHome Proxy DisableTunnel commands. Proxy server tunneling lets you provide a secure path through an untrusted network.

Additionally the proxy server may require a valid SSL certificate from the appliance to allow internet communication. In such an environment, the customer's Root Certificate Authority SSL certificate must be imported on the appliance. See Related Articles for instructions on how to import SSL certificates into the appliance.

A common problem is connectivity with or without a proxy server. The following commands can help identify the issue or at least narrow the focus of troubleshooting, commands go from simple to more complicated:

- Test DNS resolution of the following hosts:

# nslookup api.appliance.veritas.com

# nslookup receiver.appliance.veritas.com

# nslookup sort.veritas.com

# nslookup <proxy_host_name>

- curl test of connectivity, no proxy:
# curl -v https://receiver.appliance.veritas.com/receiver/telemetry

# curl -v https://api.appliance.veritas.com/AutoSupport/MyService.asmx?op=RegistrationGetOwnerDetails
 

- curl test to verify connectivity from the appliance to the proxy host:

# curl -v telnet://<proxy_hostname|proxy_IP>:<proxy_port>

- curl test of connectivity through a proxy server named myproxy-acme.dom, port 8080 and domain name acme and username acmesupport:
# curl -v -x myproxy.acme.dom:8080 -U acme\\acmesupport https://receiver.appliance.veritas.com/receiver/telemetry

# curl -v -x myproxy.acme.dom:8080 -U acme\\acmesupport https://api.appliance.veritas.com/AutoSupport/MyService.asmx?op=RegistrationGetOwnerDetails

NOTE: when a curl command cannot connect or traverse a proxy server even a command like 'curl -v https://yahoo.com' then it may mean the proxy server (or some other device on the appliance network) is inspecting TLS/SSL traffic and is not detecting an expected certificate. This is most often proven by the certificate chain in the output referring to a customer/site specific certificate. An example and solution can be found here:
Callhome proxy enable fails with "unable to find valid certification path to requested target"

Ensure all processes/services related to callhome/autosupport are running properly. Confirm by stopping and then starting these services/processes:

Log on to the NetBackup Appliance Shell Menu.
Navigate to Support > Maintenance and enter your password. 
On the Maintenance prompt, run the following command to disable the protection completely:
   /opt/Symantec/sdcssagent/IPS/sisipsoverride.sh
      Choose the type of override that you wish to perform:
       1. Override Prevention except for Self-Protection
       2. Override Prevention Completely
Choice? [1] 2

Elevate to the root user by running the 'elevate' command.

1) From elevated prompt stop crond and all autosupport services, and then stop nginx:
# systemctl stop crond
# as-collector stop
# as-analyzer stop
# as-alertmanager stop
# as-transmission stop
# systemctl stop nginx

NOTE:  For NetBackup OS v5.3+ you will need to use these commands:

# for i in crond nginx alertmanager analyzer collector transmission tomcat-vxos mongod rabbitmq-server; do echo $i; systemctl stop $i;  systemctl is-active $i; done
# for i in $(ps -ef | egrep 'alertmanager|rabbit|analyzer|collector|transmission|apache-tomcat|mongo' | grep -v grep| awk '{print $2}');do echo $i; kill -9 $i;done
# for i in rabbitmq-server mongod tomcat-vxos transmission collector analyzer alertmanager nginx crond; do echo $i; systemctl start $i;systemctl is-active $i; done
# for i in rabbitmq-server mongod tomcat-vxos transmission collector analyzer alertmanager nginx crond; do echo $i; systemctl is-active $i; done

3) In a separate NetBackup Appliance Shell Menu, stop Infraservices via NetBackup Appliance Shell Menu Main_Menu > Support > Infraservices > Stop ALL

4) From elevated prompt, confirm all are stopped, elevated prompt, run:
# ps -ef --width 135 | egrep 'collector|analyzer|alertmanager|transmission|/usr/bin/mongod|erlang|nginx.conf|tomcat' | grep -v grep | grep -v wmc | grep -v nbtelemetry

If any services/processes are still running per the 'ps' command, please contact Veritas Technical Support to speak to a Support representative.
 
5) From elevated prompt, start nginx:
# systemctl start nginx

6) From NetBackup Appliance Shell Menu Main_Menu > Support > Infraservices > Start ALL

7) From elevated prompt, start crond and autosupport services:
# systemctl start crond

# as-collector start

# as-analyzer start

# as-alertmanager start

# as-transmission start

Logs and outputs to assist Veritas Technical Support with further troubleshooting:

• During execution of any NetBackup Appliance Shell Menu commands or appliance web UI operations, note any errors reported
• Provide output of NetBackup Appliance Shell Menu Main_Menu > Settings > Alerts > Callhome Show
• From a Maintenance prompt, obtain the bottom of the /log/autosupport/transmission.log for recent content for errors:
  - Log on to the NetBackup Appliance Shell Menu.
  - Navigate to Support > Maintenance and enter your password.
  - On the Maintenance prompt, execute:
  maintenance-!> less /log/autosupport/transmission.log
           ...or grep for errors:
  maintenance-!> grep ERROR /log/autosupport/transmission.log | tail -n 20
• reproduce the issue and obtain the app_vxul logs, example command looking back 30 minutes:
  maintenance-!> vxlogview -p 409 -t 00:30:00
       ....or more specific to just callhome (note one may miss related messages with this command):
  maintenance-!> vxlogview -p 409 -o 2 -t 00:30:00
• Output of NetBackup Appliance Shell Menu Main_Menu >Network> Show Configuration
• Output from maintenance prompt: maintenance-!> cat /usr/openv/runtime_data/callhome_secret
• Output from maintenance prompt: maintenance-!> tail -n 50 /log/autosupport/transmission.lo
• DataCollect:

            1. NetBackup Appliance Shell Menu#> Support> DataCollect
            2. Open log share
            3. NetBackup Appliance Shell Menu#> Support> Logs> Share Open
            4. Retrieve DataCollect through CIFS share
            5. Open Windows Explorer
                - Right-click on “Network”
                - Select “Map network drive … ”
                - Enter \\<IP_address_or_hostname_of_Appliance>\logs\APPLIANCE as the “Folder”
                     - Check “Connect using different credentials”
                     - Click “Finish”
                - Use Appliance user name (default: “admin”) and password (default: “P@ssw0rd”) to log in to share
                - Retrieve file DataCollect.zip
            6. Close log share
                 - NetBackup Appliance Shell Menu#> Support> Logs> Share Close

Additional reference:
Callhome Proxy Enable fails in NetBackup Appliance Shell Menu with "Bad hostname" if there is a space in the configured proxy server hostname.
https://www.veritas.com/support/en_US/article.100049098

 

For VRTS Technical Support:
- 100047622 Unable to enable CallHome Proxy on NetBackup 3.2 or later Appliances
- 100047003 NetBackup appliance version 3.2 Call Home values are lost/removed when running an appliance database cleanup script and cannot be reapplied