Unable to enable proxy for CallHome, getting error "Proxy enable failed."

Problem

On 2.6.1.2 appliances unable to enable proxy for CallHome, getting an error -

<hostname>.Alerts> CallHome Proxy Enable
Proxy enable failed. Please fix the error below and enable it again CALLHOME
status: Not able to upload callhome data Error: error while setting up ssl
connection (SSL connect attempt failed with unknown errorerror:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed) at /opt/
NBUAppliance/scripts/LWP/Protocol/https/connect/Socket.pm line 23. Please
contact Veritas Support to resolve this issue.

Enabling proxy appliances running 2.6.0.х works fine and does not return the error.

The proxy server is added in the configuration:

<hostname>.Alerts> CallHome show

CallHome and Proxy Settings
+-----------------------------------------------------+
|CallHome State    | Enabled            ||
|----------------------------+------------------------||
NBInventory State | Enabled           
||---------------------------+------------------------||
   Proxy State         |   Disabled          
||---------------------------+-------------------------||
   Tunnelling           |  Enabled           
||---------------------------+-------------------------||
  Proxy Server        |  https://<proxy>   
||---------------------------+-------------------------||
   Proxy Port           |  80                
||---------------------------+-------------------------||
 Proxy UserName  |                   
|+----------------------------------------------------+

Running the command from the CLI results in the same error:

/opt/NBUAppliance/scripts/hwmon/callhome_setup.pl --proxyenable

Proxy enable failed. Please fix the error below and enable it again CALLHOME
status: Not able to upload callhome data Error: error while setting up ssl
connection (SSL connect attempt failed with unknown errorerror:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed) at /opt/
NBUAppliance/scripts/LWP/Protocol/https/connect/Socket.pm line 23. Please
contact Veritas Support to resolve this issue.
 

curl command failed due to the "certification verify failed" error -

<hostname>:/etc/ssl # curl -v --proxy https://<proxy_server>:80 https://www.symappmon.com:443

* About to connect() to proxy <proxy_server> port 80 (#0)
*   Trying <proxy_ip_address>... connected
* Connected to <proxy_server> (<proxy_ip_address>) port 80 (#0)
* Establish HTTP proxy tunnel to www.symappmon.com:443
> CONNECT www.symappmon.com:443 HTTP/1.1
> Host: www.symappmon.com:443
> User-Agent: curl/7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.3 libidn/1.10
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied OK to CONNECT request
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Closing connection #0
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: https://curl.haxx.se/docs/sslcerts.html

Another way to confirm the error is to collect the tcpdump file with following steps:
  - Open two ssh sessions connected to the appliance with problem. One is in
CLISH and another is in elevate root.
  - Run the following command to collect the tcpdump log with elevate root:

tcpdump tcp -i <interface> -s 0 -w <output file>

For example,

tcpdump tcp -i eth1 -s 0 -w ./callhome.cap

  - Run "Callhome Proxy Enable" command in CLISH.
  - Press Ctrl+c to stop the tcpdump capture and provide the log.

From the callhome.cap log (Use WireShark application to read the file),
the SSL connection is failing due to "TLSV1 Alert (Level: Fatal, Description: UnKnown CA)".

But below w3m command which also needs to setup the SSL connection with www.symappmon.com is successful -

export HTTP_PROXY=https://<proxy server>:80/w3m https://www.symappmon.com

Prompted for "Username for SymAppMon Access:".

export HTTP_PROXY=https://<proxy server>:80/w3m https://api.appliance.Veritas.com

The requested URL / was not found on this server
Apache Server at api.appliance.Veritas.com Port 443

Which is an expected output.

Error Message

From the NBAPP logs (oid for NBAPP_CALLHOME) can be seen that the issue is caused by "certificate verify failed"

 -

07/16/15 14:00:51.961 [Application] NBAPP 409 NBAPP_CALLHOME 2 PID:84775
TID:139993673299712 File ID:2 [No context] [Info] [84662] Failed to connect to callhome using NTLM Authentication
07/16/15 14:00:51.988 [Application] NBAPP 409 NBAPP_CALLHOME 2 PID:84777 TID:140290921502464 File ID:2 [No context] [Error] [84662] Registration failed with status : Can't connect to api.appliance.symantec.com:443

LWP::Protocol::https::Socket: getaddrinfo: Name or service not known at /opt/VRTSperl/lib/site_perl/5.14.2/LWP/Protocol/http.pm line 51.
.
....
07/16/15 14:00:53.427 [Application] NBAPP 409 NBAPP_CALLHOME 2 PID:84818
TID:139773378406144 File ID:2 [No context] [Info] [84662] Failed to connect to callhome using NTLM Authentication
07/16/15 14:00:53.458 [Application] NBAPP 409 NBAPP_CALLHOME 2 PID:84820 TID:140395183621888 File ID:2 [No context] [Error] [84662] Register failed with status : error while setting up ssl connection (SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed) at /opt/NBUAppliance/scripts/LWP/Protocol/https/connect/Socket.pm line 23.
.
07/16/15 14:00:53.488 [Application] NBAPP 409 NBAPP_CALLHOME 2 PID:84822 TID:139873077454592 File ID:2 [No context] [Info] [84662] Register Failed. Status is error while setting up ssl connection (SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_
CERTIFICATE:certificate verify failed) at /opt/NBUAppliance/scripts/LWP/Protocol/https/connect/Socket.pm line 23.

Note: The above error "getaddrinfo: Name or service not known" is expected and not the cause of the problem.

Cause

On NetBackup appliances, the curl command uses the certificate files under /etc/ssl/certs to verify the certification.
The curl command was failed with "certification verify failed, although VeriSign CA files exist under
/etc/ssl/certs directory.

Verified that the VeriSign files are in /etc/ssl/certs -

# ls -l /etc/ssl/certs/Veri*

The output should contain several files, here is an example -
-rw-r--r-- 1 root root 1281 Feb 24  2014 /etc/ssl/certs/VeriSign_Class_3_Public_Primary_Certification_Authority_G4.pem
-rw-r--r-- 1 root root 1732 Feb 24  2014 /etc/ssl/certs/VeriSign_Class_3_Public_Primary_Certification_Authority_G5.pem
-rw-r--r-- 1 root root 1700 Feb 24  2014 /etc/ssl/certs/VeriSign_Universal_Root_Certification_Authority.pem

The problem with enabling the proxy happens due to the firewall doing SSL interception, i.e. it works as a man-in-the-middle. To inspect encrypted connections
it splits it into two encrypted connections, but it will not be able to sign the connection between browser and proxy with the original certificate.
Thus it will create a new certificate, signed by a CA specific to the firewall.

To verify this, follow these steps:

1. Find a Windows system which is in the same subnet as the appliance and also connect to the same proxy server, try to access https://www.symappmon.com with the browser

For Firefox, click on the lock button and then “More Information…”

For Internet Explorer find the lock icon in the address field.

Internet Explorer 8 -





Internet Explorer 11 -




2. Click on “View Certificate”

The following is an example for FireFox (IE example is shown in step 1) -



3. Check the information of the Certificate.

The certificate should be issued by VeriSign, Inc and have Common name as VeriSign Class CA certificate.

For FireFox -




In the Details tab it should display more information about the VeriSign CA certificate -



For Internet Explorer also check that the certificate has been issued by VeriSign -




Certification Path should show the following -



If  the certificate is different from the above screenshots and does not contain VeriSign CA, then it is the certificate specific to the firewall and is causing this issue. 

The following is an example of a certificate specific to the firewall, it shows that the certificate starts with "CA" and does not have "VeriSign" in the name -

On 2.6.1.2 appliances, the perl code uses cacert.pem file under /opt/NBUAppliance/lib/perl5/Mozilla/CA to verify the certification.

To reproduce this defect remove all the Verisign CA from the cacert.pem file and run “CallHome Test” in CLISH without proxy.
<appliance>.Alerts> CallHome Test
Callhome test failed. Error CALLHOME status: Not able to upload callhome data
Error: Can't connect to www.symappmon.com:443 (certificate verify failed)
LWP::Protocol::https::Socket: SSL connect attempt failed with unknown
errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
failed at /opt/VRTSperl/lib/site_perl/5.14.2/LWP/Protocol/http.pm line $1.
Please contact Symantec Support to resolve the issue.

On 2.6.0.x appliances, the perl code does not do certification verify, that's why callhome proxy can be enabled successfully in these
versions even if the.firewall is intercepting the SSL connection.

Solution

To make the curl command work add this CA ceritificate to the /etc/ssl/certs folder.

To fix the problem with enabling the proxy,
add the firewall certificate to the  /opt/NBUAppliance/lib/perl5/Mozilla/CA/cacert.pem  file.