bptestbpcd resulting in socket read failed status 23 for client using backup interfaces

Problem

Have a newly configured client which cannot be backed up, it can connect to the master server and the first media server/appliance, but it cannot connect from the second media server/appliance, which fails with status 23.

Example environment:

Master Server Name: nbmaster81
  IP Address 1: <Master_Prod_NIC_IP>
  IP Address 2: <Master_Bkup_NIC_IP> (nbmaster81-bkp.domain.local / nbmaster81-bkp)

Media Server Name: nbmedia81 
  IP Address 1: <Media_Prod_NIC_IP>
  IP Address 2: <Media_Bkup_NIC_IP> (nbmedia81-bkp.domain.local / nbmedia81-bkp)

Client Server Name: nbclient81-bkp.domain.local (nbclient81-bkp)
  IP Address: <Client_Bkup_NIC_IP>

Error Message

Output from bptestbpcd command from the media server appliance:

nbmedia81:/usr/openv/netbackup/bin/admincmd # ./bptestbpcd -client nbclient81-bkp.domain.local -verbose -debug
15:43:29.768 [5720] <2> bptestbpcd: VERBOSE = 1
15:43:29.769 [5720] <8> vnet_get_user_credential_path: [vnet_vxss.c:1552] status 35 0x23
15:43:29.769 [5720] <8> vnet_check_user_certificate: [vnet_vxss_helper.c:3879] vnet_get_user_credential_path failed 35 0x23
15:43:29.769 [5720] <2> ConnectionCache::connectAndCache: Acquiring new connection for host nbmaster81, query type 223
15:43:29.830 [5720] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:43:29.831 [5720] <2> logconnections: BPDBM CONNECT FROM <Master_Prod_NIC_IP>.41895 TO <Media_Prod_NIC_IP>.1556 fd = 3
15:43:29.841 [5720] <2> db_CLIENTsend: reset client protocol version from 0 to 9
15:43:29.884 [5720] <2> db_end: Need to collect reply
15:43:29.948 [5720] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:43:29.949 [5720] <2> logconnections: BPCD CONNECT FROM <Media_Bkup_NIC_IP>.36740 TO <Client_Bkup_NIC_IP>.1556 fd = 3
15:43:29.989 [5720] <2> vnet_pbxConnect: pbxConnectEx Succeeded
15:43:29.991 [5720] <8> do_pbx_service: [vnet_connect.c:2186] via PBX VNETD CONNECT FROM <Media_Bkup_NIC_IP>.60843 TO <Client_Bkup_NIC_IP>.1556 fd = 4
15:43:29.992 [5720] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:455] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
15:43:30.004 [5720] <8> vnet_pop_byte: [vnet.c:197] errno 0 0x0
15:43:30.004 [5720] <2> vnet_pop_byte: vnet.c.199: 0: Function failed: 9 0x00000009
15:43:30.004 [5720] <2> vnet_pop_string: vnet.c.279: 0: Function failed: 9 0x00000009
15:43:30.004 [5720] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:477] status 9 0x9
15:43:30.004 [5720] <2> bpcr_vnetd_connect_forward_socket_begin_ex: vnet_vnetd_connect_forward_socket_begin() failed: 9
15:43:30.004 [5720] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_socket_begin failed: 23
15:43:30.004 [5720] <2> ConnectToBPCD: bpcd_connect_and_verify(nbclient81-bkp.domain.local, nbclient81-bkp.domain.local) failed: 23
<16>bptestbpcd main: Function ConnectToBPCD(nbclient81-bkp.domain.local) failed: 23
15:43:30.005 [5720] <16> bptestbpcd main: Function ConnectToBPCD(nbclient81-bkp.domain.local) failed: 23
<16>bptestbpcd main: socket read failed
15:43:30.015 [5720] <16> bptestbpcd main: socket read failed
<2>bptestbpcd: socket read failed
15:43:30.015 [5720] <2> bptestbpcd: socket read failed
<2>bptestbpcd: EXIT status = 23
15:43:30.015 [5720] <2> bptestbpcd: EXIT status = 23
socket read failed

Cause

This is not standard configuration, as the client cannot resolve the master server's backup network name without the production master server name being referenced, as there is nothing that links them together. The production master server name is not aliased for the backup interface name in NBDB, the client is not using the production master server name in the NetBackup configuration, and the production master server name is not referenced in the hosts file on the client.

Hence, this is clearly a name resolution issue for two reasons seen in the troubleshooting below:

1. Connection can be established and everything works when using the production master server name in the NetBackup configuration.
2. Plus, there are other clients with similar configuration that can resolve to the master, while this client cannot.

This is often seen when the backup network was not initially planned for all servers/clients and was more of an afterthought to improve network performance, which may require hosts file entries as a workaround.

Troubleshooting:

Note: For this example, the NetBackup Administrator is unable to access the client, which requires a SA (system administrator) and they are not available, hence the Logging Assistant will be utilized to collect the needed verbose/Debug logs and NBSU outputs for in-depth investigation.

    The directory path to the bptestbpcd and nbemmcmd commands is: 
      Linux/UNIX: /usr/openv/netbackup/bin/admincmd/
      Windows: install_path\NetBackup\bin\admincmd\

On the master server, check names in NetBackup Datatbase (NBDB/EMM DB) for master and server entries:

Linux/UNIX: nbemmcmd -listhosts | grep -i <master_server_name>
Windows: nbemmcmd -listhosts | findstr <master_server_name>
master  nbmaster81
server  nbmaster81

Both entries are using the production name in lowercase shortname format, not a backup interface name.

On the master server, check alias names in NetBackup Datatbase/EMM for the master:

nbemmcmd -machinealias -getaliases -machinetype master -machinename nbmaster81
NBEMMCMD, Version: 8.1
The following aliases were found for the alias: "nbmaster81"
nbmaster81
nbmaster81.domain.local
Command completed successfully.

• Only the production name in lowercase shortname and FQDN format are listed, not a backup interface name.

From the NetBackup Administration Console, run the Logging Assistant to setup verbose/Debug network related logging (bpcd & vnetd) on the client.

Reproduce the issue:

Run bptestbpcd from the media server, since bptestbpcd was failing between the media server and client interfaces.

bptestbpcd -client nbclient81-bkp.domain.local -verbose -debug

• It should fail with the expected error, "bptestbpcd: EXIT status = 23, socket read failed".

From the NetBackup Administration Console, run the Logging Assistant to run the NBSU on the client, then collect Debug logging and the NBSU output.

Note: These are usually transferred to the master server and placed into the NetBackup logging directory, under the NetBackup Logging Assistant Staging (nblastaging) directory.

Check the BPCD logs from client, where you should see this error message:

15:12:27.980 [40746] <2> vnet_proxy_build_request: request = {"bpcd_info": null, "i_am_daemon": true, "domain_constraints": {"svc_type": 2}, "service": "bpcd", "auth_only": false, "pid": 40746, "ssa": false, "local_user": "root", "host": "<Media_Bkup_NIC_IP>", "peer_host": "<Media_Bkup_NIC_IP>", "peer_client": null, "local_client": "nbclient81-bkp.domain.local", "local_dhcp_interval": 0, "peer_dhcp_info": null, "local_hostname": "nbclient81-bkp.domain.local", "secure_peer": false, "resilient": false, "nbrntd_peer_addr": null, "nbrntd_sock_addr": null, "hand_back": false, "request_timeout": 300}
15:12:27.983 [40746] <2> vnet_proxy_parse_response: response = {"status": 5949, "local_proxy_info": {}, "domain_constraints_set": {"process_hint": "", "process_hint_reason": "no local hint was selected", "process_hint_server_name": ""}, "statusmsgxml": "<?xml version=\"1.0\" encoding=\"utf-8\"?><StatusMsgDoc><StatusMsg NBUStatus=\"5949\" Severity=\"2\" VXULoid=\"486\" VXULtid=\"49\"><MsgString>A&#32;certificate&#32;entry&#32;was&#32;not&#32;found&#32;for&#32;the&#32;requested&#32;server.</MsgString></StatusMsg></StatusMsgDoc>", "statusmsg": "A certificate entry was not found for the requested server., nbu status = 5949, severity = 2"}
15:12:27.983 [40746] <16> dump_proxy_info: statusmsg: A certificate entry was not found for the requested server., nbu status = 5949, severity = 2
15:12:27.983 [40746] <16> dump_proxy_info: domain_constraints_set:
15:12:27.983 [40746] <16> dump_proxy_info: ----process_hint_server_name:
15:12:27.983 [40746] <16> dump_proxy_info: ----process_hint:
15:12:27.983 [40746] <16> dump_proxy_info: ----process_hint_reason: no local hint was selected
15:12:27.983 [40746] <16> vnet_proxy_protocol_from_legacy: proxy returned status: 5949 msg: {"status": 5949, "local_proxy_info": {}, "domain_constraints_set": {"process_hint": "", "process_hint_reason": "no local hint was selected", "process_hint_server_name": ""}, "statusmsgxml": "<?xml version=\"1.0\" encoding=\"utf-8\"?><StatusMsgDoc><StatusMsg NBUStatus=\"5949\" Severity=\"2\" VXULoid=\"486\" VXULtid=\"49\"><MsgString>A&#32;certificate&#32;entry&#32;was&#32;not&#32;found&#32;for&#32;the&#32;requested&#32;server.</MsgString></StatusMsg></StatusMsgDoc>", "statusmsg": "A certificate entry was not found for the requested server., nbu status = 5949, severity = 2"} vnet status: 5949
15:12:27.983 [40746] <16> vnet_proxy_socket_swap: vnet_proxy_protocol_from_legacy() failed: 5949
15:12:27.983 [40746] <16> daemon_proxy_proto: vnet_proxy_socket_swap() failed: vnet status 5949, nb status 5949
15:12:27.983 [40746] <16> bpcd main: daemon_proxy_proto(bpcd, 8,...) failed: 5949

Note: The VNETD logs from client will essentially echo the same error message above.

Check the master server name in the client's NBSU, which is the top SERVER/Server entry in the NetBackup configuration (bp.conf or registry):

cat -s /usr/openv/netbackup/bp.conf
SERVER = nbmaster81-bkp
...

• The master server name being used is the backup interface in lowercase shortname format, not the production name.
  • Note: If accessible, this can also be checked or updated via the client host properties in the NetBackup Administration Console.

Test or validate that communication with the client will work, using the master server's production name, as listed in the NBDB.

• From the NetBackup Administration Console, Host Properties > Clients: nbclient81-bkp.domain.local
• Add the master server's production name(nbmaster81) to client's Server List.
• Then, select it, click the Make Master button, followed by the Apply and OK buttons.

Now, re-run the bptestbpcd from the media server:

bptestbpcd -client nbclient81-bkp.domain.local -verbose -debug

• It should complete successfully with the expected status, "bptestbpcd: EXIT status = 0".

Since that worked, revert the change, by making the backup interface(nbmaster81-bkp) the master.

With SA assistance, the following messages may be experienced when trying to get client certificates:

• On a known good client(nbclient82), check client's ability to get security certificates using the master's backup interface:

  nbcertcmd -getCertificate -server nbmaster81-bkp.domain.local
Host certificate and certificate revocation list already exist for master server [nbmaster81-bkp.domain.local]

• On the problem client(nbclient81), check client's ability to get security certificates using the master's backup interface:

  nbcertcmd -getCertificate -server nbmaster81-bkp.domain.local
The target server nbmaster81-bkp.domain.local could not be authenticated.
The server name does not match any of the host names listed in the server's certificate.
Names listed in the server's certificate are:
DNS:nbmaster81.domain.local
DNS:nbmaster81
nbcertcmd: The -getCertificate operation failed for server nbmaster81-bkp.domain.local.
EXIT STATUS 8509: The specified server name was not found in the web service certificate

The directory path to the nbcertcmd command is:
Linux/UNIX: /usr/openv/netbackup/bin/
Windows: install_path\NetBackup\bin\

Check hosts file on the known good client(nbclient82), compared to problem client:

Linux/UNIX: cat hosts | grep nbmaster81
Windows: type hosts | findstr nbmaster81
<Master_Bkup_NIC_IP> nbmaster81-bkp.domain.local nbmaster81-bkp nbmaster81

• There is an entry for the backup interface IP address and names, with the production name as an alias.

Check hosts file on the problem client(nbclient81), compared to the known good client:

Linux/UNIX: cat hosts | grep nbmaster81
Windows: type hosts | findstr nbmaster81
#<Master_Bkup_NIC_IP> nbmaster81-bkp.domain.local nbmaster81-bkp nbmaster81

There is not an entry for the backup interface IP address and names, with the production name as an alias; or it has been commented out.

The directory path to the OS hosts file is: 
Linux/UNIX: /etc/hosts
Windows: %WinDir%\System32\Drivers\Etc\hosts

Solution

Recommend:

Using the same master server name consistently throughout the environment, as it is listed in the NetBackup Database.

• If using a backup network, try to use the same naming scheme on all servers and clients during initial configuration/setup.
• After making any name or network changes, be sure to clear the NetBackup host cache on applicable server(s) and/or client(s).
    Linux/UNIX: /usr/openv/netbackup/bin/bpclntcmd -clear_host_cache
    Windows: install_path\Veritas\NetBackup\bin\bpclntcmd -clear_host_cache
• Then, re-run the bptestbpcd command on the master server or media server against the client, which should complete successfully and exit with status 0.

Workaround:

On the client, add the properly formatted entry to the hosts file for the master server's backup interface, include an alias for the master server's production name.

• hosts file entry format:

  ip_address    master-bkup-name    master-bkup-alias    master-prod-alias

• Example hosts file entry:

  192.168.1.81    nbmaster81-bkp.domain.local    nbmaster81-bkp    nbmaster81