How to solve the VMware Instant Access creation issue with error 4004 after upgrading from NBFS v1.3.1 to v2.1

Problem

After upgrading from NBFS v1.3.1 to v2.1, it could fail to create VMware Instant Access. The Instant Access job could reported the error information, “Unable to query remote mount point's attributes”.

Error Message

“Unable to query remote mount point's attributes”.

Cause

The NBFS (NetBackup FlexScale) version is 1.3.1 in the customer’s environment, there are some VMware backups and VMware InstantAccess VMs created on it. After customer upgrades NBFS from v1.3.1 to v2.1, it could fail to create the new VMware InstantAccess, and the error displays “Operation failed, diagnostics report: Unable to complete Sysinfo operation.  Please see the VMkernel log file for more details.: Sysinfo error: Unable to query remote mount point's attributes. See VMkernel log for details.”

The issue could happen.

NBFS InstantAccess VMware feature depends on ESXi hosts and NFS datastores. During the NBFS upgrading process, all MSDP engines will be stopped, and the ESXi NFS datastores will lost the connection to MSDP engines for several hours. If there are the existing Instant Access VMs created, they all will be in the inactive state. The ESXi NFS datastore devices could be in the APD state (All Paths Down). After the NBFS upgrade finishes, the existing NFS datastores could not recover in normal, and the new Instant Access creation could fail due to the datastore attaching failure with the error “Unable to query remote mount point's attributes”.

Solution

If the issue happens, customer can fix it by remounting all NFS datastores on the ESXi host. Customer needs to check their vCenter or ESXi hosts, and get all ESXi hosts which involve in the VMware Instant Access created in NBFS. Then do the below steps to remount the NFS datastores on each ESXi host. After remounting the NFS datastores on all ESXi hosts, customer can create the new VMware InstantAcess.

SSH login an ESXi host.

1. Save the existing NFS datastores' records in a temporary file, for example, /tmp/esxcli_storage_nfs_list_1635041086.txt.

  esxcli storage nfs list >  /tmp/esxcli_storage_nfs_list_`date "+%s"`.txt

  2. Get the NFS datastores which the names start with "NBU_IR_" from the temporary file, and remove them on ESXi host.

  cat /tmp/esxcli_storage_nfs_list_1635041086.txt | grep "^NBU" | awk '{print $1;}' |while read line; do echo $line; esxcli storage nfs remove -v $line; done

  3. Generate the remount script and the remount parameters from the records in temporary file.

  cat /tmp/esxcli_storage_nfs_list_1635041086.txt | grep "^NBU" | awk '{print "esxcli storage nfs add -H "$2" -s "$3" -v "$1;}' > /tmp/mount_all_esxi_nfs.sh

chmod +x  /tmp/mount_all_esxi_nfs.sh

  4. Execute the remount script.

  sh /tmp/mount_all_esxi_nfs.sh

  5. Check the NFS datastores.

  esxcli storage nfs list