Search <book_title>...

NetBackup™ Web UI Cloud Administrator's Guide

Last Published: 2024-03-27

Product(s): NetBackup & Alta Data Protection (10.4)

Troubleshooting

Troubleshooting snapshot restore process for Microsoft Azure cloud

When you start a subsequent (twice) restore operation on the same VM, an error occurs during a restore operation. This error causes the following issues:

The tags from the original OS disk are not copied to the newly created restored OS disk.
User logon might fail after the VM restore due to SSH failure.

Workaround:

Check if the SSH daemon is running on the system. If not, then perform the steps in the following article.

learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/troubleshoot-ssh-connection

Filtering unsupported files and folders

If you try to restore files or folders from a partition or a file system that Snapshot Manager does not support, then you get the following error in the restore job.

Error nbcs (pid=<processs id>) Failed to restore file(s) and folder(s) from snapshot for asset <asset name>

Workaround:

You can filter any files or folders that Snapshot Manager does not support. In the bp.conf file on the primary server, enable the CP DISKMAP check by setting the following flag.

CP_DISKMAP_CHECK = true/yes

Backup from restore operation is partially successful

Backup from restore operation is partially successful when disk is full on the selected target directory. The following messages are displayed:

Dec 29, 2022 2:57:51 PM - Info nbcs (pid=2244) Granular restore(SFR) is completed
    Dec 29, 2022 2:57:51 PM - Info nbcs (pid=2244) Summary of SFR Operation - Success files/folders count: 0 , 
    Failed files/folders count: 1 , Warning files/folders 
    count: 0, Skipped files/folders count: 0
    Dec 29, 2022 2:57:51 PM - Info nbcs (pid=2244) 
    Detailed restore summary report is available on recovery target host at location: 
    /var/log/flexsnap/restore/granular-restore-09b4d44d
    .
    .
    Dec 29, 2022 2:57:51 PM - Warning bprd (pid=1977) Granular Restore from backup completed with error. 
    Copy the files manually from live access mount: 
    ip-10-239-185-241:/mnt/vpfs_shares/vmfiles/8fcc/8fcc132b-a202-49a8-b654-81ff242a718a/livemount
    Dec 29, 2022 2:57:51 PM - end Restore; elapsed time 0:01:51
    the requested operation was partially successful(1)

For a backup from restore, if a live mount is created successfully, then even if other errors are reported apart from ASSET_NOT_FOUND, it is considered as partial success. If no network devices or a file system are mounted on the target location, or the disk is full, the following messages are displayed in the job details:

 Jan 02, 2023 12:11:16 AM - Error nbcs (pid=13934) 
187776K space required for file/folder restore while 20K is total available space on /disk1

In this case, other network devices or the file system must have been mounted on the target path, hence the Snapshot Manager agent considers the free space on the device or file system. As the copy fails with space error, it is logged into the summary report. For example:

 /var/log/flexsnap/restore/granular-restore-09b4d44d in above Job details log

Workaround:

Check the summary report on the target-host location. For example,

/var/log/flexsnap/restore/granular-restore-09b4d44d
[root@ip-10-239-187-148 granular-restore-09b4d44d]# cat root-error.log
Dec 29 09:27:44: ERROR - FILE: /disk1/dl380g9-149-vm15_package.zip
[Error 28] IOError: No space left on device

If the file copy operation failed due to disk space, then create some space and copy the file from live mount.

The live mount path details can be found in the job details as follows:

Dec 29, 2022 2:57:51 PM - Warning bprd (pid=1977) Granular Restore from backup completed with error. 
Copy the files manually from live access mount: 
ip-10-239-185-241:/mnt/vpfs_shares/vmfiles/8fcc/8fcc132b-a202-49a8-b654-81ff242a718a/livemount

Partial recovery is observed when the user selects a disconnected target virtual machine

Partial recovery may be due to the following reasons:

If the target virtual machine is disconnected (no connectivity through an agent).
If any failure is seen during the copy of files or folders on the target virtual machine.
When a Windows virtual machine content is restored on a Linux target virtual machine.

In these partial recovery cases, the created instant access is not deleted and is available for the next 24 hours.

You can configure the instance access retention interval using the CLOUD_VM_IA_RETENTION_INTERVAL_IN_HOURS parameter in the bp.conf file. (Default value is 24 hours.)

Workaround:

User can perform manual steps to access the instant access share on the target host and then manfully copy the required files or folders.

(Copy files over NFS) To restore Linux image content on a Linux host:

To mount an NFS share on a Linux system, install the NFS client package using the following command:
$ sudo yum install nfs-utils

Using the following mount command, mount the Instant access on the target Linux host:

# Create a directory say /mnt/restore

$ mkdir -p /mnt/restore

# Mount the instant access 

$ mount -t nfs <InstantAccessServer:InstantAccessPath> /mnt/restore

Instant access path can be retrieved from activity manager logs which is in the following format:
<InstantAccessServer>:/mnt/vpfs_shares/vmfiles/<id>/<InstantAccessId>/livemount

(SMB access) To restore Windows image contents (with ACL) on Windows target host:

SMB credentials of MSDP storage server of source virtual machine image must be added to the Windows credential manager.
Use the given live mount to access virtual hard disks by navigating to Activity Monitor > Job details.
The virtual hard disks are listed under the folder with vhd_ as the prefix.
From the Action tab, attach the required virtual hard disk and click on OK.
Select Assign the following drive letter option to assign the letter to virtual disk to browse data and click on OK.
Navigate to the assigned drive in the previous step and copy the data manually.

(Live mount) To restore Windows image contents on a Linux target host:

Linux must have the CIFS package. Obtain the packages using the # yum install cifs-utils command.
Create the mount directory using the # mkdir <my_mount_dir> command.
Use your Samba username and password to mount the exported path as follows:
mount -t cifs -o username=<sambauser> //<InstantAccessServer>/<InstantAccessPath> <my_mount_dir>
Copy the files using the following command:
# cp <my_mount_dir>/<file_path> <target_dir_path>

Issues with single file restore from the backup of a snapshot

Issue/Error	Description	Workaround
Log path to check	For information related to restore details on the target host, check the following logs: `path/file: /tmp/flexsnap-agentless-onhost.log` `/var/log/flexsnap/restore/granular-restore-*`	To resolve the failures or any exceptions that occurred during the single file restore on Snapshot Manager, refer to the following logs on the Snapshot Manager host: `/cloudpoint/logs/flexsnap.log`
Pre-recovery check fails	When restoring files and folders to the disconnected target virtual machine, the pre-recovery check fails with the following error: Target VM state: Target VM <vm_name> has no agent configured If recovery is started, the restore operation is partially successful.	Ensure that the target virtual machine is connected to the agent configured for successful restore.
Partial recovery for source Linux VM to target Windows VM (no NFS client)	If you do not install the NFS client on the Windows target computer, a restore of files and folders from a source Linux VM is partially successful. The following error displays: Error nbcs (pid=42513) Invalid operation for asset: <asset_id> Warning bprd (pid=42045) Granular Restore from backup completed with error. Copy the files manually from live access mount: <livemount_path>. Note that live access mount is available only for 24 hrs.	Install the NFS client on the Window target computer before you perform the restore from a Linux VM to a Windows VM.
Restore job fails for deleted target VM	The restore job fails with the following error when restoring files and folders to the target VM which is deleted from the cloud environment: Error nbcs (pid=44859) Target VM not found, asset_id <asset_id>	Select a different target VM.
Creating instant access fails	If instant access is not enabled on the MSDP storage server, the creation of the instant access fails during the restore job.	Verify if instant access is supported on the MSDP media server. Run the following pre-check script: /usr/openv/pdde/vpfs/bin/ia_byo_precheck.sh
Target VM does not have the free drives to attach to the virtual disk	If the number of volumes that contain the selected file(s) is more than the number of free available drives on the target host, the operation fails.	Select a smaller number of volumes for the restore.
Not enough space: '*\\driverMapping.json	The media server where MSDP is configured has FIPS enabled.	Disable FIPS on the media server where MSDP is installed. Or, add the domain user Samba credentials to the target VM.

Issue with Azure cloud provider VMs

If any one of the disks in the VM is not initialized, download or restore of the VM 's files using instant access fails with the following error:

Jan 24, 2023 11:58:47 AM - Error NBWMC (pid=3716) Internal Error: ('failed to find operation system information, please check the source VM', ('Failed to expose 
VMDK', 1006), None)
Failed to create the instant access mount.
(4001)

libguestfs is a third-party tool that instant access uses to retrieve files from a VM backup. If a disk is not initialized, libguestfs cannot retrieve the files.

Workaround:

Initialize the disk and back up the VM. Then try again to download or restore the VM files using instant access.