Troubleshooting backup and restore failures with Backup Exec Hyper-V backups

Problem

Backup Exec Hyper-V backup job fails with the following error:

Error Message

Primary failure shown in job log :

0xa0009679 - A failure occurred querying the Writer status for a Hyper-V virtual machine. See the job log for details about the error.

V-79-40960-38521

UMI Code: V-79-40960-38523
Error Code: 0xA000967B : One or more Hyper-V services are not running.  Review and start the necessary Hyper-V services from the Windows Services Control Panel.

Note: Other errors could potentially be diagnosed and resolved by performing the steps in this solution.

Cause

One or more of the following may cause the Hyper-V backup job to fail with the above mentioned error:

1. Backup job created to back up a virtual machine (VM) running on Hyper-V 2012 / 2012 R2 may fail due to snapshot errors (i.e. VSS related issues).
2. Hyper-V Integration Services related issues:
   • Hyper-V Integration Services are not installed. 
   • There is a version mismatch between the Hyper-V host and Hyper-V Integration Services inside the VM.
   • There is a communication failure i.e. Hyper-V host is unable to communicate with the necessary VSS components inside the VM.
3. If an offline backup is being performed.

 

Solution

 

Issue 1:  Backup job is created to backup a VM running on Hyper-V 2012 (R2) / 2016 / 2019 The job runs and fails with a snapshot error.

 

Follow instructions below to gather more information which can potentially help troubleshoot this issue:

1. On the Hyper-V host for the failing VM, enable BEREMOTE debug logging:
   • Go to Windows Registry Editor
     
     Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes. 
   • Browse to - HKEY_LOCAL_MACHINE\SOFTWARE\veritas\Backup Exec for Windows\Backup Exec\Engine\Logging\
      
   • Set the CreateDebugLog value to 1 (create the value if it does not already exist).
      
2. Run the job again
    
3. Look in the BEREMOTE debug log to see if the following entries exist:
   [1780] T16:45:11.914  | VSS_VOLSNAP_ATTR_AUTORECOVER
   [1780] T16:45:11.914 [besc]- -------End of snapshot property dump------
   [1780] T16:45:11.914 [besc]- VssSnashotVolume::GetSnapshotProperties() - done. LastError_: 0x0
   [1780] T16:45:11.914 [fsys\shadow]- SHADOW::GetWriterStatus(): Start Phase POST_SNAPSHOT
   [1780] T16:45:12.004 [fsys\shadow]- [FAILED STATUS] Writer=Microsoft Hyper-V VSS Writer ID={66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE} Instance={35DAE8F2-1545-4244-A114-AAE7CE8831C3} status VSS_E_WRITERERROR_RETRYABLE (0x800423F3) state=8
   [1780] T16:45:12.004 [fsys\shadow]- Status VSS_E_WRITERERROR_RETRYABLE (0x800423F3) State 8 getting Writer Microsoft Hyper-V VSS Writer ID {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE} Instance ID {35DAE8F2-1545-4244-A114-AAE7CE8831C3} status in SHADOW::UpdateWriterStatus
   [1780] T16:45:12.004 [fsys\shadow]- SHADOW::GetWriterStatus(): End Phase POST_SNAPSHOT [result=0xe000fed1]
   [1780] T16:45:12.004 [fsys\shadow]- Status FS_WRITER_STATUS_FAILURE (0xE000FED1) returned getting Writer status during SNAPSHOT_NOTIFICATION::PostSnapshot
   [1780] T16:45:12.079 [fsys\vrtsrv]- VRTSRV:Status FS_WRITER_STATUS_FAILURE (0xE000FED1) calling FS_SnapshotNotification in VirtualServer::SnapshotNotification:521
   [1780] T16:45:12.085 [besc]- SnapshotBackupEngine - SNAPSHOT_NOTIFICATION::PostSnapshot failed with fstatus: 0xa0009679
   [1780] T16:45:12.085 [besc]- Error SnapshotBackupEngine::snap_file_systems failed.
   [1780] T16:45:12.085 [ndmp\ndmpsrvr]- ndmpdSnapshotPrepare2: Snapshot error. ErrorCode: 0xa0009679 ErrorLevel: 1 Error Src: 3
   [1780] T16:45:12.086 [ndmp\ndmpsrvr]- ndmpdSnapshotPrepare2: sending reply...
    
4. Look in the Hyper-V event logs on the Hyper-V host for addition information. 
   
   Note: The Hyper-V event logs can be found here:
   
   Figure 1:
         
   
   Review the Hyper-V VMMS Admin Event Log.  Look for events that pertain to the VM that a backup failed from and received a VSS error.  
   
   Possible Events displayed are shown in 'Figure 2' below:
   
   Figure 2
   
   
   This particular error indicates one or more of the VSS writers inside the VM had a problem which prevented the Hyper-V host from completing its processing. Look inside the VM for further information. There could be many events per attempted snapshot operation per VM and all of these events should be investigated.
    
5. Open up the Event logs inside the VM and look in the Application and System Event logs for VSS related errors.

 

For snapshot issues related to VSS:

Perform the following on the Hyper-V host and inside the VM to make sure that VSS is in a “good” state:

1. Open up a “Run as Administrator” command prompt
    
2. Issue the following command(s):
   VSSADMIN LIST WRITERS
   VSSADMIN LIST PROVIDERS
    
3. Take note if any writers are reporting a “FAILED” state or “NONRETRYABLE” error. 

 

Note: The writers inside a Windows 2003 / Windows 2008 based VM when running on a Windows 2012-R2 Hyper-V or higher host will report a “FAILED” state after being backed up using the Backup Exec Hyper-V agent (any backup application will cause the same results). This is a result of how the Hyper-V host and the Hyper-V Integration Services inside the VM generate the snapshot.

Below is a link to the Microsoft KB Article that states this is to be expected:

VSS writers report as failed on a Windows Server 2003-based or Windows Server 2008-based virtual machine - https://support.microsoft.com/kb/2952783

For Windows 2003 / Windows 2008 VMs on Hyper-V 2012-R2 this “FAILED writer condition” should not adversely impact Backup Exec’s ability to back up the VM using the Hyper-V agent or any agent based back up (ie: using RAWS inside the VM). This does, however, make it difficult to analyze the state of VSS when errors do occur.

 

 
Issue 2:  Hyper-V Integration Services related issues 

If an OS (Operating System) upgrade of the Hyper-V host is performed without upgrading the Hyper-V Integration Services inside the VM or if it has copied / migrated a VM from an older Hyper-V host OS to a newer Hyper-V host OS, chances are high the Hyper-V host will have a problem communicating with the guest during the snapshot process. 

The following PowerShell commands can help identify potential issues:


Windows 2012/2012R2/2016 Hosts

$vm_name = "<change to the name of the VM>"
$vm = gwmi -namespace root\virtualization\v2 -query "select * from Msvm_ComputerSystem where ElementName = '$vm_name'"
gwmi -namespace root\virtualization\v2 -query "Select * from Msvm_VssComponent where SystemName = '$($vm.Name)'"


Windows 2008R2 Hyper-v Hosts

$vm_name = "<change to the name of the VM>"
$vm = gwmi -namespace root\virtualization -query "select * from Msvm_ComputerSystem where ElementName = '$vm_name'"
gwmi -namespace root\virtualization -query "Select * from Msvm_VssComponent where SystemName = '$($vm.Name)'"

 

Note: The above script must be “Run as Administrator”.

The output should display:

---

__GENUS : 2
__CLASS : Msvm_VssComponent
__SUPERCLASS : CIM_LogicalDevice
__DYNASTY : CIM_ManagedElement
__RELPATH : Msvm_VssComponent.CreationClassName="Msvm_VssComponent",DeviceID="Microsoft:5CED1297-4598-4915-A5FC-AD21BB4D02A4", SystemCreationClassName="Msvm_ComputerSystem", SystemName="CA1B9EC0-F86A-4C18-AC34-F7A6BC2DF4D0"
__PROPERTY_COUNT : 38
__DERIVATION : {CIM_LogicalDevice, CIM_EnabledLogicalElement, CIM_LogicalElement, CIM_ManagedSystemElement...}
__SERVER : SERVER1
__NAMESPACE : root\virtualization\v2
__PATH : \\SERVER1\root\virtualization\v2:Msvm_VssComponent.CreationClassName="Msvm_VssComponent",DeviceID="Microsoft:5CED1297-4598-4915-A5FC-AD21BB4D02A4", SystemCreationClassName="Msvm_ComputerSystem",SystemName="CA1B9EC0-F86A-4C18-AC34-F7A6BC2DF4D0"
AdditionalAvailability : {6.EN_US}
Availability : 6
AvailableRequestedStates :
Caption : VSS
CommunicationStatus :
CreationClassName : Msvm_VssComponent
Description : Microsoft VSS Service
DetailedStatus :
DeviceID : Microsoft:5CED1297-4598-4915-A5FC-AD21BB4D02A4
ElementName : VSS
EnabledDefault : 7
EnabledState : 2
ErrorCleared :
ErrorDescription :
HealthState : 5
IdentifyingDescriptions :
InstallDate : 20140124210204.886879-000
InstanceID :
LastErrorCode :
MaxQuiesceTime :
Name : VSS
OperatingStatus :
OperationalStatus : {2.EN_US}
OtherEnabledState :
OtherIdentifyingInfo :
PowerManagementCapabilities :
PowerManagementSupported :
PowerOnHours :
PrimaryStatus :
RequestedState : 12
Status : 

StatusDescriptions : {OK.EN_US}

StatusInfo :
SystemCreationClassName : Msvm_ComputerSystem
SystemName : CA1B9EC0-F86A-4C18-AC34-F7A6BC2DF4D0
TimeOfLastStateChange :
TotalPowerOnHours :
TransitioningToState : 12
PSComputerName : SERVER1

---

 

The highlighted line is most informative, but it is only meaningful if the VM is powered on.  If it says “OK”, things should be fine.  But it is possible some text that says “no contact” may be displayed.  This means the Hyper-V host is unable to communicate the necessary VSS components inside the VM and could mean the following:

• Hyper-V Integration Services are not installed:
• There is a version mismatch between the Hyper-V host and Hyper-V Integration Services inside the VM
• There is a communication failure. In most cases, rebooting the VM resolves this problem.

 

If “no contact” is observed in the PowerShell output, perform the following steps in the order listed:

1. Ensure that the VM was powered on and running at the time the PowerShell was run.
2. Make sure that the Hyper-V Integration Services have been installed / upgraded inside the VM.
3. Ensure that the VM has been rebooted after install/upgrade of the Hyper-V Integration Services.
4. Check and confirm that the “Backup Integration” option is enabled in the VM’s Hyper-V settings options.
5. Confirm if the guest OS actually support VSS and such (For example, a guest running Linux will return “no contact”).
6. Stop and Restart VMMS on the Hyper-V host (net stop VMMS … net start VMMS).
7. Reboot the the Hyper-V host and then the VM.

 

If the issue still persists, contact Microsoft for further assistance.

 

Issue 3:  An Offline backups is being performed.

Note: Backup Exec does not ask for "online" or an "offline" backup.  It simply asks the Hyper-V subsystem to take a snapshot so that Backup Exec can move the data to the media server.  If an "offline" backup is being performed, it is Hyper-V that is making that decision, not Backup Exec.

Known causes of an “offline” backup:

1. The Hyper-V Integration Services are out-of-date or report a no-contact condition (see the PowerShell above):
   • Ensure the Hyper-V Integration Services are up to date.
   • See "If no-contact is observed in the PowerShell output....." section above to resolve the no-contact condition.
      
2. The VM has a dynamic volume inside of it.
    
3. The VM uses a storage space inside of it.
    
4. The VM is not configured with a SCSI controller. 
   Note: This has been found to be a problem for some but not all VMs.
   
   Look in the Hyper-V Admin VMMS log and see if there are any errors like this:
   
   The virtual machine 'SERVER1.org' cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID 807B2738-1A18-4256-A98B-4BC0321720FE).
   
   Add a SCSI controller to the VM’s configuration.  Any VHD/VHDX files do not need to attach/assign to the SCSI controller.

 

There could be other causes. The Hyper-V subsystem usually reports the reason why it didn’t perform an “online” backup with an entry in the Hyper-V VMMS Admin event log listed above. Check the log immediately after the VM’s snapshot was generated to see if anything of interest shows up.
 

 

Applies To

Note: Though this article was primarily written to troubleshoot issues with Backup Exec 2014 (or newer) Hyper-V Agent backups, some of the steps could also pertain to prior supported versions of Backup Exec.

 

UMI : V-79-40960-38521