Deduplication folder goes offline after multiple Deduplication folders are shared with Backup Exec servers.

Problem

The Deduplication folder goes offline after multiple Deduplication folders are shared between Backup Exec MBES and/or CAS servers.

Error Message

The local Deduplication folder appears offline in the storage tab and jobs targeted to deduplication storage folder fail with the following error:  

Robotic library hardware error. 

The robotic library has reported a general hardware error condition. Manual intervention, calibration, etc., may be required. This may be a device related hardware problem, or it may be some other hardware problem (i.e. SCSI card/bus etc.). The library and drive states have been set to offline. Please attend to this condition. 

Errors observed in ADAMM log:

PvlChanger::MapErrorCode() - offline device  

Drive = 1246 "Dedupe 0001:57" 

ERROR = 0xE000820A (E_CHG_HARDWARE) 

PvlDrive::UpdateOnlineState() 

Drive = 1247 "Dedupe 0001:58" 

ERROR = The device is offline! 

Cause

Backup Exec 24 and above, changes are made to the internal deduplication device concurrency due to which the maximum number of supported deduplication devices per Backup Exec server has changed.

This affects Deduplication Storage folders, Open Storage Technology (OST) devices and Deduplication to cloud (MSDP-C) device. 

Scenario 1 --> Fresh installation of BE 25 -  A maximum of only 1 local and 2 shared devices can remain online when Deduplication concurrent device per server limit is exceeded. OST and MSCP-C devices are not impacted, so more number of OST and MSDP-C devices can be added. 

Scenario 2 --> Upgrade from BE 23 to BE 24 - A maximum of only 1 local and 1 shared device can remain online when Deduplication concurrent device per server limit is exceeded. OST and MSCP-C devices are impacted and so, more number of OST and MSDP-C devices cannot be added. 

Scenario 3 --> Upgrade from BE 23 to BE 25 - A maximum of only 1 local and 3 shared devices can remain online when Deduplication concurrent device per server limit is exceeded. OST and MSCP-C devices are impacted and so, more number of OST and MSDP-C devices cannot be added. 

Solution

To bring the offline Deduplication folder online, try any one of the following solutions.

Solution 1 --> Un-share any one of the shared Deduplication folders and restart BE services. Once the deduplication concurrent devices are within the limit, the Deduplication folder will appear online.

OR

Solution 2 --> Create the following registry key on the affected Backup Exec server and restart Backup Exec services – 

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

HKEY_LOCAL_MACHINE\SOFTWARE\Software\Veritas\Backup Exec For Windows\Adamm 
Type= REG_DWORD
ValueName = "ExtendConcurrencyTo"
Value = 32 (or lower according to environment)

Applies to -->  Backup Exec 24, Backup Exec 25