Problem
NetBackup deduplication disk pool/disk volume intermittently goes to a DOWN state. Backups/duplications can fail with status 213, 84, 191, 800, 2074 and 2106.
Error Message
Status Code: 84 - media write error
Status Code: 191 - no images were successfully processed
Status Code: 213 - no storage units available for use
Status Code: 2074 - disk volume is down
Status Code: 2106 - disk storage server is down
Cause
The NetBackup Disk Polling Service (DPS), which is responsible for telling NetBackup whether a disk pool / disk volume is up, polls stats from the MSDP / PDDO / OST storage servers via bpstsinfo. However, DPS has a 1 minute default timeout limit. If DPS does not get a reply back within one minute, DPS treats the timeout as an error on the disk pool / disk volumes as down.
Heavy system resource load or depletion of system resources can cause a delay in the DPS reply.
Solution
Workaround:
On the Media Servers, increase the NetBackup Disk Polling Service (DPS) timeout, per steps below.
Windows Steps:
- Create the following empty file to disable DPS Proxy timeout:
- install_path\netbackup\db\config\DPS_PROXYNOEXPIRE
- Create the following two DPS configuration files and add an entry of: 1800 or the max of 3600.
- install_path\netbackup\db\config\DPS_PROXYDEFAULTSENDTMO
- install_path\netbackup\db\config\DPS_PROXYDEFAULTRECVTMO
- Note: There must be no file extensions on the DPS configuration files. For example: No '.txt' extension.
- The files should only contain the number value (1800 or 3600) and no other data within the file.
- Restart nbrmms (NetBackup Remote Manager and Monitor Service), or just stop and restart all services.
- install_path\netbackup\bin\bpdown -f -v
- install_path\netbackup\bin\bpup -f -v
UNIX/Linux Steps:
- Create the following empty file to disable DPS Proxy timeout:
# touch /usr/openv/netbackup/db/config/DPS_PROXYNOEXPIRE
# echo "3600" > /usr/openv/netbackup/db/config/DPS_PROXYDEFAULTSENDTMO
# echo "3600" > /usr/openv/netbackup/db/config/DPS_PROXYDEFAULTRECVTMO
- Restart nbrmms (NetBackup Remote Manager and Monitor Service):
# /usr/openv/netbackup/bin/nbrmms -terminate
# /usr/openv/netbackup/bin/nbrmms
- Or, stop and restart all services on the MSDP media server:
# /usr/openv/netbackup/bin/goodies/netbackup stop
# /usr/openv/netbackup/bin/goodies/netbackup start
Note: If the issue persists after one daily or nightly backup schedule window after employing the above configuration changes, remove the touch files and troubleshoot the issue further via logs:
Primary - nbemm (111), rem(219)
Media - nbrmms (222), dps(220), dsm(178), rdsm (230), nbstserv (226), bpstsinfo (legacy)
Storage Server logs
Examples:
- MSDP - spoold, spad
- DataDomain - ddfs.info
Consult OST vendor documentation for location of the OST logging.