UNIX or Linux MSDP Disk pool is down, backups and duplications failing with status 213, no storage units available for use

Problem

Backups, restores and duplications failing with status 213 when accessing a media server deduplication storage server.

Error Message

Status Code: 213 - no storage units available for use

Cause

1. Disk pool is down
2. The NetBackup MSDP media server is so busy that that the disk pool gets marked as down. 
3. The NetBackup services are not running on the media server with MSDP configured (spoold spad)
4. Queue processing is not able to run due to issues in the <MSDP install path\queue folder (transaction logs, sorted.delayed or other .delayed files)
5. Patch version of the NetBackup binaries on the MSDP media server

Solution

To trouble shoot the above possible causes:

1. Disk pool is down.

Try to put the disk pool up from the GUI.

If there is an error, record the error and search for possible causes of that error.
If the disk pool does not change to an UP state, try bringing up the disk pool using command line.

/usr/openv/NetBackup/bin/admincmd/nbdevconfig -changestate -stype PureDisk -dp <disk pool name> -state UP

2. Check to see if the status 213 is happening only at certain  high volume times.  If yes, create these files on the MSDP media server using the steps in 000012819 to adjust the DPS polling interval time.

DPS_PROXYDEFAULTSENDTMO (value of 1800 inside)
DPS_PROXYDEFAULTRECVTMO (value of 1800 inside)

Warning: if the issue persists after one daily or nightly backup schedule window after employing the above configuration changes, please remove the touchfiles and troubleshoot the issue further via logs (nbrmms, dps, spoold, spad) to determine root cause.

3. Check the services on the MSDP media server.  Verify that these are running.

spoold
spad

If they are not running, start them.  If they do not start, and there is an error, record the error and search to determine what is the usual cause for the error.

Additional information about why a services is not starting is located in these log folders on the MSDP server.

/MSDP install path/log/spoold/spoold.log
/MSDP install path/log/spad/spad.log
/MSDP install path/log/spoold/storaged.log

4. The NetBackup deduplication engine (spoold.exe) is running but has stopped responding.

Check the /MSDP install path/log/spoold/storaged.log to see if the queue processing stopped on a particular transaction log.  Look for messages like this:

WARNING [xxxx]: 25000: Transaction log /MSDP install path/queue/sorted-xxxxxxx-yyyyyyy.tlog failed: Could not process tlog entries: zero-sized object
Transaction will be retried.
ERR [xxxx]: 25004: Queue processing failed five times in a row. Queue processing will be disabled and the CR will no longer accept new backup data. Content router has been totally shut down.

Check to see if the last line in the storaged.log file contains a string similar to this.

sorted-123456-789101.delayed

If yes, check the <msdp install path>/queue folder to see what is the date and time associated with that file, if it exists.

5. Stop and restart the NetBackup services to see if that allows the MSDP pool to come back to an UP status.

6. Disable any any internal firewall that may be running.

• Look at Windows Event viewer, Application Log for indications of why it won't start.

It is possible the user running the postgresql service does not have permission to access /msdp install path/databases/pddb/data folder.

7. Queue processing is not able to run due to issues in the <MSDP install path\queue folder (transaction logs, sorted.delayed or other .delayed files)

• Review these log files from the MSDP media server.

/MSDP install path/log/spoold/spoold.log
/MSDP install path/log/spoold/storaged.log

• Check to see how many files exist in the /MSDP install path/queue folder, what is the time stamp on the oldest and newest file in that folder.  If queue processing is run, does that information change?

8. Patch version of the NetBackup binaries on the MSDP media server.

Helpful log files to enable on the NetBackup media server with MSDP configured

/MSDP install path/log/spad/spad.log
/MSDP install path/log/spoold/spoold.log
/MSDP install path/log/spoold/storaged.log

Enable logging level 5 using NetBackup GUI for bptm on the MSDP server and create this folder.

/usr/openv/netbackup\logs\bptm

Windows Event viewer, Application Log and System log. Look for errors or warnings reported for spood, spad, service control manager, or any other NetBackup service reporting an error.

Edit the /MSDP install path/etc/spa.cfg file so that the line Logging=long,thread reads:
Logging=full,thread

Edit the /usr/openv/lib/ost-plugins/pd.conf file so that these 2 lines are uncommented:

DEBUGLOG = <some_dir_path_with_space> pdplugin.log
LOGLEVEL = 10

When the error happens again, gather these and review.

/MSDP install path/log/spad/spad.log
/MSDP install path/log/spoold/spoold.log
/MSDP install path/log/spoold/storaged.log

<install path>\NetBackup\logs\bptm

pdplugin.log (location listed in bp.conf file)

/var/adm/messages/

/var/log/messages

/var/log/syslog

Note: After completing troubleshooting please turn down the logging level by reverting the changes.