Windows MSDP Disk pool is down, backups and duplications failing with status 213, no storage units available for use

Problem

Backups, restores and duplications failing with status 213 when accessing a media server deduplication storage server.

Error Message

Status Code: 213 - no storage units available for use

Cause

1. Disk pool is down
2. The NetBackup MSDP media server is so busy that that the disk pool gets marked as down. 
3. The NetBackup services are not running on the media server with MSDP configured (spoold.exe, spad.exe, postgreSQL)
4. The NetBackup deduplication engine (spoold.exe) is running but has stopped responding
5. The NetBackup Deduplication Manager (spad.exe) is running but has stopped responding
6. The database processes (postgres) is not running
7. Queue processing is not able to run due to issues in the <MSDP install path\queue folder (transaction logs, sorted.delayed or other .delayed files)
8. Patch version of the NetBackup binaries on the MSDP media server

Solution

To trouble shoot the above possible causes:

1. Disk pool is down.

Try to put the disk pool up from the GUI.

If there is an error, record the error and search for possible causes of that error.
If the disk pool does not change to an UP state, try bringing up the disk pool using command line.

2. The MSDP media server is so busy that the disk pool gets marked as down. 

Check to see if the status 213 is happening only at certain  high volume times.  If yes, create these files on the MSDP media server using the steps in  100005739 (Status Code 84, 213, 800: Disk Storage Volume is intermittently being marked down.) to adjust the DPS polling interval time.

DPS_PROXYDEFAULTSENDTMO (value of 1800 inside)
DPS_PROXYDEFAULTRECVTMO (value of 1800 inside)

Warning: if the issue persists after one daily or nightly backup schedule window after employing the above configuration changes, please remove the touchfiles and troubleshoot the issue further via logs (nbrmms, dps, spoold, spad) to determine root cause.

3. Check the services on the MSDP media server. 

Verify that these are running.

If they are not running, start them.  If they do not start, and there is an error, record the error and search to determine what is the usual cause for the error.

Additional information about why a services is not starting is located in these log folders on the MSDP server.

4. The NetBackup deduplication engine (spoold.exe) is running but has stopped responding.

Check the <MSDP install path\log\spoold\storaged.log to see if the queue processing stopped on a particular transaction log.  Look for messages like this:

Check to see if the last line in the storaged.log file contains a string similar to this.

sorted-123456-789101.delayed

If yes, check the <msdp install path>\queue folder to see what is the date and time associated with that file, if it exists.

Stop and restart the NetBackup services to see if that allows the MSDP pool to come back to an UP status.

5. The NetBackup Deduplication Manager (spad.exe) is running but has stopped responding.

• Check the <MSDP install path\log\spad\spad.log file for errors.
• Run this command to see what spad.exe may be doing.

Stop and restart the NetBackup services to see if that allows the MSDP pool to come back to an UP status.

6. The database processes (postgres) is not running

Try to restart the postgresql service.  If there is an error, and it won't start:

• Check to make sure the user running postgresSQL has permissions to the MSDP install folder.
• Disable any any internal firewall that may be running.
• Look at Windows Event viewer, Application Log for indications of why it won't start.

It is possible the user running the postgresql service does not have permission to access <msdp install path>\databases\pddb\data folder.

• Amend the security policy to allow the purediskdbuser account to have the "log on as a service" right. 
• Change the postgresql-8.3 service manually to a new domain account or to any other account that can run the services. 
• Change the postgresql-8.3 service to run as Local System.

Note: If the 2nd or 3rd solution is being implemented, then the account from step 2 and the SYSTEM account both need to be granted full NTFS permissions to the <msdp install path>\databases\pddb\data folder.

7. Queue processing is not able to run due to issues in the <MSDP install path\queue folder (transaction logs, sorted.delayed or other .delayed files)

• Review these log files from the MSDP media server.

<MSDP install path\log\spoold\spoold.log
<MSDP install path\log\spoold\storaged.log

• Check to see how many files exist in the <MSDP install path\queue folder, what is the time stamp on the oldest and newest file in that folder.  If queue processing is run, does that information change?

8. Patch version of the NetBackup binaries on the MSDP media server.

Helpful log files to enable on the NetBackup media server with MSDP configured

Enable logging level 5 using NetBackup GUI for bptm on the MSDP server and create this folder.

Windows Event viewer, Application Log and System log. Look for errors or warnings reported for spood, spad, service control manager, or any other NetBackup service reporting an error.

Edit the <MSDP install path>\etc\spa.cfg file so that the line Logging=long,thread reads:
Logging=full,thread

Edit the <NBU install path>\NetBackup\bin\ost-plugins\pd.conf file so that these two lines are uncommented:

DEBUGLOG = <some_dir_path_with_space> pdplugin.log
LOGLEVEL = 10

When the error happens again, gather these and review.

<NBU install path>\NetBackup\logs\bptm\log.mmddyy

<MSDP install path\log\spad\spad.log
<MSDP install path\log\spoold\spoold.log
<MSDP install path\log\spoold\storaged.log

pdplugin.log (location listed in pd.conf file)

Event Viewer, Application log and System Log.

Notes:-

The same investigations below can be used on a UNIX MSDP storage server.  This article is specific to windows syntax.  For more UNIX syntax, see article 100024420 (UNIX or Linux MSDP Disk pool is down, backups and duplications failing with status 213, no storage units available for use.)