All backups on tapes are failing with status 2001

Problem

A tape library that is utilized by multiple media servers is experiencing a problem. All backup operations that are scheduled on tape storage units are failing, displaying an error status of 2001, indicating that no drives are available.

Attempts to resolve the issue, such as rebooting and clearing allocations on the Primary/Master Server, have proven unsuccessful.

Error Message

Job Details show:

7 Jul, 2021 4:50:55 PM - Error nbjm (pid=5883) NBU status: 2001, EMM status: No drives are available
No drives are available for this job  (2001)

MDS log shows:

08/30/2021 12:59:33.204 [sql_select_drive] cdot_affinity_list = NULL
08/30/2021 12:59:33.204 [sql_select_drive] skip_drv_list = NULL
08/30/2021 12:59:33.204 [sql_select_drive] alloc_drv_list is empty
08/30/2021 12:59:33.204 [sql_select_drive] select_up_path = 0
08/30/2021 12:59:33.210 [sql_select_drive] drv_list has 0 drives
08/30/2021 12:59:33.210 [select_from_one_stu] Returning error = 2005001
08/30/2021 12:59:33.210 [StuTraverser::next] No stu/dv found
08/30/2021 12:59:33.210 [select_from_stu_list] Returning error = 2005001
08/30/2021 12:59:33.210 [END_MDS_DB_TRANSACTION] cur_err = 2005001, m_dbconn_stat = 0, m_dberr_stat = 0, m_closed_db_trx = 0
08/30/2021 12:59:33.211 [END_MDS_DB_TRANSACTION] rolled back database transaction
08/30/2021 12:59:33.211 [allocateTwin] EXIT INFO:
08/30/2021 12:59:33.211 [allocateTwin] skip_stu_list is empty
08/30/2021 12:59:33.211 [allocateTwin] disk_alloc_key_list is empty
08/30/2021 12:59:33.211 [allocateTwin] cdot_affinity_list is empty
08/30/2021 12:59:33.211 [allocateTwin] unload_requested_drive_cache is empty
08/30/2021 12:59:33.211 [allocateTwin] EXIT STATUS = 2005001 (EMM_ERROR_MDS_DriveUnavailable, No drives are available)

The tpconfig command shows the following:

tpconfig -emm_dev_list

Drive:                          IBMULT000-TD0000000
Index:                          1
Drive Type:                     hcart(3)
Media Server:                   mediaserver.com
Status:                         2
Saved Mode Bits:                0
Access Mode:                    82
Robot Type:                     TLD(8)
Robot Number:                   1
VH Saved Mode Bits:             0
Flags:                          1073741952  <——
PathFlags:                      0
DeviceFlags:                    0
Time Mounted:                   0
Total Time Mounted:             221
SN:                             000ABC04AB
Inquiry Info:                   IBM   ULT0000-TD0   M570
World Wide Id:                  -
NDMP Host:                      -
Scan Host:                      mediaserver.com
Assign Host:                    -

Cause

The Flags field for drives in the EMM database is displaying the value “1073741952”, which is 0x40000080 in hexadecimal. This value also appears in the MDS logs. This occurrence could potentially obstruct the allocation of drives for new jobs.

Solution

Execute the following command on all media servers that share the problematic tape library. This will remove the Flags value in the EMM database.

tpconfig -emm_dev_list  | grep Media | sort -u | awk '{print $3}' | while read ms; do  vmoprcmd -reset_ltid_restarted_bit -h $ms;done

Alternatively, if the above solution does not help, restart Device Manager (ltid) services on all media servers sharing the tape library, per Related Article (100050602) below.