Troubleshooting Failing Disks, Missing Disks and the "failed was" status

Problem

This article contains a procedure for troubleshooting VxVM managed disks reporting a "failed" or "failed was" state, as displayed by the VxVM vxdisk command.

Error Message

 

In the below example, VxVM disk named "disk4" is reported as "failed was":
 

# vxdisk -o alldgs list
DEVICE       TYPE            DISK         GROUP        STATUS
disk_0       auto:cdsdisk    -            (vxfendg)    online
disk_1       auto:cdsdisk    -            (vxfendg)    online
disk_2       auto:cdsdisk    -            (vxfendg)    online
disk_3       auto:cdsdisk    datadg01     datadg       online
disk_4       auto            -            -            error
disk_5       auto:cdsdisk    datadg03     datadg       online
disk_6       auto:cdsdisk    datadg04     datadg       online
disk_7       auto:cdsdisk    -            (sambadg)    online
disk_8       auto:cdsdisk    -            -            online
disk_9       auto:cdsdisk    -            -            online
sda          auto:none       -            -            online invalid
-            -         datadg02     datadg       failed was:disk_4

Solution

Table of Contents

1. Introduction
2. "Failed" versus "failing" disks
3. Making an emergency backup of the disk group configuration
4. Has the disk been excluded in vxvm.exclude?
5. Has the disk been overwritten by another logical volume manager solution?
6. Determining if a disk can be reattached
7. Verifying that a disk is readable to the operating system
8. Have the paths to the disk been disabled?
9. Restoring the disk group configuration using vxconfigrestore
10. Restoring the disk group configuration manually, using UDIDs and Disk IDs
11. Restarting the volume

 

1. Introduction

(Back to top)
 

VxVM managed disks may report a "failed" or "failed was" state, as displayed by the vxdisk command.

A "failed" status is a record of a disk that is no longer accessible.

This is often caused by sustained I/O errors to the disk that prevents it from being read by the operating system (OS). It may also be the result of corruption within the Veritas private region.

The private region is the portion of the disk where Veritas stores records about the disk group, such as disks, volumes, subdisks and plexes. This can be contrasted with the public region, which contains the actual volumes, including user data.

2. "Failed" versus "failing" disks

(Back to top)


The status of "failed" should not be confused with a status of "failing" This article primarily discusses the "failed" status, as reported by vxdisk.

For information on troubleshooting the "failing" status, see https://www.veritas.com/support/en_US/article.100018844

3. Making an emergency backup of the disk group configuration

(Back to top)

Before making any changes to the disks or the disk group configuration, create a backup of the disk group configuration using the vxconfigbackup command.

Vxconfigbackup does not back up any actual data that is contained within the volumes. Instead, it backs up the Veritas private region configuration database that resides on the disks associated with the diskgroup, along with some information about the disks themselves. The configuration database stores information about which disks are contained by the disk group, volume structures, plexes and subdisks.

If vxconfigbackup is not available, vxprivutil can be used to dump a copy of the configuration database.
 

More details about vxconfigbackup and vxprivutil, including syntax and examples, can be found in this article:

"Using vxconfigbackup and vxprivutil to back up the disk group configuration of the Veritas private region"
https://www.veritas.com/docs/000087431

4. Has the disk been excluded by vxvm.exclude?

(Back to top)

The /etc/vx/vxvm.exclude file maintains a list of paths, controllers and products that are excluded from Veritas Volume Manager's (VxVM's) view.

Check if the disk, or its associated paths or controllers are listed in this file.

If the value of "exclude_all" is 1, all devices will be excluded.

Figure 1 - Default contents of /etc/vx/vxvm.exclude

5. Has the disk been overwritten by another logical volume manager solution?

(Back to top)


If vxdisk shows that the disk type includes the words "LVM" or "ZFS," then the disk may have been overwritten by another logical volume manager (LVM) solution. It is also possible that there is a problem with the SAN zoning which may have caused disks to be presented to the wrong systems. Before making any further changes, ensure that the disk is not supposed to be zoned to another system.

Please be aware that if a disk has been overwritten by another logical volume manager, there is a chance that the public region of the disk might have been affected. If the public region was modified, then it is usually impossible to restore the data for that filesystem.

Once the disk is removed from the LVM solution, it will need to be initialized for Veritas use, using the vxdisksetup command. Care should be taken when initializing the disk as default private region sizes and public region starting offsets can vary per VxVM version. Please consult Veritas Technical Support for guidance. Refer to the documentation for the appropriate vendor for information about removing a disk from the control of their LVM solution.

6. Determining if a disk can be reattached

(Back to top)
 
The vxreattach command is used to reattach the original disk media name with the specified disk group. It can normally only be used if the status of the VxVM disk is shown in an "online" state by vxdisk list (see Figure 2).

Run vxreattach, using the "-c" argument, to determine if a disk can be reattached to the disk group.


Figure 2 - Using vxreattach, with the "-c" argument, to check if a reattach is possible

If vxreattach -c returns a disk group and disk media name, without returning any errors, proceed with reattaching the disk (Figure 3). If a reattach is not possible, a V-5-2-238 error will appear.


Figure 3 - Using vxreattach to reattach a disk to the disk group

7. Verifying that a disk is readable to the operating system

(Back to top)


Use native OS commands to confirm that the OS can read the disk, including the disk label.

• Use commands, such as prtvtoc, fdisk, lspv or diskinfo to read the disk label.
• Use dd to read a block from the disk.

Veritas depends on the OS device drivers to communicate with disks. If the OS is unable to read a disk, Veritas will also fail to read it. If a disk does not have a label, or the label has been corrupted, Veritas will not recognize the disk. Completing these steps will assist with identifying the source of a disk outage.
 

8. Have the paths to the disk been disabled?

(Back to top)

Use vxdmpadm to determine the status of the paths to the disks (Figure 4).

Veritas will disable a path if serious or sustained I/O errors occur. When all paths to a disk are disabled, the server will be unable to read or write to the volume. If a path has been disabled, review the syslog for events that are reported by "vxdmp," or "scsi" for I/O errors.

Although a path can be re-enabled using "vxdmpadm enable," vxdmp should automatically evaluate the status of a path in five minute intervals using a scsi inquiry. If the query is successful, the path is automatically re-enabled. If a path remains disabled beyond this interval, it is possible that I/O errors are still being detected, warranting further investigation. Paths are not automatically re-enabled If the diskgroup has been disabled, or if vxesd is stopped. The behavior of vxdmp in response to disabled paths can be modified via the DMP tunables, which can be viewed using "vxmpadm gettune."
 

Figure 4 - Example of a disabled path, as reported by vxdmpadm

To enable a disabled path, run:

# vxdmpadm enable path=<device>

In this instance, sdi and sdl are shown as disabled

# vxdmpadm enable path=sdi,sdl

9. Restoring the disk group configuration using vxconfigrestore

(Back to top)

If a vxreattach is not possible, you need to use vxconfigrestore to recover the disk group configuration, consult with Veritas Technical Support before doing so.

vxconfigrestore does not restore the actual data that is contained within the volumes. It only restores the Veritas disk group configuration database that is located within the private region of the disks. The configuration database stores information about which disks are contained by the disk group, volume structures, plexes and subdisks.
 

10. Restoring the disk group configuration manually, using UDIDs and Disk IDs

(Back to top)

If using vxconfigrestore is not possible, another method for recovering the disks is to compare the UDID or Disk ID attributes of the disks with the records that are contained with the private region configuration database.
 

11. Restarting the volume

(Back to top)


Once the original failed disk has been added back to its disk group, additional steps may be needed to recover the volume. Use the vxprint command to determine the current status of all volumes in a disk group (Figure 5).

• For mirrored volumes:
  • If at least one plex was not affected by the outage, the other plexes should be resynchronized when they are reattached to the volume.  It may be necessary to use vxrecover to initiate this process (Figure 6).
  • If all plexes were affected by the outage, it may be necessary to manually review each plex to determine which contain the most recent updates.

Warning: Do not simply force-start a mirrored volume. This may cause a plex that contains old, or corrupt, blocks to overwrite a plex that contains up-to-date data. A procedure for manually determining the most up-to-date mirror plex can be found in this article:

"Manually determining which mirror plex contains the most recent data and then resynchronizing"
https://www.veritas.com/docs/000087709

• For non-mirrored volumes:
  • It may be necessary to manually force restart the volume using vxvol after adding the disk back to the disk group (Figure 5).

Figure 5 - Using vxprint to determine the status of a volume

Figure 6 - Using vxrecover to start volume recovery and resynchronization of a volume

Figure 7 - For volumes not started by the vxrecover operation, use the vxvol command instead. Volumes can be be manually started using vxvol to start a volume and using vxprint to review any changes in the status of the volume
 

Is it possible to start all volumes in a disk group by typing:

# vxvol -g datadg startall

keywords: failed, failing, failed disk, failed disks, failing disk, failed disks,