Veritas SCSI-3 Fencing disks are not supported with Solaris 10 LDOM Guest with or without Solaris MPGROUPs

Problem

Solaris MPGROUPs must be used to present LDOM Virtual Disks in connection with Solaris 10 LDOM Guests.

Veritas does NOT support SCSI-3 Fencing disks with or without the use of MPGROUPs in LDOM Solaris 10 Guest environments.

NOTE: Veritas SCSI-3 Fencing disks have never been supported with MPGROUPs.

Clustered Volume Manager (CVM) is also impacted and cannot use traditional SCSI-3 Fencing disks for Solaris 10 Guests.

MPGROUPs is the only LDOM presentation vehicle (interface) to manage and handle Virtual disks presented to Solaris 10 LDOM Guests.

The use of MPGROUPs is NOT supported with Solaris 11.3 onwards and the Veritas Product Suite.
 

Error Message

Impact:

Commands will hang in the Solaris 10 LDOM Guest when the disk communication (access) is lost to the I/O domain. This can be a result of one or more I/O domains being rebooted or has encountered a panic.

Solaris 11.3 onwards provides additional functionality to enable Veritas Dynamic Multi-pathing (vxdmp) in LDOM environments without the use of MPGROUPs (which is NOT supported with Solaris 11 onwards).

Users will see commands hanging in the LDOM Solaris 10 Guests:

Whenever, there are disk mappings from an IO Domain to a Solaris 10 LDOM Guest via MPGROUPs, Veritas observed command hangs. This is seen with and without SCSI-3 Fencing disks.

VxVM commands which access disks (vxdisk scandisks, vxdisk list, etc) and OS native commands like “echo | format” will hang when the I/O domain is rebooted or it panics. This behaviour is common with both layered DMP device and OS native device mapping configurations.

Veritas has raised the below SR with Oracle to troubleshoot & investigate:

SR 3-28492730891 : OS native disk commands hang inside Sol 10 LDOM on IO domain failure in Native OS device + mpgroup
 

Cause

Due to the missing vdc.conf functionality limited to Solaris 11.3 onwards. Solaris 10 LDOM Guests is missing this key Solaris LDOM functionality, so Veritas is forced to use MPGROUPs to avoid hangs within the Solaris 10 LDOM guests

Solaris 11.3+ LDOM Guests are more than stable surrounding the reduction of hangs etc whilst I/O domains are panic’ing or being rebooted. 

Solaris 11 Sparc Enhancement

Solaris 11.3 SRU 18.0.6 provides the timeout-noretry-list parameter, which is not currently available with Solaris 10.

File location: /platform/sun4v/kernel/drv/vdc.conf

The vdc.conf file needs to be modified in the GUEST domain, to reduce the chances of specific commands hanging in the logical domain when access to the Control/Service I/O domain is lost.
 
Sample vdc.conf update:# cat /platform/sun4v/kernel/drv/vdc.conf
#
# Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
#
#
# Associate the driver with devid resolution.
#
ddi-devid-registrant=1;
timeout-noretry-list=0,1,2,3,4,5;
 
NOTE:

The “timeout-noretry-list” setting has been added to the vdc.conf file shown above.

The Virtual Disk IDS can be extracted from the below LDOM command (for LDOM Guest "scooby"):
 

Sample output (executed from Primary/Control I/O Domain)

# ldm list -o disk scooby
NAME
scooby
  
DISK
    NAME         VOLUME                                  TOUT   ID   DEVICE  SERVER         MPGROUP
    scoobydisk-pri scoobyboot@primary-vds0  25         0    disk@0    primary
    scoobydisk-alt scoobyboot@altio-vds0        25        1    disk@1     altio
    scoobycds-pri scoobycds@primary-vds0    25         2    disk@2    primary
    scoobycds-alt scoobycds@altio-vds0          25        3    disk@3     altio
    scoobysli-pri scoobysli@primary-vds0         25        4    disk@4     primary
    scoobysli-alt scoobysli@altio-vds0               25        5    disk@5     altio


NOTE: The Virtual Disk IDs 0 to 5 in this instance, will be populated in the /platform/sun4v/kernel/drv/vdc.conf file in relation to the "timeout-noretry-list=0,1,2,3,4,5;" attribute.

The Virtual Disk Client (VDC) for virtual disk instances 0,1,2,3,4 & 5 will NOW return an I/O error immediately (for each I/O), after the defined VDC timeout from the time the I/O domain is not accessible.

Veritas QA identified a series of product roadblocks preventing Veritas Fencing working correctly with and without SCSI-3 Fencing disks in Solaris 10 LDOM environments.

Veritas development will continue to work with Oracle Solaris Support to see what can be achieved for Solaris 10 LDOM Fencing Support in the future.

 

Solution

To use Veritas Fencing in Solaris 10 LDOMs environments, we recommend the use of Veritas CPS configurations to avoid the use of traditional SCSI-3 based Fencing disks.
 

The only solution we currently have for Solaris 10 LDOM Fencing support, is the deployment and use of Veritas CPS servers.
 

NOTE: Veritas SCSI-3 Fencing disks have never been supported with MPGROUPs and does not work correctly.