LDOM: Oracle VM Server for SPARC - Veritas DMP is supported with Solaris MPGROUPs on Solaris 10 only

Problem

LDOMs known as Oracle VM Server for Sparc provides a Solaris feature called MPGROUPs. Traditionally MPGROUPs (multipathing group) were not supported/certified with the Veritas product stack in connection with LDOM environments.

Figure 1.0

Oracle released an interoperability enhancement for Solaris 11.3 with SRU 18.0.6 and higher. This interoperability enhancement enabled Veritas Dynamic Multi-pathing (DMP) to better handle the loss of Primary (Control) and Service I/O Domains. Oracle decided not to backport this functionality to Solaris 10 at this time.

To overcome this interoperability limitation, Veritas has agreed to support Veritas DMP in connection with Solaris MPGROUPs for Solaris 10 GUEST deployments only.

The solution provided here is not supported for Solaris 11 LDOM configurations and MPGROUPS must not be used for Solaris 11 GUESTs. This article supercedes any previous Veritas LDOM related content/articles.

Cause
 

With Solaris 10 GUEST deployments the VDC (Virtual Disk Client) is sometimes unable to provide the required responses to the Veritas DMP (multi-pathing) interface. Preventing DMP from acting on SCSI failures returned by the lower layers up into the GUEST.

In the event that the GUEST domain cannot get the failed I/O(s) back, they cannot be routed through an alternate operational I/O domain.
 

VDC Timeout

In Oracle VM Server configurations the Virtual Disk Client (VDC) driver timeout is set to zero by default, which signifies infinity.

The VDC driver timeout must be defined in all cases to further minimize delays with returning a failure to the VDC client.

The VDC timeout can be defined in the /etc/system file in the GUEST domain (system wide globally parameter).

Sample file output

# cat /etc/system
.
.
set vdc:vdc_timeout=30
set vdc:vdc_read_timeout=30

 

Additionally, the VDC timeout can also be set manually at a Virtual disk level (vdisk) using the “ldm” commands shown below:

Sample syntax:

 

# ldm add-vdisk timeout=30 scoobydisk scoobyboot@primary-vds0 scooby

# ldm set-vdisk timeout=20  scoobydisk scooby


NOTE: The ldm commands are executed from the Control (primary) domain.
 

Solution

For Solaris 10 GUEST environments only. Veritas is recommending the deployment of Veritas DMP exported backend devices into a MPGROUP for both boot and data disks, which is then finally made visible into the Solaris 10 GUEST.

The MPGROUP requirement is critical for the presentation of the boot & data disk(s) presented up into the Solaris 10 GUEST domains.

NOTE: When Veritas CVM/CFS is used, it is recommended that MPGROUPs not be used, especially when fencing is also configured in the GUEST domain.

This configuration is to allow Solaris 10 Guest domains a greater chance of surviving the loss of a Control or Service (Alternate) I/O domain.

Sample LDOM commands:

Issued from the Control (Primary) domain.

Boot disk

# ldm add-vdsdev mpgroup=scrappyboot /dev/vx/dmp/emc_clariion0_47s2 scrappyboot@primary-vds0
# ldm add-vdsdev mpgroup=scrappyboot /dev/vx/dmp/emc_clariion0_47s2 scrappyboot@altio-vds0
# ldm add-vdisk timeout=30 scrappydisk scrappyboot@primary-vds0 scrappy


NOTE: Only a single Virtual disk resource is required when using MPGROUPs. The boot & data disks should only be managed by MPGROUPs with Solaris 10 GUEST deployments.

With Solaris 11 the exported disks must NOT be configured in connection with MPGROUPs.

Data Disks


The following commands can be used in connection with Solaris 10 GUEST deployments, where CVM/CFS is not configured:

Exported from Primary VDS (Virtual Disk Server) I/O domain:

# ldm add-vdsdev mpgroup=scrappy177 /dev/vx/dmp/emc_clariion1_177s2 scrappy177@primary-vds0
# ldm add-vdsdev mpgroup=scrappy253 /dev/vx/dmp/emc_clariion1_253s2 scrappy253@primary-vds0
# ldm add-vdsdev mpgroup=scrappy254 /dev/vx/dmp/emc_clariion1_254s2 scrappy254@primary-vds0


Exported from Alternate/Secondary VDS (Virtual Disk Server) I/O domain: 

# ldm add-vdsdev mpgroup=scrappy177 /dev/vx/dmp/emc_clariion1_177s2 scrappy177@altio-vds0
# ldm add-vdsdev mpgroup=scrappy253 /dev/vx/dmp/emc_clariion1_253s2 scrappy253@altio-vds0
# ldm add-vdsdev mpgroup=scrappy254 /dev/vx/dmp/emc_clariion1_254s2 scrappy254@altio-vds0

Virtual disk resources are linked to the respective MPGROUP and made visible to the GUEST domain scrappy:

# ldm add-vdisk timeout=30 scrappy-177 scrappy177@primary-vds0 scrappy
# ldm add-vdisk timeout=30 scrappy-253 scrappy253@primary-vds0 scrappy
# ldm add-vdisk timeout-30 scrappy-254 scrappy254@primary-vds0 scrappy


NOTE: The Virtual disk names need to be unique when not using MPGROUPs. Only a single Virtual disk resource is required when using MPGROUPs.

 

To support the handling of Solaris ZFS devices, the underlying device paths are presented by the corresponding MPGROUPs as a single OS device handle.
 

The DMP tunable dmp_native support must NOT be enabled on Solaris 10 environments to manage the boot disk, unless a single path is visible to the host.


# vxdmpadm gettune dmp_native_support

# vxdmpadm settune dmp_native_support=off

NOTE: A series of Veritas Volume Manager (VxVM) patches were released to ensure DMP imports ZFS devices using DMP.

Please contact Veritas support to ensure you are running the required Veritas Volume Manager (VxVM) patch level.
 

MPGROUPs remains unsupported for Solaris 11 configurations.
MPxIO is not supported with Solaris 10 or 11 LDOM configurations.

Additional Information (Solaris 11):

The solution for Solaris 11 deployments is documented below.

The key thing here is that no MPGROUPs should be configured, and the ZFS devices presented to the Solaris 11 GUEST domain need to have the dmp_native_support tunable enabled for DMP.

Solaris 11 Sparc Enhancement

 

Solaris 11.3 SRU 18.0.6 provides the timeout-noretry-list parameter, which is not currently available with Solaris 10.
 

File location: /platform/sun4v/kernel/drv/vdc.conf

The vdc.conf file needs to be modified in the GUEST domain (i.e. scooby) to reduce the chances of specific commands hanging in the logical domain when access to the Control/Service I/O domain is lost.

Sample vdc.conf update:


scooby # cat /platform/sun4v/kernel/drv/vdc.conf
#
# Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.
#
#
# Associate the driver with devid resolution.
#
ddi-devid-registrant=1;
timeout-noretry-list=0,1,2,3,4,5;
 

NOTE:
The “timeout-noretry-list” setting has been added to the vdc.conf file shown above. The Virtual Disk Client (VDC) for virtual disk instances 0,1,2,3,4 & 5 will NOW return an I/O error immediately (for each I/O), after the defined VDC timeout from the time the I/O domain is not accessible.


Impact:

Even with the VDC timeout defined, the NEW timeout-noretry-list parameter made available with Solaris 11.3 SRU 18.0.6 and higher needs to be defined for all virtual disk instances.

If not defined for Solaris 11 GUEST domains, the failed I/O may not to return to the guest domain in the event that either the Control (primary) or Service I/O domain crashes (reboots) unexpectedly.