ZFS Data corruption is possible when enabling the dmp_native_support DMP tunable on multiple Solaris Sparc servers

Problem
 

ZFS is a type of file system presenting a pooled storage model developed by SUN (Oracle). File systems can directly draw from a common storage pool (zpool).

The Solaris Operating System (OS) can use two different types of disk labels: Sun Microsystems, Inc. (SMI) labels, and Extensible Firmware Interface (EFI) labels. Traditionally, LUNs which are larger than 1TB in size, are configured with an EFI label (instead of SMI labels). ZFS configured devices will default with an EFI disk label regardless of the physical LUN size.

Veritas Volume Manager (VxVM) in connection with Veritas DMP can be used to manage ZFS pools and any ZFS file system(s) that operate on those pools. If enabled, DMP can be used as the multi-pathing driver for the management of ZFS pools.

Veritas VCS uses the Zpool agent and enables the user to specify the faster /dev/vx/dmp/<da-name>/<slice-number> path details when importing the ZFS zpool. Taking microseconds to import rather than minutes in some cases.

Figure 1.0

The above sample Solaris environment consists of two Solaris Sparc (SOL) servers with several ZFS devices visible to both servers.

Following the Veritas InfoScale product installation, it is possible to configure DMP to manage all ZFS pools by enabling the dmp_native_support DMP tunable. When this tunable is turned on, DMP enables support for ZFS on any device that does not have a VxVM label and is not in control of any third party multi-pathing (TPD) software.

In addition, turning on the dmp_native_support DMP tunable migrates any ZFS pools that are not in use onto DMP devices.

Potential Data Corruption

If the dmp_native_support DMP tunable is enabled on multiple servers at the same time and these servers have visibility to the same ZFS zpools, this parallel activity can result in DMP importing the same ZFS zpools on multiple systems at the same time.

Cause

Enabling the dmp_native_support tunable may take a long time to complete depending on the number of LUNs and zpools configured on the system. Do not abort the command.

When enabling the dmp_native_support tunable, all exported ZFS zpools are imported and subsequently exported.

The intention is to stamp the DMP import path in the ZFS header. This ensures the ZFS zpool uses the DMP /dev/vx/dmp/.... path each time the zpool is imported. This avoids the user having to specify additional zpool arguments when importing the ZFS zpool.

As the ZFS import operation of exported zpools can be performed on multiple servers at the same time for the same ZFS zpool, this parallel activity can result in the ZFS zpool corruption.

To verify the value of the dmp_native_support tunable, use the following command:

# vxdmpadm gettune dmp_native_support
Tunable                     Current Value   Default Value
--------------------------  -------------   ---------------
dmp_native_support          on              off

Enabling DMP native support

The "dmp_native_support" tunable enables DMP support for ZFS devices and can be enabled using the following command:
 

# vxdmpadm settune dmp_native_support=on

The first time this operation is performed, the command will report if any zpool is in use.

To migrate the zpool under DMP control, the zpool must be in an exported state. Once "dmp_native_support" is turned on, the ZFS pool will be migrated to DMP.


Once the ZFS zpool has been exported, using the "-d /dev/vx/dmp" path to import subsequent zpools.


# zpool import -d /dev/vx/dmp/ VERITASPOOL
 

The following "for" loop can be used to extract the "path" and "phys_path" details for a ZFS pool:
 

Sample Output:
 

# for pool in `zpool list -H -o name` ; do echo "$pool" ; zdb -C "$pool" | grep path ; done
rpool
                path: '/dev/dsk/emc0_01das0'
                phys_path: '/pseudo/vxdmp@0:n60000970000297802241533030344435-a'

ZFS pools
 

If the ZFS zpools are not in use, turning on DMP native support migrates the pools to DMP devices.

This involves importing and exporting the exported ZFS zpools.
 

If the ZFS zpools are in use, the ZFS zpools must be exported to allow the pools to be migrated under DMP control.
 

Veritas Volume Manager (VxVM) devices
 

Native support is not enabled for any device that has a VxVM label.

To make the device available for ZFS, the VxVM label must be removed using commands like "vxdisksetup".

VxVM devices can coexist with native devices under DMP control.
 

Devices that are multi-pathed with Third-party drivers (TPD)
 

If a disk is already multi-pathed with a third-party driver (TPD), DMP does not manage the devices unless you remove TPD support.

After removing TPD support, turn on the dmp_native_support tunable to migrate the devices.

If ZFS pools are constructed over TPD devices, then perform the steps to migrate the ZFS pools onto DMP devices.

Solution

To limit the possibility of corrupting the ZFS zpools, it is recommended that the dmp_native_support DMP tunable only be enabled on a single system at a time.

Once activated, enable the dmp_native_support tunable on the remaining systems.

The issue is related to the below Solaris (Oracle) incident:
Abstract: SUNBT6710718 Multi-import protection in ZFS
 

VxVM engineering have creating the following solution, VxVM should not import the exported zpools when enabling DMP (Dynamic Multipathing) native support tunable.

The native_input file should be created as follows:

# cat /var/adm/vx/native_input
auto_import_exported_pools=off

By creating the file and turning off the auto import design for exported ZFS zpools will prevent VxVM from causing any corruption.

The following Veritas CPI installer changes were also done:

The CPI patch will create the /var/adm/vx/native_input file,

1.If dmp_native_support is on, it will prompt the user to import the exported zpools and update the native_input file accordingly.

2.If dmp_native_support is off, it will print a message to make customer aware of the current status

See related article for updated functionality:

How to prevent Solaris ZFS zpool corruption by disabling DMP tunable dmp_import_exported_zpools
https://www.veritas.com/support/en_US/article.100047984

The DMP tunable dmp_import_exported_zpools tunable has been written to allow the customer to prevent VxVM from automatically importing ZFS zpools.

# vxdmpadm gettune dmp_import_exported_zpools
            Tunable               Current Value  Default Value
------------------------------    -------------  -------------
dmp_import_exported_zpools              off              off

With VxVM hot-fix 7.4.1.2201, the tunable is set to "off" by default, so VxVM will not attempt to automatically import any ZFS zpool(s) as part of enabling DMP native support.

This would ensure that only the zpools which are imported on the system are migrated to DMP and those which are exported or not.

Sample CPI messages 
 

Veritas InfoScale Foundation Install completed successfully

Tunable dmp_native_support is turned 'on' on the following system(s): fred

Retaining this tunable setting may increase the upgrade time depending on the number of LUNs and native ZFS pools configured on the system.

Would you like to retain the tunable setting? [y,n,q,?] (n) y

Would you like to import the exported zpools when dmp_native_support is enabled? [y,n,q,?] (y) n

 

Fixed in VxVM patches

vm-sol10_sparc-Patch-6.2.1.500
vm-sol11_sparc-Patch-6.2.1.500
vm-sol11_sparc-Patch-7.1.0.4200
infoscale-sol11_sparc-Patch-7.1.0.1100
vm-sol11_sparc-Patch-7.2.0.300
vm-sol11_sparc-Patch-7.3.1.100
infoscale-sol11_sparc-Patch-7.3.1.200

MPxIO is not supported in Solaris LDOM (Oracle VM for SPARC) environments, I/O domains & LDOM guests
MPxIO is not supported when Veritas DMP_NATIVE_SUPPORT is enabled

To confirm if MPxIO is disabled, type:

# stmsboot -L
stmsboot: MPXIO disabled