How to prevent Solaris ZFS zpool corruption by disabling DMP tunable dmp_import_exported_zpools

Problem

Potential Data Corruption
 

If the dmp_native_support DMP tunable is enabled on multiple servers at the same time and these servers have visibility to the same ZFS zpools, this parallel activity can result in DMP importing the same ZFS zpools on multiple systems at the same time.

This can potential lead to ZFS zpool corruption.
 

ZFS is a type of file system presenting a pooled storage model developed by SUN (Oracle). File systems can directly draw from a common storage pool (zpool).

The Solaris Operating System (OS) can use two different types of disk labels: Sun Microsystems, Inc. (SMI) labels, and Extensible Firmware Interface (EFI) labels. Traditionally, LUNs which are larger than 1TB in size, are configured with an EFI label (instead of SMI labels). ZFS configured devices will default with an EFI disk label regardless of the physical LUN size.
 

Veritas Volume Manager (VxVM) in connection with Veritas DMP can be used to manage ZFS pools and any ZFS file system(s) that operate on those pools. If enabled, DMP can be used as the multi-pathing driver for the management of ZFS pools.

Veritas VCS uses the Zpool agent and enables the user to specify the faster /dev/vx/dmp/<da-name>/<slice-number> path details when importing the ZFS zpool. Taking microseconds to import rather than minutes in some cases.
 

Figure 1.0

The above sample Solaris environment consists of two Solaris Sparc (SOL) servers with several ZFS devices visible to both servers.

Error Message

Cause

To limit the possibility of corrupting ZFS zpools, it is recommended that the dmp_native_support DMP tunable only be enabled on a single system at a time.

Once activated on a single host, then enable the dmp_native_support tunable on the remaining systems, each in turn.

The issue is related to the below Solaris (Oracle) incident:
Abstract: SUNBT6710718 Multi-import protection in ZFS

As this is not practical in all situations, VxVM engineering created a new DMP tunable dmp_import_exported_zpools.to control the automatic import of ZFS zpools.

When the tunable is turned off, VxVM will stop automatically importing zpools when enabling DMP (Dynamic Multipathing) native support tunable.

VxVM Private Hot-fix 7.4.1.2201 for Solaris 11 Sparc ensures the DMP tunable dmp_import_exported_zpools.is turned off by default.

INCIDENT NO:4006947  
TRACKING ID:4006539
 

SYMPTOM: zpool corruption occurs when DMP Native Support is enabled on two servers at the same time.
 

DESCRIPTION: When DMP Native Support is enabled then all the zpools present on the system are migrated to DMP devices. During migration even exported zpools are imported on top of DMP devices. If the same operation is carried out on 2 servers having shared storage then zpools would be imported on top of DMP devices on both the servers approximately at the same time which leads to zpool corruption.
 

RESOLUTION: Changes have been done to not import exported zpools on top of DMP devices and let the customer decide which zpools to import on which server.

Solution

The original design of VxVM was to by default import all ZFS zpools even if those zpools are exported on the system.

Example:

Imagine there are 2 systems which are sharing the same ZFS zpool. The customer use-case is some zpools would be imported on node1, whereas other zpools would be imported on node2.

When the DMP Native support is enabled on a system, VxVM would by default import all ZFS zpools regardless of where they were not intended to be imported and potentially already imported on another host.

Hence we have the same ZFS zpool(s) imported on 2 or more systems at the same time which leads to ZFS zpool corruption.

The DMP tunable dmp_import_exported_zpools.tunable has been written to allow the customer to prevent VxVM from automatically importing ZFS zpools.


# vxdmpadm gettune dmp_import_exported_zpools
            Tunable               Current Value  Default Value
------------------------------    -------------  -------------
dmp_import_exported_zpools              off              off


With VxVM hot-fix 7.4.1.2201, the tunable is set to "off" by default, so VxVM will not attempt to automatically import any ZFS zpool(s) as part of enabling DMP native support.

This would ensure that only the zpools which are imported on the system are migrated to DMP and those which are exported or not.

If the customer wants to import the exported zpools using DMP, then they would have to do so using the "zpool import -d /dev/vx/dmp <zpoolname> command.

This would help to reduce the chances of ZFS zpool corruption when enabling DMP native support.


See related articles:

Veritas Cluster Server (VCS) Zpool agent includes the enhanced functionality to specify the /dev/vx/dmp/ path details with the DeviceDir attribute
https://www.veritas.com/support/en_US/article.100033999

How to specify multiple DMPNODE names with the VCS Zpool DeviceDir Attribute for Solaris ZFS zpools
https://www.veritas.com/support/en_US/article.100052831