Veritas Cluster Server (VCS) EMC SRDF agent (5.0.20.1) now detects replication mode conflict

Problem

The Veritas Cluster Server (VCS) EMC SRDF agent has now been enhanced to detect SRDF replication mode conflicts.

The enhancement will check the SRDF H/W replication mode for the EMC DeviceGroup and compare against the VCS SRDF resource replication mode definition entry in the VCS configuration file: /etc/VRTSvcs/conf/config/main.cf.

If the two values differ, an user friendly messages will be recorded in the /var/VRTSvcs/log/engine_A.log file.
 

Example:


EMC SRDF H/W replication mode:
 

# symdg show PlutoDg | grep "RDF Mode"
        RDF Mode                               : Synchronous

 

VCS Configuration (main.cf) entry:

        SRDF SRDF-PlutoDg (
                SymHome = "/opt/emc/SYMCLI"
                GrpName = PlutoDg
                Mode = Asynchronous
                )


In this instance, the two values do not match. The H/W replication mode is set to "Synchronous" and the VCS replication mode is defined as "Asynchronous".
Prior to the VCS EMC SRDF agent enhancement, the agent was unable to detect the replication mode conflict.

Error Message

VCS EMC SRDF agent error message enhancement:
 

2014/10/22 09:22:42 VCS ERROR V-16-20017-89 (pluto) SRDF:SRDF-PlutoDg:online:SRDF replication mode conflict detected. Mode of replication configured is Asynchronous, while the device group is in Synchronous mode. Cannot proceed with online of SRDF-PlutoDg resource. Administrative intervention is required.
 

Cause

This is a product enhancement to the VCS EMC SRDF agent to minimize human error and make troubleshooting easier.

Solution

1.] Offline the VCS EMC SRDF managed Service Groups.
 

# hagrp -offline <ServiceGroup> -sys <system-name>
 

2.] Download and install version 5.0.20.1 or higher of the VCS SRDF agent
 

# installp -ac -d VRTSvcse.rte.bff VRTSvcse.rte
 

3.] Ensure the VCS SRDF agent fileset has been successfully installed on all servers.
 

# lslpp -l VRTSvcse.rte
  Fileset                      Level  State      Description
  ----------------------------------------------------------------------------
Path: /usr/lib/objrepos
  VRTSvcse.rte              5.0.20.1  COMMITTED  VERITAS Clustering Support for
                                                 EMC SRDF(tm)
                                                 Veritas-5.1.0.0-10/21/14-02:51
                                                 :05 Build Env:AIX ogma 3 5
                                                 0001D4CAD300 IBM,9115-505

Path: /etc/objrepos
  VRTSvcse.rte              5.0.20.1  COMMITTED  VERITAS Clustering Support for
                                                 EMC SRDF(tm)
                                                 Veritas-5.1.0.0-10/21/14-02:51
                                                 :05 Build Env:AIX ogma 3 5
                                                 0001D4CAD300 IBM,9115-505
 
4.] Online the VCS EMC SRDF managed Service Groups.


If an EMC SRDF replication mode conflict is present, VCS will record the enhanced user friendly messages in the VCS engine_A.log file.

Sample VCS messages
 

2014/10/22 09:22:34 VCS NOTICE V-16-1-10301 Initiating Online of Resource SRDF-PlutoDg (Owner: Unspecified, Group: srdfgrp) on System pluto
2014/10/22 09:22:39 VCS NOTICE V-16-20017-24 (pluto) SRDF:SRDF-PlutoDg:online:Group PlutoDg is enabled for role swap on failover
2014/10/22 09:22:42 VCS ERROR V-16-20017-89 (pluto) SRDF:SRDF-PlutoDg:online:SRDF replication mode conflict detected. Mode of replication configured is Asynchronous, while the device group is in Synchronous mode. Cannot proceed with online of SRDF-PlutoDg resource. Administrative intervention is required.

2014/10/22 09:32:43 VCS ERROR V-16-2-13066 (pluto) Agent is calling clean for resource(SRDF-PlutoDg) because the resource is not up even after online completed.

Note: The correct replication mode should be defined as "Synchronous" in the VCS configuration file (main.cf).


5.] Force stop the VCS configuration and edit the main.cf file.


# hastop -all -force

# vi /etc/VRTSvcs/conf/config/main.cf

<snippet>

        SRDF SRDF-PlutoDg (
                SymHome = "/opt/emc/SYMCLI"
                GrpName = PlutoDg
                Mode = Synchronous                     <<<< changed from Asynchronous
                )

<snippett>
 

6.] Validate the main.cf changes.

# hacf -verify /etc/VRTSvcs/conf/config
 

7.] Restart VCS across of the servers.

# hastart
 

8.] Once ready, attempt to online the VCS EMC SRDF managed Service Group on the required server(s).

# hagrp -online srdfgrp -sys <system-name>


Sample VCS messages

 

VCS NOTICE V-16-1-50735 Attempting to online group on system pluto
2014/10/22 11:41:32 VCS INFO V-16-1-10493 Evaluating pluto as potential target node for group srdfgrp
2014/10/22 11:41:32 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group srdfgrp on all nodes
2014/10/22 11:41:32 VCS NOTICE V-16-1-10301 Initiating Online of Resource SRDF-PlutoDg (Owner: Unspecified, Group: srdfgrp) on System pluto
2014/10/22 11:41:40 VCS NOTICE V-16-20017-4 (pluto) SRDF:SRDF-PlutoDg:online:devices in group PlutoDg are all read/write enabled; no action is required
2014/10/22 11:41:42 VCS INFO V-16-1-10298 Resource SRDF-PlutoDg (Owner: Unspecified, Group: srdfgrp) is online on pluto (VCS initiated)
2014/10/22 11:41:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource plutodg (Owner: Unspecified, Group: srdfgrp) on System pluto
2014/10/22 11:41:42 VCS WARNING V-16-10011-715 (pluto) DiskGroup:plutodg:online:Diskgroups will be imported without reservations
2014/10/22 11:41:43 VCS INFO V-16-10011-728 (pluto) DiskGroup:plutodg:online:ClearClone attribute is enabled for disk group [plutodg]. DiskGroup will be imported with '-c' option.
2014/10/22 11:41:45 VCS NOTICE V-16-10011-710 (pluto) DiskGroup:plutodg:online:vxdg import succeeded on Disk Group plutodg
2014/10/22 11:41:45 VCS NOTICE V-16-10011-724 (pluto) DiskGroup:plutodg:online:Volumes in Disk Group plutodg are started automatically as part of import command,the system level autostartvolumes flag is set to on
2014/10/22 11:41:47 VCS INFO V-16-1-10298 Resource plutodg (Owner: Unspecified, Group: srdfgrp) is online on pluto (VCS initiated)
2014/10/22 11:41:47 VCS NOTICE V-16-1-10301 Initiating Online of Resource srdfmnt (Owner: Unspecified, Group: srdfgrp) on System pluto
2014/10/22 11:41:49 VCS INFO V-16-1-10298 Resource srdfmnt (Owner: Unspecified, Group: srdfgrp) is online on pluto (VCS initiated)
2014/10/22 11:41:49 VCS NOTICE V-16-1-10447 Group srdfgrp is online on system pluto
 

Applies To

The VCS EMC SRDF agent hot-fix is only available on AIX as of Oct 2014. The enhancement will be ported to the other platforms.