Cluster Server 7.4.1 Administrator's Guide - Linux

Last Published:
Product(s): InfoScale & Storage Foundation (7.4.1)
Platform: Linux
  1. Section I. Clustering concepts and terminology
    1. Introducing Cluster Server
      1. About Cluster Server
        1.  
          How VCS detects failure
        2. How VCS ensures application availability
          1.  
            About switchover and failover
      2. About cluster control guidelines
        1.  
          Defined start, stop, and monitor procedures
        2.  
          Ability to restart the application in a known state
        3.  
          External data storage
        4.  
          Licensing and host name issues
      3. About the physical components of VCS
        1.  
          About VCS nodes
        2.  
          About shared storage
        3.  
          About networking
      4. Logical components of VCS
        1.  
          About resources and resource dependencies
        2.  
          Categories of resources
        3.  
          About resource types
        4.  
          About service groups
        5. Types of service groups
          1.  
            About failover service groups
          2.  
            About parallel service groups
          3.  
            About hybrid service groups
        6.  
          About the ClusterService group
        7.  
          About the cluster UUID
        8.  
          About agents in VCS
        9.  
          About agent functions
        10. About resource monitoring
          1.  
            How intelligent resource monitoring works
          2.  
            About Open IMF
        11. Agent classifications
          1.  
            About bundled agents
          2.  
            About enterprise agents
          3.  
            About custom agents
        12.  
          VCS agent framework
        13. About cluster control, communications, and membership
          1.  
            About the high availability daemon (HAD)
          2.  
            About the HostMonitor daemon
          3.  
            About Group Membership Services and Atomic Broadcast (GAB)
          4.  
            About Low Latency Transport (LLT)
          5.  
            About the I/O fencing module
          6.  
            About the IMF notification module
        14. About security services
          1.  
            Digital certification structure
          2.  
            Components for secure communication
        15. Components for administering VCS
          1.  
            About Veritas InfoScale Operations Manager
      5.  
        Putting the pieces together
    2. About cluster topologies
      1. Basic failover configurations
        1.  
          Asymmetric or active / passive configuration
        2.  
          Symmetric or active / active configuration
        3.  
          About N-to-1 configuration
      2. About advanced failover configurations
        1.  
          About the N + 1 configuration
        2.  
          About the N-to-N configuration
      3. Cluster topologies and storage configurations
        1.  
          About basic shared storage cluster
        2.  
          About campus, or metropolitan, shared storage cluster
        3.  
          About shared nothing clusters
        4.  
          About replicated data clusters
        5.  
          About global clusters
    3. VCS configuration concepts
      1.  
        About configuring VCS
      2.  
        VCS configuration language
      3. About the main.cf file
        1.  
          About the SystemList attribute
        2.  
          Initial configuration
        3.  
          Including multiple .cf files in main.cf
      4.  
        About the types.cf file
      5. About VCS attributes
        1.  
          About attribute data types
        2.  
          About attribute dimensions
        3.  
          About attributes and cluster objects
        4.  
          Attribute scope across systems: global and local attributes
        5.  
          About attribute life: temporary attributes
        6.  
          Size limitations for VCS objects
      6.  
        VCS keywords and reserved words
      7.  
        About managing VCS modules
      8. VCS environment variables
        1.  
          Defining VCS environment variables
        2.  
          Environment variables to start and stop VCS modules
  2. Section II. Administration - Putting VCS to work
    1. About the VCS user privilege model
      1. About VCS user privileges and roles
        1.  
          VCS privilege levels
        2.  
          User roles in VCS
        3.  
          Hierarchy in VCS roles
        4.  
          User privileges for CLI commands
        5.  
          User privileges for cross-cluster operations
        6.  
          User privileges for clusters that run in secure mode
        7.  
          About the cluster-level user
      2.  
        How administrators assign roles to users
      3.  
        User privileges for OS user groups for clusters running in secure mode
      4.  
        VCS privileges for users with multiple roles
    2. Administering the cluster from the command line
      1. About administering VCS from the command line
        1.  
          Symbols used in the VCS command syntax
        2.  
          How VCS identifies the local system
        3.  
          About specifying values preceded by a dash (-)
        4.  
          About the -modify option
        5.  
          Encrypting VCS passwords
        6. Encrypting agent passwords
          1.  
            Generating a security key
          2.  
            Encrypting the agent password
          3.  
            Granting password encryption privileges to group administrators
          4.  
            Changing the security key
      2. About installing a VCS license
        1.  
          Installing and updating license keys using vxlicinst
        2.  
          Setting or changing the product level for keyless licensing
      3. Administering LLT
        1.  
          Displaying the cluster details and LLT version for LLT links
        2.  
          Adding and removing LLT links
        3.  
          Configuring aggregated interfaces under LLT
        4.  
          Configuring destination-based load balancing for LLT
      4.  
        Administering the AMF kernel driver
      5. Starting VCS
        1.  
          Starting the VCS engine (HAD) and related processes
      6.  
        Stopping VCS
      7. Stopping the VCS engine and related processes
        1.  
          About stopping VCS without the -force option
        2.  
          About stopping VCS with options other than the -force option
        3.  
          About controlling the hastop behavior by using the EngineShutdown attribute
        4.  
          Additional considerations for stopping VCS
      8. Logging on to VCS
        1.  
          Running high availability commands (HA) commands as non-root users on clusters in secure mode
      9. About managing VCS configuration files
        1.  
          About multiple versions of .cf files
        2.  
          Verifying a configuration
        3.  
          Scheduling automatic backups for VCS configuration files
        4.  
          Saving a configuration
        5.  
          Setting the configuration to read or write
        6.  
          Displaying configuration files in the correct format
      10. About managing VCS users from the command line
        1.  
          Adding a user
        2.  
          Assigning and removing user privileges
        3.  
          Modifying a user
        4.  
          Deleting a user
        5.  
          Displaying a user
      11. About querying VCS
        1.  
          Querying service groups
        2.  
          Querying resources
        3.  
          Querying resource types
        4.  
          Querying agents
        5.  
          Querying systems
        6.  
          Querying clusters
        7.  
          Querying status
        8.  
          Querying log data files (LDFs)
        9.  
          Using conditional statements to query VCS objects
      12. About administering service groups
        1.  
          Adding and deleting service groups
        2. Modifying service group attributes
          1.  
            Modifying the SystemList attribute
        3.  
          Bringing service groups online
        4.  
          Taking service groups offline
        5.  
          Switching service groups
        6.  
          Migrating service groups
        7.  
          Freezing and unfreezing service groups
        8.  
          Enabling and disabling service groups
        9.  
          Enabling and disabling priority based failover for a service group
        10.  
          Clearing faulted resources in a service group
        11.  
          Flushing service groups
        12.  
          Linking and unlinking service groups
      13.  
        Administering agents
      14. About administering resources
        1.  
          About adding resources
        2.  
          Adding resources
        3.  
          Deleting resources
        4.  
          Adding, deleting, and modifying resource attributes
        5.  
          Defining attributes as local
        6.  
          Defining attributes as global
        7.  
          Enabling and disabling intelligent resource monitoring for agents manually
        8. Enabling and disabling IMF for agents by using script
          1.  
            Enabling and disabling IMF for all IMF-aware agents
          2.  
            Enabling and disabling IMF for a set of agents
          3.  
            Enabling and disabling AMF on a system
          4.  
            Viewing the configuration changes made by the script
          5.  
            Displaying the current IMF status of agents
        9. Linking and unlinking resources
          1.  
            Configuring atleast resource dependency
        10.  
          Bringing resources online
        11.  
          Taking resources offline
        12.  
          Probing a resource
        13.  
          Clearing a resource
      15. About administering resource types
        1.  
          Adding, deleting, and modifying resource types
        2.  
          Overriding resource type static attributes
        3.  
          About initializing resource type scheduling and priority attributes
        4.  
          Setting scheduling and priority attributes
      16.  
        Administering systems
      17. About administering clusters
        1.  
          Configuring and unconfiguring the cluster UUID value
        2.  
          Retrieving version information
        3.  
          Adding and removing systems
        4.  
          Changing ports for VCS
        5.  
          Setting cluster attributes from the command line
        6.  
          About initializing cluster attributes in the configuration file
        7.  
          Enabling and disabling secure mode for the cluster
        8.  
          Migrating from secure mode to secure mode with FIPS
      18.  
        Using the -wait option in scripts that use VCS commands
      19.  
        Running HA fire drills
      20.  
        About administering simulated clusters from the command line
    3. Configuring applications and resources in VCS
      1.  
        Configuring resources and applications
      2. VCS bundled agents for UNIX
        1.  
          About Storage agents
        2.  
          About Network agents
        3.  
          About File share agents
        4.  
          About Services and Application agents
        5.  
          About VCS infrastructure and support agents
        6.  
          About Testing agents
      3. Configuring NFS service groups
        1. About NFS
          1.  
            NFS terminology
          2.  
            About managing and configuring NFS
        2. Configuring NFS service groups
          1. Configuring for a single NFS environment
            1.  
              Creating the NFS exports service group
          2. Configuring for a multiple NFS environment
            1.  
              Creating the NFS service group for a multiple NFS environment
            2.  
              Creating the NFS exports service group for a multiple NFS environment
          3. Configuring NFS with separate storage
            1.  
              Creating the NFS service group
            2.  
              Creating the NFS storage service group
            3.  
              Creating the NFS exports service group
          4. Configuring all NFS services in a parallel service group
            1.  
              Creating the NFS service group
            2.  
              Creating the NFS exports service group
        3. Sample configurations
          1.  
            Sample configuration for a single NFS environment without lock recovery
          2.  
            Sample configuration for a single NFS environment with lock recovery
          3.  
            Sample configuration for a single NFSv4 environment
          4.  
            Sample configuration for a multiple NFSv4 environment
          5.  
            Sample configuration for a multiple NFS environment without lock recovery
          6.  
            Sample configuration for a multiple NFS environment with lock recovery
          7.  
            Sample configuration for configuring NFS with separate storage
          8.  
            Sample configuration when configuring all NFS services in a parallel service group
      4. About configuring the RemoteGroup agent
        1. About the ControlMode attribute
          1.  
            About the OnOff mode
          2.  
            About the MonitorOnly mode
          3.  
            About the OnlineOnly mode
        2. About the ReturnIntOffline attribute
          1.  
            About the RemotePartial option
          2.  
            About the RemoteOffline option
          3.  
            About the RemoteFaulted option
        3.  
          Configuring a RemoteGroup resource
        4. Service group behavior with the RemoteGroup agent
          1.  
            Bringing the Apache service group online
          2.  
            Unexpected offline of the database service group
          3.  
            Taking the Apache service group offline
          4.  
            Configuring RemoteGroup resources in parallel service groups
      5. About configuring Samba service groups
        1.  
          Sample configuration for Samba in a failover configuration
      6.  
        Configuring the Coordination Point agent
      7. About testing resource failover by using HA fire drills
        1.  
          About HA fire drills
        2.  
          About running an HA fire drill
    4. Predicting VCS behavior using VCS Simulator
      1.  
        About VCS Simulator
      2.  
        VCS Simulator ports
      3. Administering VCS Simulator from the command line interface
        1.  
          Starting VCS Simulator from the command line interface
        2.  
          Administering simulated clusters from the command line
  3. Section III. VCS communication and operations
    1. About communications, membership, and data protection in the cluster
      1. About cluster communications
        1.  
          About intra-system communications
        2. About inter-system cluster communications
          1.  
            About Group Membership Services/Atomic Broadcast (GAB)
          2.  
            About Low Latency Transport (LLT)
      2. About cluster membership
        1. Initial joining of systems to cluster membership
          1.  
            Seeding a new cluster
          2.  
            Seeding a cluster using the GAB auto-seed parameter through I/O fencing
          3.  
            Manual seeding of a cluster
          4.  
            Stale key detection to avoid a false preexisting split brain condition
        2.  
          Ongoing cluster membership
      3. About membership arbitration
        1. About membership arbitration components
          1.  
            About the fencing module
          2.  
            About coordination points
          3.  
            About preferred fencing
          4.  
            How the fencing module starts up
          5.  
            How membership arbitration works
          6.  
            How preferred fencing works
        2. About server-based I/O fencing
          1.  
            I/O fencing enhancements provided by CP server
        3. About majority-based fencing
          1.  
            How majority-based I/O fencing works
          2.  
            Deciding cluster majority for majority-based I/O fencing mechanism
        4.  
          About making CP server highly available
        5.  
          About the CP server database
        6.  
          Recommended CP server configurations
        7. About the CP server service group
          1.  
            About the Quorum agent for CP server
        8.  
          About the CP server user types and privileges
        9. About secure communication between the VCS cluster and CP server
          1.  
            How secure communication works between the CP servers and the VCS clusters using the HTTPS protocol
      4. About data protection
        1.  
          About SCSI-3 Persistent Reservation
      5.  
        About I/O fencing configuration files
      6. Examples of VCS operation with I/O fencing
        1.  
          About the I/O fencing algorithm
        2.  
          Example: Two-system cluster where one system fails
        3.  
          Example: Four-system cluster where cluster interconnect fails
        4.  
          How I/O fencing works in different event scenarios
      7. About cluster membership and data protection without I/O fencing
        1.  
          About jeopardy
        2.  
          About Daemon Down Node Alive (DDNA)
      8. Examples of VCS operation without I/O fencing
        1. Example: Four-system cluster without a low priority link
          1.  
            Cluster interconnect link failure
          2.  
            Cluster interconnect link failure followed by system failure
          3.  
            All high priority cluster interconnect links fail
        2. Example: Four-system cluster with low priority link
          1.  
            Cluster interconnect link failure
          2.  
            Cluster interconnect link failure followed by system failure
          3.  
            All high priority cluster interconnect links fail
      9.  
        Summary of best practices for cluster communications
    2. Administering I/O fencing
      1.  
        About administering I/O fencing
      2. About the vxfentsthdw utility
        1.  
          General guidelines for using the vxfentsthdw utility
        2.  
          About the vxfentsthdw command options
        3. Testing the coordinator disk group using the -c option of vxfentsthdw
          1.  
            Removing and replacing a failed disk
        4.  
          Performing non-destructive testing on the disks using the -r option
        5.  
          Testing the shared disks using the vxfentsthdw -m option
        6.  
          Testing the shared disks listed in a file using the vxfentsthdw -f option
        7.  
          Testing all the disks in a disk group using the vxfentsthdw -g option
        8.  
          Testing a disk with existing keys
        9.  
          Testing disks with the vxfentsthdw -o option
      3. About the vxfenadm utility
        1.  
          About the I/O fencing registration key format
        2.  
          Displaying the I/O fencing registration keys
        3.  
          Verifying that the nodes see the same disk
      4. About the vxfenclearpre utility
        1.  
          Removing preexisting keys
      5. About the vxfenswap utility
        1.  
          Replacing I/O fencing coordinator disks when the cluster is online
        2.  
          Replacing the coordinator disk group in a cluster that is online
        3.  
          Adding disks from a recovered site to the coordinator disk group
        4.  
          Refreshing lost keys on coordinator disks
      6. About administering the coordination point server
        1.  
          CP server operations (cpsadm)
        2.  
          Cloning a CP server
        3.  
          Adding and removing VCS cluster entries from the CP server database
        4.  
          Adding and removing a VCS cluster node from the CP server database
        5.  
          Adding or removing CP server users
        6.  
          Listing the CP server users
        7.  
          Listing the nodes in all the VCS clusters
        8.  
          Listing the membership of nodes in the VCS cluster
        9.  
          Preempting a node
        10.  
          Registering and unregistering a node
        11.  
          Enable and disable access for a user to a VCS cluster
        12.  
          Starting and stopping CP server outside VCS control
        13.  
          Checking the connectivity of CP servers
        14.  
          Adding and removing virtual IP addresses and ports for CP servers at run-time
        15.  
          Taking a CP server database snapshot
        16.  
          Replacing coordination points for server-based fencing in an online cluster
        17.  
          Refreshing registration keys on the coordination points for server-based fencing
        18. About configuring a CP server to support IPv6 or dual stack
          1.  
            Configuring a new CP server to support pure IPv6
          2.  
            Configuring an existing CP server to support IPv6 or dual stack
        19.  
          Deployment and migration scenarios for CP server
      7. About migrating between disk-based and server-based fencing configurations
        1.  
          Migrating from disk-based to server-based fencing in an online cluster
        2.  
          Migrating from server-based to disk-based fencing in an online cluster
        3. Migrating between fencing configurations using response files
          1.  
            Sample response file to migrate from disk-based to server-based fencing
          2.  
            Sample response file to migrate from server-based fencing to disk-based fencing
          3.  
            Sample response file to migrate from single CP server-based fencing to server-based fencing
          4.  
            Response file variables to migrate between fencing configurations
      8.  
        Enabling or disabling the preferred fencing policy
      9.  
        About I/O fencing log files
    3. Controlling VCS behavior
      1. VCS behavior on resource faults
        1.  
          Critical and non-critical resources
        2. VCS behavior diagrams
          1.  
            Example scenario 1: Resource with critical parent faults
          2.  
            Example scenario 2: Resource with non-critical parent faults
          3.  
            Example scenario 3: Resource with critical parent fails to come online
          4.  
            Example scenario 4: Resource with atleast resource dependency faults
      2. About controlling VCS behavior at the service group level
        1.  
          About the AutoRestart attribute
        2.  
          About controlling failover on service group or system faults
        3.  
          About defining failover policies
        4. About AdaptiveHA
          1.  
            Enabling AdaptiveHA for a service group
          2.  
            Considerations for setting FailOverPolicy to BiggestAvailable
          3.  
            Limitations on AdaptiveHA
          4.  
            Manually upgrading the VCS configuration file to the latest version
        5.  
          About system zones
        6.  
          About sites
        7.  
          Load-based autostart
        8.  
          About freezing service groups
        9.  
          About controlling Clean behavior on resource faults
        10.  
          Clearing resources in the ADMIN_WAIT state
        11.  
          About controlling fault propagation
        12. Customized behavior diagrams
          1.  
            Example scenario: Resource with a critical parent and ManageFaults=NONE
          2.  
            Example scenario: Resource with a critical parent and FaultPropagation=0
        13. About preventing concurrency violation
          1.  
            Enabling or preventing resources to start outside VCS control
          2.  
            Limitations of ProPCV
        14. VCS behavior for resources that support the intentional offline functionality
          1.  
            About the IntentionalOffline attribute
          2.  
            About the ExternalStateChange attribute
        15. VCS behavior when a service group is restarted
          1.  
            VCS behavior when non-persistent resources restart
          2.  
            VCS behavior when persistent resources transition from faulted to online
      3. About controlling VCS behavior at the resource level
        1. Resource type attributes that control resource behavior
          1.  
            About the RestartLimit attribute
          2.  
            About the OnlineRetryLimit attribute
          3.  
            About the ConfInterval attribute
          4.  
            About the ToleranceLimit attribute
          5.  
            About the FaultOnMonitorTimeouts attribute
        2. How VCS handles resource faults
          1.  
            VCS behavior when an online resource faults
          2.  
            VCS behavior when a resource fails to come online
        3.  
          VCS behavior after a resource is declared faulted
        4. VCS behavior when a resource is restarted
          1.  
            VCS behavior when online agent function fails to bring a resource online
          2.  
            VCS behavior when a resource goes offline
        5. About disabling resources
          1.  
            When to disable a resource
          2.  
            Limitations of disabling resources
          3.  
            Additional considerations for disabling resources
          4.  
            How disabled resources affect group states
      4.  
        Changing agent file paths and binaries
      5. VCS behavior on loss of storage connectivity
        1.  
          Disk group configuration and VCS behavior
        2.  
          How VCS attributes control behavior on loss of storage connectivity
        3.  
          VCS behavior when a disk group is disabled
        4.  
          Recommendations to ensure application availability
      6. Service group workload management
        1.  
          About enabling service group workload management
        2. System capacity and service group load
          1.  
            Static load versus dynamic load
          2.  
            About overload warning
        3.  
          System limits and service group prerequisites
        4.  
          About capacity and limits
      7. Sample configurations depicting workload management
        1.  
          System and Service group definitions
        2. Sample configuration: Basic four-node cluster
          1.  
            About AutoStart operation
          2.  
            About the failure scenario
          3.  
            About the cascading failure scenario
        3. Sample configuration: Complex four-node cluster
          1.  
            About the AutoStart operation
          2.  
            About the normal operation
          3.  
            About the failure scenario
          4.  
            About the cascading failure scenario
        4. Sample configuration: Server consolidation
          1.  
            About the AutoStart operation
          2.  
            About the normal operation
          3.  
            About the failure scenario
          4.  
            About the cascading failure scenario
    4. The role of service group dependencies
      1. About service group dependencies
        1. About dependency links
          1.  
            Dependency categories
          2.  
            Dependency location
          3.  
            Dependency rigidity
        2.  
          About dependency limitations
      2. Service group dependency configurations
        1. About failover parent / failover child
          1.  
            About failover parent / parallel child
          2.  
            About parallel parent / failover child
          3.  
            About parallel parent / parallel child
      3.  
        Frequently asked questions about group dependencies
      4.  
        About linking service groups
      5. About linking multiple child service groups
        1.  
          Dependencies supported for multiple child service groups
        2.  
          Dependencies not supported for multiple child service groups
      6. VCS behavior with service group dependencies
        1.  
          Online operations in group dependencies
        2.  
          Offline operations in group dependencies
        3.  
          Switch operations in group dependencies
  4. Section IV. Administration - Beyond the basics
    1. VCS event notification
      1. About VCS event notification
        1.  
          Event messages and severity levels
        2.  
          About persistent and replicated message queue
        3.  
          How HAD deletes messages
      2. Components of VCS event notification
        1. About the notifier process
          1.  
            Example of notifier command
        2. About the hanotify utility
          1.  
            Example of hanotify command
      3. About VCS events and traps
        1.  
          Events and traps for clusters
        2.  
          Events and traps for agents
        3.  
          Events and traps for resources
        4.  
          Events and traps for systems
        5.  
          Events and traps for service groups
        6.  
          SNMP-specific files
        7. Trap variables in VCS MIB
          1.  
            About severityId
          2.  
            EntityType and entitySubType
          3.  
            About entityState
      4. About monitoring aggregate events
        1.  
          How to detect service group failover
        2.  
          How to detect service group switch
      5.  
        About configuring notification
    2. VCS event triggers
      1.  
        About VCS event triggers
      2. Using event triggers
        1.  
          Performing multiple actions using a trigger
      3. List of event triggers
        1.  
          About the dumptunables trigger
        2.  
          About the globalcounter_not_updated trigger
        3.  
          About the injeopardy event trigger
        4.  
          About the loadwarning event trigger
        5.  
          About the nofailover event trigger
        6.  
          About the postoffline event trigger
        7.  
          About the postonline event trigger
        8.  
          About the preonline event trigger
        9.  
          About the resadminwait event trigger
        10.  
          About the resfault event trigger
        11.  
          About the resnotoff event trigger
        12.  
          About the resrestart event trigger
        13.  
          About the resstatechange event trigger
        14.  
          About the sysoffline event trigger
        15.  
          About the sysup trigger
        16.  
          About the sysjoin trigger
        17.  
          About the unable_to_restart_agent event trigger
        18.  
          About the unable_to_restart_had event trigger
        19.  
          About the violation event trigger
    3. Virtual Business Services
      1.  
        About Virtual Business Services
      2.  
        Features of Virtual Business Services
      3.  
        Sample virtual business service configuration
      4.  
        About choosing between VCS and VBS level dependencies
  5. Section V. Veritas High Availability Configuration wizard
    1. Introducing the Veritas High Availability Configuration wizard
      1.  
        About the Veritas High Availability Configuration wizard
      2.  
        Launching the Veritas High Availability Configuration wizard
      3.  
        Typical VCS cluster configuration in a VMware virtual environment
      4.  
        Typical VCS cluster configuration in a physical environment
    2. Administering application monitoring from the Veritas High Availability view
      1. Administering application monitoring from the Veritas High Availability view
        1.  
          Understanding the Veritas High Availability view
        2.  
          To view the status of configured applications
        3.  
          To configure or unconfigure application monitoring
        4.  
          To start or stop applications
        5.  
          To suspend or resume application monitoring
        6.  
          To switch an application to another system
        7.  
          To add or remove a failover system
        8.  
          To clear Fault state
        9.  
          To resolve a held-up operation
        10.  
          To determine application state
        11.  
          To remove all monitoring configurations
        12.  
          To remove VCS cluster configurations
      2.  
        Administering application monitoring settings
  6. Section VI. Cluster configurations for disaster recovery
    1. Connecting clusters–Creating global clusters
      1.  
        How VCS global clusters work
      2. VCS global clusters: The building blocks
        1.  
          Visualization of remote cluster objects
        2.  
          About global service groups
        3. About global cluster management
          1.  
            About the wide-area connector process
          2.  
            About the wide-area heartbeat agent
          3.  
            Sample configuration for the wide-area heartbeat agent
        4. About serialization - The Authority attribute
          1.  
            About the Authority and AutoStart attributes
        5.  
          About resiliency and "Right of way"
        6.  
          VCS agents to manage wide-area failover
        7.  
          About the Steward process: Split-brain in two-cluster global clusters
        8.  
          Secure communication in global clusters
      3. Prerequisites for global clusters
        1.  
          Prerequisites for cluster setup
        2.  
          Prerequisites for application setup
        3.  
          Prerequisites for wide-area heartbeats
        4.  
          Prerequisites for ClusterService group
        5.  
          Prerequisites for replication setup
        6.  
          Prerequisites for clusters running in secure mode
      4.  
        About planning to set up global clusters
      5. Setting up a global cluster
        1.  
          Configuring application and replication for global cluster setup
        2. Configuring clusters for global cluster setup
          1.  
            Configuring global cluster components at the primary site
          2.  
            Installing and configuring VCS at the secondary site
          3.  
            Securing communication between the wide-area connectors
          4.  
            Configuring remote cluster objects
          5.  
            Configuring additional heartbeat links (optional)
          6.  
            Configuring the Steward process (optional)
        3. Configuring service groups for global cluster setup
          1.  
            Configuring VCS service group for VVR-based replication
        4.  
          Configuring a service group as a global service group
      6. About IPv6 support with global clusters
        1.  
          Prerequisites for configuring a global cluster to support IPv6
        2.  
          Migrating an InfoScale Availability cluster from IPv4 to IPv6 when Virtual IP (ClusterAddress) is configured
        3.  
          Migrating an InfoScale Availability cluster to IPv6 in a GCO deployment
      7. About cluster faults
        1.  
          About the type of failure
        2.  
          Switching the service group back to the primary
      8. About setting up a disaster recovery fire drill
        1. About creating and configuring the fire drill service group manually
          1.  
            Creating the fire drill service group
          2.  
            Linking the fire drill and replication service groups
          3.  
            Adding resources to the fire drill service group
          4.  
            Configuring the fire drill service group
          5.  
            Enabling the FireDrill attribute
        2. About configuring the fire drill service group using the Fire Drill Setup wizard
          1.  
            Running the fire drill setup wizard
          2.  
            About configuring local attributes in the fire drill service group
        3.  
          Verifying a successful fire drill
        4.  
          Scheduling a fire drill
      9.  
        Multi-tiered application support using the RemoteGroup agent in a global environment
      10. Test scenario for a multi-tiered environment
        1.  
          About the main.cf file for cluster 1
        2.  
          About the main.cf file for cluster 2
        3.  
          About the main.cf file for cluster 3
        4.  
          About the main.cf file for cluster 4
    2. Administering global clusters from the command line
      1.  
        About administering global clusters from the command line
      2. About global querying in a global cluster setup
        1.  
          Querying global cluster service groups
        2.  
          Querying resources across clusters
        3.  
          Querying systems
        4.  
          Querying clusters
        5.  
          Querying status
        6.  
          Querying heartbeats
      3.  
        Administering global service groups in a global cluster setup
      4.  
        Administering resources in a global cluster setup
      5. Administering clusters in global cluster setup
        1.  
          Managing cluster alerts in a global cluster setup
        2.  
          Changing the cluster name in a global cluster setup
        3.  
          Removing a remote cluster from a global cluster setup
      6.  
        Administering heartbeats in a global cluster setup
    3. Setting up replicated data clusters
      1.  
        About replicated data clusters
      2.  
        How VCS replicated data clusters work
      3. About setting up a replicated data cluster configuration
        1.  
          About typical replicated data cluster configuration
        2.  
          About setting up replication
        3.  
          Configuring the service groups
        4.  
          Configuring the service group dependencies
      4. About migrating a service group
        1.  
          Switching the service group
      5.  
        About setting up a fire drill
    4. Setting up campus clusters
      1.  
        About campus cluster configuration
      2.  
        VCS campus cluster requirements
      3.  
        Typical VCS campus cluster setup
      4. How VCS campus clusters work
        1.  
          About I/O fencing in campus clusters
      5. About setting up a campus cluster configuration
        1.  
          Preparing to set up a campus cluster configuration
        2.  
          Configuring I/O fencing to prevent data corruption
        3.  
          Configuring VxVM disk groups for campus cluster configuration
        4.  
          Configuring VCS service group for campus clusters
      6.  
        Fire drill in campus clusters
      7.  
        About the DiskGroupSnap agent
      8. About running a fire drill in a campus cluster
        1.  
          Configuring the fire drill service group
        2.  
          Running a successful fire drill in a campus cluster
  7. Section VII. Troubleshooting and performance
    1. VCS performance considerations
      1. How cluster components affect performance
        1.  
          How kernel components (GAB and LLT) affect performance
        2.  
          How the VCS engine (HAD) affects performance
        3. How agents affect performance
          1.  
            Monitoring resource type and agent configuration
        4.  
          How the VCS graphical user interfaces affect performance
      2. How cluster operations affect performance
        1.  
          VCS performance consideration when booting a cluster system
        2.  
          VCS performance consideration when a resource comes online
        3.  
          VCS performance consideration when a resource goes offline
        4.  
          VCS performance consideration when a service group comes online
        5.  
          VCS performance consideration when a service group goes offline
        6.  
          VCS performance consideration when a resource fails
        7.  
          VCS performance consideration when a system fails
        8.  
          VCS performance consideration when a network link fails
        9. VCS performance consideration when a system panics
          1.  
            About GAB client process failure
          2.  
            About GAB client registration monitoring
          3.  
            About network failure and GAB IOFENCE message
          4.  
            About quick reopen
        10.  
          VCS performance consideration when a service group switches over
        11.  
          VCS performance consideration when a service group fails over
      3. About scheduling class and priority configuration
        1.  
          About priority ranges
        2.  
          Default scheduling classes and priorities
      4. VCS agent statistics
        1.  
          Tracking monitor cycle times
        2.  
          VCS attributes enabling agent statistics
      5. About VCS tunable parameters
        1. About LLT tunable parameters
          1.  
            About LLT timer tunable parameters
          2.  
            About LLT flow control tunable parameters
          3.  
            Setting LLT timer tunable parameters
        2. About GAB tunable parameters
          1.  
            About GAB load-time or static tunable parameters
          2.  
            About GAB run-time or dynamic tunable parameters
        3. About VXFEN tunable parameters
          1.  
            Configuring the VXFEN module parameters
        4.  
          About AMF tunable parameters
    2. Troubleshooting and recovery for VCS
      1. VCS message logging
        1.  
          Log unification of VCS agent's entry points
        2.  
          Enhancing First Failure Data Capture (FFDC) to troubleshoot VCS resource's unexpected behavior
        3.  
          GAB message logging
        4.  
          Enabling debug logs for agents
        5.  
          Enabling debug logs for IMF
        6.  
          Enabling debug logs for the VCS engine
        7.  
          Enable VCS logging for VxAT
        8.  
          About debug log tags usage
        9. Gathering VCS information for support analysis
          1.  
            Verifying the metered or forecasted values for CPU, Mem, and Swap
        10.  
          Gathering LLT and GAB information for support analysis
        11.  
          Gathering IMF information for support analysis
        12.  
          Message catalogs
      2. Troubleshooting the VCS engine
        1.  
          HAD diagnostics
        2.  
          HAD restarts continuously
        3.  
          DNS configuration issues cause GAB to kill HAD
        4.  
          Seeding and I/O fencing
        5.  
          Preonline IP check
      3. Troubleshooting Low Latency Transport (LLT)
        1.  
          LLT startup script displays errors
        2.  
          LLT detects cross links usage
        3.  
          LLT link status messages
        4.  
          Cluster does not form after installing and starting VCS or SFHA
      4. Troubleshooting Group Membership Services/Atomic Broadcast (GAB)
        1.  
          Delay in port reopen
        2.  
          Node panics due to client process failure
      5. Troubleshooting VCS startup
        1.  
          "VCS:10622 local configuration missing"
        2.  
          "VCS:10623 local configuration invalid"
        3.  
          "VCS:11032 registration failed. Exiting"
        4.  
          "Waiting for cluster membership."
      6. Troubleshooting issues with systemd unit service files
        1.  
          If a unit service has failed and the corresponding module is still loaded, systemd cannot unload it and so its package cannot be removed
        2.  
          If a unit service is active and the corresponding process is stopped outside of systemd, the service cannot be started again using 'systemctl start'
        3.  
          If a unit service takes longer than the default timeout to stop or start the corresponding service, it goes into the Failed state
      7.  
        Troubleshooting Intelligent Monitoring Framework (IMF)
      8. Troubleshooting service groups
        1.  
          VCS does not automatically start service group
        2.  
          System is not in RUNNING state
        3.  
          Service group not configured to run on the system
        4.  
          Service group not configured to autostart
        5.  
          Service group is frozen
        6.  
          Failover service group is online on another system
        7.  
          A critical resource faulted
        8.  
          Service group autodisabled
        9.  
          Service group is waiting for the resource to be brought online/taken offline
        10.  
          Service group is waiting for a dependency to be met.
        11.  
          Service group not fully probed.
        12.  
          Service group does not fail over to the forecasted system
        13.  
          Service group does not fail over to the BiggestAvailable system even if FailOverPolicy is set to BiggestAvailable
        14.  
          Restoring metering database from backup taken by VCS
        15.  
          Initialization of metering database fails
        16.  
          Error message appears during service group failover or switch
      9. Troubleshooting resources
        1.  
          Service group brought online due to failover
        2.  
          Waiting for service group states
        3.  
          Waiting for child resources
        4.  
          Waiting for parent resources
        5.  
          Waiting for resource to respond
        6. Agent not running
          1.  
            Invalid agent argument list.
        7.  
          The Monitor entry point of the disk group agent returns ONLINE even if the disk group is disabled
      10. Troubleshooting sites
        1.  
          Online propagate operation was initiated but service group failed to be online
        2.  
          VCS panics nodes in the preferred site during a network-split
        3.  
          Configuring of stretch site fails
        4.  
          Renaming a Site
      11. Troubleshooting I/O fencing
        1.  
          Node is unable to join cluster while another node is being ejected
        2.  
          The vxfentsthdw utility fails when SCSI TEST UNIT READY command fails
        3.  
          Manually removing existing keys from SCSI-3 disks
        4.  
          System panics to prevent potential data corruption
        5.  
          Cluster ID on the I/O fencing key of coordinator disk does not match the local cluster's ID
        6. Fencing startup reports preexisting split-brain
          1.  
            Clearing preexisting split-brain condition
        7.  
          Registered keys are lost on the coordinator disks
        8.  
          Replacing defective disks when the cluster is offline
        9.  
          The vxfenswap utility exits if rcp or scp commands are not functional
        10. Troubleshooting CP server
          1.  
            Troubleshooting issues related to the CP server service group
          2.  
            Checking the connectivity of CP server
        11. Troubleshooting server-based fencing on the VCS cluster nodes
          1.  
            Issues during fencing startup on VCS cluster nodes set up for server-based fencing
        12. Issues during online migration of coordination points
          1.  
            Vxfen service group activity after issuing the vxfenswap command
      12. Troubleshooting notification
        1.  
          Notifier is configured but traps are not seen on SNMP console.
      13. Troubleshooting and recovery for global clusters
        1.  
          Disaster declaration
        2.  
          Lost heartbeats and the inquiry mechanism
        3. VCS alerts
          1.  
            Types of alerts
          2.  
            Managing alerts
          3.  
            Actions associated with alerts
          4.  
            Negating events
          5.  
            Concurrency violation at startup
      14.  
        Troubleshooting the steward process
      15. Troubleshooting licensing
        1.  
          Validating license keys
        2. Licensing error messages
          1.  
            [Licensing] Insufficient memory to perform operation
          2.  
            [Licensing] No valid VCS license keys were found
          3.  
            [Licensing] Unable to find a valid base VCS license key
          4.  
            [Licensing] License key cannot be used on this OS platform
          5.  
            [Licensing] VCS evaluation period has expired
          6.  
            [Licensing] License key can not be used on this system
          7.  
            [Licensing] Unable to initialize the licensing framework
          8.  
            [Licensing] QuickStart is not supported in this release
          9.  
            [Licensing] Your evaluation period for the feature has expired. This feature will not be enabled the next time VCS starts
      16. Troubleshooting secure configurations
        1.  
          FIPS mode cannot be set
        2.  
          Broker does not start
        3.  
          AT initialization fails
      17. Troubleshooting wizard-based configuration issues
        1.  
          Running the 'hastop -all' command detaches virtual disks
      18. Troubleshooting issues with the Veritas High Availability view
        1.  
          Veritas High Availability view does not display the application monitoring status
        2.  
          Veritas High Availability view may freeze due to special characters in application display name
        3.  
          In the Veritas High Availability tab, the Add Failover System link is dimmed
  8. Section VIII. Appendixes
    1. Appendix A. VCS user privileges—administration matrices
      1.  
        About administration matrices
      2. Administration matrices
        1.  
          Agent Operations (haagent)
        2.  
          Attribute Operations (haattr)
        3.  
          Cluster Operations (haclus, haconf)
        4.  
          Service group operations (hagrp)
        5.  
          Heartbeat operations (hahb)
        6.  
          Log operations (halog)
        7.  
          Resource operations (hares)
        8.  
          System operations (hasys)
        9.  
          Resource type operations (hatype)
        10.  
          User operations (hauser)
    2. Appendix B. VCS commands: Quick reference
      1.  
        About this quick reference for VCS commands
      2.  
        VCS command line reference
    3. Appendix C. Cluster and system states
      1. Remote cluster states
        1.  
          Examples of cluster state transitions
      2. System states
        1.  
          Examples of system state transitions
    4. Appendix D. VCS attributes
      1.  
        About attributes and their definitions
      2.  
        Resource attributes
      3.  
        Resource type attributes
      4.  
        Service group attributes
      5.  
        System attributes
      6.  
        Cluster attributes
      7.  
        Heartbeat attributes (for global clusters)
      8.  
        Remote cluster attributes
      9.  
        Site attributes
    5. Appendix E. Accessibility and VCS
      1.  
        About accessibility in VCS
      2. Navigation and keyboard shortcuts
        1.  
          Navigation in the Java Console
        2.  
          Navigation in the Web console
      3.  
        Support for accessibility settings
      4.  
        Support for assistive technologies

Service group attributes

Table: Service group attributes lists the service group attributes.

Table: Service group attributes

Service Group Attributes

Definition

ActiveCount

(system use only)

Number of resources in a service group that are active (online or waiting to go online). When the number drops to zero, the service group is considered offline.

  • Type and dimension: integer-scalar

  • Default: Not applicable.

AdministratorGroups

(user-defined)

List of operating system user account groups that have administrative privileges on the service group.

This attribute applies to clusters running in secure mode.

  • Type and dimension: string-keylist

  • Default: {} (none)

Administrators

(user-defined)

List of VCS users with privileges to administer the group.

Note:

A Group Administrator can perform all operations related to a specific service group, but cannot perform generic cluster operations.

  • Type and dimension: string-keylist

  • Default: {} (none)

Authority

(user-defined)

Indicates whether or not the local cluster is allowed to bring the service group online. If set to 0, it is not, if set to 1, it is. Only one cluster can have this attribute set to 1 for a specific global group.

  • Type and dimension: integer-scalar

  • Default: 0

AutoClearCount

(System use only)

Indicates the number of attempts that the VCS engine made to clear the state of the service group that has faulted and does not have a failover target. This attribute is used only if the AutoClearLimit attribute is set for the service group.

AutoClearInterval

(user-defined)

Indicates the interval in seconds after which a service group that has faulted and has no failover target is cleared automatically. The state of the service group is cleared only if AutoClearLimit is set to a non-zero value.

Default: 0

AutoclearLimit

(user-defined)

Defines the number of attempts to be made to clear the Faulted state of a service group. Disables the auto-clear feature when set to zero.

AutoDisabled

(system use only)

Indicates that VCS does not know the status of a service group (or specified system for parallel service groups). This could occur because the group is not probed (on specified system for parallel groups) in the SystemList attribute. Or the VCS engine is not running on a node designated in the SystemList attribute, but the node is visible.

When VCS does not know the status of a service group on a node but you want VCS to consider the service group enabled, perform this command to change the AutoDisabled value to 0.

hagrp -autoenable grp -sys sys1

This command instructs VCS that even though VCS has marked the service group auto-disabled, you are sure that the service group is not online on sys1. For failover service groups, this is important because the service groups now can be brought online on remaining nodes.

  • Type and dimension: boolean-scalar

  • Default: 0

AutoFailOver

(user-defined)

Indicates whether VCS initiates an automatic failover if the service group faults.

The attribute can take the following values:

  • 0 - VCS does not fail over the service group.

  • 1 - VCS automatically fails over the service group if a suitable node exists for failover.

  • 2 - VCS automatically fails over the service group only if a suitable node exists in the same system zone or sites where the service group was online.

    To set the value as 2, you must have enabled HA/DR license and the service group must not be hybrid. If you have not defined system zones or sites, the failover behavior is similar to 1.

  • Type and dimension: integer-scalar

  • Default: 1 (enabled)

AutoRestart

(user-defined)

Restarts a service group after a faulted persistent resource becomes online.

The attribute can take the following values:

  • 0 - Autorestart is disabled.

  • 1 - Autorestart is enabled.

  • 2 - When a faulted persistent resource recovers from a fault, the VCS engine clears the faults on all non-persistent faulted resources on the system. It then restarts the service group.

Note:

This attribute applies only to service groups containing persistent resources.

  • Type and dimension: integer-scalar

  • Default: 1 (enabled)

AutoStart

(user-defined)

Designates whether a service group is automatically started when VCS is started.

  • Type and dimension: boolean-scalar

  • Default: 1 (enabled)

AutoStartIfPartial

(user-defined)

Indicates whether to initiate bringing a service group online if the group is probed and discovered to be in a PARTIAL state when VCS is started.

  • Type and dimension: boolean-scalar

  • Default: 1 (enabled)

AutoStartList

(user-defined)

List of systems on which, under specific conditions, the service group will be started with VCS (usually at system boot). For example, if a system is a member of a failover service group's AutoStartList attribute, and if the service group is not already running on another system in the cluster, the group is brought online when the system is started.

VCS uses the AutoStartPolicy attribute to determine the system on which to bring the service group online.

Note:

For the service group to start, AutoStart must be enabled and Frozen must be 0. Also, beginning with 1.3.0, you must define the SystemList attribute prior to setting this attribute.

  • Type and dimension: string-keylist

  • Default: {} (none)

AutoStartPolicy

(user-defined)

Sets the policy VCS uses to determine the system on which a service group is brought online during an autostart operation if multiple systems exist.

This attribute has three options:

Order (default) - Systems are chosen in the order in which they are defined in the AutoStartList attribute.

Load - Systems are chosen in the order of their capacity, as designated in the AvailableCapacity system attribute. System with the highest capacity is chosen first.

Note:

You cannot set the value Load when the cluster attribute Statistics is set to Enabled.

Priority - Systems are chosen in the order of their priority in the SystemList attribute. Systems with the lowest priority is chosen first.

  • Type and dimension: string-scalar

  • Default: Order

CapacityReserved

(system use only)

Indicates whether capacity is reserved to bring service groups online or to fail them over. Capacity is reserved only when the service group attribute FailOverPolicy is set to BiggestAvailable.

This attribute is localized.

  • Type and dimension: Boolean

  • Default: ""

Possible values:

1: Capacity is reserved.

0: Capacity is not reserved.

The value can be reset using the hagrp -flush command.

To list this attribute, use the -all option with the hagrp -display command.

ClusterFailOverPolicy

(user-defined)

Determines how a global service group behaves when a cluster faults or when a global group faults. The attribute can take the following values:

Manual - The group does not fail over to another cluster automatically.

Auto - The group fails over to another cluster automatically if it is unable to fail over within the local cluster, or if the entire cluster faults.

Connected - The group fails over automatically to another cluster only if it is unable to fail over within the local cluster.

  • Type and dimension: string-scalar

  • Default: Manual

ClusterList

(user-defined)

Specifies the list of clusters on which the service group is configured to run.

  • Type and dimension: integer-association

  • Default: {} (none)

ContainerInfo

(user-defined)

Specifies if you can use the service group with the container. Assign the following values to the ContainerInfo attribute:

  • Name: The name of the container.

  • Type: The type of container. You can set this to Zone.

  • Enabled: Specify the value as 1 to enable the container. Specify the value as 0 to disable the container. Specify the value as 2 to enable failovers from physical computers to virtual machines and from virtual machines to physical computers. Refer to the Storage Foundation and High Availability Solutions Virtualization Guide for more information on use of value 2 for the Enabled key.

You can set a per-system value or a global value for this attribute.

  • Type and dimension: string-scalar

  • Default: {}

For a given service group, you can either set the ContainerInfo attribute for the group or the ResContainerInfo attribute for the resources in the group. You can modify ContainerInfo only if the following conditions are met:

  • ResContainerInfo is not localized for any resource in the group.

  • ResContainerInfo is not set or its Name key is set to an empty string for all resources in the group.

    See Resource attributes.

You can change the attribute scope from local to global as follows:

# hagrp -local <service_group_name> <attribute_name>

You can change the attribute scope from global to local as follows:

# hagrp -global <service_group_name> <attribute_name>
<value> ... | <key> ... | {<key> <value>} ...

For more information about the -local option and the -global option, see the man pages associated with the hagrp command.

CurrentCount

(system use only)

Number of systems on which the service group is active.

  • Type and dimension: integer-scalar

  • Default: Not applicable.

DisableFaultMessages

(user-defined)

Suppresses fault and failover messages, for a group and its resources, from getting logged in the VCS engine log file. This attribute does not suppress the information messages getting logged in the log file.

The attribute can take the following values:

  • 0 - Logs all the fault and failover messages for the service group and its resources.

  • 1 - Disables the fault and failover messages of the service groups, but continues to log resource messages.

  • 2 - Disables the fault and failover messages of the service group resources, but continues to log service group messages.

  • 3 - Disables the fault and failover messages of both service groups and its resources.

DeferAutoStart

(system use only)

Indicates whether HAD defers the auto-start of a global group in the local cluster in case the global cluster is not fully connected.

  • Type and dimension: boolean-scalar

  • Default: Not applicable

Enabled

(user-defined)

Indicates if a service group can be failed over or brought online.

The attribute can have global or local scope. If you define local (system-specific) scope for this attribute, VCS prevents the service group from coming online on specified systems that have a value of 0 for the attribute. You can use this attribute to prevent failovers on a system when performing maintenance on the system.

  • Type and dimension: boolean-scalar

  • Default: 1 (enabled)

Evacuate

(user-defined)

Indicates if VCS initiates an automatic failover when user issues hastop -local -evacuate.

  • Type and dimension: boolean-scalar

  • Default: 1

Evacuating

(system use only)

Indicates the node ID from which the service group is being evacuated.

  • Type and dimension: integer-scalar

  • Default: Not applicable

EvacList

(system use only)

Contains list of pairs of low priority service groups and the systems on which they will be evacuated.

For example:

Grp1 EvacList  grp2 Sys0  grp3 Sys0  grp4 Sys4

Type and dimension: string-association

Default: Not applicable.

EvacuatingForGroup

(system use only)

Displays the name of the high priority service group for which evacuation is in progress. The service group name is visible only as long as the evacuation is in progress.

Type and dimension: string-scalar

Default: Not applicable.

Failover

(system use only)

Indicates service group is in the process of failing over.

  • Type and dimension: boolean-scalar

  • Default: Not applicable

FailOverPolicy

(user-defined)

Defines the failover policy used by VCS to determine the system to which a group fails over. It is also used to determine the system on which a service group has been brought online through manual operation.

The policy is defined only for clusters that contain multiple systems:

Priority - The system defined as the lowest priority in the SystemList attribute is chosen.

Load - The system with the highest value of AvailableCapacity is chosen.

RoundRobin - Systems are chosen based on the active service groups they host. The system with the least number of active service groups is chosen first.

BiggestAvailable - Systems are chosen based on the forecasted available capacity for all systems in the SystemList. The system with the highest available capacity forecasted is selected.

Note:

VCS selects the node in an alphabetical order when VCS detects two systems with same values set for the policy Priority, Load, RoundRobin, or BiggestAvailable.

Prerequisites for setting FailOverPolicy to BiggestAvailable:

  • The cluster attribute Statistics must be set to Enabled.

  • Veritas recommends that the cluster attribute HostMeters should contain at least one key.

  • The service group attribute Load must contain at least one key.

  • You cannot change the attribute from BiggestAvailable to some other value, when the service group attribute CapacityReserved is set to 1 because the VCS engine reserves system capacity when it determines BiggestAvailable for the service group.

    When the service group online transition completes and after the next forecast cycle, CapacityReserved is reset.

  • Type and dimension: string-scalar

  • Default: Priority

FaultPropagation

(user-defined)

Specifies if VCS should propagate the fault up to parent resources and take the entire service group offline when a resource faults.

The value 1 indicates that when a resource faults, VCS fails over the service group, if the group's AutoFailOver attribute is set to 1. If The value 0 indicates that when a resource faults, VCS does not take other resources offline, regardless of the value of the Critical attribute. The service group does not fail over on resource fault.

  • Type and dimension: boolean-scalar

  • Default: 1

FromQ

(system use only)

Indicates the system name from which the service group is failing over. This attribute is specified when service group failover is a direct consequence of the group event, such as a resource fault within the group or a group switch.

  • Type and dimension: string-association

  • Default: Not applicable

Frozen

(system use only)

Disables all actions, including autostart, online and offline, and failover, except for monitor actions performed by agents. (This convention is observed by all agents supplied with VCS.)

  • Type and dimension: boolean-scalar

  • Default: 0 (not frozen)

GroupOwner

(user-defined)

This attribute is used for VCS email notification and logging. VCS sends email notification to the person designated in this attribute when events occur that are related to the service group. Note that while VCS logs most events, not all events trigger notifications.

Make sure to set the severity level at which you want notifications to be sent to GroupOwner or to at least one recipient defined in the SmtpRecipients attribute of the NotifierMngr agent.

  • Type and dimension: string-scalar

  • Default: ""

GroupRecipients

(user-defined)

This attribute is used for VCS email notification. VCS sends email notification to persons designated in this attribute when events related to the service group occur and when the event's severity level is equal to or greater than the level specified in the attribute.

Make sure to set the severity level at which you want notifications to be sent to GroupRecipients or to at least one recipient defined in the SmtpRecipients attribute of the NotifierMngr agent.

  • Type and dimension: string-association

  • email id: The email address of the person registered as a recipient for notification.

    severity: The minimum level of severity at which notifications must be sent.

Guests

(user-defined)

List of operating system user accounts that have Guest privileges on the service group.

This attribute applies to clusters running in secure mode.

  • Type and dimension: string-keylist

  • Default: ""

IntentOnline

(system use only)

Indicates whether to keep service groups online or offline.

VCS sets this attribute to 1 if an attempt has been made to bring the service group online.

For failover groups, VCS sets this attribute to 0 when the group is taken offline.

For parallel groups, it is set to 0 for the system when the group is taken offline or when the group faults and can fail over to another system.

VCS sets this attribute to 2 for service groups if VCS attempts to autostart a service group; for example, attempting to bring a service group online on a system from AutoStartList.

  • Type and dimension: integer-scalar

  • Default: Not applicable.

IntentionalOnlineList

(system use only)

Lists the nodes where a resource that can be intentionally brought online is found ONLINE at first probe. IntentionalOnlineList is used along with AutoStartList to determine the node on which the service group should go online when a cluster starts.

  • Type and dimension: string-keylist

  • Default: Not applicable

LastSuccess

(system use only)

Indicates the time when service group was last brought online.

  • Type and dimension: integer-scalar

  • Default: Not applicable

Load

(user-defined)

Indicates the multidimensional value expressing load exerted by a service group on the system.

When the cluster attribute Statistics is enabled, the allowed key values are CPU, Mem, and Swap. The value for these keys in corresponding units as specified in MeterUnit (cluster attribute).

When the cluster attribute Statistics is not enabled, the allowed key value is Units.

  • Type and dimension: float-association

  • Default: Not applicable

The following additional considerations apply:

  • You cannot change this attribute when the service group attribute CapacityReserved is set to 1 in the cluster and when the FailOverPolicy is set to BiggestAvailable. This is because the VCS engine reserves system capacity based on the service group attribute Load.

    When the service group's online transition completes and after the next forecast cycle, CapacityReserved is reset.

  • If the FailOverPolicy is set to BiggestAvailable for a service group, the attribute Load must be specified with at least one of the following keys:

    • CPU

    • Mem

    • Swap

ManageFaults

(user-defined)

Specifies if VCS manages resource failures within the service group by calling the Clean function for the resources. This attribute can take the following values.

NONE - VCS does not call the Clean function for any resource in the group. You must manually handle resource faults.

  • Type and dimension: string-scalar

  • Default: ALL

ManualOps

(user-defined)

Indicates if manual operations are allowed on the service group.

  • Type and dimension: boolean-scalar

  • Default = 1 (enabled)

MeterWeight

(user-defined)

Represents the weight given for the cluster attribute's HostMeters key to determine a target system for a service group when more than one system meets the group attribute's Load requirements.

  • Type and dimension: integer-association

  • Default: Not applicable

Additional considerations for configuring this attribute in main.cf or changing it at run time:

  • This is an optional service group attribute. If it is not defined for a group, the VCS considers the cluster attribute MeterWeight.

  • To override this attribute at an individual group level, define it at run time or in the main.cf file. Ensure that keys are subsets of the cluster attribute HostAvailableMeters.

  • You cannot change this attribute when the service group attribute CapacityReserved is set to 1 in the cluster.

  • The values for the keys represent weights of the corresponding parameters. It should be in range of 0 to 10.

MigrateQ

(system use only)

Indicates the system from which the service group is migrating. This attribute is specified when group failover is an indirect consequence (in situations such as a system shutdown or another group faults and is linked to this group).

  • Type and dimension: string-association

  • Default: Not applicable

NumRetries

(system use only)

Indicates the number of attempts made to bring a service group online. This attribute is used only if the attribute OnlineRetryLimit is set for the service group.

  • Type and dimension: integer-scalar

  • Default: Not applicable

OnlineAtUnfreeze

(system use only)

When a node or a service group is frozen, the OnlineAtUnfreeze attribute specifies how an offline service group reacts after it or a node is unfrozen.

  • Type and dimension: integer-scalar

  • Default: Not applicable

OnlineClearParent

When this attribute is enabled for a service group and the service group comes online or is detected online, VCS clears the faults on all online type parent groups, such as online local, online global, and online remote.

  • Type and dimension: boolean-scalar

  • Default: 0

For example, assume that both the parent group and the child group faulted and both cannot failover. Later, when VCS tries again to bring the child group online and the group is brought online or detected online, the VCS engine clears the faults on the parent group, allowing VCS to restart the parent group too.

OnlineRetryInterval

(user-defined)

Indicates the interval, in seconds, during which a service group that has successfully restarted on the same system and faults again should be failed over, even if the attribute OnlineRetryLimit is non-zero. This prevents a group from continuously faulting and restarting on the same system.

  • Type and dimension: integer-scalar

  • Default: 0

OnlineRetryLimit

(user-defined)

If non-zero, specifies the number of times the VCS engine tries to restart a faulted service group on the same system on which the group faulted, before it gives up and tries to fail over the group to another system.

  • Type and dimension: integer-scalar

  • Default: 0

OperatorGroups

(user-defined)

List of operating system user groups that have Operator privileges on the service group. This attribute applies to clusters running in secure mode.

  • Type and dimension: string-keylist

  • Default: ""

Operators

(user-defined)

List of VCS users with privileges to operate the group. A Group Operator can only perform online/offline, and temporary freeze/unfreeze operations pertaining to a specific group.

  • Type and dimension: string-keylist

  • Default: ""

Parallel

(user-defined)

Indicates if service group is failover (0), parallel (1), or hybrid(2).

  • Type and dimension: integer-scalar

  • Default: 0

PathCount

(system use only)

Number of resources in path not yet taken offline. When this number drops to zero, the engine may take the entire service group offline if critical fault has occurred.

  • Type and dimension: integer-scalar

  • Default: Not applicable

PCVAllowOnline

(system use only)

Indicates whether ProPCV-enabled resources in a service group can be brought online on a node outside VCS control.

  • Type and dimension: boolean-scalar

  • Default: Not applicable

See About preventing concurrency violation.

PolicyIntention

(system use only)

Functions as a lock on service groups listed in the hagrp -online -propagate command and hagrp -offline -propagate command:

  • Type and dimension: integer-scalar

When PolicyIntention is set to a non-zero value for the service groups in dependency tree, this attribute protects the service groups from any other operation. PolicyIntention can take three values.

  • The value 0 indicates that the service group is not part of the hagrp -online -propagate operation or the hagrp -offline -propagate operation.

  • The value 1 indicates that the service group is part of the hagrp -online -propagate operation.

  • The value 2 indicates that the service group is part of the hagrp -offline -propagate operation.

PreOnline

(user-defined)

Indicates that the VCS engine should not bring online a service group in response to a manual group online, group autostart, or group failover. The engine should instead run the PreOnline trigger.

You can set a local (per-system) value or a global value for this attribute. A per-system value enables you to control the firing of PreOnline triggers on specific nodes in the cluster.

  • Type and dimension: boolean-scalar

  • Default: 0

You can change the attribute scope from local to global as follows:

# hagrp -local <service_group_name> <attribute_name>

You can change the attribute scope from global to local as follows:

# hagrp -global <service_group_name> <attribute_name>
<value> ... | <key> ... | {<key> <value>} ...

For more information about the -local option and the -global option, see the man pages associated with the hagrp command.

PreOnlining

(system use only)

Indicates that VCS engine invoked the preonline script; however, the script has not yet returned with group online.

  • Type and dimension: integer-scalar

  • Default: Not applicable

PreonlineTimeout

(user-defined)

Defines the maximum amount of time in seconds the preonline script takes to run the command hagrp -online -nopre for the group. Note that HAD uses this timeout during evacuation only. For example, when a user runs the command hastop -local -evacuate and the Preonline trigger is invoked on the system on which the service groups are being evacuated.

  • Type and dimension: integer-scalar

  • Default: 300

Prerequisites

(user-defined)

An unordered set of name=value pairs denoting specific resources required by a service group. If prerequisites are not met, the group cannot go online. The format for Prerequisites is:

Prerequisites() = {Name=Value, name2=value2}.

Names used in setting Prerequisites are arbitrary and not obtained from the system. Coordinate name=value pairs listed in Prerequisites with the same name=value pairs in Limits().

  • Type and dimension: integer-association

PreSwitch

(user-defined)

Indicates whether VCS engine should invoke PreSwitch actions in response to a manual service group switch operation.

Note:

The engine does not invoke the PreSwitch action during a group fault or when you use -any option to switch a group.

This attribute must be defined in the global group definition on the remote cluster. This attribute takes the following values:

0 - VCS engine switches the service group normally.

1 - VCS engine switches the service group based on the output of PreSwitch action of the resources.

If you set the value as 1, the VCS engine looks for any resource in the service group that supports PreSwitch action. If the action is not defined for any resource, the VCS engine switches a service group normally.

If the action is defined for one or more resources, then the VCS engine invokes PreSwitch action for those resources. If all the actions succeed, the engine switches the service group. If any of the actions fail, the engine aborts the switch operation.

The engine invokes the PreSwitch action in parallel and waits for all the actions to complete to decide whether to perform a switch operation. The VCS engine reports the action's output to the engine log. The PreSwitch action does not change the configuration or the cluster state.

  • Type and dimension: boolean-scalar

  • Default: 0

PreSwitching

(system use only)

Indicates that the VCS engine invoked the agent's PreSwitch action; however, the action is not yet complete.

  • Type and dimension: integer-scalar

  • Default: Not applicable

PrintTree

(user-defined)

Indicates whether or not the resource dependency tree is written to the configuration file. The value 1 indicates the tree is written.

Note:

For very large configurations, the time taken to print the tree and to dump the configuration is high.

  • Type and dimension: boolean-scalar

  • Default: 1

Priority

(user-defined)

Enables users to designate and prioritize the service group. VCS does not interpret the value; rather, this attribute enables the user to configure the priority of a service group and the sequence of actions required in response to a particular event.

If the cluster-level attribute value for PreferredFencingPolicy is set to Group, VCS uses this Priority attribute value to calculate the node weight to determine the surviving subcluster during I/O fencing race.

VCS assigns the following node weight based on the priority of the service group:

Priority     Node weight
1            625
2            125
3            25
4            5
0 or >=5     1

A higher node weight is associated with higher values of Priority. The node weight is the sum of node weight values for service groups which are ONLINE/PARTIAL.

See About preferred fencing.

  • Type and dimension: integer-scalar

  • Default: 0

Probed

(system use only)

Indicates whether all enabled resources in the group have been detected by their respective agents.

  • Type and dimension: boolean-scalar

  • Default: Not applicable

ProbesPending

(system use only)

The number of resources that remain to be detected by the agent on each system.

  • Type and dimension: integer-scalar

  • Default: Not applicable

ProPCV

(user-defined)

Indicates whether the service group is proactively prevented from concurrency violation for ProPCV-enabled resources.

  • Type and dimension: boolean-scalar

  • Default: 0

See About preventing concurrency violation.

Responding

(system use only)

Indicates VCS engine is responding to a failover event and is in the process of bringing the service group online or failing over the node.

  • Type and dimension: integer-scalar

  • Default: Not applicable

Restart

(system use only)

For internal use only.

  • Type and dimension: integer-scalar

  • Default: Not applicable

SourceFile

(user-defined)

File from which the configuration is read. Do not configure this attribute in main.cf.

Make sure the path exists on all nodes before running a command that configures this attribute.

Make sure the path exists on all nodes before configuring this attribute.

  • Type and dimension: string-scalar

  • Default: ./main.cf

State

(system use only)

Group state on each system:

OFFLINE - All non-persistent resources are offline.

ONLINE - All resources whose AutoStart attribute is equal to 1 are online.

FAULTED - At least one critical resource in the group is faulted or is affected by a fault.

PARTIAL - At least one, but not all, resources with Operations=OnOff is online, and not all AutoStart resources are online.

STARTING - Group is attempting to go online.

STOPPING - Group is attempting to go offline.

MIGRATING - Group is attempting to migrate a resource from the source system to the target system. This state should be seen only as a combination of multiple states such as, ONLINE|STOPPING|MIGRATING, OFFLINE|STARTING|MIGRATING, and OFFLINE|MIGRATING.

A group state may be a combination of multiple states described above. For example, OFFLINE | FAULTED, OFFLINE | STARTING, PARTIAL | FAULTED, PARTIAL | STARTING, PARTIAL | STOPPING, ONLINE | STOPPING

  • Type and dimension: integer-scalar

  • Default: Not applicable.

SysDownPolicy

(user-defined)

Determines whether a service group is autodisabled when the system is down and if the service group is taken offline when the system is rebooted or is shut down gracefully.

If SysDownPolicy contains the key AutoDisableNoOffline, the following conditions apply:

  • The service group is autodisabled when system is down, gracefully shut down, or is detected as down.

  • The service group is not taken offline when the system reboots or shuts down gracefully.

Valid values: Empty keylist or the key AutoDisableNoOffline

Default: Empty keylist

For example, if a service group with SysDownPolicy = AutoDisableNoOffline is online on system sys1, it has the following effect for various commands:

  • The hastop -local -evacuate command for sys1 is rejected

  • The hastop -sysoffline command is accepted but the service group with SysDownPolicy = AutoDisableNoOffline is not taken offline.

  • The hastop -all command is rejected.

SystemList

(user-defined)

List of systems on which the service group is configured to run and their priorities. Lower numbers indicate a preference for the system as a failover target.

Note:

You must define this attribute prior to setting the AutoStartList attribute.

  • Type and dimension: integer-association

  • Default: "" (none)

SystemZones

(user-defined)

Indicates the virtual sublists within the SystemList attribute that grant priority in failing over. Values are string/integer pairs. The string key is the name of a system in the SystemList attribute, and the integer is the number of the zone. Systems with the same zone number are members of the same zone. If a service group faults on one system in a zone, it is granted priority to fail over to another system within the same zone, despite the policy granted by the FailOverPolicy attribute.

  • Type and dimension: integer-association

  • Default: "" (none)

Note:

You cannot modify this attribute when SiteAware is set as 1 and Sites are defined.

Tag

(user-defined)

Identifies special-purpose service groups created for specific VCS products.

  • Type and dimension: string-scalar

  • Default: Not applicable.

TargetCount

(system use only)

Indicates the number of target systems on which the service group should be brought online.

  • Type and dimension: integer-scalar

  • Default: Not applicable.

TFrozen

(user-defined)

Indicates if service groups can be brought online or taken offline on nodes in the cluster. Service groups cannot be brought online or taken offline if the value of the attribute is 1.

  • Type and dimension: boolean-scalar

  • Default: 0 (not frozen)

ToQ

(system use only)

Indicates the node name to which the service is failing over. This attribute is specified when service group failover is a direct consequence of the group event, such as a resource fault within the group or a group switch.

  • Type and dimension: string-association

  • Default: Not applicable

TriggerEvent

(user-defined)

For internal use only.

  • Type and dimension: boolean-scalar

  • Default: Not applicable

TriggerPath

(user-defined)

Enables you to customize the trigger path.

If a trigger is enabled but the trigger path is "" (default), VCS invokes the trigger from the $VCS_HOME/bin/triggers directory. If you specify an alternate directory, VCS invokes the trigger from that path. The value is case-sensitive. VCS does not trim the leading spaces or trailing spaces in the Trigger Path value. If the path contains leading spaces or trailing spaces, the trigger might fail to get executed.

The path that you specify must be in the following format:

$VCS_HOME/TriggerPath/Trigger

For example, if TriggerPath is set to mytriggers/sg1, VCS looks for the preonline trigger scripts in the $VCS_HOME/mytriggers/sg1/preonline/ directory.

  • Type and dimension: string-scalar

  • Default: ""

TriggerResFault

(user-defined)

Defines whether VCS invokes the resfault trigger when a resource faults. The value 0 indicates that VCS does not invoke the trigger.

  • Type and dimension: boolean-scalar

  • Default: 1

TriggerResRestart

(user-defined)

Determines whether or not to invoke the resrestart trigger if resource restarts.

See About the resrestart event trigger.

To invoke the resrestart trigger for a specific resource, enable this attribute at the resource level.

See Resource attributes.

  • Type and dimension: boolean-scalar

  • Default: 0 (disabled)

TriggerResStateChange

(user-defined)

Determines whether or not to invoke the resstatechange trigger if resource state changes.

To invoke the resstatechange trigger for a specific resource, enable this attribute at the resource level.

See Resource attributes.

  • Type and dimension: boolean-scalar

  • Default: 0 (disabled)

TriggersEnabled

(user-defined)

Determines if a specific trigger is enabled or not.

Triggers are disabled by default. You can enable specific triggers on all nodes or on selected nodes. Valid values are VIOLATION, NOFAILOVER, PREONLINE, POSTONLINE, POSTOFFLINE, RESFAULT, RESSTATECHANGE, and RESRESTART.

To enable triggers on a node, add trigger keys in the following format:

TriggersEnabled@node1 = {POSTOFFLINE, POSTONLINE}

The postoffline trigger and postonline trigger are enabled on node1.

To enable triggers on all nodes in the cluster, add trigger keys in the following format:

TriggersEnabled = {POSTOFFLINE, POSTONLINE}

The postoffline trigger and postonline trigger are enabled on all nodes.

  • Type and dimension: string-keylist

  • Default: {}

You can change the attribute scope from local to global as follows:

# hagrp -local <service_group_name> <attribute_name>

You can change the attribute scope from global to local as follows:

# hagrp -global <service_group_name> <attribute_name>
<value> ... | <key> ... | {<key> <value>} ...

For more information about the -local option and the -global option, see the man pages associated with the hagrp command.

TypeDependencies

(user-defined)

Creates a dependency (via an ordered list) between resource types specified in the service group list, and all instances of the respective resource type.

  • Type and dimension: string-keylist

  • Default: ""

UnSteadyCount

(system use only)

Represents the total number of resources with pending online or offline operations. This is a localized attribute.

  • Type and dimension: integer

  • Default: 0

To list this attribute, use the -all option with the hagrp -display command.

The hagrp -flush command resets this attribute.

UserAssoc

(user-defined)

This is a free form string-association attribute to hold any key-value pair. "Name" and "UITimeout" keys are reserved by VCS health view. You must not delete these keys or update the values corresponding to these keys, but you can add other keys and use it for any other purpose.

  • Type and dimension: string-association

  • Default: {}

You can change the attribute scope from local to global as follows:

# hagrp -local <service_group_name> <attribute_name>

You can change the attribute scope from global to local as follows:

# hagrp -global <service_group_name> <attribute_name>
<value> ... | <key> ... | {<key> <value>} ...

For more information about the -local option and -global option, see the man pages associated with the hagrp command.

UserIntGlobal

(user-defined)

Use this attribute for any purpose. It is not used by VCS.

  • Type and dimension: integer-scalar

  • Default: 0

UserStrGlobal

(user-defined)

VCS uses this attribute in the ClusterService group. Do not modify this attribute in the ClusterService group. Use the attribute for any purpose in other service groups.

  • Type and dimension: string-scalar

  • Default: 0

UserIntLocal

(user-defined)

Use this attribute for any purpose. It is not used by VCS.

  • Type and dimension: integer-scalar

  • Default: 0

UserStrLocal

(user-defined)

Use this attribute for any purpose. It is not used by VCS.

  • Type and dimension: string-scalar

  • Default: ""

More Information

About VCS user privileges and roles

About serialization - The Authority attribute

About controlling failover on service group or system faults

About service group dependencies

About controlling Clean behavior on resource faults

About VCS user privileges and roles

???

Administering global service groups in a global cluster setup

About the resstatechange event trigger