Cluster Server 7.4.1 Administrator's Guide - Linux

Last Published:
Product(s): InfoScale & Storage Foundation (7.4.1)
Platform: Linux
  1. Section I. Clustering concepts and terminology
    1. Introducing Cluster Server
      1. About Cluster Server
        1.  
          How VCS detects failure
        2. How VCS ensures application availability
          1.  
            About switchover and failover
      2. About cluster control guidelines
        1.  
          Defined start, stop, and monitor procedures
        2.  
          Ability to restart the application in a known state
        3.  
          External data storage
        4.  
          Licensing and host name issues
      3. About the physical components of VCS
        1.  
          About VCS nodes
        2.  
          About shared storage
        3.  
          About networking
      4. Logical components of VCS
        1.  
          About resources and resource dependencies
        2.  
          Categories of resources
        3.  
          About resource types
        4.  
          About service groups
        5. Types of service groups
          1.  
            About failover service groups
          2.  
            About parallel service groups
          3.  
            About hybrid service groups
        6.  
          About the ClusterService group
        7.  
          About the cluster UUID
        8.  
          About agents in VCS
        9.  
          About agent functions
        10. About resource monitoring
          1.  
            How intelligent resource monitoring works
          2.  
            About Open IMF
        11. Agent classifications
          1.  
            About bundled agents
          2.  
            About enterprise agents
          3.  
            About custom agents
        12.  
          VCS agent framework
        13. About cluster control, communications, and membership
          1.  
            About the high availability daemon (HAD)
          2.  
            About the HostMonitor daemon
          3.  
            About Group Membership Services and Atomic Broadcast (GAB)
          4.  
            About Low Latency Transport (LLT)
          5.  
            About the I/O fencing module
          6.  
            About the IMF notification module
        14. About security services
          1.  
            Digital certification structure
          2.  
            Components for secure communication
        15. Components for administering VCS
          1.  
            About Veritas InfoScale Operations Manager
      5.  
        Putting the pieces together
    2. About cluster topologies
      1. Basic failover configurations
        1.  
          Asymmetric or active / passive configuration
        2.  
          Symmetric or active / active configuration
        3.  
          About N-to-1 configuration
      2. About advanced failover configurations
        1.  
          About the N + 1 configuration
        2.  
          About the N-to-N configuration
      3. Cluster topologies and storage configurations
        1.  
          About basic shared storage cluster
        2.  
          About campus, or metropolitan, shared storage cluster
        3.  
          About shared nothing clusters
        4.  
          About replicated data clusters
        5.  
          About global clusters
    3. VCS configuration concepts
      1.  
        About configuring VCS
      2.  
        VCS configuration language
      3. About the main.cf file
        1.  
          About the SystemList attribute
        2.  
          Initial configuration
        3.  
          Including multiple .cf files in main.cf
      4.  
        About the types.cf file
      5. About VCS attributes
        1.  
          About attribute data types
        2.  
          About attribute dimensions
        3.  
          About attributes and cluster objects
        4.  
          Attribute scope across systems: global and local attributes
        5.  
          About attribute life: temporary attributes
        6.  
          Size limitations for VCS objects
      6.  
        VCS keywords and reserved words
      7.  
        About managing VCS modules
      8. VCS environment variables
        1.  
          Defining VCS environment variables
        2.  
          Environment variables to start and stop VCS modules
  2. Section II. Administration - Putting VCS to work
    1. About the VCS user privilege model
      1. About VCS user privileges and roles
        1.  
          VCS privilege levels
        2.  
          User roles in VCS
        3.  
          Hierarchy in VCS roles
        4.  
          User privileges for CLI commands
        5.  
          User privileges for cross-cluster operations
        6.  
          User privileges for clusters that run in secure mode
        7.  
          About the cluster-level user
      2.  
        How administrators assign roles to users
      3.  
        User privileges for OS user groups for clusters running in secure mode
      4.  
        VCS privileges for users with multiple roles
    2. Administering the cluster from the command line
      1. About administering VCS from the command line
        1.  
          Symbols used in the VCS command syntax
        2.  
          How VCS identifies the local system
        3.  
          About specifying values preceded by a dash (-)
        4.  
          About the -modify option
        5.  
          Encrypting VCS passwords
        6. Encrypting agent passwords
          1.  
            Generating a security key
          2.  
            Encrypting the agent password
          3.  
            Granting password encryption privileges to group administrators
          4.  
            Changing the security key
      2. About installing a VCS license
        1.  
          Installing and updating license keys using vxlicinst
        2.  
          Setting or changing the product level for keyless licensing
      3. Administering LLT
        1.  
          Displaying the cluster details and LLT version for LLT links
        2.  
          Adding and removing LLT links
        3.  
          Configuring aggregated interfaces under LLT
        4.  
          Configuring destination-based load balancing for LLT
      4.  
        Administering the AMF kernel driver
      5. Starting VCS
        1.  
          Starting the VCS engine (HAD) and related processes
      6.  
        Stopping VCS
      7. Stopping the VCS engine and related processes
        1.  
          About stopping VCS without the -force option
        2.  
          About stopping VCS with options other than the -force option
        3.  
          About controlling the hastop behavior by using the EngineShutdown attribute
        4.  
          Additional considerations for stopping VCS
      8. Logging on to VCS
        1.  
          Running high availability commands (HA) commands as non-root users on clusters in secure mode
      9. About managing VCS configuration files
        1.  
          About multiple versions of .cf files
        2.  
          Verifying a configuration
        3.  
          Scheduling automatic backups for VCS configuration files
        4.  
          Saving a configuration
        5.  
          Setting the configuration to read or write
        6.  
          Displaying configuration files in the correct format
      10. About managing VCS users from the command line
        1.  
          Adding a user
        2.  
          Assigning and removing user privileges
        3.  
          Modifying a user
        4.  
          Deleting a user
        5.  
          Displaying a user
      11. About querying VCS
        1.  
          Querying service groups
        2.  
          Querying resources
        3.  
          Querying resource types
        4.  
          Querying agents
        5.  
          Querying systems
        6.  
          Querying clusters
        7.  
          Querying status
        8.  
          Querying log data files (LDFs)
        9.  
          Using conditional statements to query VCS objects
      12. About administering service groups
        1.  
          Adding and deleting service groups
        2. Modifying service group attributes
          1.  
            Modifying the SystemList attribute
        3.  
          Bringing service groups online
        4.  
          Taking service groups offline
        5.  
          Switching service groups
        6.  
          Migrating service groups
        7.  
          Freezing and unfreezing service groups
        8.  
          Enabling and disabling service groups
        9.  
          Enabling and disabling priority based failover for a service group
        10.  
          Clearing faulted resources in a service group
        11.  
          Flushing service groups
        12.  
          Linking and unlinking service groups
      13.  
        Administering agents
      14. About administering resources
        1.  
          About adding resources
        2.  
          Adding resources
        3.  
          Deleting resources
        4.  
          Adding, deleting, and modifying resource attributes
        5.  
          Defining attributes as local
        6.  
          Defining attributes as global
        7.  
          Enabling and disabling intelligent resource monitoring for agents manually
        8. Enabling and disabling IMF for agents by using script
          1.  
            Enabling and disabling IMF for all IMF-aware agents
          2.  
            Enabling and disabling IMF for a set of agents
          3.  
            Enabling and disabling AMF on a system
          4.  
            Viewing the configuration changes made by the script
          5.  
            Displaying the current IMF status of agents
        9. Linking and unlinking resources
          1.  
            Configuring atleast resource dependency
        10.  
          Bringing resources online
        11.  
          Taking resources offline
        12.  
          Probing a resource
        13.  
          Clearing a resource
      15. About administering resource types
        1.  
          Adding, deleting, and modifying resource types
        2.  
          Overriding resource type static attributes
        3.  
          About initializing resource type scheduling and priority attributes
        4.  
          Setting scheduling and priority attributes
      16.  
        Administering systems
      17. About administering clusters
        1.  
          Configuring and unconfiguring the cluster UUID value
        2.  
          Retrieving version information
        3.  
          Adding and removing systems
        4.  
          Changing ports for VCS
        5.  
          Setting cluster attributes from the command line
        6.  
          About initializing cluster attributes in the configuration file
        7.  
          Enabling and disabling secure mode for the cluster
        8.  
          Migrating from secure mode to secure mode with FIPS
      18.  
        Using the -wait option in scripts that use VCS commands
      19.  
        Running HA fire drills
      20.  
        About administering simulated clusters from the command line
    3. Configuring applications and resources in VCS
      1.  
        Configuring resources and applications
      2. VCS bundled agents for UNIX
        1.  
          About Storage agents
        2.  
          About Network agents
        3.  
          About File share agents
        4.  
          About Services and Application agents
        5.  
          About VCS infrastructure and support agents
        6.  
          About Testing agents
      3. Configuring NFS service groups
        1. About NFS
          1.  
            NFS terminology
          2.  
            About managing and configuring NFS
        2. Configuring NFS service groups
          1. Configuring for a single NFS environment
            1.  
              Creating the NFS exports service group
          2. Configuring for a multiple NFS environment
            1.  
              Creating the NFS service group for a multiple NFS environment
            2.  
              Creating the NFS exports service group for a multiple NFS environment
          3. Configuring NFS with separate storage
            1.  
              Creating the NFS service group
            2.  
              Creating the NFS storage service group
            3.  
              Creating the NFS exports service group
          4. Configuring all NFS services in a parallel service group
            1.  
              Creating the NFS service group
            2.  
              Creating the NFS exports service group
        3. Sample configurations
          1.  
            Sample configuration for a single NFS environment without lock recovery
          2.  
            Sample configuration for a single NFS environment with lock recovery
          3.  
            Sample configuration for a single NFSv4 environment
          4.  
            Sample configuration for a multiple NFSv4 environment
          5.  
            Sample configuration for a multiple NFS environment without lock recovery
          6.  
            Sample configuration for a multiple NFS environment with lock recovery
          7.  
            Sample configuration for configuring NFS with separate storage
          8.  
            Sample configuration when configuring all NFS services in a parallel service group
      4. About configuring the RemoteGroup agent
        1. About the ControlMode attribute
          1.  
            About the OnOff mode
          2.  
            About the MonitorOnly mode
          3.  
            About the OnlineOnly mode
        2. About the ReturnIntOffline attribute
          1.  
            About the RemotePartial option
          2.  
            About the RemoteOffline option
          3.  
            About the RemoteFaulted option
        3.  
          Configuring a RemoteGroup resource
        4. Service group behavior with the RemoteGroup agent
          1.  
            Bringing the Apache service group online
          2.  
            Unexpected offline of the database service group
          3.  
            Taking the Apache service group offline
          4.  
            Configuring RemoteGroup resources in parallel service groups
      5. About configuring Samba service groups
        1.  
          Sample configuration for Samba in a failover configuration
      6.  
        Configuring the Coordination Point agent
      7. About testing resource failover by using HA fire drills
        1.  
          About HA fire drills
        2.  
          About running an HA fire drill
    4. Predicting VCS behavior using VCS Simulator
      1.  
        About VCS Simulator
      2.  
        VCS Simulator ports
      3. Administering VCS Simulator from the command line interface
        1.  
          Starting VCS Simulator from the command line interface
        2.  
          Administering simulated clusters from the command line
  3. Section III. VCS communication and operations
    1. About communications, membership, and data protection in the cluster
      1. About cluster communications
        1.  
          About intra-system communications
        2. About inter-system cluster communications
          1.  
            About Group Membership Services/Atomic Broadcast (GAB)
          2.  
            About Low Latency Transport (LLT)
      2. About cluster membership
        1. Initial joining of systems to cluster membership
          1.  
            Seeding a new cluster
          2.  
            Seeding a cluster using the GAB auto-seed parameter through I/O fencing
          3.  
            Manual seeding of a cluster
          4.  
            Stale key detection to avoid a false preexisting split brain condition
        2.  
          Ongoing cluster membership
      3. About membership arbitration
        1. About membership arbitration components
          1.  
            About the fencing module
          2.  
            About coordination points
          3.  
            About preferred fencing
          4.  
            How the fencing module starts up
          5.  
            How membership arbitration works
          6.  
            How preferred fencing works
        2. About server-based I/O fencing
          1.  
            I/O fencing enhancements provided by CP server
        3. About majority-based fencing
          1.  
            How majority-based I/O fencing works
          2.  
            Deciding cluster majority for majority-based I/O fencing mechanism
        4.  
          About making CP server highly available
        5.  
          About the CP server database
        6.  
          Recommended CP server configurations
        7. About the CP server service group
          1.  
            About the Quorum agent for CP server
        8.  
          About the CP server user types and privileges
        9. About secure communication between the VCS cluster and CP server
          1.  
            How secure communication works between the CP servers and the VCS clusters using the HTTPS protocol
      4. About data protection
        1.  
          About SCSI-3 Persistent Reservation
      5.  
        About I/O fencing configuration files
      6. Examples of VCS operation with I/O fencing
        1.  
          About the I/O fencing algorithm
        2.  
          Example: Two-system cluster where one system fails
        3.  
          Example: Four-system cluster where cluster interconnect fails
        4.  
          How I/O fencing works in different event scenarios
      7. About cluster membership and data protection without I/O fencing
        1.  
          About jeopardy
        2.  
          About Daemon Down Node Alive (DDNA)
      8. Examples of VCS operation without I/O fencing
        1. Example: Four-system cluster without a low priority link
          1.  
            Cluster interconnect link failure
          2.  
            Cluster interconnect link failure followed by system failure
          3.  
            All high priority cluster interconnect links fail
        2. Example: Four-system cluster with low priority link
          1.  
            Cluster interconnect link failure
          2.  
            Cluster interconnect link failure followed by system failure
          3.  
            All high priority cluster interconnect links fail
      9.  
        Summary of best practices for cluster communications
    2. Administering I/O fencing
      1.  
        About administering I/O fencing
      2. About the vxfentsthdw utility
        1.  
          General guidelines for using the vxfentsthdw utility
        2.  
          About the vxfentsthdw command options
        3. Testing the coordinator disk group using the -c option of vxfentsthdw
          1.  
            Removing and replacing a failed disk
        4.  
          Performing non-destructive testing on the disks using the -r option
        5.  
          Testing the shared disks using the vxfentsthdw -m option
        6.  
          Testing the shared disks listed in a file using the vxfentsthdw -f option
        7.  
          Testing all the disks in a disk group using the vxfentsthdw -g option
        8.  
          Testing a disk with existing keys
        9.  
          Testing disks with the vxfentsthdw -o option
      3. About the vxfenadm utility
        1.  
          About the I/O fencing registration key format
        2.  
          Displaying the I/O fencing registration keys
        3.  
          Verifying that the nodes see the same disk
      4. About the vxfenclearpre utility
        1.  
          Removing preexisting keys
      5. About the vxfenswap utility
        1.  
          Replacing I/O fencing coordinator disks when the cluster is online
        2.  
          Replacing the coordinator disk group in a cluster that is online
        3.  
          Adding disks from a recovered site to the coordinator disk group
        4.  
          Refreshing lost keys on coordinator disks
      6. About administering the coordination point server
        1.  
          CP server operations (cpsadm)
        2.  
          Cloning a CP server
        3.  
          Adding and removing VCS cluster entries from the CP server database
        4.  
          Adding and removing a VCS cluster node from the CP server database
        5.  
          Adding or removing CP server users
        6.  
          Listing the CP server users
        7.  
          Listing the nodes in all the VCS clusters
        8.  
          Listing the membership of nodes in the VCS cluster
        9.  
          Preempting a node
        10.  
          Registering and unregistering a node
        11.  
          Enable and disable access for a user to a VCS cluster
        12.  
          Starting and stopping CP server outside VCS control
        13.  
          Checking the connectivity of CP servers
        14.  
          Adding and removing virtual IP addresses and ports for CP servers at run-time
        15.  
          Taking a CP server database snapshot
        16.  
          Replacing coordination points for server-based fencing in an online cluster
        17.  
          Refreshing registration keys on the coordination points for server-based fencing
        18. About configuring a CP server to support IPv6 or dual stack
          1.  
            Configuring a new CP server to support pure IPv6
          2.  
            Configuring an existing CP server to support IPv6 or dual stack
        19.  
          Deployment and migration scenarios for CP server
      7. About migrating between disk-based and server-based fencing configurations
        1.  
          Migrating from disk-based to server-based fencing in an online cluster
        2.  
          Migrating from server-based to disk-based fencing in an online cluster
        3. Migrating between fencing configurations using response files
          1.  
            Sample response file to migrate from disk-based to server-based fencing
          2.  
            Sample response file to migrate from server-based fencing to disk-based fencing
          3.  
            Sample response file to migrate from single CP server-based fencing to server-based fencing
          4.  
            Response file variables to migrate between fencing configurations
      8.  
        Enabling or disabling the preferred fencing policy
      9.  
        About I/O fencing log files
    3. Controlling VCS behavior
      1. VCS behavior on resource faults
        1.  
          Critical and non-critical resources
        2. VCS behavior diagrams
          1.  
            Example scenario 1: Resource with critical parent faults
          2.  
            Example scenario 2: Resource with non-critical parent faults
          3.  
            Example scenario 3: Resource with critical parent fails to come online
          4.  
            Example scenario 4: Resource with atleast resource dependency faults
      2. About controlling VCS behavior at the service group level
        1.  
          About the AutoRestart attribute
        2.  
          About controlling failover on service group or system faults
        3.  
          About defining failover policies
        4. About AdaptiveHA
          1.  
            Enabling AdaptiveHA for a service group
          2.  
            Considerations for setting FailOverPolicy to BiggestAvailable
          3.  
            Limitations on AdaptiveHA
          4.  
            Manually upgrading the VCS configuration file to the latest version
        5.  
          About system zones
        6.  
          About sites
        7.  
          Load-based autostart
        8.  
          About freezing service groups
        9.  
          About controlling Clean behavior on resource faults
        10.  
          Clearing resources in the ADMIN_WAIT state
        11.  
          About controlling fault propagation
        12. Customized behavior diagrams
          1.  
            Example scenario: Resource with a critical parent and ManageFaults=NONE
          2.  
            Example scenario: Resource with a critical parent and FaultPropagation=0
        13. About preventing concurrency violation
          1.  
            Enabling or preventing resources to start outside VCS control
          2.  
            Limitations of ProPCV
        14. VCS behavior for resources that support the intentional offline functionality
          1.  
            About the IntentionalOffline attribute
          2.  
            About the ExternalStateChange attribute
        15. VCS behavior when a service group is restarted
          1.  
            VCS behavior when non-persistent resources restart
          2.  
            VCS behavior when persistent resources transition from faulted to online
      3. About controlling VCS behavior at the resource level
        1. Resource type attributes that control resource behavior
          1.  
            About the RestartLimit attribute
          2.  
            About the OnlineRetryLimit attribute
          3.  
            About the ConfInterval attribute
          4.  
            About the ToleranceLimit attribute
          5.  
            About the FaultOnMonitorTimeouts attribute
        2. How VCS handles resource faults
          1.  
            VCS behavior when an online resource faults
          2.  
            VCS behavior when a resource fails to come online
        3.  
          VCS behavior after a resource is declared faulted
        4. VCS behavior when a resource is restarted
          1.  
            VCS behavior when online agent function fails to bring a resource online
          2.  
            VCS behavior when a resource goes offline
        5. About disabling resources
          1.  
            When to disable a resource
          2.  
            Limitations of disabling resources
          3.  
            Additional considerations for disabling resources
          4.  
            How disabled resources affect group states
      4.  
        Changing agent file paths and binaries
      5. VCS behavior on loss of storage connectivity
        1.  
          Disk group configuration and VCS behavior
        2.  
          How VCS attributes control behavior on loss of storage connectivity
        3.  
          VCS behavior when a disk group is disabled
        4.  
          Recommendations to ensure application availability
      6. Service group workload management
        1.  
          About enabling service group workload management
        2. System capacity and service group load
          1.  
            Static load versus dynamic load
          2.  
            About overload warning
        3.  
          System limits and service group prerequisites
        4.  
          About capacity and limits
      7. Sample configurations depicting workload management
        1.  
          System and Service group definitions
        2. Sample configuration: Basic four-node cluster
          1.  
            About AutoStart operation
          2.  
            About the failure scenario
          3.  
            About the cascading failure scenario
        3. Sample configuration: Complex four-node cluster
          1.  
            About the AutoStart operation
          2.  
            About the normal operation
          3.  
            About the failure scenario
          4.  
            About the cascading failure scenario
        4. Sample configuration: Server consolidation
          1.  
            About the AutoStart operation
          2.  
            About the normal operation
          3.  
            About the failure scenario
          4.  
            About the cascading failure scenario
    4. The role of service group dependencies
      1. About service group dependencies
        1. About dependency links
          1.  
            Dependency categories
          2.  
            Dependency location
          3.  
            Dependency rigidity
        2.  
          About dependency limitations
      2. Service group dependency configurations
        1. About failover parent / failover child
          1.  
            About failover parent / parallel child
          2.  
            About parallel parent / failover child
          3.  
            About parallel parent / parallel child
      3.  
        Frequently asked questions about group dependencies
      4.  
        About linking service groups
      5. About linking multiple child service groups
        1.  
          Dependencies supported for multiple child service groups
        2.  
          Dependencies not supported for multiple child service groups
      6. VCS behavior with service group dependencies
        1.  
          Online operations in group dependencies
        2.  
          Offline operations in group dependencies
        3.  
          Switch operations in group dependencies
  4. Section IV. Administration - Beyond the basics
    1. VCS event notification
      1. About VCS event notification
        1.  
          Event messages and severity levels
        2.  
          About persistent and replicated message queue
        3.  
          How HAD deletes messages
      2. Components of VCS event notification
        1. About the notifier process
          1.  
            Example of notifier command
        2. About the hanotify utility
          1.  
            Example of hanotify command
      3. About VCS events and traps
        1.  
          Events and traps for clusters
        2.  
          Events and traps for agents
        3.  
          Events and traps for resources
        4.  
          Events and traps for systems
        5.  
          Events and traps for service groups
        6.  
          SNMP-specific files
        7. Trap variables in VCS MIB
          1.  
            About severityId
          2.  
            EntityType and entitySubType
          3.  
            About entityState
      4. About monitoring aggregate events
        1.  
          How to detect service group failover
        2.  
          How to detect service group switch
      5.  
        About configuring notification
    2. VCS event triggers
      1.  
        About VCS event triggers
      2. Using event triggers
        1.  
          Performing multiple actions using a trigger
      3. List of event triggers
        1.  
          About the dumptunables trigger
        2.  
          About the globalcounter_not_updated trigger
        3.  
          About the injeopardy event trigger
        4.  
          About the loadwarning event trigger
        5.  
          About the nofailover event trigger
        6.  
          About the postoffline event trigger
        7.  
          About the postonline event trigger
        8.  
          About the preonline event trigger
        9.  
          About the resadminwait event trigger
        10.  
          About the resfault event trigger
        11.  
          About the resnotoff event trigger
        12.  
          About the resrestart event trigger
        13.  
          About the resstatechange event trigger
        14.  
          About the sysoffline event trigger
        15.  
          About the sysup trigger
        16.  
          About the sysjoin trigger
        17.  
          About the unable_to_restart_agent event trigger
        18.  
          About the unable_to_restart_had event trigger
        19.  
          About the violation event trigger
    3. Virtual Business Services
      1.  
        About Virtual Business Services
      2.  
        Features of Virtual Business Services
      3.  
        Sample virtual business service configuration
      4.  
        About choosing between VCS and VBS level dependencies
  5. Section V. Veritas High Availability Configuration wizard
    1. Introducing the Veritas High Availability Configuration wizard
      1.  
        About the Veritas High Availability Configuration wizard
      2.  
        Launching the Veritas High Availability Configuration wizard
      3.  
        Typical VCS cluster configuration in a VMware virtual environment
      4.  
        Typical VCS cluster configuration in a physical environment
    2. Administering application monitoring from the Veritas High Availability view
      1. Administering application monitoring from the Veritas High Availability view
        1.  
          Understanding the Veritas High Availability view
        2.  
          To view the status of configured applications
        3.  
          To configure or unconfigure application monitoring
        4.  
          To start or stop applications
        5.  
          To suspend or resume application monitoring
        6.  
          To switch an application to another system
        7.  
          To add or remove a failover system
        8.  
          To clear Fault state
        9.  
          To resolve a held-up operation
        10.  
          To determine application state
        11.  
          To remove all monitoring configurations
        12.  
          To remove VCS cluster configurations
      2.  
        Administering application monitoring settings
  6. Section VI. Cluster configurations for disaster recovery
    1. Connecting clusters–Creating global clusters
      1.  
        How VCS global clusters work
      2. VCS global clusters: The building blocks
        1.  
          Visualization of remote cluster objects
        2.  
          About global service groups
        3. About global cluster management
          1.  
            About the wide-area connector process
          2.  
            About the wide-area heartbeat agent
          3.  
            Sample configuration for the wide-area heartbeat agent
        4. About serialization - The Authority attribute
          1.  
            About the Authority and AutoStart attributes
        5.  
          About resiliency and "Right of way"
        6.  
          VCS agents to manage wide-area failover
        7.  
          About the Steward process: Split-brain in two-cluster global clusters
        8.  
          Secure communication in global clusters
      3. Prerequisites for global clusters
        1.  
          Prerequisites for cluster setup
        2.  
          Prerequisites for application setup
        3.  
          Prerequisites for wide-area heartbeats
        4.  
          Prerequisites for ClusterService group
        5.  
          Prerequisites for replication setup
        6.  
          Prerequisites for clusters running in secure mode
      4.  
        About planning to set up global clusters
      5. Setting up a global cluster
        1.  
          Configuring application and replication for global cluster setup
        2. Configuring clusters for global cluster setup
          1.  
            Configuring global cluster components at the primary site
          2.  
            Installing and configuring VCS at the secondary site
          3.  
            Securing communication between the wide-area connectors
          4.  
            Configuring remote cluster objects
          5.  
            Configuring additional heartbeat links (optional)
          6.  
            Configuring the Steward process (optional)
        3. Configuring service groups for global cluster setup
          1.  
            Configuring VCS service group for VVR-based replication
        4.  
          Configuring a service group as a global service group
      6. About IPv6 support with global clusters
        1.  
          Prerequisites for configuring a global cluster to support IPv6
        2.  
          Migrating an InfoScale Availability cluster from IPv4 to IPv6 when Virtual IP (ClusterAddress) is configured
        3.  
          Migrating an InfoScale Availability cluster to IPv6 in a GCO deployment
      7. About cluster faults
        1.  
          About the type of failure
        2.  
          Switching the service group back to the primary
      8. About setting up a disaster recovery fire drill
        1. About creating and configuring the fire drill service group manually
          1.  
            Creating the fire drill service group
          2.  
            Linking the fire drill and replication service groups
          3.  
            Adding resources to the fire drill service group
          4.  
            Configuring the fire drill service group
          5.  
            Enabling the FireDrill attribute
        2. About configuring the fire drill service group using the Fire Drill Setup wizard
          1.  
            Running the fire drill setup wizard
          2.  
            About configuring local attributes in the fire drill service group
        3.  
          Verifying a successful fire drill
        4.  
          Scheduling a fire drill
      9.  
        Multi-tiered application support using the RemoteGroup agent in a global environment
      10. Test scenario for a multi-tiered environment
        1.  
          About the main.cf file for cluster 1
        2.  
          About the main.cf file for cluster 2
        3.  
          About the main.cf file for cluster 3
        4.  
          About the main.cf file for cluster 4
    2. Administering global clusters from the command line
      1.  
        About administering global clusters from the command line
      2. About global querying in a global cluster setup
        1.  
          Querying global cluster service groups
        2.  
          Querying resources across clusters
        3.  
          Querying systems
        4.  
          Querying clusters
        5.  
          Querying status
        6.  
          Querying heartbeats
      3.  
        Administering global service groups in a global cluster setup
      4.  
        Administering resources in a global cluster setup
      5. Administering clusters in global cluster setup
        1.  
          Managing cluster alerts in a global cluster setup
        2.  
          Changing the cluster name in a global cluster setup
        3.  
          Removing a remote cluster from a global cluster setup
      6.  
        Administering heartbeats in a global cluster setup
    3. Setting up replicated data clusters
      1.  
        About replicated data clusters
      2.  
        How VCS replicated data clusters work
      3. About setting up a replicated data cluster configuration
        1.  
          About typical replicated data cluster configuration
        2.  
          About setting up replication
        3.  
          Configuring the service groups
        4.  
          Configuring the service group dependencies
      4. About migrating a service group
        1.  
          Switching the service group
      5.  
        About setting up a fire drill
    4. Setting up campus clusters
      1.  
        About campus cluster configuration
      2.  
        VCS campus cluster requirements
      3.  
        Typical VCS campus cluster setup
      4. How VCS campus clusters work
        1.  
          About I/O fencing in campus clusters
      5. About setting up a campus cluster configuration
        1.  
          Preparing to set up a campus cluster configuration
        2.  
          Configuring I/O fencing to prevent data corruption
        3.  
          Configuring VxVM disk groups for campus cluster configuration
        4.  
          Configuring VCS service group for campus clusters
      6.  
        Fire drill in campus clusters
      7.  
        About the DiskGroupSnap agent
      8. About running a fire drill in a campus cluster
        1.  
          Configuring the fire drill service group
        2.  
          Running a successful fire drill in a campus cluster
  7. Section VII. Troubleshooting and performance
    1. VCS performance considerations
      1. How cluster components affect performance
        1.  
          How kernel components (GAB and LLT) affect performance
        2.  
          How the VCS engine (HAD) affects performance
        3. How agents affect performance
          1.  
            Monitoring resource type and agent configuration
        4.  
          How the VCS graphical user interfaces affect performance
      2. How cluster operations affect performance
        1.  
          VCS performance consideration when booting a cluster system
        2.  
          VCS performance consideration when a resource comes online
        3.  
          VCS performance consideration when a resource goes offline
        4.  
          VCS performance consideration when a service group comes online
        5.  
          VCS performance consideration when a service group goes offline
        6.  
          VCS performance consideration when a resource fails
        7.  
          VCS performance consideration when a system fails
        8.  
          VCS performance consideration when a network link fails
        9. VCS performance consideration when a system panics
          1.  
            About GAB client process failure
          2.  
            About GAB client registration monitoring
          3.  
            About network failure and GAB IOFENCE message
          4.  
            About quick reopen
        10.  
          VCS performance consideration when a service group switches over
        11.  
          VCS performance consideration when a service group fails over
      3. About scheduling class and priority configuration
        1.  
          About priority ranges
        2.  
          Default scheduling classes and priorities
      4. VCS agent statistics
        1.  
          Tracking monitor cycle times
        2.  
          VCS attributes enabling agent statistics
      5. About VCS tunable parameters
        1. About LLT tunable parameters
          1.  
            About LLT timer tunable parameters
          2.  
            About LLT flow control tunable parameters
          3.  
            Setting LLT timer tunable parameters
        2. About GAB tunable parameters
          1.  
            About GAB load-time or static tunable parameters
          2.  
            About GAB run-time or dynamic tunable parameters
        3. About VXFEN tunable parameters
          1.  
            Configuring the VXFEN module parameters
        4.  
          About AMF tunable parameters
    2. Troubleshooting and recovery for VCS
      1. VCS message logging
        1.  
          Log unification of VCS agent's entry points
        2.  
          Enhancing First Failure Data Capture (FFDC) to troubleshoot VCS resource's unexpected behavior
        3.  
          GAB message logging
        4.  
          Enabling debug logs for agents
        5.  
          Enabling debug logs for IMF
        6.  
          Enabling debug logs for the VCS engine
        7.  
          Enable VCS logging for VxAT
        8.  
          About debug log tags usage
        9. Gathering VCS information for support analysis
          1.  
            Verifying the metered or forecasted values for CPU, Mem, and Swap
        10.  
          Gathering LLT and GAB information for support analysis
        11.  
          Gathering IMF information for support analysis
        12.  
          Message catalogs
      2. Troubleshooting the VCS engine
        1.  
          HAD diagnostics
        2.  
          HAD restarts continuously
        3.  
          DNS configuration issues cause GAB to kill HAD
        4.  
          Seeding and I/O fencing
        5.  
          Preonline IP check
      3. Troubleshooting Low Latency Transport (LLT)
        1.  
          LLT startup script displays errors
        2.  
          LLT detects cross links usage
        3.  
          LLT link status messages
        4.  
          Cluster does not form after installing and starting VCS or SFHA
      4. Troubleshooting Group Membership Services/Atomic Broadcast (GAB)
        1.  
          Delay in port reopen
        2.  
          Node panics due to client process failure
      5. Troubleshooting VCS startup
        1.  
          "VCS:10622 local configuration missing"
        2.  
          "VCS:10623 local configuration invalid"
        3.  
          "VCS:11032 registration failed. Exiting"
        4.  
          "Waiting for cluster membership."
      6. Troubleshooting issues with systemd unit service files
        1.  
          If a unit service has failed and the corresponding module is still loaded, systemd cannot unload it and so its package cannot be removed
        2.  
          If a unit service is active and the corresponding process is stopped outside of systemd, the service cannot be started again using 'systemctl start'
        3.  
          If a unit service takes longer than the default timeout to stop or start the corresponding service, it goes into the Failed state
      7.  
        Troubleshooting Intelligent Monitoring Framework (IMF)
      8. Troubleshooting service groups
        1.  
          VCS does not automatically start service group
        2.  
          System is not in RUNNING state
        3.  
          Service group not configured to run on the system
        4.  
          Service group not configured to autostart
        5.  
          Service group is frozen
        6.  
          Failover service group is online on another system
        7.  
          A critical resource faulted
        8.  
          Service group autodisabled
        9.  
          Service group is waiting for the resource to be brought online/taken offline
        10.  
          Service group is waiting for a dependency to be met.
        11.  
          Service group not fully probed.
        12.  
          Service group does not fail over to the forecasted system
        13.  
          Service group does not fail over to the BiggestAvailable system even if FailOverPolicy is set to BiggestAvailable
        14.  
          Restoring metering database from backup taken by VCS
        15.  
          Initialization of metering database fails
        16.  
          Error message appears during service group failover or switch
      9. Troubleshooting resources
        1.  
          Service group brought online due to failover
        2.  
          Waiting for service group states
        3.  
          Waiting for child resources
        4.  
          Waiting for parent resources
        5.  
          Waiting for resource to respond
        6. Agent not running
          1.  
            Invalid agent argument list.
        7.  
          The Monitor entry point of the disk group agent returns ONLINE even if the disk group is disabled
      10. Troubleshooting sites
        1.  
          Online propagate operation was initiated but service group failed to be online
        2.  
          VCS panics nodes in the preferred site during a network-split
        3.  
          Configuring of stretch site fails
        4.  
          Renaming a Site
      11. Troubleshooting I/O fencing
        1.  
          Node is unable to join cluster while another node is being ejected
        2.  
          The vxfentsthdw utility fails when SCSI TEST UNIT READY command fails
        3.  
          Manually removing existing keys from SCSI-3 disks
        4.  
          System panics to prevent potential data corruption
        5.  
          Cluster ID on the I/O fencing key of coordinator disk does not match the local cluster's ID
        6. Fencing startup reports preexisting split-brain
          1.  
            Clearing preexisting split-brain condition
        7.  
          Registered keys are lost on the coordinator disks
        8.  
          Replacing defective disks when the cluster is offline
        9.  
          The vxfenswap utility exits if rcp or scp commands are not functional
        10. Troubleshooting CP server
          1.  
            Troubleshooting issues related to the CP server service group
          2.  
            Checking the connectivity of CP server
        11. Troubleshooting server-based fencing on the VCS cluster nodes
          1.  
            Issues during fencing startup on VCS cluster nodes set up for server-based fencing
        12. Issues during online migration of coordination points
          1.  
            Vxfen service group activity after issuing the vxfenswap command
      12. Troubleshooting notification
        1.  
          Notifier is configured but traps are not seen on SNMP console.
      13. Troubleshooting and recovery for global clusters
        1.  
          Disaster declaration
        2.  
          Lost heartbeats and the inquiry mechanism
        3. VCS alerts
          1.  
            Types of alerts
          2.  
            Managing alerts
          3.  
            Actions associated with alerts
          4.  
            Negating events
          5.  
            Concurrency violation at startup
      14.  
        Troubleshooting the steward process
      15. Troubleshooting licensing
        1.  
          Validating license keys
        2. Licensing error messages
          1.  
            [Licensing] Insufficient memory to perform operation
          2.  
            [Licensing] No valid VCS license keys were found
          3.  
            [Licensing] Unable to find a valid base VCS license key
          4.  
            [Licensing] License key cannot be used on this OS platform
          5.  
            [Licensing] VCS evaluation period has expired
          6.  
            [Licensing] License key can not be used on this system
          7.  
            [Licensing] Unable to initialize the licensing framework
          8.  
            [Licensing] QuickStart is not supported in this release
          9.  
            [Licensing] Your evaluation period for the feature has expired. This feature will not be enabled the next time VCS starts
      16. Troubleshooting secure configurations
        1.  
          FIPS mode cannot be set
        2.  
          Broker does not start
        3.  
          AT initialization fails
      17. Troubleshooting wizard-based configuration issues
        1.  
          Running the 'hastop -all' command detaches virtual disks
      18. Troubleshooting issues with the Veritas High Availability view
        1.  
          Veritas High Availability view does not display the application monitoring status
        2.  
          Veritas High Availability view may freeze due to special characters in application display name
        3.  
          In the Veritas High Availability tab, the Add Failover System link is dimmed
  8. Section VIII. Appendixes
    1. Appendix A. VCS user privileges—administration matrices
      1.  
        About administration matrices
      2. Administration matrices
        1.  
          Agent Operations (haagent)
        2.  
          Attribute Operations (haattr)
        3.  
          Cluster Operations (haclus, haconf)
        4.  
          Service group operations (hagrp)
        5.  
          Heartbeat operations (hahb)
        6.  
          Log operations (halog)
        7.  
          Resource operations (hares)
        8.  
          System operations (hasys)
        9.  
          Resource type operations (hatype)
        10.  
          User operations (hauser)
    2. Appendix B. VCS commands: Quick reference
      1.  
        About this quick reference for VCS commands
      2.  
        VCS command line reference
    3. Appendix C. Cluster and system states
      1. Remote cluster states
        1.  
          Examples of cluster state transitions
      2. System states
        1.  
          Examples of system state transitions
    4. Appendix D. VCS attributes
      1.  
        About attributes and their definitions
      2.  
        Resource attributes
      3.  
        Resource type attributes
      4.  
        Service group attributes
      5.  
        System attributes
      6.  
        Cluster attributes
      7.  
        Heartbeat attributes (for global clusters)
      8.  
        Remote cluster attributes
      9.  
        Site attributes
    5. Appendix E. Accessibility and VCS
      1.  
        About accessibility in VCS
      2. Navigation and keyboard shortcuts
        1.  
          Navigation in the Java Console
        2.  
          Navigation in the Web console
      3.  
        Support for accessibility settings
      4.  
        Support for assistive technologies

How membership arbitration works

Upon startup of the cluster, all systems register a unique key on the coordinator disks. The key is unique to the cluster and the node, and is based on the LLT cluster ID and the LLT system ID.

See About the I/O fencing registration key format.

When there is a perceived change in membership, membership arbitration works as follows:

  • GAB marks the system as DOWN, excludes the system from the cluster membership, and delivers the membership change - the list of departed systems - to the fencing module.

  • The system with the lowest LLT system ID in the cluster races for control of the coordinator disks

    • In the most common case, where departed systems are truly down or faulted, this race has only one contestant.

    • In a split brain scenario, where two or more subclusters have formed, the race for the coordinator disks is performed by the system with the lowest LLT system ID of that subcluster. This system that races on behalf of all the other systems in its subcluster is called the RACER node and the other systems in the subcluster are called the SPECTATOR nodes.

  • During the I/O fencing race, if the RACER node panics or if it cannot reach the coordination points, then the VxFEN RACER node re-election feature allows an alternate node in the subcluster that has the next lowest node ID to take over as the RACER node.

    The racer re-election works as follows:

    • In the event of an unexpected panic of the RACER node, the VxFEN driver initiates a racer re-election.

    • If the RACER node is unable to reach a majority of coordination points, then the VxFEN module sends a RELAY_RACE message to the other nodes in the subcluster. The VxFEN module then re-elects the next lowest node ID as the new RACER.

    • With successive re-elections if no more nodes are available to be re-elected as the RACER node, then all the nodes in the subcluster will panic.

  • The race consists of executing a preempt and abort command for each key of each system that appears to no longer be in the GAB membership.

    The preempt and abort command allows only a registered system with a valid key to eject the key of another system. This ensures that even when multiple systems attempt to eject other, each race will have only one winner. The first system to issue a preempt and abort command will win and eject the key of the other system. When the second system issues a preempt and abort command, it cannot perform the key eject because it is no longer a registered system with a valid key.

    If the value of the cluster-level attribute PreferredFencingPolicy is System, Group, or Site then at the time of a race, the VxFEN Racer node adds up the weights for all nodes in the local subcluster and in the leaving subcluster. If the leaving partition has a higher sum (of node weights) then the racer for this partition will delay the race for the coordination point. This effectively gives a preference to the more critical subcluster to win the race. If the value of the cluster-level attribute PreferredFencingPolicy is Disabled, then the delay will be calculated, based on the sums of node counts.

    See About preferred fencing.

  • If the preempt and abort command returns success, that system has won the race for that coordinator disk.

    Each system will repeat this race to all the coordinator disks. The race is won by, and control is attained by, the system that ejects the other system's registration keys from a majority of the coordinator disks.

  • On the system that wins the race, the vxfen module informs all the systems that it was racing on behalf of that it won the race, and that subcluster is still valid.

  • On the system(s) that do not win the race, the vxfen module will trigger a system panic. The other systems in this subcluster will note the panic, determine they lost control of the coordinator disks, and also panic and restart.

  • Upon restart, the systems will attempt to seed into the cluster.

    • If the systems that restart can exchange heartbeat with the number of cluster systems declared in /etc/gabtab, they will automatically seed and continue to join the cluster. Their keys will be replaced on the coordinator disks. This case will only happen if the original reason for the membership change has cleared during the restart.

    • If the systems that restart cannot exchange heartbeat with the number of cluster systems declared in /etc/gabtab, they will not automatically seed, and HAD will not start. This is a possible split brain condition, and requires administrative intervention.

    • If you have I/O fencing enabled in your cluster and if you have set the GAB auto-seeding feature through I/O fencing, GAB automatically seeds the cluster even when some cluster nodes are unavailable.

      See Seeding a cluster using the GAB auto-seed parameter through I/O fencing.

    Note:

    Forcing a manual seed at this point will allow the cluster to seed. However, when the fencing module checks the GAB membership against the systems that have keys on the coordinator disks, a mismatch will occur. vxfen will detect a possible split brain condition, print a warning, and will not start. In turn, HAD will not start. Administrative intervention is required.

    See Manual seeding of a cluster.