Storage Foundation for Oracle® RAC 7.3.1 Administrator's Guide - Linux

Last Published:
Product(s): InfoScale & Storage Foundation (7.3.1)
  1. Section I. SF Oracle RAC concepts and administration
    1. Overview of Storage Foundation for Oracle RAC
      1. About Storage Foundation for Oracle RAC
        1.  
          Benefits of SF Oracle RAC
      2.  
        How SF Oracle RAC works (high-level perspective)
      3. Component products and processes of SF Oracle RAC
        1. Communication infrastructure
          1.  
            Data flow
          2.  
            Communication requirements
        2. Cluster interconnect communication channel
          1. About Low Latency Transport (LLT)
            1.  
              How LLT supports RDMA capability for faster interconnects between applications
          2.  
            Group Membership Services/Atomic Broadcast
        3.  
          Low-level communication: port relationship between GAB and processes
        4. Cluster Volume Manager (CVM)
          1.  
            CVM architecture
          2.  
            CVM communication
          3.  
            CVM recovery
          4.  
            Configuration differences with VxVM
          5. About Flexible Storage Sharing
            1.  
              Flexible Storage Sharing use cases
            2.  
              Limitations of Flexible Storage Sharing
        5. Cluster File System (CFS)
          1.  
            CFS architecture
          2.  
            CFS communication
          3.  
            CFS file system benefits
          4.  
            CFS configuration differences
          5.  
            CFS recovery
          6.  
            Comparing raw volumes and CFS for data files
        6. Cluster Server (VCS)
          1.  
            VCS architecture
          2. VCS communication
            1.  
              About the IMF notification module
          3. About resource monitoring
            1.  
              How intelligent resource monitoring works
          4.  
            Cluster configuration files
        7.  
          About I/O fencing
        8. Oracle RAC components
          1. Oracle Clusterware/Grid Infrastructure
            1.  
              Co-existence with VCS
          2.  
            Oracle Cluster Registry
          3. Application resources
            1.  
              Resource profiles
            2.  
              Oracle Clusterware/Grid Infrastructure node applications
            3.  
              Voting disk
        9. Oracle Disk Manager
          1.  
            ODM architecture
          2.  
            Veritas ODM performance enhancements
          3.  
            ODM communication
        10. RAC extensions
          1.  
            Veritas Cluster Server membership manager
          2.  
            Oracle and cache fusion traffic
      4.  
        Periodic health evaluation of SF Oracle RAC clusters
      5. About Virtual Business Services
        1.  
          Features of Virtual Business Services
        2.  
          Sample virtual business service configuration
      6.  
        About Veritas InfoScale Operations Manager
      7.  
        About Veritas Services and Operations Readiness Tools (SORT)
    2. Administering SF Oracle RAC and its components
      1. Administering SF Oracle RAC
        1.  
          Setting the environment variables for SF Oracle RAC
        2. Starting or stopping SF Oracle RAC on each node
          1.  
            Starting SF Oracle RAC using the script-based installer
          2.  
            Starting SF Oracle RAC manually on each node
          3.  
            Stopping SF Oracle RAC using the script-based installer
          4.  
            Stopping SF Oracle RAC manually on each node
        3.  
          Applying Oracle patches on SF Oracle RAC nodes
        4.  
          Migrating Pluggable Databases (PDB) between Container Databases (CDB)
        5.  
          Installing Veritas Volume Manager, Veritas File System, or ODM patches on SF Oracle RAC nodes
        6.  
          Applying operating system updates on SF Oracle RAC nodes
        7.  
          Adding storage to an SF Oracle RAC cluster
        8.  
          Recovering from storage failure
        9.  
          Backing up and restoring Oracle database using Veritas NetBackup
        10.  
          Enhancing the performance of SF Oracle RAC clusters
        11.  
          Administering SmartIO
        12.  
          Creating snapshots for offhost processing
        13.  
          Managing database storage efficiently using SmartTier
        14.  
          Optimizing database storage using Thin Provisioning and SmartMove
        15.  
          Scheduling periodic health checks for your SF Oracle RAC cluster
        16.  
          Using environment variables to start and stop VCSMM modules
        17.  
          Verifying the nodes in an SF Oracle RAC cluster
      2. Administering VCS
        1.  
          About managing VCS modules
        2.  
          Viewing available Veritas device drivers
        3.  
          Starting and stopping VCS
        4.  
          Environment variables to start and stop VCS modules
        5.  
          Adding and removing LLT links
        6.  
          Configuring aggregated interfaces under LLT
        7.  
          Displaying the cluster details and LLT version for LLT links
        8.  
          Configuring destination-based load balancing for LLT
        9.  
          Enabling and disabling intelligent resource monitoring for agents manually
        10.  
          Administering the AMF kernel driver
      3. Administering I/O fencing
        1.  
          About administering I/O fencing
        2. About the vxfentsthdw utility
          1.  
            General guidelines for using the vxfentsthdw utility
          2.  
            About the vxfentsthdw command options
          3. Testing the coordinator disk group using the -c option of vxfentsthdw
            1.  
              Removing and replacing a failed disk
          4.  
            Performing non-destructive testing on the disks using the -r option
          5.  
            Testing the shared disks using the vxfentsthdw -m option
          6.  
            Testing the shared disks listed in a file using the vxfentsthdw -f option
          7.  
            Testing all the disks in a disk group using the vxfentsthdw -g option
          8.  
            Testing a disk with existing keys
        3. About the vxfenadm utility
          1.  
            About the I/O fencing registration key format
          2.  
            Displaying the I/O fencing registration keys
          3.  
            Verifying that the nodes see the same disk
        4. About the vxfenclearpre utility
          1.  
            Removing preexisting keys
        5. About the vxfenswap utility
          1.  
            Replacing I/O fencing coordinator disks when the cluster is online
          2.  
            Replacing the coordinator disk group in a cluster that is online
          3.  
            Adding disks from a recovered site to the coordinator disk group
          4.  
            Refreshing lost keys on coordinator disks
        6.  
          Enabling or disabling the preferred fencing policy
        7.  
          About I/O fencing log files
        8.  
          Migrating from disk-based fencing to server-based fencing using the installer
        9.  
          Migrating from server-based fencing to disk-based fencing using the installer
      4. Administering the CP server
        1.  
          Refreshing registration keys on the coordination points for server-based fencing
        2.  
          Replacing coordination points for server-based fencing in an online cluster
        3.  
          Migrating from non-secure to secure setup for CP server and SF Oracle RAC cluster communication
      5. Administering CFS
        1.  
          Adding CFS file systems to a VCS configuration
        2.  
          Resizing CFS file systems
        3.  
          Verifying the status of CFS file system nodes and their mount points
      6. Administering CVM
        1.  
          Listing all the CVM shared disks
        2.  
          Establishing CVM cluster membership manually
        3. Changing the CVM master manually
          1.  
            Considerations for changing the master manually
          2.  
            Errors during CVM master switching
        4.  
          Importing a shared disk group manually
        5.  
          Deporting a shared disk group manually
        6.  
          Starting shared volumes manually
        7.  
          Verifying if CVM is running in an SF Oracle RAC cluster
        8.  
          Verifying CVM membership state
        9.  
          Verifying the state of CVM shared disk groups
        10.  
          Verifying the activation mode
      7. Administering Flexible Storage Sharing
        1.  
          About Flexible Storage Sharing disk support
        2.  
          About the volume layout for Flexible Storage Sharing disk groups
        3.  
          Setting the host prefix
        4.  
          Exporting a disk for Flexible Storage Sharing
        5.  
          Setting the Flexible Storage Sharing attribute on a disk group
        6.  
          Using the host disk class and allocating storage
        7.  
          Administering mirrored volumes using vxassist
        8.  
          Displaying exported disks and network shared disk groups
        9.  
          Tuning LLT for memory and performance in FSS environments
        10. Backing up and restoring disk group configuration data
          1.  
            Backing up and restoring Flexible Storage Sharing disk group configuration data
      8. Administering SF Oracle RAC global clusters
        1.  
          About setting up a disaster recovery fire drill
        2. About configuring the fire drill service group using the Fire Drill Setup wizard
          1.  
            Running the fire drill setup wizard
          2.  
            About configuring local attributes in the fire drill service group
        3.  
          Verifying a successful fire drill
        4.  
          Scheduling a fire drill
        5.  
          Sample fire drill service group configuration
  2. Section II. Performance and troubleshooting
    1. Troubleshooting SF Oracle RAC
      1. About troubleshooting SF Oracle RAC
        1. Gathering information from an SF Oracle RAC cluster for support analysis
          1.  
            Gathering configuration information using SORT Data Collector
          2.  
            Gathering SF Oracle RAC information for support analysis
          3.  
            Gathering VCS information for support analysis
          4.  
            Gathering LLT and GAB information for support analysis
          5.  
            Gathering IMF information for support analysis
        2. SF Oracle RAC log files
          1.  
            Collecting important CVM logs
        3.  
          About SF Oracle RAC kernel and driver messages
        4. VCS message logging
          1.  
            GAB message logging
          2.  
            About debug log tags usage
          3.  
            Enabling debug logs for agents
          4.  
            Enabling debug logs for the VCS engine
          5.  
            Enabling debug logs for IMF
          6.  
            Message catalogs
      2.  
        Restarting the installer after a failed network connection
      3.  
        Installer cannot create UUID for the cluster
      4.  
        Troubleshooting SF Oracle RAC pre-installation check failures
      5.  
        Troubleshooting LLT health check warning messages
      6. Troubleshooting I/O fencing
        1.  
          SCSI reservation errors during bootup
        2.  
          The vxfentsthdw utility fails when SCSI TEST UNIT READY command fails
        3.  
          Node is unable to join cluster while another node is being ejected
        4.  
          System panics to prevent potential data corruption
        5.  
          Cluster ID on the I/O fencing key of coordinator disk does not match the local cluster's ID
        6. Fencing startup reports preexisting split-brain
          1.  
            Clearing preexisting split-brain condition
        7.  
          Registered keys are lost on the coordinator disks
        8.  
          Replacing defective disks when the cluster is offline
        9.  
          Troubleshooting I/O fencing health check warning messages
        10. Troubleshooting CP server
          1.  
            Troubleshooting issues related to the CP server service group
          2.  
            Checking the connectivity of CP server
        11. Troubleshooting server-based fencing on the SF Oracle RAC cluster nodes
          1.  
            Issues during fencing startup on SF Oracle RAC cluster nodes set up for server-based fencing
        12. Issues during online migration of coordination points
          1.  
            Vxfen service group activity after issuing the vxfenswap command
      7. Troubleshooting Cluster Volume Manager in SF Oracle RAC clusters
        1.  
          Restoring communication between host and disks after cable disconnection
        2.  
          Shared disk group cannot be imported in SF Oracle RAC cluster
        3.  
          Error importing shared disk groups in SF Oracle RAC cluster
        4.  
          Unable to start CVM in SF Oracle RAC cluster
        5.  
          CVM group is not online after adding a node to the SF Oracle RAC cluster
        6.  
          CVMVolDg not online even though CVMCluster is online in SF Oracle RAC cluster
        7.  
          Shared disks not visible in SF Oracle RAC cluster
      8. Troubleshooting CFS
        1.  
          Incorrect order in root user's <library> path
      9. Troubleshooting interconnects
        1.  
          Example entries for mandatory devices
      10. Troubleshooting Oracle
        1.  
          Error when starting an Oracle instance in SF Oracle RAC
        2.  
          Clearing Oracle group faults
        3.  
          Oracle log files show shutdown called even when not shutdown manually
        4.  
          DBCA fails while creating an Oracle RAC database
        5.  
          Oracle's clusterware processes fail to start
        6.  
          Oracle Clusterware fails after restart
        7.  
          Troubleshooting the Virtual IP (VIP) configuration in an SF Oracle RAC cluster
        8.  
          Troubleshooting Oracle Clusterware health check warning messages in SF Oracle RAC clusters
      11. Troubleshooting ODM in SF Oracle RAC clusters
        1.  
          File System configured incorrectly for ODM shuts down Oracle
      12.  
        Troubleshooting Flex ASM in SF Oracle RAC clusters
    2. Prevention and recovery strategies
      1.  
        Verification of GAB ports in SF Oracle RAC cluster
      2.  
        Examining GAB seed membership
      3.  
        Manual GAB membership seeding
      4.  
        Evaluating VCS I/O fencing ports
      5.  
        Verifying normal functioning of VCS I/O fencing
      6. Managing SCSI-3 PR keys in SF Oracle RAC cluster
        1.  
          Evaluating the number of SCSI-3 PR keys on a coordinator LUN, if there are multiple paths to the LUN from the hosts
        2.  
          Detecting accidental SCSI-3 PR key removal from coordinator LUNs
      7.  
        Identifying a faulty coordinator LUN
    3. Tunable parameters
      1.  
        About SF Oracle RAC tunable parameters
      2. About GAB tunable parameters
        1.  
          About GAB load-time or static tunable parameters
        2.  
          About GAB run-time or dynamic tunable parameters
      3. About LLT tunable parameters
        1.  
          About LLT timer tunable parameters
        2.  
          About LLT flow control tunable parameters
        3.  
          Setting LLT timer tunable parameters
      4. About VXFEN tunable parameters
        1.  
          Configuring the VXFEN module parameters
      5.  
        Tuning guidelines for campus clusters
  3. Section III. Reference
    1. Appendix A. List of SF Oracle RAC health checks
      1.  
        LLT health checks
      2.  
        I/O fencing health checks
      3.  
        PrivNIC health checks in SF Oracle RAC clusters
      4.  
        Oracle Clusterware health checks in SF Oracle RAC clusters
      5.  
        CVM, CFS, and ODM health checks in SF Oracle RAC clusters
    2. Appendix B. Error messages
      1.  
        About error messages
      2.  
        VxVM error messages
      3. VXFEN driver error messages
        1.  
          VXFEN driver informational message
        2.  
          Node ejection informational messages

About LLT timer tunable parameters

Table: LLT timer tunable parameters lists the LLT timer tunable parameters. The timer values are set in .01 sec units. The command lltconfig - T query can be used to display current timer values.

Table: LLT timer tunable parameters

LLT parameter

Description

Default

When to change

Dependency with other LLT tunable parameters

peerinact

LLT marks a link of a peer node as "inactive," if it does not receive any packet on that link for this timer interval. Once a link is marked as "inactive," LLT will not send any data on that link.

1600

  • Change this value for delaying or speeding up node/link inactive notification mechanism as per client's notification processing logic.

  • Increase the value for planned replacement of faulty network cable /switch.

  • In some circumstances, when the private networks links are very slow or the network traffic becomes very bursty, increase this value so as to avoid false notifications of peer death.

    Set the value to a high value for planned replacement of faulty network cable or faulty switch.

The timer value should always be higher than the peertrouble timer value.

rpeerinact

Mark RDMA channel of a RDMA link as "inactive", if the node does not receive any packet on that link for this timer interval. Once RDMA channel is marked as "inactive", LLT does not send any data on the RDMA channel of that link, however, it may continue to send data over non-RDMA channel of that link until peerinact expires. You can view the status of the RDMA channel of a RDMA link using lltstat -nvv -r command. This parameter is supported only on selected versions of Linux.

700

Decrease the value of this tunable for speeding up the RDMA link failure recovery. If the links are unstable, and they are going up and down frequently then do not decrease this value.

This timer value should always be greater than peertrouble timer value and less than peerinact value.

peertrouble

LLT marks a high-pri link of a peer node as "troubled", if it does not receive any packet on that link for this timer interval. Once a link is marked as "troubled", LLT will not send any data on that link till the link is up.

200

  • In some circumstances, when the private networks links are very slow or nodes in the cluster are very busy, increase the value.

  • Increase the value for planned replacement of faulty network cable /faulty switch.

This timer value should always be lower than peerinact timer value. Also, It should be close to its default value.

peertroublelo

LLT marks a low-pri link of a peer node as "troubled", if it does not receive any packet on that link for this timer interval. Once a link is marked as "troubled", LLT will not send any data on that link till the link is available.

400

  • In some circumstances, when the private networks links are very slow or nodes in the cluster are very busy, increase the value.

  • Increase the value for planned replacement of faulty network cable /faulty switch.

This timer value should always be lower than peerinact timer value. Also, It should be close to its default value.

heartbeat

LLT sends heartbeat packets repeatedly to peer nodes after every heartbeat timer interval on each highpri link.

50

In some circumstances, when the private networks links are very slow (or congested) or nodes in the cluster are very busy, increase the value.

This timer value should be lower than peertrouble timer value. Also, it should not be close to peertrouble timer value.

heartbeatlo

LLT sends heartbeat packets repeatedly to peer nodes after every heartbeatlo timer interval on each low pri link.

100

In some circumstances, when the networks links are very slow or nodes in the cluster are very busy, increase the value.

This timer value should be lower than peertroublelo timer value. Also, it should not be close to peertroublelo timer value.

timetoreqhb

If LLT does not receive any packet from the peer node on a particular link for "timetoreqhb" time period, it attempts to request heartbeats (sends 5 special heartbeat requests (hbreqs) to the peer node on the same link) from the peer node. If the peer node does not respond to the special heartbeat requests, LLT marks the link as "expired" for that peer node. The value can be set from the range of 0 to (peerinact -200). The value 0 disables the request heartbeat mechanism.

1400

Decrease the value of this tunable for speeding up node/link inactive notification mechanism as per client's notification processing logic.

Disable the request heartbeat mechanism by setting the value of this timer to 0 for planned replacement of faulty network cable /switch.

In some circumstances, when the private networks links are very slow or the network traffic becomes very bursty, don't change the value of this timer tunable.

This timer is set to 'peerinact - 200' automatically every time when the peerinact timer is changed.

reqhbtime

This value specifies the time interval between two successive special heartbeat requests. See the timetoreqhb parameter for more information on special heartbeat requests.

40

Veritas recommends that you do not change this value.

Not applicable

timetosendhb

LLT sends out of timer context heartbeats to keep the node alive when LLT timer does not run at regular interval. This option specifies the amount of time to wait before sending a heartbeat in case of timer not running.

If this timer tunable is set to 0, the out of timer context heartbeating mechanism is disabled.

200

Disable the out of timer context heart-beating mechanism by setting the value of this timer to 0 for planned replacement of faulty network cable /switch.

In some circumstances, when the private networks links are very slow or nodes in the cluster are very busy, increase the value

This timer value should not be more than peerinact timer value. Also, it should not be close to the peerinact timer value.

sendhbcap

This value specifies the maximum time for which LLT will send contiguous out of timer context heartbeats.

18000

Veritas recommends that you do not change this value.

NA

oos

If the out-of-sequence timer has expired for a node, LLT sends an appropriate NAK to that node. LLT does not send a NAK as soon as it receives an oos packet. It waits for the oos timer value before sending the NAK.

10

Do not change this value for performance reasons. Lowering the value can result in unnecessary retransmissions/negative acknowledgement traffic.

You can increase the value of oos if the round trip time is large in the cluster (for example, campus cluster).

Not applicable

retrans

LLT retransmits a packet if it does not receive its acknowledgement for this timer interval value.

10

Do not change this value. Lowering the value can result in unnecessary retransmissions.

You can increase the value of retrans if the round trip time is large in the cluster (for example, campus cluster).

Not applicable

service

LLT calls its service routine (which delivers messages to LLT clients) after every service timer interval.

100

Do not change this value for performance reasons.

Not applicable

arp

LLT flushes stored address of peer nodes when this timer expires and relearns the addresses.

0

This feature is disabled by default.

Not applicable

arpreq

LLT sends an arp request when this timer expires to detect other peer nodes in the cluster.

3000

Do not change this value for performance reasons.

Not applicable