Search <book_title>...

Veritas InfoScale™ 7.4 Installation Guide - Solaris

Last Published: 2019-02-26

Product(s): InfoScale & Storage Foundation (7.4)

Platform: Solaris

Setting up the private network

This topic applies to VCS, SFHA, SFCFS, SFCFSHA, SF Oracle RAC, and SF Sybase CE.

VCS requires you to set up a private network between the systems that form a cluster. You can use either NICs or aggregated interfaces to set up private network.

You can use network switches instead of hubs. However, Oracle Solaris systems assign the same MAC address to all interfaces by default. Thus, connecting two or more interfaces to a network switch can cause problems.

For example, consider the following case where:

The IP address is configured on one interface and LLT on another
Both interfaces are connected to a switch (assume separate VLANs)

The duplicate MAC address on the two switch ports can cause the switch to incorrectly redirect IP traffic to the LLT interface and vice versa. To avoid this issue, configure the system to assign unique MAC addresses by setting the eeprom(1M) parameter local-mac-address to true.

The following products make extensive use of the private cluster interconnects for distributed locking:

Storage Foundation Cluster File System (SFCFS)
Storage Foundation for Oracle RAC (SF Oracle RAC)

Veritas recommends network switches for the SFCFS and the SF Oracle RAC clusters due to their performance characteristics.

Refer to the Cluster Server Administrator's Guide to review VCS performance considerations.

Figure: Private network setups: two-node and four-node clusters shows two private networks for use with VCS.

Figure: Private network setups: two-node and four-node clusters

You need to configure at least two independent networks between the cluster nodes with a network switch for each network. You can also interconnect multiple layer 2 switches for advanced failure protection. Such connections for LLT are called cross-links.

Figure: Private network setup with crossed links shows a private network configuration with crossed links between the network switches.

Figure: Private network setup with crossed links

Veritas recommends one of the following two configurations:

Use at least two private interconnect links and one public link. The public link can be a low priority link for LLT. The private interconnect link is used to share cluster status across all the systems, which is important for membership arbitration and high availability. The public low priority link is used only for heartbeat communication between the systems.
If your hardware environment allows use of only two links, use one private interconnect link and one public low priority link. If you decide to set up only two links (one private and one low priority link), then the cluster must be configured to use I/O fencing, either disk-based or server-based fencing configuration. With only two links, if one system goes down, I/O fencing ensures that other system can take over the service groups and shared file systems from the failed node.

To set up the private network

Install the required network interface cards (NICs).
Create aggregated interfaces if you want to use these to set up private network.
Connect the Veritas InfoScale private Ethernet controllers on each system.
Use crossover Ethernet cables, switches, or independent hubs for each Veritas InfoScale communication network. Note that the crossover Ethernet cables are supported only on two systems.
Ensure that you meet the following requirements:
- The power to the switches or hubs must come from separate sources.
- On each system, you must use two independent network cards to provide redundancy.
- If a network interface is part of an aggregated interface, you must not configure the network interface under LLT. However, you can configure the aggregated interface under LLT.
- When you configure Ethernet switches for LLT private interconnect, disable the spanning tree algorithm on the ports used for the interconnect.
During the process of setting up heartbeat connections, consider a case where a failure removes all communications between the systems.
Note that a chance for data corruption exists under the following conditions:
- The systems still run, and
- The systems can access the shared storage.
Configure the Ethernet devices that are used for the private network such that the autonegotiation protocol is not used. You can achieve a more stable configuration with crossover cables if the autonegotiation protocol is not used.
To achieve this stable configuration, do one of the following:
- Edit the /etc/system file to disable autonegotiation on all Ethernet devices system-wide.
- Create a qfe.conf or bge.conf file in the /kernel/drv directory to disable autonegotiation for the individual devices that are used for private network.
Refer to the Oracle Ethernet driver product documentation for information on these methods.
Test the network connections. Temporarily assign network addresses and use telnet or ping to verify communications.
LLT uses its own protocol, and does not use TCP/IP. So, you must ensure that the private network connections are used only for LLT communication and not for TCP/IP traffic. To verify this requirement, unplumb and unconfigure any temporary IP addresses that are configured on the network interfaces.
The installer configures the private network in the cluster during configuration.
You can also manually configure LLT.