Working with Link Aggregation

In This Section:

Link aggregation, also known as interface bonding, joins multiple physical interfaces to be one virtual interface: a bond interface. Configure a bond interface for High Availability (redundancy), or for load sharing (increased throughput).

For more about Link Aggregation, see the R76 ClusterXL Administration Guide.

Link Aggregation Overview

Link aggregation, also known as interface bonding, joins multiple physical interfaces together into a virtual interface, known as a bond interface. A bond interface can be configured for High Availability redundancy or for load sharing, which increases connection throughput above that which is possible using one physical interface.

For more about Link Aggregation, see the R76 ClusterXL Administration Guide.

Link Aggregation Terminology

Link Aggregation (Interface Bonding): Networking technology that binds multiple physical interfaces together into one virtual interface.
Bond: A group of physical interfaces that operate together as one virtual interface and share an IP address and MAC address. A bond is identified by the cluster by its Bond ID (for example: bond0).
Bond Interface: The logical representation of the bond.
Slave (enslaved interface): A physical interface that is a member of a bond. Slaves do not have an IP Address and in some cases share the same MAC address.

How Link Aggregation Works

A bond contains a minimum of one and may contain up to eight slave interfaces. All slave interfaces contained in a bond share a common IP address and may share the same MAC address. We recommend that each cluster member contain the same quantity of identical slave interfaces.

You can configure Link Aggregation using one of the following strategies:

High Availability (Active/Backup): Ensures redundancy in the event of interface or link failure. This option also provides switch redundancy.
Load Sharing (Active/Active): All interfaces are active, but handle different connections simultaneously. Traffic is balanced amongst slave interfaces to maximize throughput. The Load Sharing option does not support switch redundancy.

High Availability Overview

Clusters, by definition, provide redundancy and high availability at the gateway level. Link Aggregation, however, adds interface and switch redundancy by providing automatic failover to a standby interface card within the same VSX Gateway.

In a High Availability deployment, only one interface is active at a time. If an interface or connection fails, the bond fails over to a standby slave interface. Bonding High Availability failover occurs in one of these cases:

An active interface detects a link state failure in a monitored interface.
ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) keep-alive packets.

Fully Meshed Redundancy via Interface Bonding

The Link Aggregation High Availability mode, when deployed with ClusterXL, enables a higher level of reliability by providing granular redundancy in the network. This granular redundancy is achieved by using a fully meshed topology, which provides for independent backups for both NICs and switches.

In this scenario:

Member-1 and Member-2 are cluster members in the High Availability mode
S-1 and S-2 are switches
C-1, C-2, C-3 and C-4 are network connections

Load Sharing Overview

Load sharing provides the ability to spread traffic over multiple slave interfaces, in addition to providing interface redundancy. All interfaces are always active.

Traffic is balanced between interfaces in a manner similar to the way load sharing balances traffic between cluster members. Load sharing operates according to either the IEEE 802.3ad or the XOR standard.

In Load Sharing mode, each individual connection is assigned to a specific slave interface. For a specific connection, only the designated slave interface is active. In the event of a failure of the designated slave interface, the traffic fails over to another interface, adding that connection's to the traffic it is already handling.

Bond Failover

Either of the following failure scenarios can induce bond failover:

An active interface detects a link state failure in another monitored interface
ClusterXL detects a failure in sending or receiving Cluster Control Protocol (CCP) keep-alive packets

Either of these occurrences will induce a failover, either to another slave interface within the bond, or between cluster members, depending on the circumstances.

Note - The bond failover operation requires a network interface card that supports the Media-Independent Interface (MII) standard.

Link State Initiated Failover

Link-state initiated failover occurs in this sequence:

The active slave interface detects a down link state.
The bond initiates failover to a standby interface. Since this is a failover within the bond, the status of the other cluster member is unaffected.
When the number of available slave interfaces is fewer than the critical minimum number of interfaces, failover to other cluster members occurs.
If the standby interface continues to detect a link failure, and the initial interface is still down, failover to other cluster members occurs.

CCP Initiated Failover

CCP failover occurs only when other cluster members are not down, in this sequence.

ClusterXL detects a problem sending or receiving of CCP packets.
ClusterXL initiates an internal bond failover.
ClusterXL monitors CCP packet transmission and reception. If additional problems are detected within three minutes, the system initiates a failover to another cluster member.

Failover Support for VLANs

ClusterXL monitors VLAN IDs for connectivity failure or miscommunication, and initiates failover when necessary. By default, both the highest and the lowest VLAN IDs are monitored for failure. This is done by sending ClusterXL Control Protocol (CCP) packets on round-trip paths at a set interval.

You can configure VSX to monitor all VLANs.

When a failure is detected, a log of the failure is recorded in SmartView Tracker.

Monitoring the Highest and Lowest VLAN IDs

By default, the highest and lowest VLAN IDs indicate the status of the physical connection. These VLAN IDs are always monitored and a connectivity failure in either initiates a failover. In most deployments this is the desired setting, as it supports the primary purpose of the feature (detecting a connectivity failure) and the traffic generated on the network is light. However, this setting only detects VLAN configuration problems on the switch for the highest and lowest VLAN IDs.

Bond Interface & Interface Limitations

You can define a maximum of 4096 interfaces on a Gaia server or appliance. The total number of bond interfaces in use is the sum of bonds plus the number of slave interfaces contained in each bond.
Up to eight interfaces can be defined in a Link Aggregation deployment for each bond interface.

Configuring High Availability Mode

This section explains how to configure High Availability on a bond interface. Run the CLI commands from the VSX Gateway (VS0) context. For a cluster configuration, run these commands on each cluster member.

Use the active-backup value for the mode parameter to configure High Availability.

Configuring the High Availability Bond

This is a workflow of CLI commands to configure Link Aggregation in High Availability mode.

When you are enslaving configured interfaces, make sure that these interfaces are not used in other configurations.

To configure High Availability:

Create the High Availability bond. Run:
add bonding group <bond id>

set bonding group <bond id> mode active-backup
Define the slave interfaces. Run add bonding group <bond id> interface <IF name>
Do this command again for all of the slave interfaces.
Make sure that the bond is configured correctly. Run show bonding group <bond id>
To show more information about the bond, from Expert mode run cat /proc/net/bonding/<bond id>
Open SmartDashboard and configure the cluster object. .
- For a new Link Aggregation installation, create a new cluster object.
- For updating an existing configuration, update the interface topology.

Updating the Interface Topology

When you are updating an existing configuration to Link Aggregation, it is necessary to reconfigure the relevant objects to connect to the newly created bond. This includes Virtual Systems, Virtual Routers and Virtual Switches. You can perform these actions using SmartDashboard. In most cases, these definitions can be found in the object Properties window.

For large existing VSX deployments containing many Domain Management Servers and virtual devices, use the vsx_util change_interfaces command to reconfigure existing object topologies. For example, in a Multi-Domain Security Management deployment with 200 Domains, each with many virtual devices, it is faster to use vsx_util change_interfaces. This command automatically replaces the interface with the new bond on all relevant objects.

Reconfiguring the Bond using SmartDashboard

To configure the newly created bond:

Open SmartDashboard.
From the Network Objects tree, double-click the VSX Gateway or cluster object.
From the navigation tree, click Physical Interfaces.
Click Add.
The Physical Interface Properties window opens.
1. Enter the bond name.
1. If the bond is a VLAN trunk, select VLAN Trunk.
2. Click OK.
From the navigation tree, click Topology.
Do these steps for each interface that you are adding to the bond.
1. Double-click the interface.
  The Interface Properties window opens.
2. From Interface, select the bond interface.
3. Click OK.
Install the policy.
Delete the slave interfaces of the newly created bond that you are not using.
You can also replace a bond interface with one that is being used.

Reconfiguring Topology with vsx_util change_interfaces

To reconfigure objects with vsx_util change_interfaces:

Important - In a Multi-Domain Security Management environment, all Domain Management Servers must be unlocked in order for this operation to succeed.

Close SmartDashboard for all Multi-Domain Security Management Domain Management Servers using the designated interface.
On the management server, enter the Expert Mode and execute the vsx_util change_interfaces command.
Enter the Security Management Server, or Multi-Domain Security Management main Domain Management Server IP address.
Enter the administrator name and password as requested.
Enter the VSX cluster object name.
Select Apply changes to the management database and to the VSX Gateway/Cluster members immediately.
When prompted, select the interface to be replaced.
When prompted, select the replacement bond interface.
If you wish to replace additional interfaces, enter "y" when prompted and repeat steps 6 and 7.
To complete the process, enter "n".

Configuring Load Sharing Mode

This section explains how to configure Load Sharing on a bond interface. Run the CLI commands from the VSX Gateway (VS0) context. For a cluster configuration, run these commands on each cluster member.

Configure one of these Load Sharing modes for the bond interface:

Round robin
XOR
802.3ad

Configuring the Load Sharing Bond

This is a workflow of CLI commands to configure Link Aggregation in Load Sharing mode.

When you are enslaving configured interfaces, make sure that these interfaces are not used in other configurations.

To configure Load Sharing:

Create the High Availability bond. Run:
add bonding group <bond id>

set bonding group <bond id> mode <round-robin|xor|8023AD>
Define the slave interfaces. Run add bonding group <bond id> interface <IF name>
Do this command again for all of the slave interfaces.
Define the number of critical interfaces.
For configurations that use Performance Pack, configure the core affinities.
Make sure that the bond is configured correctly. Run show bonding group <bond id>
To show more information about the bond, from Expert mode run cat /proc/net/bonding/<bond id>
Open SmartDashboard and configure the cluster object.
- For a new Link Aggregation installation, create a new cluster object.
- For updating an existing configuration, update the interface topology.

Setting Critical Required Interfaces

Note - The Critical Required Interfaces feature is supported for ClusterXL only.

A bond in Load Sharing mode is considered to be down when fewer than a critical minimum number of slave interfaces remain up. When not explicitly defined, the critical minimum number of interfaces in a bond of n interfaces is n-1. Failure of a second interface will cause the entire bond to be considered down, even if the bond contains more than two interfaces.

If a smaller number of interfaces will be able to handle the expected traffic, you can increase redundancy by explicitly defining the number of critical interfaces. Divide your maximal expected traffic speed by the speed of your interfaces and round up to a whole number to determine an appropriate number of critical interfaces.

To explicitly define the number of critical interfaces, create and edit the following file:

$FWDIR/conf/cpha_bond_ls_config.conf

Each line of the file should be of the following syntax:

For example, if bond0 has seven interfaces and bond1 has six interfaces, file contents could be:

bond0 5

bond1 3

In this case bond0 would be considered down when three of its interfaces have failed. bond1 would be considered down when four of its interfaces have failed.

Setting Affinities

If you are running Performance Pack in a multi-core system, after you define bonds, set affinities manually. Use the -s parameter of the sim affinity command, see the R76 Performance Pack Administration Guide.

Note - sim affinity commands take effect only if the Performance Pack is enabled and actually running. Performance Pack begins running when you install a policy for the first time.

For optimal performance, set affinities according to the following guidelines:

Run sim affinity using the -s option.
Whenever possible, dedicate one processing core to each interface. See sk33520.
If there are more interfaces than cores, one or more cores handle two interfaces. Use interface pairs of the same position with internal and external bonds.
1. To view interface positions in a bond, run:
  cat /proc/net/bonding/<bond name>.
2. Note the sequence of the interfaces in the output, and compare this for the two bonds (external bond and its respective internal bond). Interfaces that appear in the same position in the two bonds are interface pairs and set to be handled by one processing core.
For example, you might have four processing cores (0-3) and six interfaces (0-5), distributed among two bonds:

bond0	bond1
eth0	eth3
eth1	eth4
eth2	eth5

Two of the cores will need to handle two interfaces each. An optimal configuration might be:

bond0		bond1
eth0	core 0	eth3	core 0
eth1	core 1	eth4	core 1
eth2	core 2
		eth5	core 3

Configuring Cisco Switches for Load Sharing

These are sample configuration commands for Cisco switches.

For 802.3ad:

Switch#conf t

Switch(config)#port-channel load-balance src-dst-ip

Switch(config)#interface FastEthernet <all the participating interfaces>

Switch(config-if)#channel-group 1 mode active

Switch(config-if)#channel-protocol lacp

Switch(config-if)#exit

Switch(config)#interface port-channel 1

Switch(config-if)#switchport access vlan <the wanted vlan number>

Switch(config-if)#end

Switch#write

For XOR:

Switch#conf t

Switch(config)#port-channel load-balance src-dst-ip

Switch(config)#interface FastEthernet <all the participating interfaces>

Switch(config-if)#channel-group 1 mode on

Switch(config-if)#exit

Switch (config)#interface port-channel 1

Switch(config-if)#switchport access vlan <the wanted vlan number>

Switch(config-if)#end

Switch#write

Troubleshooting Bonded Interfaces

Troubleshooting Workflow

Check the status of the bond. From Expert mode, run cat/proc/net/bonding/<bond id>
If there is a problem, check if the physical link is down, as follows:
1. Execute the following command:
  cphaconf show_bond <bond-name>
2. Look for a slave interface that reports the status of the link as no.
3. Check the cable connections and other hardware.
4. Check the port configuration on the switch.
Check if a cluster member is down, by running:
cphaprob state

If any of the cluster members have a Firewall State other than active, see Monitoring Cluster Status (cphaprob state) in the R76 ClusterXL Administration Guide.
For further information regarding bond status and failovers, view logs in SmartView Tracker. Any interface bond status change is logged and can be viewed in SmartView Tracker.

Connectivity Delays on Switches

When using certain switches, connectivity delays may occur during some internal bond failovers. With the various features that are now included on some switches, it can take close to a minute for a switch to begin servicing a newly connected interface. The following are suggestions for reducing the startup time after link failure.

Disable auto-negotiation on the relevant interface.
On some Cisco switches, enable PortFast, as detailed below.

Note - PortFast is not applicable if the bond group on the switch is configured as Trunk.

Warning Regarding Use of PortFast

The PortFast feature should never be used on ports that connect to other switches or hubs. It is important that the Spanning Tree complete the initialization procedure in these situations. Otherwise, these connections may cause physical loops where packets are continuously forwarded (or even multiply) in such a way that network will ultimately fail.

Sample Configuration of PortFast on a Cisco Switch

The following are the commands necessary to enable PortFast on a Gigabit Ethernet 1/0/15 interface of a Cisco 3750 switch running IOS.

Enter configuration mode:
cisco-3750A#conf t
Specify the interface to configure:
cisco-3750A(config)#interface gigabitethernet1/0/15
Set PortFast on this interface:
cisco-3750A(config-if)#spanning-tree portfast

Gaia CLI Reference

This section contains a summary of the Gaia CLI commands that configure Link Aggregation.

Link Aggregation - CLI (bonding)

This section is a quick reference for Link Aggregation commands. The next sections include procedures for different tasks, including explanations of the configuration options.

Use these commands to configure link aggregation.

Syntax:

{add | delete} bonding group <bondID> interface <IFName>

set bonding [group <bondID>] [primary <IFName>] [mii-interval <ms>] [up-delay <ms> | down-delay <ms>] [mode {round-robin | active-backup | xor [xmit-hash-policy {layer2 | layer3+4}]| 8023AD [lacp-rate {slow | fast}]}]

show bonding group {<bondID> | groups}

Parameters

Parameter	Description
bondID	ID of bond, an integer between 1 and 1024
IFName	Name of interface to add to the bond
`primary`	Name of primary interface in the bond
`mii-interval`	Frequency that the system polls the Media Independent Interface (MII) to get status
`up-delay` `down-delay`	Waiting time to confirm the interface status before taking the specified action (0-5000 ms, default = 200 ms)
`mode`	Bond operating mode
`lacp-rate`	Link Aggregation Control Protocol packet transmission rate: `slow` - LACPDU packet sent every 30 seconds `fast` - LACPDU packet sent every second
`xmit-hash-policy`	Algorithm for interface selected by TCP/IP layer

Example

set bonding group 666 20 eth2

show bonding groups

Output

Bonding Interface: 20

    Bond Configuration

        xmit_hash_policy Not configured

        down-delay 200

        primary Not configured

        mode round-robin

        up-delay 200

        mii-interval 100

        lacp_rate Not configured

        Bond Interfaces

            eth2

            eth3

Creating or Deleting a Bond Interface

To add a new bond interface:

add bonding group <bondID>

Example:

add bonding group 777

To delete a bond interface:

Remove all interfaces from the bond.
Run: delete bonding group <bondID>

Defining the Bond Operating Mode

Define how interfaces are activated in a bond:

round-robin - Interfaces activated in order by ID (default)
active-backup - On active interface down, failover to primary interface first, and to other interfaces if primary is down
xor - Interface activation by TCP/IP layer (layer2 or layer3+4).
You can set the LACP packet transmission rate for xor mode or 8023AD mode. After you set one of these Load Sharing modes, enter this option: lacp-rate {slow | fast}
where slow is every 30 seconds, and fast is every one second.
8023AD - Link Aggregation Control Protocol load shares traffic by dynamic interface activation, with full interface monitoring between gateway and switch. In this mode only, you can set the algorithm for interface selection, according to the specified TCP/IP layer: xmit-hash-policy {layer2 | layer3+4}

To define the bond operating mode:

set bonding group <BondID> mode <mode> [option]

Example:

set bonding group 777 mode xor xmit-hash-policy layer3+4

Defining Interfaces

A bond interface typically contains between two and eight slave interfaces. This section shows how to add and remove a slave interface. The slave interface must not have IP addresses assigned to it.

To add a slave interface to a bond:

add bonding group <bondID> interface <IFName>

Example:

add bonding group 777 interface eth4

Note - Do not change the bond state manually. This is done automatically by the bonding driver.

To delete a slave interface from a bond:

delete bonding group <bondID> interface <IFName>

Example:

delete bonding group 777 interface eth4

Note - You must delete all non-primary slave interfaces before you remove the primary slave interface.

Defining the Primary Slave Interface

With the Active-Backup operating mode, the system automatically fails over to the primary slave interface, if available. If the primary interface is not available, the system fails over to a different slave interface. By default, the first slave interface that you define is the primary interface. You must define the slave interfaces and set the operating mode as Active-Backup before doing this procedure.

Note - You must delete all non-primary slave interfaces before you remove the primary slave interface.

To define the primary slave interface:

set bonding group <bondID> mode active-backup primary <IFName>

Example

add bonding group 777 interface eth4

set bonding group 777 mode active-backup primary eth4

Defining the Media Monitoring Interval

This sets the frequency of requests sent to the Media Independent Interface (MII) to confirm that a slave interface is up. The valid range is 1-5000 ms. The default is 100 ms.

To configure the monitoring interval:

set bonding group <bondID> mii-interval <ms>

Example:

set bonding group 777 mii-interval 500

To disable monitoring:

set bonding group <bondID> mii-interval 0

Defining the UP and Down Delay Times

This parameter defines the waiting time, in milliseconds, to confirm the slave interface status before taking the specified action. Valid values are 0 to 5000 ms. The default is 200 ms.

To configure the UP and Down delay times:

set bonding group <bondID> down-delay <ms>

set bonding group <bondID> up-delay <ms>

Example:

set bonding group 777 down-delay 500

Defining Load Sharing Parameters

When using Load Sharing modes (XOR or 802.3ad), you can configure these parameters:

LACP Rate - Set the Link Aggregation Control Protocol packet transmission rate. Valid values are slow (every 30 seconds) and fast (every 1 second).
Transmit Hash Policy (802.3ad only) - Set the algorithm for interface selection according to the specified TCP/IP layer. Valid values are layer2 (uses XOR of the physical interface MAC address) and layer3+4 (users upper layer protocol data).

To set the LACP rate:

set bonding group <bondID> lacp-rate {slow | fast}

Example: set bonding group 777 mode 8023AD lacp-rate slow

To set the Transmit Hash Policy:

set bonding group <bondID> xmit-hash-policy <layer>

Example: set bonding group 777 mode xor xmit-hash-policy layer2

Making Sure that Link Aggregation is Working

To make sure that a Link Aggregation is working for a bond interface, run this command in expert mode:

cat /proc/net/bonding/<bondID>

Example with output:

cat /proc/net/bonding/bond666

Ethernet Channel Bonding Driver: v3.2.4 (January 28, 2008)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth2

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 100

Down Delay (ms): 200

Slave Interface: eth2

MII Status: up

Link Failure Count: 2

Permanent HW addr: 00:50:56:94:11:de