Configuring High Availability

Background

ClusterClosed Two Quantum Spark Appliances connected to each other for High Availability. maintains connections in the organization's network when there is a failure in one of the Cluster Members. The cluster provides redundancy.

In the Device view > Advanced section > High Availability page you can create a cluster of two appliances for high availability.

After you configure a cluster, you can select to Enable or Disable the cluster.

Notes:

  • This release supports only the High Availability Cluster mode:

    One Cluster MemberClosed A Security Gateway that is part of a cluster. is Active. The other Cluster Member is Standby.

    • In versions R81.10.15 and higher:

      The cluster supports these recovery modes (which Cluster Member to select as Active during a cluster fail-back, when the cluster returns to normal operation after a cluster failover):

      • Active up

        This is the default.

        The Cluster Member that is currently in the Active state, remains in this state.

        The other Cluster Member that returns to normal operation, remains in the Standby state.

      • Primary up

        The Cluster Member with higher priority is the first one to be configured. The primary Cluster Member that has the highest priority becomes the new Active.

        The state of the previously Active Cluster Member changes to Standby.

    • In versions R81.10.00 - R81.10.10:

      The cluster supports only the "Active up" recovery mode.

      The Cluster Member that is currently in the Active state, remains in this state.

      The other Cluster Member that returns to normal operation, remains in the Standby state.

  • After you configure the cluster, when you connect to the Cluster Virtual IP address, the cluster automatically redirects you to the current Active Cluster Member.

    To log in to specific Cluster Member, you must connect to the physical IP address of that Cluster Member.

Limitations

  • You cannot create a cluster when you have a switch defined in the network settings on the appliance. If necessary, change network settings in the Device > Local Network page.

    Starting from R81.10.15, cluster in Bridge Mode is supported.

  • In versions R81.10.10 and lower, it is not supported to configure a cluster of Quantum Spark Appliances when the Internet connection is a Bond interface.

  • Cluster requires Static IP addresses on the physical cluster interfaces.

  • Cluster does not support pure IPv6 addresses on cluster interfaces (you must also configure IPv4 addresses).

  • All cluster configuration is done through the Active Cluster Member. The WebUI of the Standby Cluster Member only has some options available for fine tuning - basic network settings, and logs (a cluster managed by Quantum Spark Portal cluster also has Cloud Services).

Prerequisites

  • In WebUI > Device > Local Network, delete switch configurations before you start to configure a cluster.

  • The appliances in a cluster must have the same hardware, firmware (version and build), and licenses.

    Note - Connect the sync cables only after you complete the First Time Configuration Wizard and remove the switch on both appliances. No additional configuration is required on the members.

Best Practice - Designate the same LAN port for the Sync interface. The default Sync interface is LAN2/SYNC. For appliance models 1600, 1800, 1900 and 2000, we recommend that you configure a bond of two interfaces for synchronization.

Configuration Workflow

  1. Complete the First Time Configuration Wizard on both appliances.

    In the Local Network page of the wizard, clear the checkbox Enable switch on LAN ports.

  2. Configure network settings on the appliance that is the primary Cluster Member.

  3. Connect cables between the Sync interfaces on the appliances.

    Note - Sync ports can also be connected through a switch.

  4. Configure the primary Cluster Member.

    1. Connect to the WebUI on the appliance.

    2. From the left navigation panel, click Device.

    3. In the Advanced section, click the High Availability page.

    4. Click Configure Cluster.

      The New Cluster Wizard opens.

    5. On the page Step 1: Gateway Priority:

      1. Select this option:

        • In versions R81.10.15 and higher:

          Configure first member.

        • In versions R81.10.00 - R81.10.10

          Configure as primary member.

      2. Click Next.

    6. On the page Step 2: SIC Settings:

      Important - The configuration on the second Cluster Member must match the configuration on the primary Cluster Member.

      1. In the Sync Interface section, configure the required settings for the synchronization interfaces:

        • In the field Sync interface (master), select the first (main) synchronization interface. Default: LAN2

        • In the field Second sync interface, select the second synchronization interface.

          Best Practice - For large appliances such as the 1600, 1800, 1900, and 2000, we highly recommend that you select a second sync interface.

          This creates a bond interface called SYNCBOND that includes both the first and second synchronization interfaces.

      2. In the Advanced sub-section, you can override the default settings:

        1. In the field Operation mode, you can select the working mode between the synchronization interfaces of the Cluster Members:

          • Select Health check if the synchronization interfaces on the Cluster Members are connected through a switch.

          • Select Link state (this is the default) if the synchronization interfaces on the Cluster Members are connected directly to each other.

        2. In the field Sync IP address, you can configure a different IPv4 address of the synchronization interface on the primary Cluster Member. Default: 10.231.149.1

        3. In the Sync IP subnet field, you can configure a different IPv4 address of the synchronization subnet. Default: 255.255.255.0

        4. In the field Other member sync IP address, you can configure a different IPv4 address of the synchronization interface on the second Cluster Member. Default: 10.231.149.2

        5. In the field Synchronization mode, you can select the working mode for the cluster synchronization:

          • Optimized sync

            This is the default.

            This mode synchronizes most of kernel tables to ensure smooth cluster failover.

            This mode does not synchronize large kernel tables (such as "Connections").

          • Sync is enabled

            This mode synchronizes all the kernel tables.

            Important - Depending on the number of concurrent connections and the enabled Software Blades, this mode can increase the load on the CPU.

          • Sync is disabled

            This mode disables the synchronization.

        6. In the field High Availability mode, you can configure the Cluster Member recovery method - which Cluster Member to select as Active during a cluster fail-back (when the cluster returns to normal operation after a cluster failover):

          Important:

          • This mode must be the same on both Cluster Members.

          • Changing this mode may cause a cluster failover.

          • Active up

            This is the default.

            The Cluster Member that is currently in the Active state, remains in this state.

            The other Cluster Member that returns to normal operation, remains in the Standby state.

          • Primary up

            The Cluster Member with higher priority is the first one to be configured. The primary Cluster Member that has the highest priority becomes the new Active.

            The state of the previously Active Cluster Member changes to Standby.

      3. In the Secure Internal Communication section, in the fields Password and Confirm, enter a one-time password for connecting the two Cluster Members to each other.

        Note - You cannot use these characters in a password or shared secret: { } [ ] ` ~ | ‘ " \ (maximum number of characters: 255)

      4. Click Next.

      Important - The configuration on the second Cluster Member must match the configuration on the primary Cluster Member.

      1. In the Secure Internal Communication section, in the fields Password and Confirm, enter a one-time password for connecting the two Cluster Members to each other.

        Notes:

        • You cannot use these characters in a password or shared secret: { } [ ] ` ~ | ‘ " \ (maximum number of characters: 255)

        • You must enter the same one-time password when you configure the second Cluster Member.

      2. In the Advanced section, you can override the default settings:

        1. In the field Sync interface, you can select a synchronization interface. Default: LAN2

        2. In the field Sync IP address, you can configure the IPv4 address of the synchronization interface on the primary Cluster Member. Default: 10.231.149.1

        3. In the field Sync IP subnet, you can configure the IPv4 address of the synchronization subnet. Default: 255.255.255.0

        4. In the field Other member sync IP address, you can configure the IPv4 address of the synchronization interface on the second Cluster Member. Default: 10.231.149.2

      3. Click Next.

    7. On the page Step 3: Gateway Interfaces (<X> out of <Y>):

      On these pages, you configure the "internal" and "external" cluster interfaces.

      Note - The physical IP addresses of cluster interfaces and the cluster Virtual IP address must be in the same subnet unless you are configuring a Single Routable IP Cluster.

      1. Select Enable High Availability on interface (this is the default).

        If you enable the high availability on an interface, the primary Cluster Member monitors it and if there is a failure, it automatically fails over to the second Cluster Member.

        If you clear this option, then you can also clear the option Monitor interface state (fail over when interface is down) to stop the cluster monitoring completely.

      2. In the field Cluster IP address, configure the applicable cluster Virtual IPv4 address. All hosts and network devices on the corresponding connected network must send their traffic to this Virtual IP address as their default gateway.

      3. In the field Subnet mask, configure the applicable IPv4 subnet mask for the cluster Virtual IP address.

      4. In the field This physical IP address, the wizard shows the IPv4 address configured on the interface.

      5. In the field Peer physical IP address, configure the applicable IPv4 address.

      6. Click Next.

      1. Select Enable High Availability on interface (this is the default).

        If you enable the high availability on an interface, the primary Cluster Member monitors it and if there is a failure, it automatically fails over to the second Cluster Member.

        If you clear this option, then you can also clear the option Monitor interface state (fail over when interface is down) to stop the cluster monitoring completely.

      2. In the field Cluster IP address, configure the applicable cluster Virtual IPv4 address. All hosts and network devices on the corresponding connected network must send their traffic to this Virtual IP address as their default gateway.

      3. In the field Subnet mask, configure the applicable IPv4 subnet mask for the cluster Virtual IP address.

      4. In the field Primary physical IP address, the wizard shows the IPv4 address configured on the interface.

      5. In the field second physical IP address, configure the applicable IPv4 address.

      6. Click Next.

    8. Click Finish.

    Note - At the top of the page, the Peer gateway field shows "is not defined". This status changes after you finish configuring the second Cluster Member.

    Watch the Video (for R81.10.00 - R81.10.10 versions)

  5. Configure the second Cluster Member.

    1. Connect to the WebUI on the appliance.

    2. From the left navigation panel, click Device.

    3. In the Advanced section, click the High Availability page.

    4. Click Configure Cluster.

      The New Cluster Wizard opens.

    5. On the page Step 1: Gateway Priority:

      1. Select this option:

        • R81.10.15 and higher:

          Configure as peer member.

        • R81.10.00 - R81.10.10

          Configure as second member.

      2. Click Next.

    6. On the page Step 2: SIC Settings:

      Important - The configuration on the second Cluster Member must match the configuration on the primary Cluster Member.

      1. In the Sync Interface section, configure the required settings for the synchronization interfaces:

        • In the field Sync interface (master), select the first (main) synchronization interface. Default: LAN2

        • In the field Second sync interface, select the second synchronization interface.

          Best Practice - For large appliances such as the 1600, 1800, 1900, and 2000, we highly recommend that you select a second sync interface.

          This creates a bond interface called SYNCBOND that includes both the first and second synchronization interfaces.

      2. In the Advanced sub-section, you can override the default settings:

        1. In the field Operation mode, you can select the working mode between the synchronization interfaces of the Cluster Members:

          • Select Health check if the synchronization interfaces on the Cluster Members are connected through a switch.

          • Select Link state (this is the default) if the synchronization interfaces on the Cluster Members are connected directly to each other.

        2. In the field Sync IP address, you can configure a different IPv4 address of the synchronization interface on the primary Cluster Member. Default: 10.231.149.1

        3. In the Sync IP subnet field, you can configure a different IPv4 address of the synchronization subnet. Default: 255.255.255.0

        4. In the field Other member sync IP address, you can configure a different IPv4 address of the synchronization interface on the second Cluster Member. Default: 10.231.149.2

        5. In the field High Availability mode, you can configure the Cluster Member recovery method - which Cluster Member to select as Active during a cluster fail-back (when the cluster returns to normal operation after a cluster failover):

          Important:

          • This mode must be the same on both Cluster Members.

          • Changing this mode may cause a cluster failover.

          • Active up

            This is the default.

            The Cluster Member that is currently in the Active state, remains in this state.

            The other Cluster Member that returns to normal operation, remains in the Standby state.

          • Primary up

            The Cluster Member with higher priority is the first one to be configured. The primary Cluster Member that has the highest priority becomes the new Active.

            The state of the previously Active Cluster Member changes to Standby.

      3. In the Secure Internal Communication section, in the field Password enter the same one-time password you configured for the primary Cluster Member.

      Important - The configuration on the second Cluster Member must match the configuration on the primary Cluster Member.

      1. In the Secure Internal Communication section, in the field Password enter the same one-time password you configured for the primary Cluster Member.

      2. In the Advanced section, you can override the default settings:

        1. In the field Sync interface, you can select a synchronization interface. Default: LAN2

        2. In the field Sync IP address, you can configure the IPv4 address of the synchronization interface on the primary Cluster Member. Default: 10.231.149.1

        3. In the field Sync IP subnet, you can configure the IPv4 address of the synchronization subnet. Default: 255.255.255.0

        4. In the field Other member sync IP address, you can configure the IPv4 address of the synchronization interface on the second Cluster Member. Default: 10.231.149.2

      3. Click Next.

    7. Click Establish Trust.

      The second Cluster Member fetches the settings from the primary Cluster Member and applies them.

    8. Click Finish.

    Watch the Video

Viewing Cluster Interfaces

  1. Connect to the WebUI on a Cluster Member:

    https://<IP Address of the Cluster Member>:4434

    Best Practice - After the cluster is successfully configured, connect to https://<Virtual IP Address of the Cluster>:4434. This redirects you to the WebUI Home > System page for the active Cluster Member.

  2. From the left navigation panel, click Device.

  3. In the Advanced section, click the High Availability page.

  4. The table List of Configured Interfaces shows information about the cluster interfaces:

    Column

    Description

    Name

    Name of the interface.

    Status

    Cluster status of the interface:

    Status

    Description

    High Availability

    Two physical interfaces in 2 Cluster Members act as a single interface toward the network, using a single virtual IP address.

    Note - In this cluster solution, each interface has a local IP address in addition to the shared single virtual IP address.

    Sync

    Two physical interfaces must be defined as Sync interfaces and connected between the members to allow proper failover as needed. The default is to use LAN2/Sync physical port.

    Non HA

    This status is also called private.

    The physical interface in this member does not participate in High Availability functions.

    Monitored

    This status is also called private monitored.

    The physical interface on this Cluster Member is not coupled with another interface on the other Cluster Member as in High Availability interface mode.

    The interface's status is still monitored, and if a problem occurs, the Cluster Member fails over to the other Cluster Member.

    IP Address

    Cluster Virtual IP address configured on the interface.

    Member IP Address

    Physical IP address configured on the interface.

Viewing the Cluster Status

  1. Connect to the WebUI on a Cluster Member:

    https://<IP Address of the Cluster Member>:4434

    Best Practice - After the cluster is successfully configured, connect to https://<Virtual IP Address of the Cluster>:4434. This redirects you to the WebUI Home > System page for the active Cluster Member.

  2. From the left navigation panel, click Device.

  3. In the Advanced section, click the High Availability page.

  4. Click View diagnostics.

Failing Over Manually

  1. Connect to the WebUI on the primary Cluster Member:

    https://<IP Address of the Primary Cluster Member>:4434

  2. From the left navigation panel, click Device.

  3. In the Advanced section, click the High Availability page.

  4. Click Force Member Down.

    A confirmation message appears.

  5. Click Yes.

  6. Cluster State:

    • The primary Cluster Member is now Down.

    • The second Cluster Member is now Active.

  7. The primary Cluster Member logs you out from WebUI because it has to reload it (to show only the supported pages).

Notes:

  • Only one Cluster Member can be down at a time.

    For the Cluster Member in the Down state, the Force Member Down button becomes Disable Manual Failover.

  • If you want the primary Cluster Member to handle the traffic, you must fall back from the second Cluster Member to the primary Cluster Member.

  1. Connect to the WebUI on the primary Cluster Member:

    https://<IP Address of the Primary Cluster Member>:4434

  2. From the left navigation panel, click Device.

  3. In the Advanced section, click the High Availability page.

  4. Click Disable Manual Failover.

    A confirmation message shows.

  5. Click Yes.

In Primary Up mode, the original primary Cluster Member is now the Active Cluster Member.

In Active Up mode, run Disable Manual Failover to make the member Standby.

Resetting Cluster Configuration

  1. Connect to the WebUI on one of the Cluster Members:

    https://<IP Address of the Cluster Member>:4434

  2. From the left navigation panel, click Device.

  3. In the Advanced section, click the High Availability page.

  4. Click Reset Cluster Configuration.

Important - This deletes all cluster configuration settings from both Cluster Members. You must run the New Cluster Wizard again to configure the cluster.

Upgrading a Cluster Manually

Notes:

  • Only manual local upgrade is supported.

  • Upgrade each Cluster Member individually.

  • Start the upgrade on the Standby Cluster Member.

  • After the upgrade, the appliance remains the Standby. You must failover and then upgrade the new Standby member.

  • Start the upgrade on the new Standby Cluster Member (former Active).

  1. Upgrade the current Standby Cluster Member:

    1. Connect to the WebUI on a Cluster Member:

      https://<IP Address of the Cluster Member>:4434

    2. From the left navigation panel, click Device.

    3. In the Advanced section, click the High Availability page.

    4. At the top of this page, examine the cluster state.

      If the current cluster state shows "This gateway (<...>) is standby", then continue to the next step.

      Otherwise, connect to the other Cluster Member

    5. In the System section, click the System Operations page.

    6. Click Manual Upgrade.

      The Upgrade Software Wizard opens.

    7. Follow the wizard instructions.

    8. After the upgrade, this appliance remains the Standby.

  2. Upgrade the new Standby Cluster Member (former Active Cluster Member):

    1. Connect to the WebUI on the Cluster Member:

      https://<IP Address of the High Availability>:4434

    2. From the left navigation panel, click Device.

    3. In the Advanced section, click the High Availability page.

    4. At the top of this page, examine the cluster state.

      Wait for the current cluster state to show "This gateway (<...>) is standby", and then continue to the next step.

    5. In the System section, click the System Operations page.

    6. Click Manual Upgrade.

      The Upgrade Software Wizard opens.

    7. Follow the wizard instructions.

    8. After the upgrade, this appliance remains as the Standby.

Single Routable IP Cluster

You can configure a Single Routable IP cluster where the virtual IP address is in a different subnet than the physical IP addresses of the Cluster Members. Only the virtual IP address is routable. Traffic sent from Cluster Members to internal or external networks is hidden behind the cluster Virtual IP address.

Advantages of using different subnets:

  • Use only one public IP address for the cluster.

  • Hide physical Cluster Members' IP addresses behind the cluster Virtual IP address.

  • Create a cluster in an existing subnet that has a limited number of available IP addresses.

  • The Internet connection must be of type Static.

  • The IP address of the Internet connection must be a fake, non-routable on the same subnet as the Internet Connection of the other member. For example, the IP address of the Internet connection of the first member is 4.4.4.4 with subnet of 255.255.255.0, and the IP address of the second member is subnet 255.255.255.0.

  • When first configuring the Internet connection, you must configure a default gateway. This gateway IP address must be fake as well and in the same subnet as the Cluster Members' IP addresses. In our example, 4.4.4.1.

  • You must turn off probe monitoring:

    1. Click Edit to open the Edit Internet Connection window > Connection Monitoring tab.

    2. Clear all probing checkboxes.

  • You must turn off SD-WAN (supported starting from R81.10.10).

  1. Configure the primary and second Cluster Members as for a regular cluster (see Configuration Workflow) but with these differences:

    1. After you configure the second member:

      1. Go back to the primary (Active) member and click Edit.

      2. Set the Default gateway as the default gateway of the Virtual IP address subnet.

    2. For each Cluster Member, in the Connection Monitoring tab, click the checkboxes to restore the probing options.

    3. If SD-WAN is supported, turn it on.

  2. Click Save to save your changes.

The related route to the Virtual IP address subnet shows in the Routing Table.

Cluster Managed by Quantum Spark Management

You can configure a cluster in which both gateways are managed by the Quantum Spark Management service in Infinity Portal.

Connect to Quantum Spark Management after you configure the cluster.

A cluster supported by Quantum Spark Management is very similar to a Locally Managed cluster. One cluster member is Active, and the other cluster member is Standby. To change the status of the Active member, click Force Member Down.

Connecting a Cluster Gateway to Spark Management

Prerequisites:

  • An account in the Infinity Portal with the Spark Management application. See the Quantum Spark Management Administration Guide.

  • Both gateways must have the same hardware, firmware (version and build), and licenses.

  • The firmware version must be R81.10.15 and higher.

  • The cluster is configured on the gateway level (see Configuration Workflow).

For more information on Cloud Services, see the Configuring Cloud Services page.

The cluster is configured locally and not yet connected to Spark Management.

  1. In Spark Management:

    1. Navigate to the Gateways page

    2. Create a new gateway object.

    3. In the Gateway type field, select Spark Cluster.

    4. Enter a name and select a plan.

    5. Create the cluster members objects in Spark Management. Each member represents a physical gateway member of the cluster.

      Notes:

      • You can create each member from the New Gateway Wizard page or later on the Cluster object in the General tab.

      • If you add the members later, click Save after you add both members.

    6. Click Finish. You are redirected to the cluster’s General tab

    7. Copy the HA activation key.

  2. On the active member of the cluster local WebUI:

    1. Navigate to Home > Cloud Services.

    2. Select the option Manage with Spark Management.

    3. Paste the HA activation key you copied from the Spark Management in the applicable field.

    4. Both members begin the Cloud Services activation process. In a few minutes the process completes and both members appear as connected.

The gateway was added and cluster configuration was completed on the local WebUI.

  1. In Spark Management:

    1. Navigate to the Gateways page

    2. Create a new gateway object.

    3. In the Gateway type field, select Spark Cluster.

    4. Enter a name and select a plan. This is usually the same plan that is used by the single gateway.

    5. Create the cluster members objects in Spark Management

      • For the first member, click Search and select the relevant gateway.

      • For the second member, create a new gateway object to represent the second member of the cluster.

        Notes:

        • You can create each member from the New Gateway Wizard page or later on the Cluster object in the General tab.

        • If you add the members later, click Save after you add both members.
    6. Click Finish. You are redirected to the cluster’s General tab

    7. Copy the HA activation key.

  2. On the active member of the cluster local WebUI:

    1. Navigate to Home > Cloud Services.

    2. Change the Cloud Management mode to Off and click Save.

    3. Select the Manage with Spark Management option.

    4. Paste the HA activation key you copied from the Spark Management into the applicable field.

    5. Both members begin the Cloud Activation process. In a few minutes the process completes and both members and appear as connected.

Enable the new Cloud capabilities for Extended Monitoring on a cluster that is managed locally.

  1. On the active member local WebUI:

    1. Navigate to Home > Cloud Services.

    2. Select the Use cloud capabilities option and follow these steps:

      1. Step 1 - Create an Infinity Portal account to connect the cluster. If you already have an account, skip this step.

      2. Step 2 - Get the token:

        1. Click the Get token link.

        2. Log in to Infinity Portal with your credentials. If your user is affiliated with more than a single account, select the relevant account.

        3. Copy the token.

      3. Step 3 – Paste the activation token into the applicable field.

    3. Both members begin the Cloud Activation process. In a few minutes the process completes and both members and appear as connected.

A single gateway, with enabled Cloud capabilities for Extended Monitoring, is converted into a cluster. The gateway was added and cluster configuration was completed on the local WebUI.

  • Option 1 – Reconnect to the Cloud as a cluster (Simple):

    1. Navigate to Home > Cloud Services.

    2. Change the Cloud Management mode to Off and click Save.

    3. Select the Use cloud capabilities option and follow these steps:

      1. Step 1 - Create an Infinity Portal account to connect the cluster. If you already have an account, skip this step.

      2. Step 2 - Get the token:

        1. Click the Get token link.

        2. Log in to Infinity Portal with your credentials. If your user is affiliated with more than a single account, select the relevant account.

        3. Copy the token.

      3. Step 3 – Paste the activation token into the applicable field.

    4. Both members begin the Cloud Activation process. In a few minutes the process completes and both members and appear as connected.

      Notes:

      • In this method the logs history available locally on the appliance is not retained. The reason is that the gateway initiates a new unique connection with Cloud Services. To retain the logs, use Option 2.

      • To view the history, the data still exists when logging into to Spark Management application in the Infinity Portal

  • Option 2 – Extend the single gateway to a cluster in Spark Management.

    Note - Even when you use the Cloud Capabilities option, Spark Management configuration for the gateway still exists.

    Follow the steps to convert a single gateway to a cluster:

    1. In Spark Management:

      1. Navigate to the Gateways page

      2. Create a new gateway object.

      3. In the Gateway type field, select Spark Cluster.

      4. Enter a name and select a plan. This is usually the same plan that is used by the single gateway.

      5. Create the cluster members objects in Spark Management

        • For the first member, click Search and select the relevant gateway.

        • For the second member, create a new gateway object to represent the second member of the cluster.

          Notes:

          • You can create each member from the New Gateway Wizard page or later on the Cluster object in the General tab.

          • If you add the members later, click Save after you add both members.
      6. Click Finish. You are redirected to the cluster’s General tab

      7. Copy the HA activation key.

    2. On the active member of the cluster local WebUI:

      1. Navigate to Home > Cloud Services.

      2. Change the Cloud Management mode to Off and click Save.

      3. Select the Manage with Spark Management option.

      4. Paste the HA activation key you copied from the Spark Management into the applicable field.

      5. Both members begin the Cloud Activation process. In a few minutes the process completes and both members and appear as connected.