ClusterXL Optimal Service Upgrade

Use the Optimal Service Upgrade feature to upgrade a Security Gateway or VSX cluster from R75.40VS to R77 and future major releases. This feature upgrades the cluster with a minimum loss of connectivity.

When you upgrade the cluster, two cluster members are used to process the network traffic. New connections that are opened during the upgrade procedure are maintained after the upgrade is finished. Connections that were opened on the old version are discarded after the upgrade.

You can also use the Optimal Service Upgrade feature to upgrade a VSX cluster from R67.10 to R77. When you use this feature to upgrade from VSX R67.10, download the R67.10 upgrade Hotfix and install it on one VSX cluster member. For more about upgrading to R67.10, see the R67.10 Release Notes.

For more about the Optimal Service Upgrade and to download the R67.10 upgrade Hotfix, go to sk74300.

Upgrade Workflow from R75.40VS

Use the Optimal Service Upgrade to upgrade a cluster from R75.40VS to a later version, without loss of connectivity.

• OLD cluster member - Cluster member before the upgrade.
• NEW cluster member - Cluster member that has been upgraded.

  	Note - Do not use this workflow to upgrade a VSX cluster from R67.10.
  Diagram of Cluster Members		Summary		Step
  		Cluster with four members (OLD).
  		Leave one cluster member connected to the network (OLD) and disconnect all other cluster members. The connected cluster member continues to process old connections. For upgrades to R77.30, make sure that the cluster ID (the value of the cluster_id parameter) is the same on all cluster members. For upgrades to R77.20 or an earlier version, make sure that the value of the fwha_mac_magic parameter is the same on all cluster members.		1     2
  		Upgrade the cluster members that are disconnected from the network (NEW). For upgrades to R77.30 or a later version, make sure that the cluster ID (the value of the cluster_id parameter) is the same on all the upgraded cluster members. Change it, if necessary. For upgrades to R77.20 or an earlier version, make sure that the value of the fwha_mac_magic parameter on all the upgraded cluster members is the same. Change it, if necessary.		3 4
  		Connect one upgraded (NEW) cluster member to the network. On the active (OLD) cluster member, turn off fwaccel on all Virtual Systems. This allows the active (OLD) cluster member synchronize all delayed connections with the upgraded (NEW) cluster member. Note: If there are a lot of connections on the Virtual Systems, turning off fwaccel will cause all the connections to be forwarded to the firewall. In this case, run the cpstop command to turn off the firewall. On the active (OLD) cluster member, start the Optimal Service Upgrade procedure.		5 6             7
  		On the upgraded cluster member (NEW) that you connected to the network, start the Optimal Service Upgrade procedure. The upgraded cluster member begins to process new connections.		8
  		Check the number of active connection on the old cluster member. When this cluster member almost stops processing connections, stop the Optimal Service Upgrade procedure on it. Disconnect the old cluster member from the network.		9     10
  		Reconnect the other upgraded cluster members to the network.		11
  		Upgrade the old cluster member. Connect all the cluster members to the network. Install the policy.		12 13 14

Upgrading the Cluster from R75.40VS

Two cluster members are used to maintain connectivity, while you upgrade all the other cluster members.

To use the Optimal Service Upgrade to upgrade the cluster members:

1. Disconnect all cluster members from the network, except for one cluster member.

   Make sure that the management interfaces are not connected to the network.

2. On the old cluster member (connected to the network), configure kernel parameters:
   • Upgrade to R77.30:

     Run: cphaconf cluster_id get

     Make sure all cluster members have the same cluster ID. If the cluster ID value is different on a cluster member, run this command to configure the correct value: cphaconf cluster_id set <value>

   • Upgrade to R77.20 and lower:

     Make sure all cluster members use the same value for the fwha_mac_magic parameter. Run: fw ctl get int fwha_mac_magic

     The default value for the fwha_mac_magic parameter is 254. If your configuration uses a different value, on each member, run: fw ctl set int fwha_mac_magic <value>

   For more about the cluster_id and fwha_mac_magic parameters, see the R77ClusterXL Administration Guide and sk25977.

3. Install R77 on all the cluster members that are not connected to the network.
4. Make sure that all the cluster members use the same kernel parameter values:
   • Upgrade to R77.30 and higher: Make sure all cluster members have the same cluster ID. On each member, run: cphaconf cluster_id get
     If a member has a different ID, run: cphaconf cluster_id set <value>
   • Upgrade to R77.20 and lower: Make sure all cluster members have the same value for this parameter: fw ctl get int fwha_mac_magic
     If a member has a different value, run: fw ctl set int fwha_mac_magic <value>
5. Prepare the old cluster member for synchronization of old connections with the upgraded cluster member:
   1. On the old cluster member, turn off fwaccel - run: fwaccel off -a
   2. On the old cluster member, start the Optimal Serve Upgrade - run: cphaosu start
6. Reconnect the SYNC interface of one new cluster member to the network.
7. Move traffic to the new cluster member that is connected to the network. Do these steps:
   1. Make sure the new cluster member is in ready state.
   2. Connect the other new cluster member interfaces to the network.
   3. On the new cluster member, run cphaosu start
   4. On the old cluster member, run cphaosu stat

      The network traffic statistics are shown.

   5. When the old cluster member does not have many connections, run cphaosu finish
8. On the new cluster member, run cphaosu finish
9. Disconnect the old cluster member from the network.
10. Reconnect the other new cluster members to the network one at a time. Do these steps on each cluster member:
    1. Run cphastop
    2. Connect the new cluster member to the network.
    3. Run cphastart
    4. In SmartDashboard, change the version of the cluster object to R77 and install the Policy.
11. Upgrade the old cluster member and reconnect it to the network.
12. If the cluster has two members: In SmartDashboard, change the version to R77.
13. Install the Policy.

Upgrade Workflow from R67.10 VSX

Use the Optimal Service Upgrade to upgrade a VSX cluster from R67.10 to a later version, without loss of connectivity. When you upgrade the cluster, use two cluster members to process the network traffic.

• OLD cluster member - The R67.10 VSX Gateway on which you install the Optimal Service Upgrade Hotfix.
• NEW cluster member - VSX Gateway that is upgraded to R77 and processes new connections.

Diagram of Cluster Members	Summary
	VSX cluster with four R67.10 VSX Gateways (OLD).
	Install the Optimal Service Upgrade Hotfix on the cluster member that will stay connected to the network during the upgrade.	1
	Leave the cluster with the Hotfix connected to the network, and disconnect all other cluster members from the network. For upgrades to R77.30, make sure that the cluster ID (the value of the cluster_id parameter) is the same on all cluster members. For upgrades to R77.20 or an earlier version, make sure that the value of the fwha_mac_magic parameter is the same on all cluster members.	2   3
	Upgrade the cluster members that are disconnected from the network (NEW). For upgrades to R77.30 or a later version, make sure the cluster ID (the value of the cluster_id parameter) is the same on all the upgraded cluster members. Change it, if necessary. For upgrades to R77.20 or an earlier version, make sure that the value of the fwha_mac_magic parameter on all the upgraded cluster members is the same. Change it, if necessary.	4 5
	Connect one upgraded (NEW) cluster member to the network. On the active (OLD) cluster member, turn off fwaccel on all Virtual Systems. This allows the active (OLD) cluster member synchronize all delayed connections with the upgraded (NEW) cluster member. Note: If there are a lot of connections on the Virtual Systems, turning off fwaccel will cause all the connections to be forwarded to the firewall. In this case, run the cpstop command to turn off the firewall. On the active (OLD) cluster member, start the Optimal Service Upgrade procedure.	6 7         8
	On the upgraded cluster member (NEW) that you connected to the network, start the Optimal Service Upgrade procedure. The upgraded cluster member begins to process new connections.	9
	Check the number of active connection on the old cluster member. When this cluster member almost stops processing connections, stop the Optimal Service Upgrade procedure on it. Disconnect the old cluster member from the network.	10     11
	Reconnect the other upgraded cluster members to the network.	12
	Upgrade the old cluster member. Connect all the cluster members to the network. Install the policy.	13 14 15

Upgrading the VSX Cluster from R67.10

Two cluster members are used to maintain connectivity, while you upgrade all the other cluster members.

To use the Optimal Service Upgrade to upgrade the R67.10 VSX cluster members:

1. Install the Optimal Service Upgrade Hotfix on a cluster member. This is the old cluster member with Hotfix. For instructions and download links, refer to sk74300.
2. Disconnect all old cluster members from the network, except for one cluster member.

   Make sure that the management interfaces are not connected to the network.

3. On the old cluster member, configure kernel parameters:
   • Upgrade to R77.30:

     Run: cphaconf cluster_id get

     If the cluster ID value is not as expected, run: cphaconf cluster_id set <value>

     Make sure all cluster members have the same cluster ID. If a member has a different ID, run this set command to configure the correct value.

   • Upgrade to R77.20 and lower:

     Make sure all cluster members use the same value for the fwha_mac_magic parameter. Run: fw ctl get int fwha_mac_magic

     The default value for the fwha_mac_magic parameter is 254. If your configuration uses a different value, on each member, run: fw ctl set int fwha_mac_magic <value>

   For more about the cluster_id and fwha_mac_magic parameters, see the
   R77 ClusterXL Administration Guide
   and sk25977.

4. Install R77 on all the cluster members that are not connected to the network.
5. Prepare the old cluster member for synchronization of old connections with the upgraded cluster member:
   1. On the old cluster member, turn off fwaccel - run: fwaccel off -a
   2. On the old cluster member, start the Optimal Serve Upgrade - run: cphaosu start
6. Reconnect the SYNC interface of one new cluster member to the network.
7. Move traffic to the new cluster member that is connected to the network. Do these steps:
   1. Make sure the new cluster member is in ready state.
   2. Connect the other new cluster member interfaces to the network.
   3. On the new cluster member, run cphaosu start
   4. On the old cluster member, run cphaosu stat

      The network traffic statistics are shown.

   5. When the old cluster member does not have many connections, run cphaosu finish
8. On the new cluster member, run cphaosu finish
9. Disconnect the old cluster member from the network.
10. Reconnect the other new cluster members to the network one at a time. Do these steps on each cluster member:
    1. Run cphastop
    2. Connect the new cluster member to the network.
    3. Run cphastart
11. Upgrade the old cluster member and reconnect it to the network.

Troubleshooting the Upgrade

Use these cphaosu commands if there are problems during the upgrade process.

• If it is necessary to rollback the update, run cphaosu cancel on the new member. The old member processes all the traffic.
• After you run cpshaosu finish on the old member, you can continue to process the old traffic on the old member and the new traffic on the new member. Run cphaosu restart on the old member.

Limitations

1. Upgrade procedure should be implemented when there is minimal network traffic.
2. If there is a member failure during the upgrade, the Optimal Service Upgrade procedure does not provide redundancy.
3. Do not apply configuration changes during the upgrade process.
4. These connections do not survive the upgrade process:
   1. Complex connections, for example:

      DCE RPC SUN RPC Back Web DHCP IIOP

      FreeTel WinFrame NCP VPN

   2. Dynamic routing
   3. Bridge mode (L2) configurations

Connectivity Upgrade

Before you run Connectivity Upgrade, please see:

• sk107042 ClusterXL upgrade methods and paths
• sk101209 R77.20 Known Limitations
• sk104860 R77.30 Known Limitations

Check Point Connectivity Upgrade synchronizes existing connections to maintain connectivity during a cluster upgrade to R77.20 and later R77 versions from these versions:

• VSX Virtual System Load Sharing R75.40VS
• VSX Virtual System Load Sharing R76
• VSX High Availability R75.40VS
• VSX High Availability R76
• ClusterXL High Availability R75.40VS
• ClusterXL High Availability R76

Upgrades to R77.30 from these versions are also supported:

• R77
• R77.10
• R77.20

During Connectivity Upgrade, some connection flow information is synchronized to maintain connectivity.

Notes - Software Blade information does not get synchronized. If a connection needs to be inspected by a Software Blade, and this Software Blade is configured in SmartDashboard to Prefer Connectivity Over Security, then the connection is accepted without the inspection. Otherwise, the connection is dropped. All member gateways must have the same number of CoreXL Firewall instances. All member gateways must run the same 32-bit or 64-bit kernel edition.

Upgrading VSX High Availability Cluster

Before you upgrade:

• Make sure that the cluster has 2 members, where one of them is the Active member and the other member is the Standby
• Get the sync interface IP address and the cluster member ID of the Active cluster member

To check the cluster member's status and to get its IP address and the cluster member ID:

Run the cphaprob stat command on each of the gateways. For more information about the cphaprob stat command, see the R77 Command Line Interface (CLI) Reference Guide.

To upgrade the cluster:

1. Upgrade the Standby cluster member with a clean install. For more information about the clean install, see sk97552.
2. On the upgraded cluster member, run these commands:
   1. cphaprob stat

      Make sure the status is Ready.

   2. cphacu start <Sync IP of Active_GW> <Member ID of Active_GW>

      The Connectivity Upgrade runs, and shows this message when it finishes: Connectivity upgrade status: Ready for Failover

3. On the old Active cluster member, run these commands:
   1. cphaprob stat

      Make sure the local member is in Active or Active Attention state, and the upgraded member is in Down state.

   2. fwaccel off -a

      Turns off fwaccel on all Virtual Systems so that the delayed connections are synchronized to the upgraded member that is now in Ready state.

   3. cpstop

      The connections fail over to the upgraded member.

4. On the upgraded cluster member, run: cphaprob stat

   Make sure that it is now in Active state.

5. On the new Active cluster member, run: cphacu stat

   Make sure that it handles the traffic. See cphacu stat.

6. Upgrade the former Active cluster member with a clean install. For more information about the clean install, see sk97552.

   Make sure to reboot the gateway after the upgrade.

To make sure all cluster members are up and in VSX High Availability mode:

On each cluster member, run: cphaprob stat

If the state of a cluster member is HA not started, run cphastart on it. For more information about the cphastart and the cphaprob commands, see the R77 Command Line Interface (CLI) Reference Guide.

Upgrading ClusterXL High Availability With Connectivity Upgrade

Before you upgrade:

• Make sure that the cluster has 2 members, where one of them is the Active member and the other member is the Standby
• Get the sync interface IP address and the cluster member ID of the Active cluster member

To check the cluster member's state and to get its IP address and the cluster member ID:

Run the cphaprob stat command on the cluster member. For more information about the cphaprob stat command, see the R77 Command Line Interface (CLI) Reference Guide.

To upgrade the cluster:

1. Upgrade the standby cluster member according to the R77 Installation and Upgrade Guide (Gaia).

   Make sure to reboot the gateway after the upgrade.

2. In SmartDashboard, do these:
   1. In the Gateway Cluster General Properties window, change the Cluster version to the upgraded one.
   2. In the Install Policy window, go to Installation Mode > Install on each selected gateway independently section and clear For Gateway Clusters install on all the members, if it fails do not install at all.
   3. Install the security policy on the cluster.

   Note - The policy successfully installs on the standby cluster member and fails to install on the Active cluster member. This is expected. Ignore the warning.

3. On the Active cluster member, run: cphaprob stat

   Make sure the status is Active or Active Attention, and record the Sync IP and the Member ID of the cluster member.

4. On the upgraded cluster member, run these commands:
   1. cphaprob stat

      Make sure that the cluster member is in Ready state

   2. cphacu start <Sync IP of Active_member> <Member ID of Active_member>

      The Connectivity Upgrade runs and shows this message when it finishes: Connectivity upgrade status: Ready for Failover

   3. cphacu stat

      Make sure that the Active cluster member handles the traffic.

5. On the Active cluster member, run these commands:
   1. cphaprob stat

      Make sure the local member is in Active or Active Attention state, and the upgraded member is in Down state.

   2. fwaccel off -a

      Turns off fwaccel on all Virtual Systems so that the delayed connections are synchronized to the upgraded member that is now in Ready state.

   3. cpstop

      The connections fail over to the upgraded cluster member.

6. On the upgraded cluster member, run: cphaprob stat

   Make sure that it is now in the Active state.

7. On the new upgraded cluster member, run: cphacu stat

   Make sure it handles the traffic.

8. Upgrade the former Active cluster member.

   Make sure to reboot it after the upgrade.

9. Install Policy.

After the cluster upgrade is complete, the Cluster Control Protocol is in the broadcast mode. To return it to the multicast mode, on all cluster members, run: cphaconf set_ccp multicast

Connectivity Upgrade Commands

cphacu start

Description Runs Connectivity Upgrade on a cluster member with a specified IP address and a cluster member ID.

Note - To get the IP address and the member ID of a cluster member, run the cphaprob stat command on it. For more information about the cphaprob stat command, see the R77 Command Line Interface (CLI) Reference Guide.

Syntax

cphacu start <IP Address> <Member ID>

Output

cphacu start command outputs this information:

• Performing Full Sync on VSID <VSID number>
• Connectivity Upgrade Status -
  • Disabled - Connectivity Upgrade is not running on this cluster member
  • Enabled, ready for failover - Connectivity Upgrade completed successfully, and the Active member can now do the failover
  • Not enabled since member is Active - Connectivity Upgrade cannot run, because this member is Active
  • Full sync for connectivity upgrade is still in progress. Wait until full sync finishes
• The peer member is handling the traffic - Shows which cluster member currently handles the traffic and the version of the Cluster Control Protocol for each member
• Connection table - Shows the summary of the connections table for each Virtual System

Example 1 - VSX High Availability

[Expert@HostName]# cphacu start 192.0.2.1 1 Starting Connectivity Upgrade...   Performing Full Sync ==================== Performing Full Sync on VSID 0 Performing Full Sync on VSID 1 Performing Full Sync on VSID 2 Performing Full Sync on VSID 3 Performing Full Sync on VSID 4 Performing Full Sync on VSID 5 Performing Full Sync on VSID 6   =============================================================== Full Sync ended (Delta Sync is enabled) For delayed connections (Templates) to be synchronized it is recommended to turn off SecureXL on the old member before doing a failover. Run: 'fwaccel off' on the old member. ===============================================================   Connectivity upgrade status: Enabled, ready for failover ========================================================   The peer member is handling the traffic ======================================= Version of the local member: 2907 Version of the peer member : 2502   Connection table ================ VS HOST NAME ID #VALS #PEAK #SLINKS 0 localhost connections 8158 39 39 45 1 localhost connections 8158 0 2 0 2 localhost connections 8158 0 0 0 3 localhost connections 8158 0 0 0 4 localhost connections 8158 0 0 0 5 localhost connections 8158 0 0 0 6 localhost connections 8158 0 1 0

Example 2 - ClusterXL High Availability

[Expert@HostName]# cphacu start 192.0.2.1 1 Starting Connectivity Upgrade...   Performing Full Sync ==================== Performing Full Sync     =========================================================================== Full Sync ended (Delta Sync is enabled) For delayed connections (Templates) to be synchronized it is recommended to turn off SecureXL on the old member before doing a failover. Run: 'fwaccel off' on the old member. ===========================================================================   Connectivity upgrade status: Enabled, ready for failover ========================================================   The peer member is handling the traffic ======================================= Version of the local member: 2907 Version of the peer member : 2502   Connection table ================ HOST            NAME              ID #VALS #PEAK #SLINKS localhost       connections       8158    39    39      45

cphacu stat

Description Shows the status of Connectivity Upgrade.

Syntax

cphacu stat

Example 1 - VSX High Availability

[Expert@HostName]# cphacu stat   Connectivity upgrade status: Disabled =====================================   The peer member is handling the traffic ======================================= Version of the local member: 2907 Version of the peer member : 2502   Connection table ================ VS HOST NAME ID #VALS #PEAK #SLINKS 0 localhost connections 8158 16 56 16 1 localhost connections 8158 0 3 0 2 localhost connections 8158 0 0 0 3 localhost connections 8158 0 0 0 4 localhost connections 8158 0 0 0 5 localhost connections 8158 0 0 0 6 localhost connections 8158 0 1 0

Example 2 - ClusterXL High Availability

[Expert@HostName]# cphacu stat   Connectivity upgrade status: Disabled =====================================   The peer member is handling the traffic ======================================= Version of the local member: 2907 Version of the peer member : 2502   Connection table ================ HOST                  NAME                  ID #VALS #PEAK #SLINKS localhost             connections           8158    16    56      16