Minimum Downtime Upgrade of a VSX Cluster

Best Practice - Use the Central Deployment in SmartConsole Check Point GUI application used to manage a Check Point environment - configure Security Policies, configure devices, monitor products and events, install updates, and so on.. For more information, see the R81.20 Security Management Administration Guide > Chapter Managing Gateways > Section Central Deployment of Hotfixes and Version Upgrades.

Warning - This is the behavior when you upgrade a VSX Virtual System Extension. Check Point virtual networking solution, hosted on a computer or cluster with virtual abstractions of Check Point Security Gateways and other network devices. These Virtual Devices provide the same functionality as their physical counterparts. Cluster Two or more Security Gateways that work together in a redundant configuration - High Availability, or Load Sharing. from R80.40 / R81 / R81.10, on which CoreXL Performance-enhancing technology for Security Gateways on multi-core processing platforms. Multiple Check Point Firewall instances are running in parallel on multiple CPU cores. Dynamic Balancing was not disabled explicitly, to R81.20 and then install the R81.20 Jumbo Hotfix Accumulator Collection of hotfixes combined into a single package. Acronyms: JHA, JHF, JHFA. (see sk164155 > Limitation PMTR-114499): CoreXL Dynamic Balancing will be enabled by default. Any previously configured manual affinity settings for interfaces / daemons will be overridden. As a workaround, follow this upgrade action plan to make sure CoreXL Dynamic Balancing stays disabled by default, and manual affinity settings are not overridden (if they exist): Upgrade the VSX Cluster Members to R81.20 and reboot. Connect to the command line on each VSX Cluster Member Security Gateway that is part of a cluster.. Log in to the Expert mode. Back up the $FWDIR/conf/dynamic_split.conf file: cp -v $FWDIR/conf/dynamic_split.conf{,_BKP} Edit the $FWDIR/conf/dynamic_split.conf file: vi $FWDIR/conf/dynamic_split.conf In this parameter, configure the value "1" (one): OFF_BY_DEFAULT_ON_VSX=1 Save the changes in the file and exit the editor. Install the R81.20 Jumbo Hotfix Software package installed on top of the current software version to fix a wrong or undesired behavior, and to add a new behavior. Accumulator on each Traditional VSX Cluster Member and reboot. Important - Perform the installation of a Jumbo Hotfix Accumulator as an upgrade in a Traditional VSX Cluster.

Important - Before you upgrade a VSX Cluster: Step Instructions 1 Back up your current configuration (see Backing Up and Restoring). Important - Back up both the Management Server Check Point Single-Domain Security Management Server or a Multi-Domain Security Management Server. and the VSX Cluster Members. Follow sk100395. 2 See the Upgrade Options and Prerequisites. 3 Upgrade the Management Server and Log Servers. 4 See Planning a Cluster Upgrade. 5 Schedule a full maintenance window to make sure you can make all the custom configurations again after the upgrade.

The procedure below describes an example VSX Cluster with three VSX Cluster Members M1, M2, and M3.

However, you can use it for clusters that consist of two or more Cluster Members.

Procedure:

1. On the Management Server, upgrade the configuration of the VSX Cluster object to R81.20

   Step	Instructions
   1	Connect to the command line on the Security Management Server Dedicated Check Point server that runs Check Point software to manage the objects and policies in a Check Point environment within a single management Domain. Synonym: Single-Domain Security Management Server. or Multi-Domain Server Dedicated Check Point server that runs Check Point software to host virtual Security Management Servers called Domain Management Servers. Synonym: Multi-Domain Security Management Server. Acronym: MDS. that manages this VSX Cluster.
   2	Log in to the Expert mode.
   3	On a Multi-Domain Server, go to the context of the Main Domain Management Server that manages this VSX Cluster object: mdsenv <IP Address or Name of Main Domain Management Server>
   4	Upgrade the configuration of the VSX Cluster object to R81.20: vsx_util upgrade This command is interactive.
   	Enter these details to log in to the management database: IP address of the Security Management Server or Main Domain Management Server that manages this VSX Cluster Management Server administrator's username Management Server administrator's password
   	Select your VSX Cluster.
   	Select R81.20.
   	For auditing purposes, save the vsx_util log file: On a Security Management Server: /opt/CPsuite-R81.20/fw1/log/vsx_util_YYYYMMDD_HH_MM.log On a Multi-Domain Server: /opt/CPmds-R81.20/customers/<Name_of_Domain>/CPsuite-R81.20/fw1/log/vsx_util_YYYYMMDD_HH_MM.log
   5	Connect with SmartConsole to the R81.20 Security Management Server or Main Domain Management Server that manages this VSX Cluster.
   6	From the left navigation panel, click Gateways & Servers.
   7	Open the VSX Cluster object.
   8	From the left tree, click the General Properties page.
   9	Make sure in the Platform section, the Version field shows R81.20.
   10	Click Cancel (do not click OK). Note - If you click OK, the Management Server pushes the VSX configuration to the VSX Cluster. Because the VSX Cluster is not upgraded yet, this operation would fail.

2. On each VSX Cluster Member, change the CCP mode to Broadcast

   Important - This step does not apply to R80.30 with Linux kernel 3.10 (run the "uname -r" command).

   Best Practice - To avoid possible problems with switches around the cluster during the upgrade, we recommend to change the Cluster Control Protocol (CCP) mode to Broadcast.

   Step	Instructions
   1	Connect to the command line on each VSX Cluster Member.
   2	Log in to the Expert mode.
   3	Change the CCP mode to Broadcast: cphaconf set_ccp broadcast Notes: This change does not require a reboot. This change applies immediately and survives reboot.
   4	Make sure the CCP mode is set to Broadcast: cphaprob -a if

3. On the VSX Cluster Member M3, upgrade to R81.20 with CPUSE, or perform a Clean Install of R81.20

   Important - You must reboot the VSX Cluster Member after the upgrade or clean install.

   Installation Method	Instructions
   Upgrade to R81.20 with CPUSE Check Point Upgrade Service Engine for Gaia Operating System. With CPUSE, you can automatically update Check Point products for the Gaia OS, and the Gaia OS itself.	See Installing Software Packages on Gaia. Follow the applicable action plan for the local or central installation. In local installation, select the R81.20 package and perform Upgrade. See sk92449 for detailed steps.
   Clean Install Installation of a Check Point Operating System from scratch on a computer. of R81.20 with CPUSE	Follow these steps: See Installing Software Packages on Gaia. Follow the applicable action plan for the local or central installation. In local installation, select the R81.20 package and perform Clean Install. See sk92449 for detailed steps. Important - In the Gaia Check Point security operating system that combines the strengths of both SecurePlatform and IPSO operating systems. First Time Configuration Wizard, for the Management Connection IP address, you must use the same IP address as was used by the previous VSX Cluster Member (prior to the upgrade). Run the "vsx_util reconfigure" command on the Management Server to push the VSX configuration to this VSX Cluster Member. See the R81.20 VSX Administration Guide > Chapter Command Line Reference > Section vsx_util > Section vsx_util reconfigure. Important - You must enter the same Activation Key you entered during the Gaia First Time Configuration Wizard of this VSX Cluster Member. Configure the required settings on this VSX Cluster Member: OS configuration (for example, DNS, NTP, DHCP, Dynamic Routing, DHCP Relay, and so on). Settings manually defined in various configuration files. Applicable Check Point configuration files.
   Clean Install of R81.20 from scratch	Follow these steps: Follow Installing a VSX Cluster - only the step "Install the VSX Cluster Members". Important - In the Gaia First Time Configuration Wizard, for the Management Connection IP address, you must use the same IP address as was used by the previous VSX Cluster Member (prior to the upgrade). Run the "vsx_util reconfigure" command on the Management Server to push the VSX configuration to this VSX Cluster Member. See the R81.20 VSX Administration Guide > Chapter Command Line Reference > Section vsx_util > Section vsx_util reconfigure. Important - You must enter the same Activation Key you entered during the Gaia First Time Configuration Wizard of this VSX Cluster Member. Configure the required settings on this VSX Cluster Member: OS configuration (for example, DNS, NTP, DHCP, Dynamic Routing, DHCP Relay, and so on). Settings manually defined in various configuration files. Applicable Check Point configuration files.

4. On the VSX Cluster Member M2, upgrade to R81.20 with CPUSE, or perform a Clean Install of R81.20

   Important - You must reboot the VSX Cluster Member after the upgrade or clean install.

   Installation Method	Instructions
   Upgrade to R81.20 with CPUSE	See Installing Software Packages on Gaia. Follow the applicable action plan for the local or central installation. In local installation, select the R81.20 package and perform Upgrade. See sk92449 for detailed steps.
   Clean Install of R81.20 with CPUSE	Follow these steps: See Installing Software Packages on Gaia. Follow the applicable action plan for the local or central installation. In local installation, select the R81.20 package and perform Clean Install. See sk92449 for detailed steps. Important - In the Gaia First Time Configuration Wizard, for the Management Connection IP address, you must use the same IP address as was used by the previous VSX Cluster Member (prior to the upgrade). Run the "vsx_util reconfigure" command on the Management Server to push the VSX configuration to this VSX Cluster Member. See the R81.20 VSX Administration Guide > Chapter Command Line Reference > Section vsx_util > Section vsx_util reconfigure. Important - You must enter the same Activation Key you entered during the Gaia First Time Configuration Wizard of this VSX Cluster Member. Configure the required settings on this VSX Cluster Member: OS configuration (for example, DNS, NTP, DHCP, Dynamic Routing, DHCP Relay, and so on). Settings manually defined in various configuration files. Applicable Check Point configuration files.
   Clean Install of R81.20 from scratch	Follow these steps: Follow Installing a VSX Cluster - only the step "Install the VSX Cluster Members". Important - In the Gaia First Time Configuration Wizard, for the Management Connection IP address, you must use the same IP address as was used by the previous VSX Cluster Member (prior to the upgrade). Run the "vsx_util reconfigure" command on the Management Server to push the VSX configuration to this VSX Cluster Member. See the R81.20 VSX Administration Guide > Chapter Command Line Reference > Section vsx_util > Section vsx_util reconfigure. Important - You must enter the same Activation Key you entered during the Gaia First Time Configuration Wizard of this VSX Cluster Member. Configure the required settings on this VSX Cluster Member: OS configuration (for example, DNS, NTP, DHCP, Dynamic Routing, DHCP Relay, and so on). Settings manually defined in various configuration files. Applicable Check Point configuration files.

5. In SmartConsole, establish SIC with the VSX Cluster Member M3

   Important - This step is required only if you performed a Clean Install of R81.20 on this VSX Cluster Member.

   Step	Instructions
   1	Connect with SmartConsole to the R81.20 Security Management Server or Main Domain Management Server that manages this VSX Cluster.
   2	From the left navigation panel, click Gateways & Servers.
   3	Open the cluster object.
   4	From the left tree, click Cluster Members.
   5	Select the object of this VSX Cluster Member.
   6	Click Edit.
   7	On the General tab, click the Communication button.
   8	Click Reset.
   9	In the One-time password field, enter the same Activation Key you entered during the First Time Configuration Wizard of the Cluster Member.
   10	In the Confirm one-time password field, enter the same Activation Key again.
   11	Click Initialize.
   12	The Trust state field must show Trust established.
   13	Click Close to close the Communication window.
   14	Click OK to close the Cluster Member Properties window.
   15	Click OK to close the Gateway Cluster Properties window.
   16	Publish the SmartConsole session.

6. In SmartConsole, establish SIC with the VSX Cluster Member M2

   Important - This step is required only if you performed a Clean Install of R81.20 on this VSX Cluster Member.

   Step	Instructions
   1	Connect with SmartConsole to the R81.20 Security Management Server or Main Domain Management Server that manages this VSX Cluster.
   2	From the left navigation panel, click Gateways & Servers.
   3	Open the cluster object.
   4	From the left tree, click Cluster Members.
   5	Select the object of this VSX Cluster Member.
   6	Click Edit.
   7	On the General tab, click the Communication button.
   8	Click Reset.
   9	In the One-time password field, enter the same Activation Key you entered during the First Time Configuration Wizard of the Cluster Member.
   10	In the Confirm one-time password field, enter the same Activation Key again.
   11	Click Initialize.
   12	The Trust state field must show Trust established.
   13	Click Close to close the Communication window.
   14	Click OK to close the Cluster Member Properties window.
   15	Click OK to close the Gateway Cluster Properties window.
   16	Publish the SmartConsole session.

7. In SmartConsole, install the Access Control Policy

   Step	Instructions
   1	Connect with SmartConsole to the R81.20 Security Management Server or Main Domain Management Server that manages this VSX Cluster.
   2	From the left navigation panel, click Gateways & Servers.
   3	Click Install Policy.
   4	In the Install Policy window: In the Policy field, select the default policy for this VSX Cluster object. This policy is called: <Name of VSX Cluster object>_VSX In the Install Mode section, configure these two options: Select Install on each selected gateway independently. Clear For gateway clusters, if installation on a cluster member fails, do not install on that cluster. Click Install.
   5	The policy installation: Succeeds on the upgraded VSX Cluster Members M2 and M3. Fails on the old VSX Cluster Member M1 with a warning. Ignore this warning.

8. On each VSX Cluster Member, examine the VSX configuration and cluster state

   Step	Instructions
   1	Connect to the command line on each VSX Cluster Member.
   2	Log in to the Expert mode.
   3	Examine the VSX configuration: vsx stat -v Important: Make sure all the configured Virtual Devices are loaded. Make sure all Virtual Systems and Virtual Routers have SIC Secure Internal Communication. The Check Point proprietary mechanism with which Check Point computers that run Check Point software authenticate each other over SSL, for secure communication. This authentication is based on the certificates issued by the ICA on a Check Point Management Server. Trust and policy.
   4	Examine the cluster state in one of these ways: In Gaia Clish The name of the default command line shell in Check Point Gaia operating system. This is a restricted shell (role-based administration controls the number of commands available in the shell). (R80.20 and higher), run: set virtual-system 0 show cluster state In the Expert mode, run: vsenv 0 cphaprob state Important: The cluster states of the upgraded VSX Cluster Members M2 and M3 are Ready. The cluster state of the old VSX Cluster Member M1 is: In R80.20 and higher - Active(!). In R80.10 and lower - Active Attention.

9. On the old VSX Cluster Member M1, stop all Check Point services

   Step	Instructions
   1	Connect to the command line on the VSX Cluster Member M1.
   2	Stop all Check Point services: cpstop Notes: This forces a controlled cluster failover from the old VSX Cluster Member M1 to one of the upgraded VSX Cluster Members. At this moment, all connections that were initiated through the old VSX Cluster Member M1 are dropped (because VSX Cluster Members with different software versions cannot synchronize).

10. On the upgraded VSX Cluster Members M2 and M3, examine the cluster state

    Step	Instructions
    1	Connect to the command line on each Cluster Member M2 and M3.
    2	Examine the cluster state in one of these ways: In Gaia Clish, run: show cluster state In the Expert mode, run: cphaprob state Important: One of the VSX Cluster Members (M2 or M3) changes its cluster state to Active. The other VSX Cluster Member (M2 or M3) changes its cluster state to Standby.

11. On the old VSX Cluster Member M1, upgrade to R81.20 with CPUSE, or perform a Clean Install of R81.20

    Important - You must reboot the VSX Cluster Member after the upgrade or clean install.

    Installation Method	Instructions
    Upgrade to R81.20 with CPUSE	See Installing Software Packages on Gaia. Follow the applicable action plan for the local or central installation. In local installation, select the R81.20 package and perform Upgrade. See sk92449 for detailed steps.
    Clean Install of R81.20 with CPUSE	Follow these steps: See Installing Software Packages on Gaia. Follow the applicable action plan for the local or central installation. In local installation, select the R81.20 package and perform Clean Install. See sk92449 for detailed steps. Important - In the Gaia First Time Configuration Wizard, for the Management Connection IP address, you must use the same IP address as was used by the previous VSX Cluster Member (prior to the upgrade). Run the "vsx_util reconfigure" command on the Management Server to push the VSX configuration to this VSX Cluster Member. See the R81.20 VSX Administration Guide > Chapter Command Line Reference > Section vsx_util > Section vsx_util reconfigure. Important - You must enter the same Activation Key you entered during the Gaia First Time Configuration Wizard of this VSX Cluster Member. Configure the required settings on this VSX Cluster Member: OS configuration (for example, DNS, NTP, DHCP, Dynamic Routing, DHCP Relay, and so on). Settings manually defined in various configuration files. Applicable Check Point configuration files.
    Clean Install of R81.20 from scratch	Follow these steps: Follow Installing a VSX Cluster - only the step "Install the VSX Cluster Members". Important - In the Gaia First Time Configuration Wizard, for the Management Connection IP address, you must use the same IP address as was used by the previous VSX Cluster Member (prior to the upgrade). Run the "vsx_util reconfigure" command on the Management Server to push the VSX configuration to this VSX Cluster Member. See the R81.20 VSX Administration Guide > Chapter Command Line Reference > Section vsx_util > Section vsx_util reconfigure. Important - You must enter the same Activation Key you entered during the Gaia First Time Configuration Wizard of this VSX Cluster Member. Configure the required settings on this VSX Cluster Member: OS configuration (for example, DNS, NTP, DHCP, Dynamic Routing, DHCP Relay, and so on). Settings manually defined in various configuration files. Applicable Check Point configuration files.

12. In SmartConsole, establish SIC with the VSX Cluster Member M1

    Important - This step is required only if you performed a Clean Install of R81.20 on this VSX Cluster Member.

    Step	Instructions
    1	Connect with SmartConsole to the R81.20 Security Management Server or Main Domain Management Server that manages this VSX Cluster.
    2	From the left navigation panel, click Gateways & Servers.
    3	Open the cluster object.
    4	From the left tree, click Cluster Members.
    5	Select the object of this VSX Cluster Member.
    6	Click Edit.
    7	On the General tab, click the Communication button.
    8	Click Reset.
    9	In the One-time password field, enter the same Activation Key you entered during the First Time Configuration Wizard of the Cluster Member.
    10	In the Confirm one-time password field, enter the same Activation Key again.
    11	Click Initialize.
    12	The Trust state field must show Trust established.
    13	Click Close to close the Communication window.
    14	Click OK to close the Cluster Member Properties window.
    15	Click OK to close the Gateway Cluster Properties window.
    16	Publish the SmartConsole session.

13. In SmartConsole, install the policy

    Step	Instructions
    1	Connect with SmartConsole to the R81.20 Security Management Server or Main Domain Management Server that manages this VSX Cluster.
    2	From the left navigation panel, click Gateways & Servers.
    3	Install the default policy on the VSX Cluster object: Click Install Policy. In the Policy field, select the default policy for this VSX Cluster object. This policy is called: <Name of VSX Cluster object>_VSX In the Install Mode section, select these two options: Install on each selected gateway independently For gateway clusters, if installation on a cluster member fails, do not install on that cluster Click Install. The default policy install successfully on all the VSX Cluster Members.
    4	Install the Threat Prevention Policy on the VSX Cluster object: Click Install Policy. In the Policy field, select the applicable Threat Prevention Policy for this VSX Cluster object. Click Install. The Threat Prevention Policy must install successfully on all the VSX Cluster Members.

14. On each VSX Cluster Member, examine the VSX configuration and cluster state

    Step	Instructions
    1	Connect to the command line on each VSX Cluster Member.
    2	Log in to the Expert mode.
    3	Examine the VSX configuration: vsx stat -v Important: Make sure all the configured Virtual Devices are loaded. Make sure all Virtual Systems and Virtual Routers have SIC Trust and policy.
    4	Examine the cluster state in one of these ways: In Gaia Clish, run: set virtual-system 0 show cluster state In the Expert mode, run: vsenv 0 cphaprob state Important: All VSX Cluster Members must show the same information about the states of all VSX Cluster Members. In the High Availability mode, one VSX Cluster Member must be in the Active state, and all other VSX Cluster Members must be in Standby state. In the Virtual System Load Sharing mode, all VSX Cluster Members must be in the Active state. All Virtual Systems must show the same information about the states of all Virtual Systems.
    5	Examine the cluster interfaces in one of these ways: In Gaia Clish, run: set virtual-system 0 show cluster members interfaces all In the Expert mode, run: vsenv 0 cphaprob -a if

15. Test the functionality

    Step	Instructions
    1	Connect with SmartConsole to the R81.20 Security Management Server or each Target Domain Management Server that manages the Virtual Systems on this VSX Cluster.
    2	From the left navigation panel, click Logs & Monitor > Logs.
    3	Examine the logs from the Virtual Systems on this VSX Cluster to make sure they inspect the traffic as expected.

For more information, see the:

• R81.20 VSX Administration Guide.

• R81.20 CLI Reference Guide.