Upgrading ElasticXL Environment from R82 to R82.10 - VSNext mode

This section describes the steps for upgrading an ElasticXL environment with Zero Downtime - as a Multi-Version Cluster (MVC).

This procedure supports only this upgrade path for an ElasticXL Security Group:

Source Version and Mode

Target Version and Mode

R82 in the VSNext mode

R82.10 in the VSNext mode

Important:

Important Notes for Security Groups:

  • Before you upgrade the ElasticXL, you must upgrade the Management Server that manages the ElasticXL.

    See the R82.10 Installation and Upgrade Guide.

  • This procedure applies to ElasticXL in the Gateway mode and the VSNext mode.

    In the VSNext mode, you must run all the commands in the context of VS0:

    • To change the context in Gaia gClish, run: set virtual-system 0

    • To change the context in the Expert mode, run: vsenv 0

  • During the upgrade process, it is:

    • Forbidden to change the configuration of the ElasticXL and its Security Group Members.

  • In this upgrade procedure, you divide all Security Group Members in the ElasticXL into two or more logical groups.

    In the procedure below, we use two logical groups denoted below as "A" and "B".

    You upgrade one logical group of the Security Group Members at one time.

    The other logical group(s) of the Security Group Members continues to handle traffic.

    Each logical group should contain the same number of Security Group Members - as close as possible.

    Environment

    Description

    Single Site

    • There are 3 Security Group Members in the ElasticXL.

    • The Logical Group "A" contains the Security Group Members 1_1 and 1_2.

    • The Logical Group "B" contains the Security Group Member 1_3.

    Single Site

    • There are 2 Security Group Members in the ElasticXL.

    • The Logical Group "A" contains the Security Group Member 1_1.

    • The Logical Group "B" contains the Security Group Member 1_2.

    Dual Site

    • There are 6 Security Group Members in the ElasticXL.

    • The Logical Group "A" contains the Security Group Member on Site 1 from 1_1 to 1_3.

    • The Logical Group "B" contains the Security Group Member on Site 2 from 2_1 to 2_3.

  • In a Dual Site environment:

    • We recommend to upgrade all Security Group Members in the ElasticXL on one Site and then upgrade all Security Group Members in the ElasticXL on the next Site.

    • To prevent a fail-over between Sites during the upgrade, we recommend these steps for the ElasticXL:

      1. Connect to the command line on the ElasticXL.

      2. If your default shell is the Expert mode (/bin/bash), then go to Gaia gClish:

        gclish

      3. Get the current Active/Standby mode and write it down (you need it at the end of the procedure):

        show cluster configuration high-availability mode

        Available Modes:

        Mode ID

        Mode Title

        Mode Description

        0

        Active/Standby - Active Up

        No primary Site.

        The currently Active Site stays Active until one of these changes:

        • The currently Active Site changes its state to "DOWN".

          If the formerly Active Site changes its state to "UP" again, it becomes the Standby Site.

        • The quality grade of the currently Standby Site becomes greater than the quality grade of the currently Active Site.

        1

        Active/Standby - Primary Up

        The Active Site always stays Active until one of these changes:

        • The currently Active Site changes its state to "DOWN".

          If the formerly Active Site changes its state to "UP" again, it becomes the Active Site again.

        • The quality grade of the currently Standby Site becomes greater than the quality grade of the currently Active Site.

        2

        Not available

        Not supported.

        This mode is reserved.

        3

        Standby Chassis VSLS Mode

        Not supported the VSNext mode.

        In the Traditional VSX mode, provides Virtual System Load Sharing.

      4. Decide which of the Site should be Active for this procedure.

      5. Change the Active/Standby mode to "Active/Standby - Primary Up" (value "1"):

        set cluster configuration high-availability mode 1

      6. Change the Site Priority - enter the number of the currently Active Site:

        set cluster configuration high-availability vs site_priority {1 | 2}

      7. Upgrade all Security Group Members on the Standby Site.

      8. On the upgraded Standby Site, change the Active/Standby mode to "Active/Standby - Primary Up" (value "1"):

        set cluster configuration high-availability mode 1

      9. On the upgraded Site change the Site Priority - enter the number of the currently Active Site:

        set cluster configuration high-availability vs site_priority {1 | 2}

      10. Upgrade all Security Group Members on the new Standby Site (former Active Site).

      11. Change the Active/Standby mode to the previous mode:

        set cluster configuration high-availability mode {0 | 1}

Required software packages:

Download the required software packages from sk183506:

  1. The required CPUSE Deployment Agent.

  2. The required Take of the Jumbo Hotfix Accumulator for the current version.

  3. The R82.10 Upgrade Package for Scalable Platforms.

Workflow:

  1. On the Management Server - Upgrade to the required version that can manage an R82.10 Security Group (see sk113113).

  2. On the Management Server - Change the software version in the object of each Virtual Gateway.

  3. On the Management Server - Prepare the policy package for the object of each Virtual Gateway.

  4. On the Security Group - Install the required CPUSE Deployment Agent package for the Security Group.

  5. On the Security Group - Install the required Jumbo Hotfix Accumulator for the current version (using two logical groups of Security Group Members).

  6. On the Security Group - Upgrade to R82.10 (using two logical groups of Security Group Members).

  7. On the Security Group - Make sure the upgrade was successful.

Procedure:

Upgrade the Security Management Server / Multi-Domain Security Management Server to the required version that can manage an R82.10 Security Group.

See the R82.10 Release Notes or sk113113.

Important - You must perform this step for each Security Gateway object that represents each Virtual Gateway.

Step

Instructions

A

Connect with SmartConsole to the Management Server that manages the Virtual Gateway.

B

From the left navigation panel, click Gateways & Servers.

C

Double-click the Security Gateway object.

D

In the left panel, click the General Properties page.

E

 Change the version to R82.10.

F

 Click OK.

G

Publish the SmartConsole session.

Important:

  1. You must perform this step for the Security Gateway object that represents the Security Group.

  2. You must perform this step for each Security Gateway object that represents each Virtual Gateway.

Note - See the Check Point Management API Reference (at the top, select the correct version) .

You can run the required API command on the Management Server in several ways (locally in CLI, remotely in CLI, remotely over HTTP).

The steps below show only the local command "mgmt_cli" in the Expert mode.

Step

Instructions

A

Connect to the command line on the Management Server.

B

Log in to the Expert mode.

C

Run this API command to prepare the policy package for the installation:

Note - On a Multi-Domain Security Management Server, enter the IP address of the correct Domain Management Server.

mgmt_cli -r true -d <IP Address of Management Server> install-policy policy-package <Name of Policy Package> targets <Name of Security Gateway Object> prepare-only true --sync false

Note - The SMO Image Cloning feature automatically clones all the required software packages to the Security Group Members during their boot. When you install or remove software packages gradually on Security Group Members, it is necessary to disable this feature, so that after a reboot the updated Security Group Members do not clone the software packages from the existing non-updated Security Group Members. Note - Enable the SMO Image Cloning feature only before you need to add a new Security Group Member and disable it immediately after it joins the Security Group.

Step

Instructions

A

Connect to the command line on the Security Group.

B

If your default shell is the Expert mode, then go to Gaia gClish:

gclish

C

Examine the state of the SMO Image Cloning feature:

show cluster configuration image auto-clone state

D

Disable the SMO Image Cloning feature, if it is enabled:

set cluster configuration image auto-clone state off

E

Examine the state of the SMO Image Cloning feature again:

show cluster configuration image auto-clone state

Important - You must do this step even if you upgrade again after a rollback procedure on the Security Group.

Step

Instructions

A

Transfer the required CPUSE Deployment Agent package (from sk92449) to the Security Group (into some directory, for example /var/log/).

B

Connect to the command line on the Security Group.

C

If your default shell is the Expert mode, then go to Gaia gClish:

gclish

D

Upgrade the CPUSE Deployment Agent:

installer agent install /<Full Path>/<Name of CPUSE Deployment Agent Package>

Example:

installer agent install /var/log/DeploymentAgent_XXXXXXXXX.tgz

E

Make sure all Security Group Members have the same build of the CPUSE Deployment Agent:

show installer status build

Important:

  • You must install the required Jumbo Hotfix Accumulator on all Security Group Members in the Security Group.

  • Before you install the Jumbo Hotfix Accumulator, you must make sure you disabled the SMO Image Cloning feature on the Security Group.

Step

Instructions

A

See the required Jumbo Hotfix Accumulator in sk183506.

B

Follow the Jumbo Hotfix Accumulator installation instructions in the R82 Scalable Platforms Administration Guide >

Chapter "Common Procedures for Scalable Platforms" >

Section "Installing and Uninstalling a Hotfix on Security Group Members".

Step

Instructions

A

Make sure you have the applicable CPUSE Offline package:

R82.10 Upgrade Package for Scalable Platforms

B

Transfer the CPUSE Offline package to the Security Group (into some directory, for example /var/log/).

C

Connect to the command line on the Security Group.

D

If your default shell is the Expert mode (/bin/bash), then go to Gaia gClish:

gclish

E

If you enabled MDPS (sk138672) on the Security Group, then go to the Management Plane:

set mdps environment mplane

F

Import the CPUSE Offline package from the hard disk:

installer import local /<Full Path>/<Name of the CPUSE Offline Package>

Example:

[Global] HostName-s01-01 > installer import local /var/log/Check_Point_R82.10_Gaia_Install_and_Upgrade.tar

G

Show the imported CPUSE packages:

show installer packages imported

H

Make sure the imported CPUSE package can be installed on this Security Group:

installer verify [Press Tab]

installer verify <Number of CPUSE Package> member_ids all

Example:

[Global] HostName-s01-01 > installer verify 2 member_ids all
... ...
Update Service Engine
+-----------------------------------------------------------+
|Member ID    |Status                                       |
+-----------------------------------------------------------+
|1_01 (local) |Upgrade is allowed.                          |
|1_02         |Upgrade is allowed.                          |
|1_03         |Upgrade is allowed.                          |
+-----------------------------------------------------------+
[Global] HostName-s01-01 >

Step

Instructions

A

Connect in one of these ways:

  • Connect to one of the Security Group Members in the Logical Group "A" through the console.

  • Connect to one of the Security Group Members in the Logical Group "B" over SSH.

B

Go to the context of one of the Security Group Members in the Logical Group "A":

member <Member ID>

Example:

member 1_1

C

If your default shell is Gaia gClish (/etc/gcli.sh), then go to the Expert mode:

expert

D

Go to the context of the Virtual Gateway with ID 0 (VS0):

vsenv 0

E

Set the Security Group Members in the Logical Group "A" to the state "DOWN" one by one:

Best Practice - To ensure correct traffic failover within the Security Group, change the state of Security Group Members in the logical group one by one.

Warning - During this upgrade procedure, you must use only the "g_clusterXL_admin" command in the Expert mode to change the state of Security Group Members.

Syntax:

g_clusterXL_admin –b <ID of first selected SGM in Group "A"> down

g_clusterXL_admin –b <ID of second selected SGM in Group "A"> down

and so on

Example:

g_clusterXL_admin –b 1_1 down

g_clusterXL_admin –b 1_2 down

F

Examine the state of the Security Group Members:

asg monitor

The state of the Security Group Members must be:

  • In the Logical Group "A" - "DOWN".

  • In the Logical Group "B" - "ACTIVE".

Warning - During this upgrade procedure, do not use the "insights" command to monitor the state of Security Group Members.

You are still working in the Expert mode in the Logical Group "A".

Step

Instructions

A

Go from the Expert mode to Gaia gClish:

  • If your default shell is the Expert mode (/bin/bash), then run:

    gclish

  • If your default shell is Gaia gClish (/etc/gcli.sh), then run:

    exit

B

Go to the context of the Virtual Gateway with ID 0 (VS0):

vsenv 0

C

Upgrade the Security Group Members in the Logical Group "A":

installer upgrade [Press the Tab key]

installer upgrade <Number of CPUSE Package> member_ids <SGM IDs in Group "A">

Example:

[Global] HostName-s01-01 > installer upgrade 2 member_ids 1_1-1_2
... ...
Update Service Engine
+-----------------------------------------------------------+
|Member ID    |Status                                       |
+-----------------------------------------------------------+
|1_01 (local) |Package is ready for installation            |
|1_02         |Package is ready for installation            |
+-----------------------------------------------------------+
The machines (1_02,1_02) will automatically reboot after upgrade.
Do you want to continue? ([y]es / [n]o) y
[Global] HostName-s01-01 >

D

Go from Gaia gClish to the Expert mode:

  • If your default shell is the Expert mode (/bin/bash), then run:

    exit

  • If your default shell is Gaia gClish (/etc/gcli.sh), then run:

    expert

E

Monitor the Security Group Members in the Logical Group "A" until they boot:

asg monitor

The state of the Security Group Members must be:

  • In the Logical Group "A" - "DOWN (READY)".

  • In the Logical Group "B" - "ACTIVE".

Warning - During this upgrade procedure, do not use the "insights" command to monitor the state of Security Group Members.

F

Make sure each Virtual Gateway installed its policy (Security Group Members pulled their policy automatically from the Management Server during their boot):

vsx stat -v

You are still working in the Expert mode in the Logical Group "A".

Step

Instructions

A

Go to the context of the Virtual Gateway with ID 0 (VS0):

vsenv 0

B

Set the Security Group Members in the Logical Group "A" to the state "UP" one by one:

Best Practice - To ensure correct traffic failover within the Security Group, change the state of Security Group Members in the logical group one by one.

Warning - During this upgrade procedure, you must use only the "g_clusterXL_admin" command in the Expert mode to change the state of Security Group Members.

Syntax:

g_clusterXL_admin –b <ID of first selected SGM in Group "A"> up

g_clusterXL_admin –b <ID of second selected SGM in Group "A"> up

and so on

Example:

g_clusterXL_admin –b 1_1 up

g_clusterXL_admin –b 1_2 up

C

Examine the state of the Security Group Members:

asg monitor

The state of the Security Group Members must be:

  • In the upgraded Logical Group "A" - "ACTIVE(R82.10)".

  • In the Logical Group "B" - "ACTIVE(<previous version>)".

Warning - During this upgrade procedure, do not use the "insights" command to monitor the state of Security Group Members.

Note - Because of the Multi-Version Cluster (MVC) mechanism design, the upgraded Security Group Members also process the traffic.

Important:

When the Security Group works in the VSX mode, the state of the upgraded Security Group Members may remain "DOWN" because the Critical Device "DSD" reports its state as "problem" (run the Gaia gClish command "show cluster info pnotes").

This issue may occur if before the upgrade you configured static CoreXL Affinity with the command "fw ctl affinity -s -d -fwkall <Number of CPUs>" (to see the current configuration, run the command "fw ctl affinity -l -x").

Follow these workaround steps:

  1. Connect to the command line on the Security Group.

  2. If the default shell is the Expert mode, then go to Gaia gClish:

    gclish

  3. Disable the CoreXL Dynamic Balancing:

    set dynamic-balancing state disable

  4. Reboot the upgraded Security Group Members:

    reboot -b <SGM IDs>

  5. Examine the state of the upgraded Security Group Members:

    show cluster info pnotes

    asg monitor

Important - In the Multi-Version Cluster mode, it is not necessary to choose all of the remaining Security Group Members that you did not upgrade yet.

You can repeat the previous steps for different logical groups of Security Group Members until you upgrade all Security Group Members in the Security Group.

Step

Instructions

A

Connect in one of these ways:

  • Connect to one of the Security Group Members in the Logical Group "B" through the console.

  • Connect to one of the Security Group Members in the Logical Group "A" over SSH.

B

Go to the context of one of the Security Group Members in the Logical Group "A":

member <Member ID>

Example:

member 1_3

C

If your default shell is Gaia gClish (/etc/gcli.sh), then go to the Expert mode:

expert

D

Go to the context of the Virtual Gateway with ID 0 (VS0):

vsenv 0

E

Set the Security Group Members in the Logical Group "A" to the state "DOWN" one by one:

Best Practice - To ensure correct traffic failover within the Security Group, change the state of Security Group Members in the logical group one by one.

Warning - During this upgrade procedure, you must use only the "g_clusterXL_admin" command in the Expert mode to change the state of Security Group Members.

Syntax:

g_clusterXL_admin –b <ID of first selected SGM in Group "B"> down

g_clusterXL_admin –b <ID of second selected SGM in Group "B"> down

and so on

Example:

g_clusterXL_admin –b 1_3 down

Note - At this time, the Security Group runs only with the upgraded Security Group Members.

You can perform the required tests on your network to make sure the new version works as expected.

If there are any issues:

  1. Set the Security Group Members in the Logical Group "B" to the state "UP".

  2. Set the Security Group Members in the upgraded Logical Group "A" to the state "DOWN".

You are still working in the Expert mode in the Logical Group "B".

Step

Instructions

A

Go to the context of one of the Security Group Members in the Logical Group "B":

member <Member ID>

Example:

member 1_3

B

Go from the Expert mode to Gaia gClish:

  • If your default shell is the Expert mode (/bin/bash), then run:

    gclish

  • If your default shell is Gaia gClish (/etc/gcli.sh), then run:

    exit

C

Go to the context of the Virtual Gateway with ID 0 (VS0):

set virtual-system 0

D

Upgrade the Security Group Members in the Logical Group "B":

installer upgrade [Press the Tab key]

installer upgrade <Number of CPUSE Package> member_ids <SGM IDs in Group "B">

Example:

[Global] HostName-s01-01 > installer upgrade 2 member_ids 1_3
... ...
Update Service Engine
+-----------------------------------------------------------+
|Member ID    |Status                                       |
+-----------------------------------------------------------+
|1_03 (local) |Package is ready for installation            |
+-----------------------------------------------------------+
The machines (1_03) will automatically reboot after upgrade.
Do you want to continue? ([y]es / [n]o) y
[Global] HostName-s01-01 >

E

Go from Gaia gClish to the Expert mode:

  • If your default shell is the Expert mode (/bin/bash), then run:

    exit

  • If your default shell is Gaia gClish (/etc/gcli.sh), then run:

    expert

F

Monitor the Security Group Members in the Logical Group "B" until they boot:

asg monitor

The state of the Security Group Members must be:

  • In the Logical Group "A" - "ACTIVE(R82.10)".

  • In the Logical Group "B" - "DOWN(READY)(R82.10)".

Warning - During this upgrade procedure, do not use the "insights" command to monitor the state of Security Group Members.

You are still working in the Expert mode in the Logical Group "B".

Step

Instructions

A

Go to the context of the Virtual Gateway with ID 0 (VS0):

vsenv 0

B

Set the Security Group Members in the Logical Group "B" to the state "UP" one by one:

Best Practice - To ensure correct traffic failover within the Security Group, change the state of Security Group Members in the logical group one by one.

Warning - During this upgrade procedure, you must use only the "g_clusterXL_admin" command in the Expert mode to change the state of Security Group Members.

Syntax:

g_clusterXL_admin –b <ID of first selected SGM in Group "B"> up

g_clusterXL_admin –b <ID of second selected SGM in Group "B"> up

and so on

Example:

g_clusterXL_admin –b 1_3 up

Step

Instructions

A

Connect to the command line on the Security Group.

B

If your default shell is Gaia gClish (/etc/gcli.sh), then go to the Expert mode:

expert

C

Examine the state of the Security Group Members:

asg monitor

Each Security Group Member must have the state "ACTIVE".

D

Run this command (see hcp):

hcp -m all -r all