Rolling Back a Failed Upgrade of a Security Group from R82 - Zero Downtime (MVC)

This section describes the steps to roll back a failed upgrade of a Security Group from R82 with Zero Downtime - as a Multi-Version Cluster (MVC).

For upgrade procedures, see:

This section describes the steps for rolling back a failed upgrade of a Security Group to R82.

This procedure supports only these downgrade paths for Security Groups:

from R82 to R81.10
from R82 to R81

Warnings:

Multi-Version Cluster (Zero Downtime) downgrade from R82 to R81.10 / R81 is not supported if a Security Group has Bond interfaces in the 802.3ad (LACP) mode on Uplink ports (Known Limitation PMTR-88191).
Before you follow the downgrade procedure, revert all changes in the topology you made after the upgrade procedure. For example, after the upgrade you added / removed interfaces, you changed the configuration of interfaces, you added / removed Security Group Members in the Security Group.

Rolling Back If Only Some of the Security Group Members Were Upgraded - Zero Downtime

Important - Use this rollback procedure if you upgraded only some (not all) Security Group Members in the Security Group.

Step

Instructions

Connect to the command line on the Security Group.

If your default shell is the Expert mode (/bin/bash), then go to the Gaia gClish:

gclish

Disable the SMO Image Cloning feature:

Note - The SMO Image Cloning feature automatically clones all the required software packages to the Security Group Members during their boot. When you install or remove software packages gradually on Security Group Members, it is necessary to disable this feature, so that after a reboot the updated Security Group Members do not clone the software packages from the existing non-updated Security Group Members.

Examine the state of the SMO Image Cloning feature:

show cluster configuration image auto-clone state

See show cluster configuration image.

Disable the SMO Image Cloning feature, if it is enabled:

set cluster configuration image auto-clone state off

See set cluster configuration image.

Examine the state of the SMO Image Cloning feature:

show cluster configuration image auto-clone state

Go to the Expert mode:

If your default shell is the Expert mode (/bin/bash):

exit

If your default shell is Gaia gClish (/etc/gcli.sh):

expert

Go to the context of one of the Security Group Members that were upgraded to R82:

member <Member ID>

See member.

Example:

member 1_1

On each Security Group Member that was upgraded to R82, restore the Gaia snapshot that was automatically created before the upgrade.

In Dual Site, follow these steps to restore a Gaia snapshot:

Procedure

Do these steps on each upgraded Security Group Member on the Standby Site:

Go to the context of the upgraded Security Group Member:

member <ID of Security Group Member>

Example:

member 2_5

Go to Gaia Clish (do not use the Gaia gClish):

clish

Restore the Gaia snapshot that was saved automatically before the upgrade.

set snapshot revert AutoSnapShot_<Original-Version>_<Take>

Example:

set snapshot revert AutoSnapShot_AutoSnapShot_R8120_47

Wait for the Security Group Member to complete the reboot.

After you revert each upgraded Security Group Member on the Standby Site, examine the state of all Security Group Members (write down the ID of the Standby Site):

asg stat

See asg stat.

Note - By design, the state of the reverted Security Group Members will be "DOWN".

Change the state of all Security Group Members ‎to "UP" on the Standby Site:

g_clusterXL_admin -b {chassis1 | chassis2} up‎

See g_clusterXL_admin.

Where:

chassis1 - Applies to all Security Group Members on Site 1
chassis2 - Applies to all Security Group Members on Site 2

Make sure all Security Group Members are in the state "ACTIVE":

asg stat

Manually fail over from the Active Site to the Standby Site:

g_clusterXL_admin -b {chassis1 | chassis2} down‎

Where:

chassis1 - Applies to all Security Group Members on Site 1
chassis2 - Applies to all Security Group Members on Site 2

Examine the state of all Security Group Members:

asg stat

Repeat Steps 6, a-d for Dual Site on each upgraded Security Group Member on the former Active Site.

At the end of this procedure all Security Group Members must be in the state "ACTIVE".

In Single Site, follow these steps to restore a Gaia snapshot:

Procedure

Do these steps on each upgraded Security Group Member:

Go to the context of the upgraded Security Group Member:

member <ID of Security Group Member>

Example:

member 1_5

Go to Gaia Clish (do not use the Gaia gClish):

clish

Restore the Gaia snapshot that was saved automatically before the upgrade.

set snapshot revert AutoSnapShot_<Original-Version>_<Take>

Example:

set snapshot revert AutoSnapShot_AutoSnapShot_R8120_47

Wait for the Security Group Member to complete the reboot.

After you revert each upgraded Security Group Member, examine the state of all Security Group Members:

asg stat

See asg stat.

Note - By design, the state of the reverted Security Group Members will be "DOWN".

Change the state of all reverted Security Group Members ‎to "UP":

g_clusterXL_admin -b <SGM ID 1>,<SGM ID 2>,...,<SGM ID N> up‎

See g_clusterXL_admin.

Make sure all Security Group Members are in the state "ACTIVE":

asg stat

Make sure the downgrade was successful.

Use one of these tools:

The "insights" tool (see insights):

insights

The HCP tool (see hcp):

hcp --help

Rolling Back the Whole Security Group - Zero Downtime

Use this rollback procedure if you upgraded allSecurity Group Members in the Security Group and it is necessary to keep the current connections.

Important:

This procedure does not interrupt the traffic and does not require down time.
In this rollback procedure, you divide all upgraded Security Group Members in a specific Security Group into two logical groups - denoted below as "A" and "B".

You revert one logical group of the Security Group Members at one time.

The other logical group of the Security Group Members continues to handle traffic.

Each logical group should contain the same number of Security Group Members - as close as possible.

Example 1:
- There are 8 Security Group Members in the Security Group.
- The Logical Group "A" contains Security Group Members from 1_1 to 1_4.
- The Logical Group "B" contains Security Group Members from 1_5 to 1_8.
Example 2:
- There are 5 Security Group Members in the Security Group.
- The Logical Group "A" contains Security Group Members from 1_1 to 1_3.
- The Logical Group "B" contains Security Group Members 1_4 and 1_5.

Procedure

Step

Instructions

Connect to the command line on the Security Group.

If your default shell is the Expert mode (/bin/bash), then go to the Gaia gClish:

gclish

Disable the SMO Image Cloning feature:

Examine the state of the SMO Image Cloning feature:

show cluster configuration image auto-clone state

See show cluster configuration image.

Disable the SMO Image Cloning feature, if it is enabled:

set cluster configuration image auto-clone state off

See set cluster configuration image.

Examine the state of the SMO Image Cloning feature:

show cluster configuration image auto-clone state

Go to the Expert mode:

If your default shell is the Expert mode (/bin/bash):

exit

If your default shell is Gaia gClish (/etc/gcli.sh):

expert

Go to the context of one of the Security Group Members that were upgraded to R82:

member <Member ID>

See member.

Example:

member 1_1

On each Security Group Member that was upgraded to R82, restore the Gaia snapshot that was automatically created before the upgrade.

In Dual Site, follow these steps to restore a Gaia snapshot:

Procedure

Do these steps on each upgraded Security Group Member on the Standby Site:

Go to the context of the upgraded Security Group Member:

member <ID of Security Group Member>

Example:

member 2_5

Go to Gaia Clish (do not use the Gaia gClish):

clish

Restore the Gaia snapshot that was saved automatically before the upgrade.

set snapshot revert AutoSnapShot_<Original-Version>_<Take>

Example:

set snapshot revert AutoSnapShot_AutoSnapShot_R8120_47

Wait for the Security Group Member to complete the reboot.

After you revert each upgraded Security Group Member on the Standby Site, examine the state of all Security Group Members (write down the ID of the Standby Site):

asg stat

See asg stat.

Note - By design, the state of the reverted Security Group Members will be "DOWN".

Change the state of all Security Group Members ‎to "UP" on the Standby Site:

g_clusterXL_admin -b {chassis1 | chassis2} up‎

See g_clusterXL_admin.

Where:

chassis1 - Applies to all Security Group Members on Site 1
chassis2 - Applies to all Security Group Members on Site 2

Make sure all Security Group Members are in the state "ACTIVE":

asg stat

Manually fail over from the Active Site to the Standby Site:

g_clusterXL_admin -b {chassis1 | chassis2} down‎

Where:

chassis1 - Applies to all Security Group Members on Site 1
chassis2 - Applies to all Security Group Members on Site 2

Examine the state of all Security Group Members:

asg stat

Repeat Steps 6, a-d for Dual Site on each upgraded Security Group Member on the former Active Site.

At the end of this procedure all Security Group Members must be in the state "ACTIVE".

In Single Site, follow these steps to restore a Gaia snapshot:

Procedure

Divide all Security Group Members into two logical groups - Logical Group "A" (for example 1_1-1_4) and Logical Group "B" (for example, 1_5-1_8‎).

Do these steps on each upgraded Security Group Member in the Logical Group "A":

Go to the context of the upgraded Security Group Member:

member <ID of Security Group Member>

Example:

member 1_1

Go to Gaia Clish (do not use the Gaia gClish):

clish

Restore the Gaia snapshot that was saved automatically before the upgrade.

set snapshot revert AutoSnapShot_<Original-Version>_<Take>

Example:

set snapshot revert AutoSnapShot_AutoSnapShot_R8120_47

Wait for the Security Group Member to complete the reboot.

After you revert each upgraded Security Group Member in the Logical Group "A", examine the state of all Security Group Members:

asg stat

Note - By design, the state of the reverted Security Group Members will be "DOWN".

See asg stat.

Change the state of all reverted Security Group Members ‎in the Logical Group "A" to "UP":

g_clusterXL_admin -b <SGM IDs in Logical Group "A"> up‎

See g_clusterXL_admin.

Example:

g_clusterXL_admin -b 1_1-1_4 up‎

Examine the state of all Security Group Members:

asg stat

Manually fail over from the Security Group Members ‎in the Logical Group "B" (not reverted yet) to the Security Group Members ‎in the Logical Group "A" (already reverted):

g_clusterXL_admin -b <SGM IDs in Logical Group "B"> down‎

Example:

g_clusterXL_admin -b 1_5-1_8 down‎

Examine the state of all Security Group Members:

asg stat

Repeat Steps 6, a-d for Single Site on each upgraded Security Group Member in the Logical Group "B".

At the end of this procedure all Security Group Members must be in the state "ACTIVE".

Make sure the downgrade was successful.

Use one of these tools:

The "insights" tool (see insights):

insights

The HCP tool (see hcp):

hcp --help