Maestro Auto-Scaling

Overview of Auto-Scaling

The Maestro Auto-Scaling feature assigns available Security Appliances (Scale Units) to a Security Group A logical group of Security Appliances that provides Active/Active cluster functionality. A Security Group can contain one or more Security Appliances. Security Groups work separately and independently from each other. To the production networks, a Security Group appears a single Security Gateway. Every Security Group contains: (A) Applicable Uplink ports, to which your production networks are connected; (B) Security Appliances (the Quantum Maestro Orchestrator determines the applicable Downlink ports automatically); (C) Applicable management port, to which the Check Point Management Server is connected. when certain conditions are met.

You configure these conditions on the Quantum Maestro Orchestrator A scalable Network Security System that connects multiple Check Point Security Appliances into a unified system. Synonyms: Orchestrator, Quantum Maestro Orchestrator, Maestro Hyperscale Orchestrator. Acronym: MHO. for each Security Group.

Prerequisites for Auto-Scaling

A Maestro Security Group must contain Security Appliances of the same model.

You must enable SMO Image Cloning in the Security Group.

Run in Gaia gClish The name of the global command line shell in Check Point Gaia operating system for Security Appliances connected to Check Point Quantum Maestro Orchestrators. Commands you run in this shell apply to all Security Appliances in the Security Group. on the Security Group:

set smo image auto-clone state on

show smo image auto-clone state

Important - Various procedures for installing software packages require you to disable SMO Image Cloning. After you install the required software packages, you must enable SMO Image Cloning again.

The Security Group must have internet connectivity for the Scale Unit to fetch the license from the User Center.

Limitations of Auto-Scaling

In R81.20, it is not supported to configure Auto-Scaling Settings if a Maestro Security Group contains different appliance models.
If the CPU utilization on the Security Group is high, the Orchestrator might consider the Security Group Members as "Expired".

Auto-Scaling Terms

Term	Definition
KPI	Key Performance Indicator
Scale Unit	A Security Appliance that can automatically be assigned to a Security Group, if a minimum of one "Scale Up" policy rule is met, or automatically removed from a Security Group if all "Scale Down" policy rules are met.
Scale Up policy	A set of rules configured based on the Security Group's KPIs. If a minimum of one rule is matched (for a consecutive amount of seconds), a Scale Unit (of the same hardware) is assigned to that Security Group.
Scale Down Policy	A set of rules configured based on the Security Group's KPIs. If a minimum of one rule is matched (for a consecutive amount of seconds), a Scale Unit is marked as a release candidate.
Release Candidate	A Scale Unit that is currently assigned to a Security Group is ready to be moved to another Security Group if one of its "Scale Up" policy rules is met.

Configuration of Auto-Scaling

Configure Auto-Scaling on the Orchestrator in Gaia Portal or Gaia Clish.

Configuration in Gaia Portal

Connect to the Gaia Portal on the Orchestrator.
From the left tree, click Orchestrator.
In the left pane Unassigned Gateways / assigned Gateways (if there is more than one appliance assigned to the Security Group), right-click a Security Appliance and click Set Scale Unit.
In the middle pane Topology, right-click the Security Group and click Set Security Group configuration.
Click the tab Auto-Scaling settings.

Configure the Scale up policy rules:

Click Add.

Select the applicable values in the fields:

Field	Available Values
When	Each Member Measures the value according to "Each Member" KPIs. "Asks" for new Security Appliances to be added to the Security Group when "Each Member" passes the threshold configured in the rule. Security Group (Average) Measures the value according to "Security Group (Average)" KPIs. "Asks" for new Security Appliances to be added to the Security Group when the "Security Group (Average)" passes the threshold configured in the rule.
With	CPU Utilization (%) Throughput (Gbps) Packets (p/s) Connections The KPI for the rule.
More than	Configure the threshold between: 1 and 100 for the CPU utilization metric. 1 and 1099511627776 (1TB) for other metrics.
For consecutive period of	Configure the duration between 1 and 86400 seconds (1 day).

Click OK.
In the Attach up to field, configure the number of Security Appliances to attach to this Security Group.

Configure the Scale down policy rules:

Click Add.

Select the applicable values in the fields:

Field	Available Values
When	Security Group (Average) Measures the value according to "Security Group (Average)" KPIs. Marks the scale unit that is currently occupied by the Security Group (if there is such a one) as a release candidate when the "Security Group (Average)" is less than the threshold configured in the rule. This means the Security Group can release the current scale unit.
With	CPU Utilization (%) Throughput (Gbps) Packets (p/s) Connections The KPI for the rule.
Less than	Configure the threshold between: 1 and 100 for the CPU utilization metric. 1 and 1099511627776 (1TB) for other metrics.
For consecutive period of	Configure the duration between 1 and 86400 seconds (1 day).

Click OK.
In the Detach single scale unit field, configure the number of Security Appliances to detach from this Security Group.

Click OK.

Configuration in Gaia Clish

Connect to the command line on the Orchestrator.
Log in to Gaia Clish.

Configure the Scale up policy rules:

add maestro auto-scale security-group-id <Security Group ID> scale-up-policy when {Each-Member | Security-Group-Average} with {CPU | bps | pps | connections} more-than <Threshold> period <Duration>

Configure the number of Security Appliances to attach to this Security Group when conditions match a minimum of one of the configured scale up rules:

set maestro auto-scale security-group-id <Security Group ID> scale-units-to-attach <1-4>

Configure the Scale down policy rules:

add maestro auto-scale security-group id <Security Group ID> scale-down-policy when <Security-Group-Average> with {CPU | bps | pps | connections} less-than <Threshold> period <Duration>

Enable Auto-Scaling in this Security Group:

set maestro auto-scale security-group-id <Security Group ID> state enable

Configure a scale Security Appliance on a specific site based on the Security Appliance serial number:

Important - If this Security Appliance is the only one in the Security Group (and Site), then do not configure it as a scale Security Appliance.

set maestro auto-scale site-id <Site ID> scale-unit-serial <Serial Number> state on

The applicable commands in Gaia Clish:

add maestro auto-scale
set maestro auto-scale
show maestro auto-scale
delete maestro auto-scale

Monitoring of Auto-Scaling

To view the current Security Groups and KPIs

Connect to the command line on the Orchestrator.
Log in.

Run this command:

In Gaia Clish:

show maestro auto-scale interactive-status

In the Expert mode:

mas_cli --interactive

Example output:

-------------------------------------------------------------------------------

Security Group: 1

    Site: 1
      Average:            | CPU Utilization: 9%         | Pckt/s: 8         | Bytes/s: 5652      | Connections: 384
      Average (w/o 1 SU): | CPU Utilization: 13%        | Pckt/s: 13        | Bytes/s: 8478      | Connections: 576

          Member: 1       | CPU Utilization: 7%         | Pckt/s: 12        | Bytes/s: 8562      | Connections: 372
          Member: 2       | CPU Utilization: 9%         | Pckt/s: 4         | Bytes/s: 2656      | Connections: 412      [Scale Unit]
          Member: 3       | CPU Utilization: 11%        | Pckt/s: 10        | Bytes/s: 5739      | Connections: 369      [Scale Unit]

-------------------------------------------------------------------------------

To view the current Auto-Scaling rules

Connect to the command line on the Orchestrator.
Log in.

Run this command:

In Gaia Clish:

show maestro auto-scale interactive-rules

In the Expert mode:

mas_cli --interactive-rules

Example output:

-------------------------------------------------------------------------------

Security Group: 1

  Scale Up Rules:
   ID 1: When Each member with CPU utilization exceeds 70% For 60 seconds | No Match

  Scale Down Rules:
   ID 1: When Security Group with CPU utilization less than 30% For 60 seconds | Match for 16 seconds

-------------------------------------------------------------------------------

Troubleshooting of Auto-Scaling

Auto-Scaling Service

To troubleshoot issues with Auto-Scaling, make sure that the masd service is running on the Orchestrator and Security Group Members:

On the Orchestrator:

Connect to the command line on the Orchestrator.
Log in to the Expert mode.

Get the status of the masd service:

service masd status

The output must be:

masd is running...

On the Security Group:

Connect to the command line on the Security Group.
Log in to the Expert mode.

Get the status of the masd service:

service masd status

The output must be:

masd is running...

Auto-Scaling Log File

Examine the logs of the masd daemon on the Orchestrator and Security Group Members:

On the Orchestrator:

Connect to the command line on the Orchestrator.
Log in to the Expert mode.

Examine the masd log file in real time:

tail -F /var/log/masd.elg

On the Security Group:

Connect to the command line on the Security Group.
Log in to the Expert mode.

Examine the masd log file in real time:

tail -F /var/log/masd.elg