Maestro Auto-Scaling

Overview

The Maestro Auto-Scaling feature assigns available Security Appliances (Scale Units) to a Security Group when certain conditions are met.

You configure these conditions on the Quantum Maestro Orchestrator for each Security Group.

Prerequisites

  • A Maestro Security Group must contain Security Appliances of the same model.

  • You must enable SMO Image Cloning in the Security Group.

    Run in Gaia gClish on the Security Group:

    set smo image auto-clone state on

    show smo image auto-clone state

    Important - Various procedures for installing software packages require you to disable SMO Image Cloning. After you install the required software packages, you must enable SMO Image Cloning again.

  • The Security Group must have internet connectivity for the Scale Unit to fetch the license from the User Center.

Limitations

  • In R81.20, it is not supported to configure Auto-Scaling Settings if a Maestro Security Group contains different appliance models.

  • If the CPU utilization on the Security Group is high, the Orchestrator might consider the Security Group Members as "Expired".

Terms

Term

Definition

KPI

Key Performance Indicator

Scale Unit

A Security Appliance that can automatically be assigned to a Security Group, if a minimum of one "Scale Up" policy rule is met, or automatically removed from a Security Group if all "Scale Down" policy rules are met.

Scale Up policy

A set of rules configured based on the Security Group's KPIs.

If a minimum of one rule is matched (for a consecutive amount of seconds), a Scale Unit (of the same hardware) is assigned to that Security Group.

Scale Down Policy

A set of rules configured based on the Security Group's KPIs.

If a minimum of one rule is matched (for a consecutive amount of seconds), a Scale Unit is marked as a release candidate.

Release Candidate

A Scale Unit that is currently assigned to a Security Group is ready to be moved to another Security Group if one of its "Scale Up" policy rules is met.

Configuration

Configure Auto-Scaling on the Orchestrator in Gaia Portal or Gaia Clish.

  1. Connect to the Gaia Portal on the Orchestrator.

  2. From the left tree, click Orchestrator.

  3. In the left pane Unassigned Gateways / assigned Gateways (if there is more than one appliance assigned to the Security Group), right-click a Security Appliance and click Set Scale Unit.

  4. In the middle pane Topology, right-click the Security Group and click Set Security Group configuration.

  5. Click the tab Auto-Scaling settings.

  6. Configure the Scale up policy rules:

    1. Click Add.

    2. Select the applicable values in the fields:

      Field

      Available Values

      When

      Each Member

      Measures the value according to "Each Member" KPIs.

      "Asks" for new Security Appliances to be added to the Security Group when "Each Member" passes the threshold configured in the rule.

      Security Group (Average)
      Measures the value according to "Security Group (Average)" KPIs.

      "Asks" for new Security Appliances to be added to the Security Group when the "Security Group (Average)" passes the threshold configured in the rule.

      With

      CPU Utilization (%)

      Throughput (Gbps)

      Packets (p/s)

      Connections

      The KPI for the rule.

      More than

      Configure the threshold between:

      • 1 and 100 for the CPU utilization metric.

      • 1 and 1099511627776 (1TB) for other metrics.

      For consecutive period of

      Configure the duration between 1 and 86400 seconds (1 day).

    3. Click OK.

    4. In the Attach up to field, configure the number of Security Appliances to attach to this Security Group.

  7. Configure the Scale down policy rules:

    1. Click Add.

    2. Select the applicable values in the fields:

      Field

      Available Values

      When

      Security Group (Average)

      Measures the value according to "Security Group (Average)" KPIs.

      Marks the scale unit that is currently occupied by the Security Group (if there is such a one) as a release candidate when the "Security Group (Average)" is less than the threshold configured in the rule. This means the Security Group can release the current scale unit.

      With

      CPU Utilization (%)

      Throughput (Gbps)

      Packets (p/s)

      Connections

      The KPI for the rule.

      Less than

      Configure the threshold between:

      • 1 and 100 for the CPU utilization metric.

      • 1 and 1099511627776 (1TB) for other metrics.

      For consecutive period of

      Configure the duration between 1 and 86400 seconds (1 day).

    3. Click OK.

    4. In the Detach single scale unit field, configure the number of Security Appliances to detach from this Security Group.

  8. Click OK.

  1. Connect to the command line on the Orchestrator.

  2. Log in to Gaia Clish.

  3. Configure the Scale up policy rules:

    add maestro auto-scale security-group-id <Security Group ID> scale-up-policy when {Each-Member | Security-Group-Average} with {CPU | bps | pps | connections} more-than <Threshold> period <Duration>

  4. Configure the number of Security Appliances to attach to this Security Group when conditions match a minimum of one of the configured scale up rules:

    set maestro auto-scale security-group-id <Security Group ID> scale-units-to-attach <1-4>

  5. Configure the Scale down policy rules:

    add maestro auto-scale security-group id <Security Group ID> scale-down-policy when <Security-Group-Average> with {CPU | bps | pps | connections} less-than <Threshold> period <Duration>

  6. Enable Auto-Scaling in this Security Group:

    set maestro auto-scale security-group-id <Security Group ID> state enable

  7. Configure a scale Security Appliance on a specific site based on the Security Appliance serial number:

    Important - If this Security Appliance is the only one in the Security Group (and Site), then do not configure it as a scale Security Appliance.

    set maestro auto-scale site-id <Site ID> scale-unit-serial <Serial Number> state on

The applicable commands in Gaia Clish:

  • add maestro auto-scale

  • set maestro auto-scale

  • show maestro auto-scale

  • delete maestro auto-scale

Monitoring on Orchestrator

  1. Connect to the command line on the Orchestrator.

  2. Log in to Gaia Clish.

  3. Run:

    show maestro auto-scale security-group-id <Security Group ID> state

  1. Connect to the command line on the Orchestrator.

  2. Log in to Gaia Clish.

  3. Run:

    show maestro auto-scale scale-units

  1. Connect to the command line on the Orchestrator.

  2. Log in to Gaia Clish.

  3. Run:

    show maestro auto-scale security-group-id <Security Group ID> scale-up-rules

  1. Connect to the command line on the Orchestrator.

  2. Log in to Gaia Clish.

  3. Run:

    show maestro auto-scale security-group-id <Security Group ID> scale-down-rules

  1. Connect to the command line on the Orchestrator.

  2. Log in.

  3. Run this command:

    • In Gaia Clish:

      show maestro auto-scale interactive-status

    • In the Expert mode:

      mas_cli --interactive

Example output:

-------------------------------------------------------------------------------

Security Group: 1

    Site: 1
      Average:            | CPU Utilization: 9%         | Pckt/s: 8         | Bytes/s: 5652      | Connections: 384
      Average (w/o 1 SU): | CPU Utilization: 13%        | Pckt/s: 13        | Bytes/s: 8478      | Connections: 576

          Member: 1       | CPU Utilization: 7%         | Pckt/s: 12        | Bytes/s: 8562      | Connections: 372
          Member: 2       | CPU Utilization: 9%         | Pckt/s: 4         | Bytes/s: 2656      | Connections: 412      [Scale Unit]
          Member: 3       | CPU Utilization: 11%        | Pckt/s: 10        | Bytes/s: 5739      | Connections: 369      [Scale Unit]

-------------------------------------------------------------------------------

  1. Connect to the command line on the Orchestrator.

  2. Log in.

  3. Run this command:

    • In Gaia Clish:

      show maestro auto-scale interactive-rules

    • In the Expert mode:

      mas_cli --interactive-rules

Example output:

-------------------------------------------------------------------------------

Security Group: 1

  Scale Up Rules:
   ID 1: When Each member with CPU utilization exceeds 70% For 60 seconds | No Match

  Scale Down Rules:
   ID 1: When Security Group with CPU utilization less than 30% For 60 seconds | Match for 16 seconds

-------------------------------------------------------------------------------

Troubleshooting

To troubleshoot issues with Auto-Scaling, make sure that the masd service is running on the Orchestrator and Security Group Members:

  1. On the Orchestrator:

    1. Connect to the command line on the Orchestrator.

    2. Log in to the Expert mode.

    3. Get the status of the masd service:

      service masd status

      The output must be:

      masd is running...

  2. On the Security Group:

    1. Connect to the command line on the Security Group.

    2. Log in to the Expert mode.

    3. Get the status of the masd service:

      service masd status

      The output must be:

      masd is running...

Examine the logs of the masd daemon on the Orchestrator and Security Group Members:

  • On the Orchestrator:

    1. Connect to the command line on the Orchestrator.

    2. Log in to the Expert mode.

    3. Examine the masd log file in real time:

      tail -F /var/log/masd.elg

  • On the Security Group:

    1. Connect to the command line on the Security Group.

    2. Log in to the Expert mode.

    3. Examine the masd log file in real time:

      tail -F /var/log/masd.elg

To make sure that the Orchestrator correctly applied the Auto-Scaling rules you configured in Gaia Portal or Gaia Clish:

  1. Connect to the command line on the Orchestrator.

  2. Log in to the Expert mode.

  3. Get the list of rules:

    mas_cli --interactive-rules

  4. Compare the rules you see in the command line with the rules you configured on the Orchestrator.

To see if Security Group Members are reporting their KPIs to the Orchestrator:

  1. Connect to the command line on the Orchestrator.

  2. Log in to the Expert mode.

  3. Get the KPI status:

    mas_cli --interactive