Authentication between Maestro Orchestrators

Warnings:

  • To avoid synchronization issues between Orchestrators, do not configure Security Groups before making sure all Orchestrators are authenticated.

  • Make sure only one administrator configures the authentication between Orchestrators at any time.

    This prevents an issue when different administrators configure the authentication on different Orchestrators.

Important - After you configure authentication between Orchestrators, you must configure it again in these cases:

  • One of the Orchestrators was installed from scratch (clean install) or restored to factory defaults.

  • An existing authenticated Orchestrator was replaced with another Orchestrator.

  • Site ID or Orchestrator ID changed on an existing authenticated Orchestrator.

Introduction

Starting in R82, you can configure mutual authentication between all Maestro Orchestrators on your Maestro Sites to make sure their communication is secure and encrypted over Internal Sync ports (sync in the same Maestro Site) and External Sync ports (sync between Maestro Sites).

This authentication is based on SSH keys and SSL certificates. These SSL certificates are valid for one year. Orchestrators renew these SSL certificates automatically.

This authentication is a two-way mesh process - each Orchestrator authenticates all other Orchestrators.

Explanation for a Dual Site Maestro configuration:

Site

Orchestrator

Authentication with

other Orchestrators

1

1_1

  • 1_2

  • 2_1

  • 2_2

 

1_2

  • 1_1

  • 2_1

  • 2_2

2

2_1

  • 1_1

  • 1_2

  • 2_2

 

2_2

  • 1_1

  • 1_2

  • 2_1

Note - This popup appears when you connect to Gaia Portal on an Orchestrator, and not all Orchestrators are authenticated in this Maestro environment:

<Number> orchestrators are pending authentication, would you like to authenticate them now?

If you click Yes, the Authentication window opens.

Authentication statuses in Gaia Portal and CLI

Status

Description

Authenticated

The local Orchestrator that shows this status has authenticated the remote Orchestrator.

Unknown

The local Orchestrator that shows this status cannot determine the status for the remote Orchestrator.

Unreachable

The local Orchestrator that shows this status cannot connect to the remote Orchestrator over the External Sync port (between Maestro Sites) or over the Internal Sync port (on the same Maestro Site).

Untrusted

The local Orchestrator that shows this status does not trust the remote Orchestrator.

Configuring new authentication between Maestro Orchestrators

The procedure below is for a Dual Site Maestro environment with four Orchestrators.

The same steps apply in a Single Site Maestro environment with two Orchestrators.

Important:

If you enabled Two-Factor Authentication in Gaia settings on Orchestrators, then before you establish trust between Orchestrators, you must disable the Two-Factor Authentication in Gaia settings on each Orchestrator.

After you establish trust between Orchestrators, enable the Two-Factor Authentication again in Gaia settings on Orchestrators.

Step

Instructions

1

Verify the fingerprint of each Orchestrator in one of these ways:

  • To verify the fingerprint of an Orchestrator in Gaia Portal:

    1. Connect to Gaia Portal of the Orchestrator.

    2. In the left panel, click Orchestrator Management > Security Groups.

    3. In the top right corner, click Authentication.

    4. In the section Orchestrator ID: X_Y (local), refer to the field Fingerprint.

  • To verify the fingerprint of an Orchestrator in Gaia Clish:

    1. Connect to the command line on the Orchestrator.

    2. Log in to Gaia Clish.

    3. Run:

      show ssh pubkey host localhost

2

Connect to Gaia Portal on one of the Orchestrators.

For example, Orchestrator 1_1.

3

In the left panel, click Orchestrator Management > Security Groups.

4

In the top right corner, click Authentication.

5

In the section Orchestrator ID: 1_2:

  1. Click Trust.

  2. In the Trust Orchestrator window, click Yes.

  3. In the Admin Password Required window, enter the Expert mode password and click OK to approve the connection.

    This password is only used for the initial authentication process and is not stored after the authentication process completes.

  4. In the Authentication Summary, click OK.

This establishes trust:

  • between 1_2 and 1_1.

6

In the section Orchestrator ID: 2_1:

  1. Click Trust.

  2. In the Trust Orchestrator window, click Yes.

  3. In the Admin Password Required window, enter the Expert mode password and click OK to approve the connection.

    This password is only used for the initial authentication process and is not stored after the authentication process completes.

  4. In the Authentication Summary window, click OK.

This establishes trust:

  • between 2_1 and 1_1.

  • between 2_1 and 1_2.

7

In the section Orchestrator ID: 2_2:

  1. Click Trust.

  2. In the Trust Orchestrator window, click Yes.

  3. In the Admin Password Required window, enter the Expert mode password and click OK to approve the connection.

    This password is only used for the initial authentication process and is not stored after the authentication process completes.

  4. In the Authentication Summary window, click OK.

This establishes trust:

  • between 2_2 and 1_1.

  • between 2_2 and 1_2.

  • between 2_2 and 2_1.

8

If you established trust with a new Orchestrator that was added to an existing Maestro environment,

then on that new Orchestrator, run in the Expert mode:

orchd restart

Important:

If you enabled Two-Factor Authentication in Gaia settings on Orchestrators, then before you establish trust between Orchestrators, you must disable the Two-Factor Authentication in Gaia settings on each Orchestrator.

After you establish trust between Orchestrators, enable the Two-Factor Authentication again in Gaia settings on Orchestrators.

Step

Instructions

1

Verify the fingerprint of each Orchestrator in one of these ways:

  • To verify the fingerprint of an Orchestrator in Gaia Portal:

    1. Connect to Gaia Portal of the Orchestrator.

    2. In the left panel, click Orchestrator Management > Security Groups.

    3. In the top right corner, click Authentication.

    4. In the section Orchestrator ID: X_Y (local), refer to the field Fingerprint.

  • To verify the fingerprint of an Orchestrator in Gaia Clish:

    1. Connect to the command line on the Orchestrator.

    2. Log in to Gaia Clish.

    3. Run:

      show ssh pubkey host localhost

2

Connect to the command line on one of the Orchestrators.

For example, Orchestrator 1_1

3

Establish trust with the "establish_mho_authentication" command.

Note - You can run this command in Gaia Clish or in the Expert mode.

To see the built-in help, run:

establish_mho_authentication -h

To establish trust between all Orchestrators, run:

establish_mho_authentication all

To establish trust between the current Orchestrator and the specified Orchestrators, run

(enter the required IDs separated by commas without spaces):

establish_mho_authentication 1_1,1_2,2_1,2_2

Forcing existing authentication between Maestro Orchestrators

This action is helpful in these cases:

  • You revoked trust with an Orchestrator and need to establish trust again.

  • There are communication issues between authenticated Orchestrators.

Step

Instructions

1

Connect to Gaia Portal on one of the Orchestrators.

2

In the left panel, click Orchestrator Management > Security Groups.

3

In the top right corner, click Authentication.

4

In the card of each Orchestrator, click > click Force Re-authentication.

5

In the Refresh Authentication window, click Yes.

Step

Instructions

1

Connect to the command line on one of the Orchestrators.

3

Establish trust with the "establish_mho_authentication" command.

Note - You can run this command in Gaia Clish or in the Expert mode.

To see the built-in help, run:

establish_mho_authentication -h

To establish trust between all Orchestrators, run:

establish_mho_authentication all

To establish trust between the current Orchestrator and the specified Orchestrators, run (enter the required IDs separated by commas without spaces):

establish_mho_authentication 1_1,1_2,2_1,2_2

4

If you established trust with a new Orchestrator that was added to an existing Maestro environment,

then on that new Orchestrator, run in the Expert mode:

orchd restart

Revoking existing authentication between Maestro Orchestrators

Warning - For security reasons, perform this procedure before you remove an authenticated Orchestrator from the Maestro environment.

This procedure revokes the existing authentication of the selected Orchestrator on all other Orchestrators.

The example procedures below are for revoking trust with the Orchestrator 1_2.

Step

Instructions

1

Connect to Gaia Portal on the Orchestrator 1_1.

(You can connect to any Orchestrator that will remain in this Maestro environment.)

2

In the left panel, click Orchestrator Management > Security Groups.

3

In the top right corner, click Authentication.

4

In the section Orchestrator ID: 1_2, click > click Revoke from all.

5

In the Revoke Orchestrator 1_2 window, click Yes.

6

In the Authentication Summary window, click OK.

Part 1 of 2 - Revoking trust on Orchestrator 1_1

This step revokes trust on Orchestrator 1_1 with Orchestrator 1_2.

(You can connect to any Orchestrator that will remain in this Maestro environment.)

Step

Instructions

1

Connect to Gaia Portal on the Orchestrator 1_1.

2

In the left panel, click Orchestrator Management > Security Groups.

3

In the top right corner, click Authentication.

4

In the section Orchestrator ID: 1_2, click > click Revoke.

5

In the Revoke Orchestrator 1_2 window, click Yes.

6

In the Authentication Summary window, click OK.

Part 2 of 2 - Revoking trust on Orchestrator 1_2

This step revokes trust on Orchestrator 1_2 with all other Orchestrators:

Step

Instructions

1

Connect to Gaia Portal on the Orchestrator 1_2.

2

In the left panel, click Orchestrator Management > Security Groups.

3

In the top right corner, click Authentication.

4

In the section Orchestrator ID: 1_1:

  1. Click Revoke.

  2. In the Revoke Orchestrator 1_1 window, click Yes.

  3. In the Authentication Summary window, click OK.

5

In the section Orchestrator ID: 2_1:

  1. Click Revoke.

  2. In the Revoke Orchestrator 2_1 window, click Yes.

  3. In the Authentication Summary window, click OK.

6

In the section Orchestrator ID: 2_2:

  1. Click Revoke.

  2. In the Revoke Orchestrator 2_2 window, click Yes.

  3. In the Authentication Summary window, click OK.

Part 1 of 2 - Revoking trust on Orchestrator 1_1

This step revokes trust on Orchestrator 1_1 with Orchestrator 1_2.

Step

Instructions

1

Connect to the command line on the Orchestrator 1_1.

2

If your default shell is the Expert mode (/bin/bash), then go to Gaia Clish:

clish

3

Revoke trust with Orchestrator 1_2:

delete maestro authenticated mho 1_2

4

Save the changes:

save config

Part 2 of 2 - Revoking trust on Orchestrator 1_2

This step revokes trust on Orchestrator 1_2 with Orchestrator 1_1:

Step

Instructions

1

Connect to the command line on the Orchestrator 1_2.

2

If your default shell is the Expert mode (/bin/bash), then go to Gaia Clish:

clish

3

Revoke trust with Orchestrator 1_1:

delete maestro authenticated mho 1_1

4

Save the changes:

save config

Blocking authentication with a Maestro Orchestrator

This procedure configures the selected Orchestrator as untrusted.

You can do this to prevent an untrusted Orchestrator from accidentally becoming trusted in the future.

Step

Instructions

1

Connect to Gaia Portal on one of the Orchestrators.

2

In the left panel, click Orchestrator Management > Security Groups.

3

In the top right corner, click Authentication.

4

In the applicable Orchestrator card, click Do not trust.

The fingerprint is added to this file:

/etc/blocked_fingerprints

5

In the Block fingerprint window, click Yes to confirm.

6

To unblock:

  1. Click the section Untrusted Orchestrators to expand it.

  2. In the applicable line, click Trust.

  3. In the Unblock fingerprint window, click Yes to confirm.

Troubleshooting

  1. Run the "orch_stat" command:

    Step

    Instructions

    1

    Connect to the command line on each Orchestrator.

    2

    Log in to the Expert mode.

    3

    Run the command:

    orch_stat -Av

  2. Run the HealthCheck Point (HCP) tool (see sk171436) and refer to the test "Maestro Orchestrator Authentication":

    Step

    Instructions

    1

    Connect to the command line on each Orchestrator.

    2

    Log in to the Expert mode.

    3

    Run the test:

    hcp -r "Maestro Orchestrator Authentication"

    4

    View the detailed report:

    hcp --show-last