Working with High Availability

In This Section:

Overview of High Availability

High Availability is redundancy and database backup for management servers. Synchronized servers have the same policies, rules, user definitions, network objects, and system configuration settings.

Multi-Domain Management implements High Availability at these levels:

Multi-Domain Server High Availability is an Active/Active redundancy solution that uses two or more fully synchronized Multi-Domain Servers for continuous redundancy. All Multi-Domain Servers are Active. You can log into and work with the primary or secondary Multi-Domain Servers.
Domain Management Server High Availability is both a redundancy and a Load Sharing solution for Domains. You create a Domain Management Server on two or more Multi-Domain Servers. These Domain Management Servers synchronize fully for continuous redundancy.
One Domain Management Server is Active and the others are Standby. Each Multi-Domain Server can have both Active and Standby Domain Servers. You can configure the Active Domain Management Server on different Multi-Domain Servers for effective load sharing.

All High Availability deployments include one Primary Multi-Domain Server and one or more Secondary servers. Synchronization occurs automatically when administrators publish sessions with changes to Policies, objects or configuration settings.

Primary and Secondary Multi-Domain Servers

The order in which you install Multi-Domain Servers is significant. You must define the first physical server as a Primary Multi-Domain Server in the First Time Wizard. You must define all other Multi-Domain Servers as Secondary in the First Time Wizard.

Active and Standby Domain Management Servers

You can only use the Active Domain Management Server to manage Domain Security Gateways, networks, Security Policies objects and system configuration. Standby Domain Management Servers synchronize fully for redundancy. You can connect to a Standby Domain Management Server in the Read Only mode to look at current object configurations and Rule Base.

In the standard configuration, there is only one Active Domain Management Server for each Domain. All others are Standby Domain Management Servers. If the Active Domain Management Server fails, you must manually change a Standby Domain Management Server to Active.

Important notes about backing up and restoring in Management High Availability environment:

To back up and restore a consistent environment, make sure to collect and restore the backups and snapshots from all servers in the High Availability environment at the same time. (This does not apply to Multi-Domain Log Servers.)
Make sure other administrators do not make changes in SmartConsole until the backup operation is completed.

For more information:

About Gaia Backup and Gaia Snapshot, see the R80.30 Gaia Administration Guide.
About the migrate export and migrate import commands, see the R80.30 CLI Reference Guide.
About the mds_backup and mds_restore commands, see the R80.30 Multi-Domain Management Administration Guide.
About Virtual Machine Snapshots, see the vendor documentation.

Multi-Site High Availability Deployment Example

This example shows a Multi-Site, High Availability deployment with two Multi-Domain Servers and one Multi-Domain Log Server. A real-life deployment will have many more assets.

Each Multi-Domain Server has two Domains configured for Load Sharing, where a different Domain Management Server is Active at each location. Administrators can connect to all Multi-Domain Servers. For best performance, connect to the Multi-Domain Server nearest to your geographical location.

Item	Description
1	London Multi-Domain Server with an Active Domain Management Server for London and a Standby Domain Management Server for Tokyo
2	Multi-Domain Log Server with Domain Log Servers for London and Tokyo
3	Tokyo Multi-Domain Server with an Active Domain Management Server for Tokyo and a Standby Domain Management Server for London
4	Tokyo network
5	London network
6	Internet
	Active Domain Management Server
	Standby Domain Management Server
	Domain Log Server

This illustration shows the configuration grid in the SmartConsole Multi Domain view for the example deployment:

The system automatically creates the Global Domain when you install Multi-Domain Management.

Creating a Secondary Multi-Domain Server

This section shows you how to create a new secondary Multi-Domain Server.

Important: Before you start this procedure, make sure to define the physical server as the correct server type (Secondary Multi-Domain Server, or Multi-Domain Log Server) during installation. An incorrect definition can cause deployment failure.

To create a new, secondary Multi-Domain Server:

If you did not do so, install a new R80.30 secondary Multi-Domain Server.
Follow the procedures in the R80.30 Installation and Upgrade Guide. Make sure to define this server as a secondary Multi-Domain Server in the First Time Wizard. Connect to the Primary Multi-Domain Server with SmartConsole and go the Multi-Domain > Domains view.
In the Multi-Domain navigation toolbar, click New > Multi-Domain Server.
Enter a unique name for this Multi-Domain Server.
To get the IP address automatically, the name must be in the DNS.
Enter the IPv4 address or click Resolve IP to get the IP address from the DNS.
Select the platform operating system, software version, and hardware type.
Click Connect to establish SIC trust.

The new Multi-Domain Server automatically synchronizes with all existing Multi-Domain Servers and Multi-Domain Log Servers. The synchronization operation can take some time to complete, during which a notification indicator shows in the task information area.

Domain Management Server High Availability and Load Sharing

This section includes procedures for configuring the Multi-Domain Management environment for secondary Multi-Domain Servers and a Multi-Domain Log Server. When you install Multi-Domain Management for the first time, select Primary Multi-Domain Server in the First Time Wizard. For High Availability and Load Sharing, select Secondary Multi-Domain Server in the First Time Wizard.

Each Domain has one Active and one or more Standby Domain Management Servers. For example, if a deployment has three Multi-Domain Servers, each Domain can have one Active and two Standby Domain Management Servers. This lets the Domains load be shared between several physical Multi-Domain Servers.

Example of Domain Management Server High Availability with Load Sharing:

By default, the Primary Domain Management Server is Active. All other Domain Management Servers for that Domain are Standbys. You can change a Standby Domain Management Server to Active as necessary.

All Domain management operations, such as working with Security Policies, users, networks and other objects, occur on the Active Domain Management Server. Standby Domain Management Servers automatically synchronize with the Active Domain Management Server. Security Gateways can get a Security Policy and a Certificate Revocation List (CRL) from either the Active or Standby Domain Management Servers.

Creating a Secondary Domain Management Server

When you first create a Domain, you also define the Primary Domain Management Server. Use this procedure to create Secondary Domain Management Servers for existing Domains.

To create a secondary Domain Management Server:

Connect to the Multi-Domain Server with SmartConsole.
In the Domains view, right-click the empty cell at the intersection of the applicable Multi-Domain Server and Domain in the grid.
Select New Domain Server.
In the Domain Server window, configure the Domain Management Server name and IP address.

Domain Management Server synchronization starts automatically and can take some time to complete.

Note - You cannot change settings for an existing Domain Management Server. You must first delete the Domain Management Server and then create a new one.

To delete a secondary Domain Management Server configuration, right-click the applicable cell and select Delete.

Synchronization

In a multi-domain environment, the Multi-Domain Servers work in active-active mode. All Multi-Domain Servers are active and synchronize each other.

The Domains managed by the Multi-Domain Server work in active-standby mode, where the Active Domain Server synchronizes all the standby Domain Servers.

The system automatically synchronizes periodically and when an administrator publishes changes to the configuration.

How Synchronization Works

During synchronization, the system performs these steps without user intervention:

On periodic synchronization:

The Active exports the delta data between the Active server and the Standby server to compressed files.
The compressed files are transferred to the Standby server.
The Standby Server replays the delta data from the uncompressed files.

On manual synchronization:

The Active Server exports the public data to compressed files.
The compressed files are transferred to the Standby Server.
The Standby server overrides the existing data with the uncompressed files.

The data that is transferred during synchronization includes:

Postgres database
Solr
ICA database
Configuration files
Domain licenses and contracts. Multi-Domain server licenses and contracts are not transferred.

Initial Synchronization

Initial synchronization occurs automatically when you create a secondary Multi-Domain Server, Multi-Domain Log Server, or Domain Management Server. The system generates a task to copy all databases and system information from the connected server to the new server.

Multi-Domain Server and Multi-Domain Log Server synchronization tasks show in the Task Information area, in the Multi-Domain Server SmartConsole. Domain synchronization tasks show in the Domain SmartConsole.

Periodic Synchronization

Multi-Domain Servers synchronize with all other peers and Multi-Domain Log Servers. Periodic synchronization occurs automatically, and when an administrator publishes a session. Private (non-published) sessions do not synchronize.

Periodic synchronizations are incremental. Only database changes synchronize with peers. Active Domain Management Servers synchronize to the standby Domain Management Servers.

Manual Synchronization

Manual synchronization is a full synchronization that overwrites all data on the peers. It disconnects all connected clients and overrides active sessions and running tasks.

When changes made in a session are published on the Active server (made public), the changes are synchronized to the Standby server. Unpublished, private sessions are not synchronized.

Best practice - Use this option with caution, and only in cases of synchronization error. We recommend that you publish changes before initiating full sync.

For Domain Management Servers, you can only run a manual synchronization from the active Domain Management Server to the standby peers.

Manually Synchronizing a Multi-Domain Server

You can manually synchronize the connected Multi-Domain Server with a peer Multi-Domain Server.

To manually synchronize Multi-Domain Servers:

Click the Synchronization Status area at the bottom of the SmartConsole window.
In the High Availability Status window, select a peer Multi-Domain Server to synchronize.
Click Sync Peer.

Synchronization starts immediately and the status shows in the window. The synchronization operation can take many minutes to complete.

Warning: Use manual synchronization with caution. This can overwrite all data on the peer Multi-Domain Server if they do not synchronize correctly.

Manually Synchronizing Domain Management Servers

You can manually synchronization a Standby Domain Management Server with the Active Domain Management Server on a different Multi-Domain Server.

To manually synchronize Domain Management Servers for a Domain:

Open SmartConsole for the active Domain Management Server.
Click Menu > High Availability.
In the High Availability Status window, click Actions > Sync Peer.

Synchronization starts immediately and the status shows in the window. The synchronization operation can take many minutes to complete.

Multi-Domain Server ICA Database Synchronization

When you create a new secondary Multi-Domain Server, the Internal Certificate Authority (ICA) on the Primary Multi-Domain Server generates a certificate when you establish SIC trust. The ICA can generate a certificate for a new administrator, if required by the authentication method. In a High Availability deployment with more than one Multi-Domain Server, the system synchronizes the ICA databases as necessary.

Changing the Active Domain Management Server

If the current Active Domain Management Server is responsive, use this procedure to set a different Domain Management Server to Active.

To change an Active Domain Management Server:

Right-click the cell for a Standby Domain Management Server, and then select Connect to Domain Management Server.
In the Domain SmartConsole instance, click Menu > High Availability.
In the High Availability Status window, click a Standby Domain Management Server Actions > Set Active.
Close SmartConsole and re-connect to the newly Active Domain SmartConsole.

The Standby Domain Management Server changes to Active. The Standby Domain Management Servers automatically synchronize, and a confirmation message shows in the High Availability Status window. The synchronization operation can take many minutes to complete.

To manually set the Active Domain Management Server to Standby

Right-click the cell for the Active Domain Management Server, and select Connect to Domain Management Server.
Click Menu > Management High Availability.
In the High Availability Status window, click Actions > Set Standby.
Confirm when prompted.

The Active Domain Management Server changes to Standby. Continue the procedure to set a different Domain Management Server to Active. Until you do this, Domain SmartConsole clients open in the Read Only mode and you cannot work with Domain objects or Policies.

Note - SmartConsole clients connected to the Active Domain Management Server will be disconnected during the procedure for changing the Active Domain Management Server.

Looking at High Availability Status

To see Multi-Domain Server and Multi-Domain Log Server High Availability status:

Select Management High Availability from the SmartConsole menu.

The High Availability Status window shows all Multi-Domain Servers and Multi-Domain Log Servers in your environment, together with their synchronization status.

Icon	Status
	Multi-Domain Server (that you are connected to) - Synchronization OK
	Multi-Domain Server Synchronization OK
	Multi-Domain Log Server Synchronization OK
	Multi-Domain Server - Not synchronized - No connection with peer

To see Domain Management Server High Availability status:

Connect to a Domain with SmartConsole.
By default, SmartConsole connects to the Active Domain Management Server.

Select Management High Availability from the SmartConsole menu.

The High Availability Status window shows the status of all Domain Management Servers for the selected Domain. You can manually synchronize the peer servers with the Domain Management Server to which you are connected. You can also connect with SmartConsole to a peer Domain Management Server in the Read Only mode.

Icon	Status
	Active Domain Management Server - Synchronization OK
	Standby Domain Management Server - Synchronization OK
	Domain Log Server - Synchronization OK
	Domain Management Server not synchronized - No connection with peer
	Domain Management Server synchronization in process or has a problem

Note - Domain servers' status is reflected also in the Domains view in the SmartConsole connected to the Multi-Domain Server. For more information on synchronization status, see the R80.30 Security Management Administration Guide.

Failure Recovery

In many cases, you can recover a failed Primary Multi-Domain Server in a High Availability deployment. To do this, promote an existing Secondary Multi-Domain Server to become the Primary. Promote a Secondary Domain Management Server to become Primary Domain Management Server. You can then install and configure a new secondary Multi-Domain Server.

Important: Use Domain Management Server promotion only to recover a failed Multi-Domain Server.

Connecting to a Secondary Multi-Domain Server

To connect to a secondary Multi-Domain Server:

Make sure that all functional, Secondary Multi-Domain Servers and Multi-Domain Log Servers are up and running.
Connect to a secondary Multi-Domain Server with SmartConsole.
If the Global Domain Management Server to be promoted to Primary is not Active, change it to Active:
1. In the Domains view, right-click the Global Domain, and then click Connect to Domain.
  A SmartConsole instance opens for the Global Domain.
2. Go to Menu > Management High Availability.
3. In the High Availability Status window, click Actions > Set Active for the connected Global Domain.

Promoting the Secondary Multi-Domain Server to Primary

This procedure is necessary because there are no automatic steps to promote a Secondary Multi-Domain Server when the Primary Multi-Domain Server fails.

To promote a Secondary Multi-Domain Server to Primary:

Make sure that the Global Domain is active on the secondary Multi-Domain Server.
Run these commands on the secondary Multi-Domain Server to be promoted:
cpprod_util FwSetPrimary 1
cpprod_util CPPROD_SetValue PROVIDER-1 Primary 4 1 1
cpprod_util CPPROD_SetValue SIC ICAState 4 3 1
ckp_regedit -d //SOFTWARE//CheckPoint//SIC OTP
ckp_regedit -d //SOFTWARE//CheckPoint//SIC ICAip

These commands update the required parameters in the registry on the secondary Multi-Domain Server.
Remove all instances of the old Primary Management object. To see all of the instances, right-click the object and select Where Used.
Note - If the environment includes more than one secondary Multi-Domain Server and at least one Multi-Domain Log Server, you must run SIC reset in mdsconfig on these servers.
Intall database.
Delete the primary Multi-Domain Server from SmartConsole in the Domains view.

Restoring Domain Management Servers

Follow these instructions for each Domain on the failed Primary Domain Management Server.

Important - To use this procedure, there must be at least one Active Domain Management Server on a different Multi-Domain Server.

To restore the Domain Management Servers:

In SmartConsole Domain view, select a Domain Management Server to promote to Primary Domain Management Server.
If the selected Domain Management Server is Standby, change it to Active:
1. Open the selected Domain Management Server in SmartConsole.
2. Go to Menu > Management High Availability.
3. In the High Availability Status window, click Actions > Set Active.
4. Close SmartConsole.
Run these commands on the Multi-Domain Server command line to change the active Domain Management Server from Secondary to Primary:
> mdsenv <domain_server_name>
> promote_util

These steps set the Multi-Domain Server context to the specified Domain Management Server.
Open the newly promoted Domain Management Server in SmartConsole.
Find (with Where Used) and delete all instances of the failed Domain Management Server, including the failed Domain Management Server itself.
Publish the changes.
If necessary, manually synchronize the Domain Management Servers.
Re-assign Global Policies and install Policies on all Security Gateways.
If the promoted Domain Management Server is using a High Availability Domain Management Server license, replace it with a standard Domain Management Server license.

To make Domain Management Server Active when there is no corresponding peer and the High Availability Status window is not available, run these commands:

# mdsenv <domain_name>
# mgmt_cli make-server-active force true --domain <domain_name> --user <user_name> --password <password>

These commands set the Domain Management Server to the Active state. Do this for all Domain Management Servers that do not have a High Availability peer.

Finishing the Promotion

To restore your High Availability deployment, run these commands:

mdsstop
mdsstart

Deleting a Secondary Multi-Domain Server or Multi-Domain Log Server

To delete a secondary Multi-Domain Server:

Move each Active Domain Management Server on the secondary Multi-Domain Server to another Domain Management Server.
Connect to the command line on the Multi-Domain Server to be deleted and run: mdsstop
In SmartConsole, right-click the secondary Multi-Domain Server, and then select Delete Multi-Domain Server.
Confirm the action and click OK.
Publish the change.

Note - This procedure deletes all standby and non-primary Domain Management Servers on the Secondary Multi-Domain Server. You cannot delete the Primary or Active Domain Management Server.

Re-Establishing SIC Trust for a Secondary Multi-Domain Server

Important - You can only re-establish SIC trust on a Secondary Multi-Domain Server or Multi-Domain Log Servers. There is no option to establish SIC trust on the Primary Multi-Domain Server.

It is occasionally necessary to re-establish trust between a Primary and secondary Multi-Domain Server or Multi-Domain Log Server. This can occur for many reasons, including:

Changes to the IP address of the Primary Multi-Domain Server, Secondary Multi-Domain Server or Multi-Domain Log Server
Failure and recovery of the Primary Multi-Domain Server
Promotion of a Secondary Multi-Domain Server to Primary Multi-Domain Server
Internal Certificate Authority (ICA) failure on the Primary Multi-Domain Server

To re-establish SIC trust:

Open a command line interface to the Secondary Multi-Domain Server or Multi-Domain Log Server.
Log in and run: mdsconfig
Enter the number for Secure Internal Communication, and then press Enter.
Enter y to confirm.
Enter and confirm the activation key.
Enter the number for Exit.
Wait for Check Point processes to stop and automatically restart.
In the SmartConsole Multi-Domain view, double-click a Secondary Multi-Domain Server or Multi-Domain Log Server object.
In the Multi-Domain Server window, click Connect.
In the Initialize SIC window, enter activation key that you entered in step 5 above.
If successful, the Certificate State field shows Trust established.