Working with High Availability

In This Section:

Overview of High Availability

High Availability is redundancy and database backup for management servers. Synchronized servers have the same policies, rules, user definitions, network objects, and system configuration settings.

Multi-Domain Management implements High Availability at these levels:

Multi-Domain Server High Availability is an Active/Active redundancy solution that uses two or more fully synchronized Multi-Domain Servers for continuous redundancy. All Multi-Domain Servers are Active. You can log into and work with the primary or secondary Multi-Domain Servers.
Domain Server High Availability is both a redundancy and a Load Sharing solution for Domains. You create a Domain Server on two or more Multi-Domain Servers. These Domain Servers synchronize fully for continuous redundancy.
One Domain Server is Active and the others are Standby. Each Multi-Domain Server can have both Active and Standby Domain Servers. You can configure the Active Domain Server on different Multi-Domain Servers for effective load sharing.

All High Availability deployments include one Primary Multi-Domain Server and one or more Secondary servers. Synchronization occurs automatically when administrators publish sessions with changes to Policies, objects or configuration settings.

Primary and Secondary Multi-Domain Servers

The order in which you install Multi-Domain Servers is significant. You must define the first physical server as a Primary Multi-Domain Server in the First Time Wizard. You must define all other Multi-Domain Servers as Secondary in the First Time Wizard.

Active and Standby Domain Servers

You can only use the Active Domain Server to manage Domain Security Gateways, networks, Security Policies objects and system configuration. Standby Domain Servers synchronize fully for redundancy. You can connect to a Standby Domain Server in the Read Only mode to look at current object configurations and Rule Base.

In the standard configuration, there is only one Active Domain Server for each Domain. All others are Standby Domain Servers. If the Active Domain Server fails, you must manually change a Standby Domain Server to Active.

Important notes about backing up and restoring in Management High Availability environment:

To back up and restore a consistent environment, make sure to collect and restore the backups and snapshots from all servers in the High Availability environment at the same time. (This does not apply to Multi-Domain Log Servers.)
Make sure other administrators do not make changes in SmartConsole until the backup operation is completed.

For more information:

About Gaia Backup and Gaia Snapshot, see the R80.10 Gaia Administration Guide.
About the migrate export and migrate import commands, see the R80.10 CLI Reference Guide.
About the mds_backup and mds_restore commands, see the R80.10 Multi-Domain Management Administration Guide.
About Virtual Machine Snapshots, see the vendor documentation.

Multi-Site High Availability Deployment Example

This example shows a Multi-Site, High Availability deployment with two Multi-Domain Servers and one Multi-Domain Log Server. A real-life deployment will have many more assets.

Each Multi-Domain Server has two Domains configured for Load Sharing, where a different Domain Server is Active at each location. Administrators can connect to all Multi-Domain Servers. For best performance, connect to the Multi-Domain Server nearest to your geographical location.

Item	Description
1	London Multi-Domain Server with an Active Domain Server for London and a Standby Domain Server for Tokyo
2	Multi-Domain Log Server with Domain Log Servers for London and Tokyo
3	Tokyo Multi-Domain Server with an Active Domain Server for Tokyo and a Standby Domain Server for London
4	Tokyo network
5	London network
6	Internet
	Active Domain Server
	Standby Domain Server
	Domain Log Server

This illustration shows the configuration grid in the SmartConsole Multi Domain view for the example deployment:

The system automatically creates the Global Domain when you install Multi-Domain Management.

Creating a Secondary Multi-Domain Server

This section shows you how to create a new secondary Multi-Domain Server.

Important: Before you start this procedure, make sure to define the physical server as the correct server type (Secondary Multi-Domain Server or Multi-Domain Log Server) during installation. An incorrect definition can cause deployment failure.

To create a new, secondary Multi-Domain Server:

If you did not do so, install a new R80.10 secondary Multi-Domain Server.
Follow the procedures in the R80.10 Installation and Upgrade Guide. Make sure to define this server as a secondary Multi-Domain Server in the First Time Wizard. Connect to the Primary Multi-Domain Server with SmartConsole and go the Domains view.
In the Multi-Domain navigation toolbar, click New > Multi-Domain Server.
Enter a unique name for this Multi-Domain Server.
To get the IP address automatically, the name must be in the DNS.
Enter the IPv4 address or click Resolve IP to get the IP address from the DNS.
Select the platform operating system, software version, and hardware type.
Click Connect to establish SIC trust.

The new Multi-Domain Server automatically synchronizes with all existing Multi-Domain Servers and Multi-Domain Log Servers. The synchronization operation can take some time to complete, during which a notification indicator shows in the task information area.

Domain Server High Availability and Load Sharing

This section includes procedures for configuring the Multi-Domain Management environment for secondary Multi-Domain Servers and a Multi-Domain Log Server. When you install Multi-Domain Management for the first time, select Primary Multi-Domain Server in the First Time Wizard. For High Availability and Load Sharing, select Secondary Multi-Domain Server in the First Time Wizard.

Each Domain has one Active and one or more Standby Domain Servers. For example, if a deployment has three Multi-Domain Servers, each Domain can have one Active and two Standby Domain Servers. This lets the Domains load be shared between several physical Multi-Domain Servers.

Example of Domain Server High Availability with Load Sharing:

By default, the Primary Domain Server is Active. All other Domain Servers for that Domain are Standbys. You can change a Standby Domain Server to Active as necessary.

All Domain management operations, such as working with Security Policies, users, networks and other objects, occur on the Active Domain Server. Standby Domain Servers automatically synchronize with the Active Domain Server. Security Gateways can get a Security Policy and a Certificate Revocation List (CRL) from either the Active or Standby Domain Servers.

Creating a Secondary Domain Server

When you first create a Domain, you also define the Primary Domain Server. Use this procedure to create Secondary Domain Servers for existing Domains.

To create a secondary Domain Server:

Connect to the Multi-Domain Server with SmartConsole.
In the Domains view, right-click the empty cell at the intersection of the applicable Multi-Domain Server and Domain in the grid.
Select New Domain Server.
In the Domain Server window, configure the Domain Server name and IP address.

Domain Server synchronization starts automatically and can take some time to complete.

Note - You cannot change settings for an existing Domain Server. You must first delete the Domain Server and then create a new one.

To delete a secondary Domain Server configuration, right-click the applicable cell and select Delete.

Synchronization

In a multi-domain environment, the Multi-Domain Servers work in active-active mode. All Multi-Domain Servers are active and synchronize each other.

The Domains managed by the Multi-Domain Server work in active-standby mode, where the Active Domain Server synchronizes all the standby Domain Servers.

The system automatically synchronizes periodically and when an administrator publishes changes to the configuration.

Initial Synchronization

Initial synchronization occurs automatically when you create a secondary Multi-Domain Server, Multi-Domain Log Server, or Domain Server. The system generates a task to copy all databases and system information from the connected server to the new server.

Multi-Domain Server and Multi-Domain Log Server synchronization tasks show in the Task Information area, in the Multi-Domain Server SmartConsole. Domain synchronization tasks show in the Domain SmartConsole.

Periodic Synchronization

Multi-Domain Servers synchronize with all other peers and Multi-Domain Log Servers. Periodic synchronization occurs automatically, and when an administrator publishes a session. Private (non-published) sessions do not synchronize.

Periodic synchronizations are incremental. Only database changes synchronize with peers. Active Domain Servers synchronize to the standby Domain Servers.

Manual Synchronization

Manual synchronization is a full synchronization that overwrites all data on the peers. It disconnects all connected clients and overrides active sessions and running tasks.

When changes made in a session are published on the Active server (made public), the changes are synchronized to the Standby server. Unpublished, private sessions are not synchronized.

Best practice - Use this option with caution, and only in cases of synchronization error. We recommend that you publish changes before initiating full sync.

For Domain Servers, you can only run a manual synchronization from the active Domain Server to the standby peers.

Manually Synchronizing a Multi-Domain Server

You can manually synchronize the connected Multi-Domain Server with a peer Multi-Domain Server.

To manually synchronize Multi-Domain Servers:

Click the Synchronization Status area at the bottom of the SmartConsole window.
In the High Availability Status window, select a peer Multi-Domain Server to synchronize.
Click Sync Peer.

Synchronization starts immediately and the status shows in the window. The synchronization operation can take many minutes to complete.

Warning: Use manual synchronization with caution. This can overwrite all data on the peer Multi-Domain Server if they do not synchronize correctly.

Manually Synchronizing Domain Servers

You can manually synchronization a Standby Domain Server with the Active Domain Server on a different Multi-Domain Server.

To manually synchronize Domain Servers for a Domain:

Open SmartConsole for the active Domain Server.
Click Menu > High Availability.
In the High Availability Status window, click Actions > Sync Peer.

Synchronization starts immediately and the status shows in the window. The synchronization operation can take many minutes to complete.

Multi-Domain Server ICA Database Synchronization

When you create a new secondary Multi-Domain Server, the Internal Certificate Authority (ICA) on the Primary Multi-Domain Server generates a certificate when you establish SIC trust. The ICA can generate a certificate for a new administrator, if required by the authentication method. In a High Availability deployment with more than one Multi-Domain Server, the system synchronizes the ICA databases as necessary.

Changing the Active Domain Server

If the current Active Domain Server is responsive, use this procedure to set a different Domain Server to Active.

To change an Active Domain Server:

Right-click the cell for a Standby Domain Server, and then select Connect to Domain Server.
In the Domain SmartConsole instance, click Menu > High Availability.
In the High Availability Status window, click a Standby Domain Server Actions > Set Active.
Close SmartConsole and re-connect to the newly Active Domain SmartConsole.

The Standby Domain Server changes to Active. The Standby Domain Servers automatically synchronize, and a confirmation message shows in the High Availability Status window. The synchronization operation can take many minutes to complete.

To manually set the Active Domain Server to Standby

Right-click the cell for the Active Domain Server, and select Connect to Domain Server.
Click Menu > Management High Availability.
In the High Availability Status window, click Actions > Set Standby.
Confirm when prompted.

The Active Domain Server changes to Standby. Continue the procedure to set a different Domain Server to Active. Until you do this, Domain SmartConsole clients open in the Read Only mode and you cannot work with Domain objects or Policies.

Note - SmartConsole clients connected to the Active Domain Server will be disconnected during the procedure for changing the Active Domain Server.

Looking at High Availability Status

To see Multi-Domain Server and Multi-Domain Log Server High Availability status:

Select Management High Availability from the SmartConsole menu.

The High Availability Status window shows all Multi-Domain Servers and Multi-Domain Log Servers in your environment, together with their synchronization status.

Icon	Status
	Multi-Domain Server (that you are connected to) - Synchronization OK
	Multi-Domain Server Synchronization OK
	Multi-Domain Log Server Synchronization OK
	Multi-Domain Server - Not synchronized - No connection with peer

To see Domain Server High Availability status:

Connect to a Domain with SmartConsole.
By default, SmartConsole connects to the Active Domain Server.

Select Management High Availability from the SmartConsole menu.

The High Availability Status window shows the status of all Domain Servers for the selected Domain. You can manually synchronize the peer servers with the Domain Server to which you are connected. You can also connect with SmartConsole to a peer Domain Server in the Read Only mode.

Icon	Status
	Active Domain Server - Synchronization OK
	Standby Domain Server - Synchronization OK
	Domain Log Server - Synchronization OK
	Domain Server not synchronized - No connection with peer
	Domain Server synchronization in process or has a problem

Note - Domain servers status is reflected also in the Domains view in the SmartConsole connected to the Multi-Domain Server. For more information on synchronization status see the R80.10 Security Management Administrator Guide.

Failure Recovery

In many cases, you can recover a failed Primary Multi-Domain Server in a High Availability deployment. To do this, promote an existing Secondary Multi-Domain Server to become the Primary. Promote a Secondary Domain Server to become Primary Domain Server. You can then install and configure a new secondary Multi-Domain Server.

Important: Use Domain Server promotion only to recover a failed Multi-Domain Server.

Connecting to a Secondary Multi-Domain Server

To connect to a secondary Multi-Domain Server:

Make sure that all functional, Secondary Multi-Domain Servers and Multi-Domain Log Servers are up and running.
Connect to a secondary Multi-Domain Server with SmartConsole.
If the Global Domain Server to be promoted to Primary is not Active, change it to Active:
1. In the Domains view, right-click the Global Domain, and then click Connect to Domain.
  A SmartConsole instance opens for the Global Domain.
2. Go to Menu > Management High Availability.
3. In the High Availability Status window, click Actions > Set Active for the connected Global Domain.

Promoting the Secondary Multi-Domain Server to Primary

This procedure is necessary because there are no automatic steps to promote a Secondary Multi-Domain Server when the Primary Multi-Domain Server fails.

To promote a Secondary Multi-Domain Server to Primary:

Run these commands on the Secondary Multi-Domain Server to be promoted:
cpprod_util FwSetPrimary 1
cpprod_util CPPROD_SetValue PROVIDER-1 Primary 4 1 1
cpprod_util CPPROD_SetValue SIC ICAState 4 3 1
ckp_regedit -d //SOFTWARE//CheckPoint//SIC OTP
ckp_regedit -d //SOFTWARE//CheckPoint//SIC ICAip

These commands update the Secondary Multi-Domain Server registry.
Connect to the Check Point Database tool with the Secondary Multi-Domain Server IP address.
C:\Program Files (x86)\CheckPoint\SmartConsole\R80.10\
PROGRAM\GuiDBedit.exe /mds
On the Tables tab, select Other and then select (or search for) Multi-Domain Servers.
Delete the failed Domain Server object from the Object Name column.
Select the Multi-Domain Server to promote.
Double-click the Primary field in the bottom pane.
Change the value to true.
Save the database (File > Save All or Ctrl-s).

Restoring Domain Servers

Follow these instructions for each Domain on the failed Primary Domain Server.

Important - To use this procedure, there must be at least one Active Domain Server on a different Multi-Domain Server.

To restore the Domain Servers:

In SmartConsole Domain view, select a Domain Server to promote to Primary Domain Server.
If the selected Domain Server is Standby, change it to Active:
1. Open the selected Domain Server in SmartConsole.
2. Go to Menu > Management High Availability.
3. In the High Availability Status window, click Actions > Set Active.
4. Close SmartConsole.
Run these commands on the Multi-Domain Server command line to change the active Domain Server from Secondary to Primary:
> mdsenv <domain_server_name>
> promote_util

These steps set the Multi-Domain Server context to the specified Domain Server.
Open the newly promoted Domain Server in SmartConsole.
Find (with Where Used) and delete all instances of the failed Domain Server, including the failed Domain Server itself.
Publish the changes.
If necessary, manually synchronize the Domain Servers.
Re-assign Global Policies and install Policies on all Security Gateways.
If the promoted Domain Server is using a High Availability Domain Server license, replace it with a standard Domain Server license.

To make Domain Server Active when there is no corresponding peer and the High Availability Status window is not available, run these commands:

# mdsenv <domain_name>
# mgmt_cli make-server-active force true --domain <domain_name> --user <user_name> --password <password>

These commands set the Domain Server to the Active state. Do this for all Domain Servers that do not have a High Availability peer.

Finishing the Promotion

To restore your High Availability deployment, run these commands:

mdsstop
mdsstart

Deleting a Secondary Multi-Domain Server or Multi-Domain Log Server

To delete a secondary Multi-Domain Server:

Move each Active Domain Server on the secondary Multi-Domain Server to another Domain Server.
Connect to the command line on the Multi-Domain Server to be deleted and run: mdsstop
In SmartConsole, right-click the secondary Multi-Domain Server, and then select Delete Multi-Domain Server.
Confirm the action and click OK.
Publish the change.

Note - This procedure deletes all standby and non-primary Domain Servers on the Secondary Multi-Domain Server. You cannot delete the Primary or Active Domain Server.

Re-Establishing SIC Trust for a Secondary Multi-Domain Server

Important - You can only re-establish SIC trust on a Secondary Multi-Domain Server or Multi-Domain Log Servers. There is no option to establish SIC trust on the Primary Multi-Domain Server.

It is occasionally necessary to re-establish trust between a Primary and secondary Multi-Domain Server or Multi-Domain Log Server. This can occur for many reasons, including:

Changes to the IP address of the Primary Multi-Domain Server, Secondary Multi-Domain Server or Multi-Domain Log Server
Failure and recovery of the Primary Multi-Domain Server
Promotion of a Secondary Multi-Domain Server to Primary Multi-Domain Server
Internal Certificate Authority (ICA) failure on the Primary Multi-Domain Server

To re-establish SIC trust:

Open a command line interface to the Secondary Multi-Domain Server or Multi-Domain Log Server.
Log in and run: mdsconfig
Enter the number for Secure Internal Communication, and then press Enter.
Enter y to confirm.
Enter and confirm the activation key.
Enter the number for Exit.
Wait for Check Point processes to stop and automatically restart.
In the SmartConsole Multi-Domain view, double-click a Secondary Multi-Domain Server or Multi-Domain Log Server object.
In the Multi-Domain Server window, click Connect.
In the Initialize SIC window, enter activation key that you entered in step 5 above.
If successful, the Certificate State field shows Trust established.