High Availability

Overview

Multi-Domain Security Management High Availability gives uninterrupted management redundancy for all Domains. Multi-Domain Security Management High Availability operates at these levels:

• Multi-Domain Server High Availability - Multiple Multi-Domain Servers are, by default, automatically synchronized with each other. You can connect to any Multi-Domain Server to do Domain management tasks. One Multi-Domain Server is designated as the Active Multi-Domain Server. Other Multi-Domain Servers are designated as Standby Multi-Domain Servers.

  You can only do Global policy and global object management tasks using the active Multi-Domain Server. In the event that the active Multi-Domain Server is unavailable, you must change one of the standby Multi-Domain Servers to active.

• Domain Management Server High Availability - Multiple Domain Management Servers give Active/Standby redundancy for Domain management. One Domain Management Server for each Domain is Active. The other, fully synchronized Domain Management Servers for that Domain, are standbys. In the event that the Active Domain Management Server becomes unavailable, you must change one of the standby Domain Management Servers to active.

You can also use ClusterXL to give High Availability redundancy to your Domain Security Gateways. You use SmartDashboard to configure and manage Security Gateway High Availability for Domain Management Servers.

Multi-Domain Server High Availability

Multiple Multi-Domain Server Deployments

You can create multiple backup Multi-Domain Servers on different computers. A Multi-Domain Server can host either active or standby Domain Management Servers.

By default, when changes are made to Domain Management Servers, the system can automatically synchronize the active Domain Management Server with the standby Domain Management Servers. Alternatively, you can configure Domain Management Server synchronization to occur at specified events, such as every time a Domain policy is saved, or when it is installed onto one or more Domain Security Gateways. You can also synchronize Domain Management Servers manually.

Multi-Domain Server Status

When initially deploying a Multi-Domain Servers, the first Multi-Domain Server that you define becomes the Primary Multi-Domain Server. All subsequent Multi-Domain Servers are known as Secondary Multi-Domain Servers. There is no functional difference between a Primary and a Secondary Multi-Domain Server. You cannot, however, delete the Primary Multi-Domain Server.

By default, the Primary Multi-Domain Server is also the Active Multi-Domain Server. All other Multi-Domain Servers are Standby. This distinction is important, because certain tasks can only be done on the active Multi-Domain Server.

• You must use the active Multi-Domain Server to open the Global SmartDashboard with Read/Write permissions.
• Only the active Multi-Domain Server can operate as the Multi-Domain Server Internal Certificate Authority (ICA).

You can select another Multi-Domain Server to be the Active Multi-Domain Server. This is useful if the current active Multi-Domain Server is unavailable. You can see the status of Multi-Domain Servers in the High Availability - Multi-Domain Server Contents view.

To change a Multi-Domain Server from Standby to Active:

1. In the SmartDomain Manager Selection Bar, select High Availability.
2. Right-click a standby Multi-Domain Server and select Change Over from the Options menu.

Multi-Domain Server Clock Synchronization

All Multi-Domain Server system clocks must be synchronized. This is because the database synchronization method uses the time that transactions are recorded to determine the most recent action.

The transaction times are recorded using UTC (Universal Time Coordinated) on Multi-Domain Servers system clocks. You can synchronize Multi-Domain Server clocks using synchronization utilities. We strongly recommend that you update system clocks frequently to compensate for clock drift. Database synchronization requires that the Multi-Domain Server clocks be synchronized to the nearest second.

Whenever a new Multi-Domain Server is defined, it must receive a certificate and communication must be established. The Multi-Domain Server also needs to be synchronized with the other Multi-Domain Servers. The SmartDomain Manager guides the user through the stages of performing this initial synchronization.

The Multi-Domain Server Databases

The Multi-Domain Server hosts these databases:

• Domain Management Server databases
• Multi-Domain Security Management System database
• Global objects database

The content and synchronization method of each database is described below.

Multi-Domain Security Management System Database

The Multi-Domain Security Management system database contains data objects that define Multi-Domain Servers, Domains, Domain Management Servers, Security Gateways, licenses, administrators, GUI clients, and Global Policies. This database is automatically synchronized between Multi-Domain Servers.

This database architecture and automatic synchronization lets administrators use different Multi-Domain Servers to do their management tasks. Changes made to one Multi-Domain Server are synchronized automatically to all other Multi-Domain Servers.

If one Multi-Domain Server is down or disconnected from other Multi-Domain Servers, you can continue to use any other Multi-Domain Servers that are online. Once the Multi-Domain Server reconnects, it will synchronize automatically.

ICA Database for Multi-Domain Servers

This database holds certificates for Multi-Domain Servers, administrators and CRLs (certificate revocation lists). The Multi-Domain Server ICA is used for secure communication with other Multi-Domain Servers. This database is synchronized whenever the Global Policy database is synchronized. Only the Active Multi-Domain Server can issue and revoke certificates for other Multi-Domain Servers. When a Standby Multi-Domain Server becomes Active, its ICA also becomes "Active."

Domain Management Server Databases

Each Domain Management Server includes the following data:

1. Domain network objects
2. Domain Security Gateway definitions
3. Domain Security Policies
4. Domain Blade and feature configuration
5. Domain Certificate Authority (CA)
6. Other Domain-specific settings

How Synchronization Works

Multi-Domain Server Database Synchronization

By default, Multi-Domain Server database synchronization occurs automatically whenever an object is changed. The Multi-Domain Server databases are synchronized for the specific object change. For example, if you add a new administrator to the system, all Multi-Domain Servers will be updated with this information.

Multi-Domain Server ICA Database Synchronization

When a new Multi-Domain Server is added to the deployment, the active Multi-Domain Server ICA must issue it a certificate. If a new administrator is added to the system, the Multi-Domain Server ICA may issue a certificate to the new administrator, depending on the administrator's authentication method. The Multi-Domain Server ICA database is updated. If there is more than one Multi-Domain Server in the system, the Multi-Domain Server ICA databases must be synchronized to reflect these additions.

Global Policies Database Synchronization

Global Policies data synchronization occurs either when you save the global policy or after a specified event. See Automatic Synchronization for Global Policies Databases for details. Unlike the system database synchronization, which is per object, the entire contents of the Global Policies database are synchronized.

Domain Management Server Database Synchronization

Domain Management Server database synchronization occurs for each Domain separately. Domain Management Servers for each Domain are synchronized when a Domain policy is saved, or at another defined event (for details about synchronization settings, see Automatic Domain Management Server Synchronization). The entire contents of the Domain Management Server database are synchronized.

Different Domains may have different synchronization settings. This means that different Domain Management Servers synchronize according to the specific settings for that Domain only. When information is changed or updated for a Domain, all Domain Management Servers must receive the new information. For example, if a Security Gateway is added to a Domain network, and the Security Gateway receives a certificate from the Domain ICA, this information must be synchronized between all of the Domain Management Servers.

Full Synchronization Between Multi-Domain Servers

All synchronizations tasks occur according to specified synchronization settings or conditions, even if they occur on the same platforms.

Configuring Synchronization

Using SmartDomain Manager to Synchronize Multi-Domain Servers

High Availability is managed using the SmartDomain Manager High Availability View. You can perform all management High Availability tasks and view the status of these actions after a configurable delay.

The Sync Status displays synchronization statuses for Multi-Domain Servers and Domain Management Servers. Synchronization takes a while to update the status. The default is 5 minutes.

Multi-Domain Server synchronization status is applicable for the Global Policies database. The ICA database is synchronized automatically when new certificates are created for administrators, Multi-Domain Servers or Multi-Domain Log Servers. When the database contents change because of operations in the Global SmartDashboard, synchronization starts during the next Global Policies database synchronization.

Sync Status values:

• Unknown — No information received about this Domain Management Server or Multi-Domain Server synchronization status. This is temporary status shows until the first synchronization is complete.
• Never synced — This Domain Management Server or Multi-Domain Server was not synchronized with the other Domain Management Server/Multi-Domain Server to which the SmartDomain Manager is connected.
• Synchronized — This Domain Management Server or Multi-Domain Server is synchronized with the other Domain Management Server/Multi-Domain Server to which the SmartDomain Manager is connected.
• Lagging — The data of this Domain Management Server or Multi-Domain Server is less updated than the data of the other Domain Management Server/Multi-Domain Server to which the SmartDomain Manager is connected.
• Advanced —The data of this Domain Management Server or Multi-Domain Server is more updated than the data of the other Domain Management Server/Multi-Domain Server to which the SmartDomain Manager is connected.
• Collision — The data of this Domain Management Server or Multi-Domain Server conflicts with the data of the other Domain Management Server/Multi-Domain Server to which the SmartDomain Manager is connected.

Domain Management Server High Availability

Domain Management Server High Availability gives redundancy for a Domain network. At any given time, one Domain Management Server is active, while any one or more Domain Management Servers for the same Domain are in the standby mode. Data synchronization between these Domain Management Servers greatly improves fault tolerance and lets administrators seamlessly activate a standby Domain Management Server as needed. Active Domain Management Server and standby Domain Management Servers must be hosted on different Multi-Domain Servers.

You can create all redundant Domain Management Servers at the same time, or add additional Domain Management Servers at a later time. Once the Domain Management Servers have been initialized and synchronized, there is no functional difference between them.

You do not have to assign all active or all standby Domain Management Servers to the same Multi-Domain Server. A Multi-Domain Server can host a mixture of active and standby Domain Management Servers, allowing you to distribute the traffic load.

You make security policy changes using the active Domain Management Server using the Domain Management Server SmartDashboard. By default, standby Domain Management Servers are automatically synchronized with the active Domain Management Server. You can optionally configure the system to use manual synchronization.

Active versus Standby

All management operations such as editing and installing the Security Policy and modifying users and objects, are done using the Active Domain Management Server. If the active Domain Management Server is unavailable, you must change one of the Standby Domain Management Servers to active.

Standby Domain Management Servers are synchronized to the Active Domain Management Server, and therefore, are kept up to date with all changes in the databases and Security Policy. Gateways can fetch the Security Policy and retrieve a Certificate Revocation List (CRL) from any Domain Management Server.

The terms "Active" and "Standby" are not the same as the terms "Primary Domain Management Server" and "Secondary Domain Management Server," which have to do with the chronological order of creation. Either Domain Management Server can be set up to be Active or Standby. Initially, the Primary Domain Management Server (the first one created) is the Active one, but later on the administrator can manually change this as needed.

Adding a Secondary Domain Management Server

When you add a secondary Domain Management Server, the system does these tasks automatically:

1. Creates duplicate Domain Management Servers on another Multi-Domain Server.
2. Copies the Certificate Authority (CA) files from the primary Domain Management Server to the secondary Domain Management Servers.
3. Starts the secondary Domain Management Server.
4. Exchanges the activation key between the Domain Management Servers.
5. Initializes SIC communication between the Domain Management Servers.
6. Synchronizes the secondary Domain Management Server with the primary Domain Management Server. At this stage, both Domain Management Servers are running (if the primary Domain Management Server is down, the system will automatically try to start it).

   If the operation fails at stage 3 or 4, the administrator can complete these stages manually.

See Mirroring Domain Management Servers with mdscmd for instructions on mirroring Domain Management Servers using the CLI.

Domain Management Server Backup Using a Security Management Server

You can use a Security Management Server to backup Domain Management Servers in a high availability deployment. This Security Management Server can operate as an Active or Standby management.

You can only backup one Domain Management Server to a Security Management Server. If you need to backup multiple Domain Management Servers, you must back each one to a different Security Management Server.

For example:

• A backup Security Management Server is the standby management server and the Domain Management Server is the active management server. If the Domain Management Server is unavailable, the Security Management Server becomes the Active management.
• The Domain Management Server operates as the standby management and the backup Security Management Server is the Active management. If the backup Security Management Server is unavailable, the Domain Management Server becomes the Active management.

In either case, you must change one Domain Management Server to active to assign a global policy.

You must define GUI clients and administrators locally on the Security Management Server. The backup process cannot export this data from a Domain Management Server to a Security Management Server.

Creating a Backup Security Management Server

To create a backup Security Management Server from a fresh installation:

1. Do a fresh Security Management Server installation, defining the Security Management Server as a secondary Security Management Server.
2. Use cpconfig to configure the following:
   1. Select an activation key that will be used to establish SIC trust between the Security Management Server and Domain Management Server.
   2. Define GUI Clients and Administrators.
3. In the Domain Management Server SmartDashboard, create a network object that will represent the secondary backup Security Management Server.
   1. Select Manage > Network Objects > Check Point > New > Host
   2. In the Check Point Host window, select Secondary Management Station under Check Point Products. This automatically selects the Log Server.
4. From the object created in step 3 establish secure communication with the secondary backup Security Management Server.
5. From SmartDashboard access the Policy menu, select Management High Availability and press the Synchronize button.

To setup a backup Security Management Server from an existing Security Management Server:

1. Migrate the existing Security Management Server to the Domain Management Server.

   See "Upgrading Multi-Domain Security Management" in the R76 Installation and Upgrade Guide.

2. Perform a fresh Security Management Server installation as a secondary Security Management Server on an existing or new machine.
3. Using cpconfig to select an activation key that will be used to establish secure internal communication (SIC) between the Domain Management Server and Security Management.
4. Create a network object in the Domain Management Server that will represent the secondary backup Security Management Server.
   1. Select Manage > Network Objects > Check Point > New > Host
   2. In the Check Point Host window, check Secondary Management Station under Check Point Products. This automatically selects Log Server as well.
5. From the object created in step 4 establish secure communication with the secondary backup Security Management Server.
6. From SmartDashboard access the Policy menu, select Management High Availability and press the Synchronize button.

Configuration

Adding another Multi-Domain Server

These steps are described in greater detail in the section Creating a Primary Multi-Domain Server.

1. Synchronize the system clock of the new Multi-Domain Server computer with all other Multi-Domain Servers computers' system clocks.
2. Run the Multi-Domain Server installation script to install the Multi-Domain Server.
3. When prompted if this is a primary Multi-Domain Server, enter No.
4. During the configuration phase, add a Multi-Domain Server license, and enter the SIC Activation Key. This Activation Key is required to send the SIC certificate to the new Multi-Domain Server from the primary Multi-Domain Server.
5. In the SmartDomain Manager connected to the first Multi-Domain Server, define a new Multi-Domain Server. Assign it the IP address of the Leading Interface you selected for it in the configuration phase. Send the new Multi-Domain Server a certificate by the Initialize Communication option. Use the same Activation Key you entered in the configuration of the new Multi-Domain Server.
6. Do an "Initial synchronization" for this Multi-Domain Server when prompted. Your new Multi-Domain Server is now ready for use.

Creating a Mirror of an Existing Multi-Domain Server

Mirroring an existing Multi-Domain Server creates an exact duplicate that Multi-Domain Server.

To mirror an existing Multi-Domain Server:

1. Set up route tables.
2. Synchronize the system clock of the computer on which you will install the Multi-Domain Server with all other Multi-Domain Servers.
3. Install and create a new Multi-Domain Server. Define the new Multi-Domain Server using the SmartDomain Manager.
4. Do an initial synchronization. See Initializing Synchronization.
5. To complete the synchronization, run this command:

mdscmd mirrormanagement <-s source_mds <-t target_mds>
[-m <ServerName> -u user -p password]

-s source_mds stands for the primary Multi-Domain Server name
-t target_mds stands for the mirror Multi-Domain Server name
-m ServerName stands for another Multi-Domain Server logged into to do this action, 
and -u user -p password are the login user name and password. 
Note that -m, -u and -p are optional, but if used, must be used together.

This command synchronizes the data of all Domain Management Servers maintained by the source Multi-Domain Server. In fact, a duplicate (Mirror) Domain Management Server will be created for each Domain Management Server in the original Multi-Domain Server. For further details, review this command in Commands and Utilities.

First Multi-Domain Server Synchronization

This step can be performed in the Multi-Domain Server Configuration window while creating the Multi-Domain Server. Or it can be done later after the Multi-Domain Server is created, through the SmartDomain Manager High Availability View, as follows:

1. Verify that the Multi-Domain Server Sync Status is Never synced.
2. Ensure that SIC has been established between the Multi-Domain Servers.
3. Right-click the Multi-Domain Server, then select Initialize Synchronization, or select Initialize Synchronization from the Manage menu. The Status Report window is displayed, showing whether synchronization initialization succeeded or failed.

Restarting Multi-Domain Server Synchronization

If you have already started Multi-Domain Server synchronization and it failed to complete successfully, you can restart the synchronization using the High Availability View - Multi-Domain Server Contents mode.

You can either select a single Multi-Domain Server and synchronize it with the Multi-Domain Server you logged into, or select a group of Multi-Domain Servers and synchronize all of them with each other.

To Synchronize a Single Multi-Domain Server with Another Multi-Domain Server

1. Select the Multi-Domain Server you want to synchronize with the Multi-Domain Server you logged into. Check that its Sync Status is other than Never synced or Unknown.
2. Right-click the Multi-Domain Server and select Synchronize, or select Synchronize from the Manage menu.

To Synchronize a Group of Multi-Domain Servers

Choose Select and Synchronize from the Manage menu. The Multi-Domain Server Synchronization window is displayed, in which you to select which Multi-Domain Servers are to be synchronized.

Changing a Standby Multi-Domain Server to an Active Multi-Domain Server

If the Multi-Domain Server status is Standby, you can use the Change Over command to change its status to Active. Once you change the status there is a delay (by default 5 minutes) until the status is updated.

To Change the Active Multi-Domain Server

1. Male sure that you are not logged into the Global SmartDashboard (except in Read-only mode).
2. Select the Multi-Domain Server you want to make Active.
3. Select Change Over from the Manage menu.
4. The status will be changed to Active. The statuses of all other Multi-Domain Server in the system will be Standby.

Automatic Synchronization for Global Policies Databases

The Global Policies database synchronization method is selected in the Global SmartDashboard (Policy > Global properties > Management High Availability menu).

The following options are available:

On Save - after the Save operation in the Global SmartDashboard, the database is synchronized to other Multi-Domain Servers.

Scheduled - you can select a scheduled synchronization (for example, once a day at a certain time). Use local time for the scheduled event.

On Save and Scheduled can be selected simultaneously, or none of the options can be selected.

Add a Secondary Domain Management Server

Add a Domain Management Server through the SmartDomain Manager. A Domain must have at least one Domain Management Server before a secondary Domain Management Server can be added to it. The secondary Domain Management Server must be created on a different Multi-Domain Server. Ensure that the primary Domain Management Server SmartDashboard is closed.

To add a secondary Domain Management Server:

1. In the SmartDomain Manager Domain View, select a Domain, then select Add Domain Management Server or Domain Log Server from the Manage menu, or right-click the Domain and select Add Domain Management Server or Add Domain Log Server.
2. You are required to complete the fields shown. Enter a name for the Domain Management Server which does not contain any spaces. Select a Multi-Domain Server to host this Domain Management Server.
3. Enter the license information.

Mirroring Domain Management Servers with mdscmd

Use the mdscmd mirrormanagement command to mirror all Domain Management Servers on one Multi-Domain Server to another Multi-Domain Server. In the current version, the new mirror Domain Management Servers will be created even for Domains that already have two or more Domain Management Servers.

If you want to limit mirror Domain Management Server creation to Domains that have only one Domain Management Server (or any other number of Domain Management Servers), use the new -c flag. The full command syntax is:

mdscmd mirrormanagement -s <source_server> -t <target_server>
[-c <max_total_number>] [-m Security Management Server

server -u user -p password]

where <max_total_number> is the maximum resulting total number of Domain Management Servers per Domain. For example, to mirror Domain Management Servers only for Domains that have only one Domain Management Server, run:

mdscmd mirrormanagement -s FirstServer -t SecondServer -c 2

Automatic Domain Management Server Synchronization

When you create a secondary Domain Management Server it automatically synchronizes with the active Domain Management Server database. To keep these two Domain Management Servers regularly synchronized, we recommend that you configure automatic synchronization using SmartDashboard. You can select the synchronization method from the Policy > Management High Availability menu. For detailed instructions on synchronizing management stations, see.

Synchronize ClusterXL Security Gateways

The Security Gateway synchronization feature provides the mechanism for synchronizing the states of two Security Gateways. High Availability for Security Gateways is described in the R76 ClusterXL Administration Guide. High Availability for encrypted connections is described in the R76 VPN Administration Guide.

Failure Recovery

In many cases, you can recover a failed Multi-Domain Server in a High Availability deployment. To do this, you promote a Secondary Multi-Domain Server to become the Primary. You can also promote Secondary Domain Management Servers to become Primary Domain Management Servers.

Use these procedures to recover from a failed Multi-Domain Server.

Connecting to a Secondary Multi-Domain Server

To connect to a secondary Multi-Domain Server:

1. Make sure that all functional, Secondary Multi-Domain Servers and Multi-Domain Log Servers are up and running.
2. Connect to a secondary Multi-Domain Server with the SmartDomain Manager.
3. If the Multi-Domain Server that to be promoted to Primary is not active, change it to active now:
   1. Go to the High Availability > MDS Level HA view.
   2. Right-click the secondary Multi-Domain Server and select Change Over to Active.
4. Run these commands on all functional Multi-Domain Servers and Multi-Domain Log Servers:

   # mdsenv

   # cp $MDSDIR/conf/mdsdb/Customers.C cp
$MDSDIR/conf/mdsdb/Customers.prepromote

5. Run these commands on the Multi-Domain Server to be promoted to Primary:

   # mdsenv
# mcd
# enable_mds_deletion <failed_MDS_object_name>

Promoting the Secondary Multi-Domain Server to Primary

This procedure is necessary because there are no automatic steps to promote a Secondary Multi-Domain Server when the Primary Multi-Domain Server fails.

To promote a Secondary Multi-Domain Server to Primary:

1. Run these commands on the Secondary Multi-Domain Server to be promoted:

   # cpprod_util FwSetPrimary 1
# cpprod_util CPPROD_SetValue PROVIDER-1 Primary 4 1 1
# cpprod_util CPPROD_SetValue SIC ICAState 4 3 1
# ckp_regedit -d //SOFTWARE//CheckPoint//SIC OTP
# ckp_regedit -d //SOFTWARE//CheckPoint//SIC ICAip

   These commands update the Secondary Multi-Domain Server registry.

2. Connect to the Check Point Database tool with the Secondary Multi-Domain Server IP address.

   C:\Program Files (x86)\CheckPoint\SmartConsole\R76\
PROGRAM\GuiDBedit.exe /mds

3. On the Tables tab, select Other and then select (or search for) Multi-Domain Servers.

   

4. Delete the failed Domain Management Server object from the Object Name column.
5. Select the Multi-Domain Server to be promoted.
6. Double-click the Primary field in the bottom pane.

   

7. Change the value to true.
8. Save the database (File > Save All or Ctl-s).

Restoring Domain Management Servers

Do these steps for each Domain on the failed Primary Domain Management Server.

To restore the Domain Management Servers:

1. Select a Domain Management Server to be the Primary Domain Management Server.
2. If the selected Domain Management Server is a standby, open it in SmartDashboard.

   When prompted, change the Domain Management Server status to Active and then close SmartDashboard.

3. Change the active Domain Management Server from Secondary to Primary:
   1. Run:

      > mdsenv <domain_server_name>

   2. Run:

      > promote_util

   These steps set the Multi-Domain Server context to the specified Domain Management Server.

4. Open SmartDashboard for the newly promoted Domain Management Server.
5. Find (with Where Used) and delete all instances of the failed Domain Management Server, including the failed Domain Management Server itself.
6. Save the policy.
7. If necessary, manually synchronize the Domain Management Servers.
8. Re-assign Global Policies and install policies on all Security Gateways.
9. If the promoted Domain Management Server is using a HA Domain Management Server license, replace it with a regular Domain Management Server license.

Finishing the Promotion

When you delete the failed Multi-Domain Server, all of its Domain Management Servers, Global Policy assignments and many network objects no longer show in the SmartDomain Manager. To resolve this issue, do this procedure on all Multi-Domain Servers.

You can optionally install a new replacement Multi-Domain Server to replace the failed one.

To restore your High Availability deployment:

Run these commands:

# mdsstop

# mv $MDSDIR/conf/mdsdb/cp-deleted.C $MDSDIR/conf/mdsdb/
cp-deleted.C.prepromote

# cp $MDSDIR/conf/mdsdb/customers.C $MDSDIR/conf/mdsdb/
Domains.C.afterpromote

# cp $MDSDIR/conf/mdsdb/customers.C.prepromote $MDSDIR/
conf/mdsdb/Domains.C

# mdsstart