Monitoring Cluster Status in SmartConsole

Background

To see the applicable logs in SmartConsole Check Point GUI application used to manage a Check Point environment - configure Security Policies, configure devices, monitor products and events, install updates, and so on.:

• From the left navigation panel, click Logs & Monitor > Logs.

To get logs about changes in the state of Cluster Two or more Security Gateways that work together in a redundant configuration - High Availability, or Load Sharing. Members:

1. From the left navigation panel, click Gateways & Servers.

2. Open the cluster object.

3. From the left tree, click ClusterXL and VRRP.

4. Open the cluster object.

5. In the Tracking field, select Log.

6. Click OK.

7. Install the Access Control Policy on the cluster object.

ClusterXL Log Messages

This section uses these conventions:

1. Square brackets are used to indicate place holders, which are substituted by relevant data when an actual log message is issued (for example, "[NUMBER]" is replaced by a numeric value).

2. Angle brackets are used to indicate alternatives, one of which is used in actual log messages.

   The different alternatives are separated with a vertical line (for example, "<up|down>" indicates that either "up" or "down" is used).

3. These placeholders are frequently used:

   • ID: A unique Cluster Member Security Gateway that is part of a cluster. identifier, starting from "1". This corresponds to the order, in which Cluster Members are sorted in the cluster object on the Cluster Members page.

   • IP: Any unique IP address that belongs to the Cluster Member.

   • MODE: The cluster mode Configuration of Cluster Members to work in these redundant modes: (1) One Cluster Member processes all the traffic - High Availability or VRRP mode (2) All traffic is processed in parallel by all Cluster Members - Load Sharing. (for example, New HA, LS Multicast, and so on).

   • STATE: The state of the member (for example, active, down, standby).

   • DEVICE: The name of a Critical Device A special software device on each Cluster Member, through which the critical aspects for cluster operation are monitored. When the critical monitored component on a Cluster Member fails to report its state on time, or when its state is reported as problematic, the state of that member is immediately changed to Down. The complete list of the configured critical devices (pnotes) is printed by the 'cphaprob -ia list' command or 'show cluster members pnotes all' command. Synonyms: Pnote, Problem Notification. (for example, Interface Active Check, fwd).

General Logs

Log	Description
Starting <ClusterXL|State Synchronization>.	Indicates that ClusterXL Cluster of Check Point Security Gateways that work together in a redundant configuration. The ClusterXL both handles the traffic and performs State Synchronization. These Check Point Security Gateways are installed on Gaia OS: (1) ClusterXL supports up to 5 Cluster Members, (2) VRRP Cluster supports up to 2 Cluster Members, (3) VSX VSLS cluster supports up to 13 Cluster Members. Note: In ClusterXL Load Sharing mode, configuring more than 4 Cluster Members significantly decreases the cluster performance due to amount of Delta Sync traffic. (or State Synchronization Technology that synchronizes the relevant information about the current connections (stored in various kernel tables on Check Point Security Gateways) among all Cluster Members over Synchronization Network. Due to State Synchronization, the current connections are not cut off during cluster failover., for 3rd party clusters) was successfully started on the reporting Cluster Member. This message is usually issued after a member boots, or after an explicit call to cphastart.
Stopping <ClusterXL|State Synchronization>.	Informs that ClusterXL (or State Synchronization) was deactivated on this Cluster Member. The Cluster Member is no longer a part of the cluster (even if configured to be so), until ClusterXL is restarted.
Unconfigured cluster Computers changed their MAC Addresses. Please reboot the cluster so that the changes take affect.	This message is usually issued when a Cluster Member is shut down State of a Cluster Member during a failure when one of the Critical Devices reports its state as "problem": In ClusterXL, applies to the state of the Security Gateway component; in 3rd-party / OPSEC cluster, applies to the state of the State Synchronization mechanism. A Cluster Member in this state does not process any traffic passing through cluster., or after an explicit call to the cphastop.

State Logs

Log	Description
Mode inconsistency detected: member [ID] ([IP]) will change its mode to [MODE]. Please re-install the security policy on the cluster.	This message should rarely happen. It indicates that another Cluster Member has reported a different cluster mode than is known to the local Cluster Member. This is usually the result of a failure A hardware or software problem that causes a Security Gateway to be unable to serve as a Cluster Member (for example, one of cluster interface has failed, or one of the monitored daemon has crashed). Cluster Member that suffered from a failure is declared as failed, and its state is changed to Down (a physical interface is considered Down only if all configured VLANs on that physical interface are Down). to install the Access Control Policy on all Cluster Members. To correct this problem, install the Access Control Policy again. Note - The cluster continues to operate after a mode inconsistency is detected by altering the mode of the reporting Cluster Member to match the other Cluster Members. However, we highly recommend to re-install the policy as soon as possible.
State change of member [ID] ([IP]) from [STATE] to [STATE] was cancelled, since all other members are down. Member remains [STATE].	When a Cluster Member needs to change its state (for example, when an Active State of a Cluster Member that is fully operational: (1) In ClusterXL, this applies to the state of the Security Gateway component (2) In 3rd-party / OPSEC cluster, this applies to the state of the cluster State Synchronization mechanism. Cluster Member encounters a problem and needs to change its state to "Down"), it first queries the other Cluster Members for their state. If all other Cluster Members are down, this Cluster Member cannot change its state to a non-active one (otherwise the cluster cannot function). Thus, the reporting Cluster Member continues to function, despite its problem (and usually reports its state as "Active(!)").
member [ID] ([IP]) <is active|is down|is stand-by|is initializing> ([REASON]).	This message is issued whenever a Cluster Member changes its state. The log text specifies the new state of the Cluster Member.

Critical Device Logs

Log	Description
[DEVICE] on member [ID] ([IP]) status OK ([REASON])	The Critical Device is working normally.
[DEVICE] on member [ID] ([IP]) detected a problem ([REASON]).	Either an error was detected by the Critical Device, or the Critical Device has not reported its state for a number of seconds (as set by the "timeout" option of the Critical Device)
[DEVICE] on member [ID] ([IP]) is initializing ([REASON]).	Indicates that the Critical Device has registered itself with the Critical Device mechanism, but has not yet determined its state.
[DEVICE] on member [ID] ([IP]) is in an unknown state ([STATE ID]) ([REASON]).	This message should not normally appear. Contact Check Point Support.

Interface Logs

Log	Description
interface [INTERFACE NAME] of member [ID] ([IP]) is up.	Indicates that this interface is working normally - it is able to receive and transmit Cluster Control Protocol Proprietary Check Point protocol that runs between Cluster Members on UDP port 8116, and has the following roles: (1) State Synchronization (Delta Sync), (2) Health checks (state of Cluster Members and of cluster interfaces): Health-status Reports, Cluster-member Probing, State-change Commands, Querying for cluster membership. Note: CCP is located between the Check Point Firewall kernel and the network interface (therefore, only TCPdump should be used for capturing this traffic). Acronym: CCP. (CCP) packets on the expected subnet.
interface [INTERFACE NAME] of member [ID] ([IP]) is down (receive <up|down>, transmit <up|down>).	This message is issued whenever an interface encounters a problem, either in receiving or transmitting Cluster Control Protocol (CCP) packets. Note that in this case the interface may still be working properly, as far as the OS is concerned, but is unable to communicate with other Cluster Members.
interface [INTERFACE NAME] of member [ID] ([IP]) was added.	Indicates that a new interface was registered with the Cluster Member (meaning that Cluster Control Protocol (CCP) packets arriving on this interface). Usually, this message is the result of activating an interface (such as issuing the "ifconfig up" command). The interface is now included in the ClusterXL reports (in the output of the applicable CLI commands. Note that the interface may still be reported as "Disconnected", in case it was configured as such for ClusterXL.
interface [INTERFACE NAME] of member [ID] ([IP}) was removed.	Indicates that an interface was detached from the Cluster Member, and is therefore no longer monitored by ClusterXL.

Reason Strings

Log	Description
member [ID] ([IP]) reports more interfaces up.	This text can be included in a Critical Device log message describing the reasons for a problem report: another Cluster Member has more interfaces reported to be working, than the local Cluster Member does. Usually, this means that the local Cluster Member has a faulty interface, and that its peer Cluster Member can do a better job as a Cluster Member. The local Cluster Member changes it state to "Down", leaving the peer Cluster Member specified in the message to handle traffic.
member [ID] ([IP]) has more interfaces - check your disconnected interfaces configuration in the <discntd.if file|registry>	This message is issued when Cluster Members in the same cluster have a different number of interfaces. A Cluster Member with fewer interfaces than the maximal number in the cluster (the reporting Cluster Member) may not be working properly, as it is missing an interface required to operate against a cluster IP address, or a synchronization network A set of interfaces on Cluster Members that were configured as interfaces, over which State Synchronization information will be passed (as Delta Sync packets ). The use of more than one Synchronization Network for redundancy is not supported because the CPU load will increase significantly due to duplicate tasks performed by all configured Synchronization Networks. Synonyms: Sync Network, Secured Network, Trusted Network.. If some of the interfaces on the other Cluster Member are redundant, and should not be monitored by ClusterXL, they should be explicitly designated as "Non-Monitored". See Defining Non-Monitored Interfaces.
[NUMBER] interfaces required, only [NUMBER] up.	ClusterXL has detected a problem with one or more of the monitored interfaces. This does not necessarily mean that the Cluster Member changes its state to "Down", as the other Cluster Members may have less operational interfaces. In such a condition, the Cluster Member with the largest number of operational interfaces will remain up, while the others go down.