Viewing Delta Synchronization

Heavily loaded clusters and clusters with geographically separated members pose special challenges.

High connection rates, and large distances between the members can lead to delays that affect the operation of the cluster Two or more Security Gateways that work together in a redundant configuration - High Availability, or Load Sharing..

Monitor the operation of the State Synchronization Technology that synchronizes the relevant information about the current connections (stored in various kernel tables on Check Point Security Gateways) among all Cluster Members over Synchronization Network. Due to State Synchronization, the current connections are not cut off during cluster failover. mechanism in highly loaded and distributed clusters.

Perform these troubleshooting steps:

Examine the Delta Sync Synchronization of kernel tables between all working Cluster Members - exchange of CCP packets that carry pieces of information about different connections and operations that should be performed on these connections in relevant kernel tables. This Delta Sync process is performed directly by Check Point kernel. While performing Full Sync, the Delta Sync updates are not processed and saved in kernel memory. After Full Sync is complete, the Delta Sync packets stored during the Full Sync phase are applied by order of arrival. statistics counters:

Shell	Command
Gaia Clish The name of the default command line shell in Check Point Gaia operating system. This is a restricted shell (role-based administration controls the number of commands available in the shell).	`show cluster statistics sync`
Expert mode	`cphaprob syncstat`

Change the values of the applicable synchronization global configuration parameters.

Reset the Delta Sync statistics counters:

Shell	Command
Gaia Check Point security operating system that combines the strengths of both SecurePlatform and IPSO operating systems. Clish	`show cluster statistics sync reset`
Expert mode	`cphaprob -reset syncstat`

Examine the Delta Sync statistics to see if the problem is solved.
Solve any identified problem.

Example output of the "show cluster statistics sync" and "cphaprob syncstat" commands from a Cluster Member:

Delta Sync Statistics

Sync status: OK

Drops:

Lost updates.................................  0

Lost bulk update events......................  0

Oversized updates not sent...................  0

Sync at risk:

Sent reject notifications....................  0

Received reject notifications................  0

Sent messages:

Total generated sync messages................  26079

Sent retransmission requests.................  0

Sent retransmission updates..................  0

Peak fragments per update....................  1

Received messages:

Total received updates.......................  3710

Received retransmission requests.............  0

Sync Interface:

Name.........................................  eth1

Link speed...................................  1000Mb/s

Rate.........................................  46000 [Bps]

Peak rate....................................  46000 [Bps]

Link usage...................................   0%

Total........................................  376827[KB]

Queue sizes (num of updates):

Sending queue size...........................  512

Receiving queue size.........................  256

Fragments queue size.........................  50

Timers:

Delta Sync interval (ms).....................  100

Reset on Sun Sep  8 16:09:15 2019 (triggered by fullsync).

Each section of the output is described below.

The "Sync status:" section

This section shows the status of the Delta Sync mechanism. One of these:

Sync status: OK
Sync status: Off - Full-sync failure
Sync status: Off - Policy installation failure
Sync status: Off - Cluster module not started
Sync status: Off - SIC failure
Sync status: Off - Full-sync checksum error
Sync status: Off - Full-sync received queue is full
Sync status: Off - Release version mismatch
Sync status: Off - Connection to remote member timed-out
Sync status: Off - Connection terminated by remote member
Sync status: Off - Could not start a connection to remote member
Sync status: Off - cpstart
Sync status: Off - cpstop
Sync status: Off - Manually disabled sync
Sync status: Off - Was not able to start for more than X second
Sync status: Off - Boot
Sync status: Off - Connectivity Upgrade (CU)
Sync status: Off - cphastop
Sync status: Off - Policy unloaded
Sync status: Off - Hibernation
Sync status: Off - OSU deactivated
Sync status: Off - Sync interface down
Sync status: Fullsync in progress
Sync status: Problem (Able to send sync packets, unable to receive sync packets)
Sync status: Problem (Able to send sync packets, saving incoming sync packets)
Sync status: Problem (Able to send sync packets, able to receive sync packets)
Sync status: Problem (Unable to send sync packets, unable to receive sync packets)
Sync status: Problem (Unable to send sync packets, saving incoming sync packets)
Sync status: Problem (Unable to send sync packets, able to receive sync packets)

The "Drops:" section

This section shows statistics for drops on the Delta Sync network.

Table: Description of the output fields
Field	Description
Lost updates	Shows how many Delta Sync updates this Cluster Member Security Gateway that is part of a cluster. considers as lost (based on sequence numbers in CCP packets). If this counter shows a value greater than 0, this Cluster Member lost Delta Sync updates. Possible mitigation: Increase the size of the Sending Queue and the size of the Receiving Queue: Increase the size of the Sending Queue, if the counter Received reject notification is increasing. Increase the size of the Receiving Queue, if the counter Received reject notification is not increasing.
Lost bulk update events	Shows how many times this Cluster Member missed Delta Sync updates. (bulk update = twice the size of the local receiving queue) This counter increases when this Cluster Member receives a Delta Sync update with a sequence number much greater than expected. This probably indicates some networking issues that cause massive packet drops. This counter increases when the amount of missed Delta Sync updates is more than twice the local Receiving Queue Size. Possible mitigation: If the counter's value is steady, this might indicate a one-time synchronization problem that can be resolved by running manual Full Sync Process of full synchronization of applicable kernel tables by a Cluster Member from the working Cluster Member(s) when it tries to join the existing cluster. This process is meant to fetch a ‎"snapshot" of the applicable kernel tables of already Active Cluster Member(s). Full Sync is performed during the initialization of Check Point software (during boot process, the first time the Cluster Member runs policy installation, during 'cpstart', during 'cphastart'). Until the Full Sync process completes successfully, this Cluster Member remains in the Down state, because until it is fully synchronized with other Cluster Members, it cannot function as a Cluster Member. Meanwhile, the Delta Sync packets continue to arrive, and the Cluster Member that tries to join the existing cluster, stores them in the kernel memory until the Full Sync completes. The whole Full Sync process is performed by fwd daemons on TCP port 256 over the Sync network (if it fails over the Sync network, it tries the other cluster interfaces). The information is sent by fwd daemons in chunks, while making sure they confirm getting the information before sending the next chunk. Also see "Delta Sync".. See sk37029. If the counter's value keeps increasing, probable there are some networking issues. Increase the sizes of both the Receiving Queue and Sending Queue.
Oversized updates not sent	Shows how many oversized Delta Sync updates were discarded before sending them. This counter increases when Delta Sync update is larger than the local Fragments Queue Size. Possible mitigation: If the counter's value is steady, increase the size of the Sending Queue. If the counter's value keeps increasing, contact Check Point Support.

The "Sync at risk:" section

This section shows statistics that the Sending Queue is at full capacity and rejects Delta Sync retransmission It is possible that Delta Sync packets will be lost or corrupted during the Delta Sync operations. In such cases, it is required to make sure the Delta Sync packet is re-sent. The Cluster Member requests the sending Cluster Member to retransmit the lost/corrupted Delta Sync packet. Each Delta Sync packet has a sequence number. The sending member has a queue of sent Delta Sync packets. Each Cluster Member has a queue of packets sent from each of the peer Cluster Members. If, for any reason, a Delta Sync packet was not received by a Cluster Member, it can ask for a retransmission of this packet from the sending member. The Delta Sync retransmission mechanism is somewhat similar to a TCP Window and TCP retransmission mechanism. When a member requests retransmission of Delta Sync packet, which no longer exists on the sending member, the member prints a console messages that the sync is not complete. requests.

Table: Description of the output fields
Field	Description
Sent reject notifications	Shows how many times this Cluster Member rejected Delta Sync retransmission requests from its peer Cluster Members, because this Cluster Member does not hold the requested Delta Sync update anymore.
Received reject notification	Shows how many reject notifications this Cluster Member received from its peer Cluster Members.

The "Sent updates:" section

This section shows statistics for Delta Sync updates sent by this Cluster Member to its peer Cluster Members.

Table: Description of the output fields
Field	Description
Total generated sync messages	Shows how many Delta Sync updates were generated. This counts the Delta Sync updates, Retransmission Requests, Retransmission Acknowledgments, and so on.
Sent retransmission requests	Shows how many times this Cluster Member asked its peer Cluster Members to retransmit specific Delta Sync update(s). Retransmission requests are sent when certain Delta Sync updates (with a specified sequence number) are missing, while the sending Cluster Member already received Delta Sync updates with advanced sequences. Note - Compare the number of Sent retransmission requests to the Total generated sync messages of the other Cluster Members. A large counter's value can imply connectivity problems. If the counter's value is unreasonably high (more than 30% of the Total generated sync messages of other Cluster Members), contact Check Point Support equipped with the entire output and a detailed description of the network topology and configuration.
Sent retransmission updates	Shows how many times this Cluster Member retransmitted specific Delta Sync update(s) at the requests from its peer Cluster Members.
Peak fragments per update	Shows the peak amount of fragments in the Fragments Queue on this Cluster Member (usually, should be 1).

The "Received updates:" section

This section shows statistics for Delta Sync updates that were received by this Cluster Member from its peer Cluster Members.

Table: Description of the output fields
Field	Description
Total received updates	Shows the total number of Delta Sync updates this Cluster Member received from its peer Cluster Members. This counts only Delta Sync updates (not Retransmission Requests, Retransmission Acknowledgments, and others).
Received retransmission requests	Shows how many retransmission requests this Cluster Member received from its peer Cluster Members. A large counter's value can imply connectivity problems. If the counter's value is unreasonably high (more than 30% of the Total generated sync messages on this Cluster Member), contact Check Point Support equipped with the entire output and a detailed description of the network topology and configuration.

The "Queue sizes (num of updates):" section

This section shows the sizes of the Delta Sync queues.

Table: Description of the output fields
Field	Description
Sending queue size	Shows the size of the cyclic queue, which buffers all the Delta Sync updates that were already sent until it receives an acknowledgment from the peer Cluster Members. This queue is needed for retransmitting the requested Delta Sync updates. Each Cluster Member has one Sending Queue. Default: 512 Delta Sync updates, which is also the minimal value.
Receiving queue size	Shows the size of the cyclic queue, which buffers the received Delta Sync updates in two cases: When Delta Sync updates are missing, this queue is used to hold the remaining received Delta Sync updates until the lost Delta Sync updates are retransmitted (Cluster Members must keep the order, in which they save the Delta Sync updates in the kernel tables). This queue is used to re-assemble a fragmented Delta Sync update. Each Cluster Member has one Receiving Queue. Default: 256 Delta Sync updates, which is also the minimal value.
Fragments queue size	Shows the size of the queue, which is used to prepare a Delta Sync update before moving it to the Sending Queue. Notes: This queue must be smaller than the Sending Queue. This queue must be significantly smaller than the Receiving Queue. Default: 50 Delta Sync updates, which is also the minimal value.

The "Timers:" section

This section shows the Delta Sync timers.

Field	Description
Delta Sync interval (ms)	Shows the interval at which this Cluster Member sends the Delta Sync updates from its Sending Queue. The base time unit is 100ms (or 1 tick). Default: 100 ms, which is also the minimum value. See Increasing the Sync Timer.

Field

Description

Delta Sync interval (ms)

Shows the interval at which this Cluster Member sends the Delta Sync updates from its Sending Queue.

The base time unit is 100ms (or 1 tick).

Default: 100 ms, which is also the minimum value.

See Increasing the Sync Timer.