Viewing Delta Synchronization

In the R81.10.X releases, this command is available starting from the R81.10.15 version.

Heavily loaded clusters and clusters with geographically separated members pose special challenges.

High connection rates, and large distances between the members can lead to delays that affect the operation of the cluster.

Monitor the operation of the State Synchronization mechanism in highly loaded and distributed clusters.

Perform these troubleshooting steps:

Examine the Delta Sync statistics counters:

cphaprob syncstat

Change the values of the applicable synchronization global configuration parameters.

Reset the Delta Sync statistics counters:

cphaprob -reset syncstat

Examine the Delta Sync statistics to see if the problem is solved.
Solve any identified problem.

Example output of the "cphaprob syncstat" command from a Cluster Member:

Delta Sync Statistics

Sync status: OK

Drops:
Lost updates.................................  0
Lost bulk update events......................  0
Oversized updates not sent...................  0

Sync at risk:
Sent reject notifications....................  0
Received reject notifications................  0

Sent messages:
Total generated sync messages................  26079
Sent retransmission requests.................  0
Sent retransmission updates..................  0
Peak fragments per update....................  1

Received messages:
Total received updates.......................  3710
Received retransmission requests.............  0

Sync Interface:
Name.........................................  SYNCBOND
Link speed...................................  1000Mb/s
Rate.........................................  46000 [Bps]
Peak rate....................................  46000 [Bps]
Link usage...................................   0%
Total........................................  376827[KB]

Queue sizes (num of updates):
Sending queue size...........................  512
Receiving queue size.........................  256
Fragments queue size.........................  50

Timers:
Delta Sync interval (ms).....................  100

Reset on Sun Sep  8 16:09:15 2019 (triggered by fullsync).

Each section of the output is described below.

The "Sync status:" section

This section shows the status of the Delta Sync mechanism. One of these:

Sync status: OK
Sync status: Off - Full-sync failure
Sync status: Off - Policy installation failure
Sync status: Off - Cluster module not started
Sync status: Off - SIC failure
Sync status: Off - Full-sync checksum error
Sync status: Off - Full-sync received queue is full
Sync status: Off - Release version mismatch
Sync status: Off - Connection to remote member timed-out
Sync status: Off - Connection terminated by remote member
Sync status: Off - Could not start a connection to remote member
Sync status: Off - cpstart
Sync status: Off - cpstop
Sync status: Off - Manually disabled sync
Sync status: Off - Was not able to start for more than X second
Sync status: Off - Boot
Sync status: Off - Connectivity Upgrade (CU)
Sync status: Off - cphastop
Sync status: Off - Policy unloaded
Sync status: Off - Hibernation
Sync status: Off - OSU deactivated
Sync status: Off - Sync interface down
Sync status: Fullsync in progress
Sync status: Problem (Able to send sync packets, unable to receive sync packets)
Sync status: Problem (Able to send sync packets, saving incoming sync packets)
Sync status: Problem (Able to send sync packets, able to receive sync packets)
Sync status: Problem (Unable to send sync packets, unable to receive sync packets)
Sync status: Problem (Unable to send sync packets, saving incoming sync packets)
Sync status: Problem (Unable to send sync packets, able to receive sync packets)

The "Drops:" section

This section shows statistics for drops on the Delta Sync network.

Table: Description of the output fields
Field	Description
Lost updates	Shows how many Delta Sync updates this Cluster Member considers as lost (based on sequence numbers in CCP packets). If this counter shows a value greater than 0, this Cluster Member lost Delta Sync updates. Possible mitigation: Increase the size of the Sending Queue and the size of the Receiving Queue: Increase the size of the Sending Queue, if the counter Received reject notification is increasing. Increase the size of the Receiving Queue, if the counter Received reject notification is not increasing.
Lost bulk update events	Shows how many times this Cluster Member missed Delta Sync updates. (bulk update = twice the size of the local receiving queue) This counter increases when this Cluster Member receives a Delta Sync update with a sequence number much greater than expected. This probably indicates some networking issues that cause massive packet drops. This counter increases when the amount of missed Delta Sync updates is more than twice the local Receiving Queue Size. Possible mitigation: If the counter's value is steady, this might indicate a one-time synchronization problem that can be resolved by running manual Full Sync. If the counter's value keeps increasing, probable there are some networking issues. Increase the sizes of both the Receiving Queue and Sending Queue.
Oversized updates not sent	Shows how many oversized Delta Sync updates were discarded before sending them. This counter increases when Delta Sync update is larger than the local Fragments Queue Size. Possible mitigation: If the counter's value is steady, increase the size of the Sending Queue. If the counter's value keeps increasing, contact Check Point Support.

The "Sync at risk:" section

This section shows statistics that the Sending Queue is at full capacity and rejects Delta Sync retransmission requests.

Table: Description of the output fields
Field	Description
Sent reject notifications	Shows how many times this Cluster Member rejected Delta Sync retransmission requests from its peer Cluster Members, because this Cluster Member does not hold the requested Delta Sync update anymore.
Received reject notification	Shows how many reject notifications this Cluster Member received from its peer Cluster Members.

The "Sent updates:" section

This section shows statistics for Delta Sync updates sent by this Cluster Member to its peer Cluster Members.

Table: Description of the output fields
Field	Description
Total generated sync messages	Shows how many Delta Sync updates were generated. This counts the Delta Sync updates, Retransmission Requests, Retransmission Acknowledgments, and so on.
Sent retransmission requests	Shows how many times this Cluster Member asked its peer Cluster Members to retransmit specific Delta Sync update(s). Retransmission requests are sent when certain Delta Sync updates (with a specified sequence number) are missing, while the sending Cluster Member already received Delta Sync updates with advanced sequences. Note - Compare the number of Sent retransmission requests to the Total generated sync messages of the other Cluster Members. A large counter's value can imply connectivity problems. If the counter's value is unreasonably high (more than 30% of the Total generated sync messages of other Cluster Members), contact Check Point Support equipped with the entire output and a detailed description of the network topology and configuration.
Sent retransmission updates	Shows how many times this Cluster Member retransmitted specific Delta Sync update(s) at the requests from its peer Cluster Members.
Peak fragments per update	Shows the peak amount of fragments in the Fragments Queue on this Cluster Member (usually, should be 1).

The "Received updates:" section

This section shows statistics for Delta Sync updates that were received by this Cluster Member from its peer Cluster Members.

Table: Description of the output fields
Field	Description
Total received updates	Shows the total number of Delta Sync updates this Cluster Member received from its peer Cluster Members. This counts only Delta Sync updates (not Retransmission Requests, Retransmission Acknowledgments, and others).
Received retransmission requests	Shows how many retransmission requests this Cluster Member received from its peer Cluster Members. A large counter's value can imply connectivity problems. If the counter's value is unreasonably high (more than 30% of the Total generated sync messages on this Cluster Member), contact Check Point Support equipped with the entire output and a detailed description of the network topology and configuration.

The "Queue sizes (num of updates):" section

This section shows the sizes of the Delta Sync queues.

Table: Description of the output fields
Field	Description
Sending queue size	Shows the size of the cyclic queue, which buffers all the Delta Sync updates that were already sent until it receives an acknowledgment from the peer Cluster Members. This queue is needed for retransmitting the requested Delta Sync updates. Each Cluster Member has one Sending Queue. Default: 512 Delta Sync updates, which is also the minimal value.
Receiving queue size	Shows the size of the cyclic queue, which buffers the received Delta Sync updates in two cases: When Delta Sync updates are missing, this queue is used to hold the remaining received Delta Sync updates until the lost Delta Sync updates are retransmitted (Cluster Members must keep the order, in which they save the Delta Sync updates in the kernel tables). This queue is used to re-assemble a fragmented Delta Sync update. Each Cluster Member has one Receiving Queue. Default: 256 Delta Sync updates, which is also the minimal value.
Fragments queue size	Shows the size of the queue, which is used to prepare a Delta Sync update before moving it to the Sending Queue. Notes: This queue must be smaller than the Sending Queue. This queue must be significantly smaller than the Receiving Queue. Default: 50 Delta Sync updates, which is also the minimal value.

The "Timers:" section

This section shows the Delta Sync timers.

Field	Description
Delta Sync interval (ms)	Shows the interval at which this Cluster Member sends the Delta Sync updates from its Sending Queue. The base time unit is 100ms (or 1 tick). Default: 100 ms, which is also the minimum value. See Increasing the Sync Timer.

Field

Description

Delta Sync interval (ms)

Shows the interval at which this Cluster Member sends the Delta Sync updates from its Sending Queue.

The base time unit is 100ms (or 1 tick).

Default: 100 ms, which is also the minimum value.

See Increasing the Sync Timer.