Viewing Delta Synchronization

Heavily loaded clusters and clusters with geographically separated members pose special challenges.

High connection rates, and large distances between the members can lead to delays that affect the operation of the cluster.

Monitor the operation of the State Synchronization mechanism in highly loaded and distributed clusters.

Perform these troubleshooting steps:

  1. Examine the Delta Sync statistics counters:

    Shell

    Command

    Gaia Clish

    show cluster statistics sync

    Expert mode

    cphaprob syncstat

  2. Change the values of the applicable synchronization global configuration parameters.

  3. Reset the Delta Sync statistics counters:

    Shell

    Command

    Gaia Clish

    show cluster statistics sync reset

    Expert mode

    cphaprob -reset syncstat

  4. Examine the Delta Sync statistics to see if the problem is solved.

  5. Solve any identified problem.

Example output of the "show cluster statistics sync" and "cphaprob syncstat" commands from a Cluster Member:

Delta Sync Statistics
 
Sync status: OK
 
Drops:
Lost updates.................................  0
Lost bulk update events......................  0
Oversized updates not sent...................  0
 
Sync at risk:
Sent reject notifications....................  0
Received reject notifications................  0
 
Sent messages:
Total generated sync messages................  26079
Sent retransmission requests.................  0
Sent retransmission updates..................  0
Peak fragments per update....................  1
 
Received messages:
Total received updates.......................  3710
Received retransmission requests.............  0
 
Sync Interface:
Name.........................................  eth1
Link speed...................................  1000Mb/s
Rate.........................................  46000 [Bps]
Peak rate....................................  46000 [Bps]
Link usage...................................   0%
Total........................................  376827[KB]
 
Queue sizes (num of updates):
Sending queue size...........................  512
Receiving queue size.........................  256
Fragments queue size.........................  50
 
Timers:
Delta Sync interval (ms).....................  100
 
Reset on Sun Sep  8 16:09:15 2019 (triggered by fullsync).

Each section of the output is described below.

This section shows the status of the Delta Sync mechanism. One of these:

  • Sync status: OK

  • Sync status: Off - Full-sync failure

  • Sync status: Off - Policy installation failure

  • Sync status: Off - Cluster module not started

  • Sync status: Off - SIC failure

  • Sync status: Off - Full-sync checksum error

  • Sync status: Off - Full-sync received queue is full

  • Sync status: Off - Release version mismatch

  • Sync status: Off - Connection to remote member timed-out

  • Sync status: Off - Connection terminated by remote member

  • Sync status: Off - Could not start a connection to remote member

  • Sync status: Off - cpstart

  • Sync status: Off - cpstop

  • Sync status: Off - Manually disabled sync

  • Sync status: Off - Was not able to start for more than X second

  • Sync status: Off - Boot

  • Sync status: Off - Connectivity Upgrade (CU)

  • Sync status: Off - cphastop

  • Sync status: Off - Policy unloaded

  • Sync status: Off - Hibernation

  • Sync status: Off - OSU deactivated

  • Sync status: Off - Sync interface down

  • Sync status: Fullsync in progress

  • Sync status: Problem (Able to send sync packets, unable to receive sync packets)

  • Sync status: Problem (Able to send sync packets, saving incoming sync packets)

  • Sync status: Problem (Able to send sync packets, able to receive sync packets)

  • Sync status: Problem (Unable to send sync packets, unable to receive sync packets)

  • Sync status: Problem (Unable to send sync packets, saving incoming sync packets)

  • Sync status: Problem (Unable to send sync packets, able to receive sync packets)

This section shows statistics for drops on the Delta Sync network.

Table: Description of the output fields

Field

Description

Lost updates

Shows how many Delta Sync updates this Cluster Member considers as lost (based on sequence numbers in CCP packets).

If this counter shows a value greater than 0, this Cluster Member lost Delta Sync updates.

Possible mitigation:

Increase the size of the Sending Queue and the size of the Receiving Queue:

  • Increase the size of the Sending Queue, if the counter Received reject notification is increasing.

  • Increase the size of the Receiving Queue, if the counter Received reject notification is not increasing.

Lost bulk update events

Shows how many times this Cluster Member missed Delta Sync updates.

(bulk update = twice the size of the local receiving queue)

This counter increases when this Cluster Member receives a Delta Sync update with a sequence number much greater than expected. This probably indicates some networking issues that cause massive packet drops.

This counter increases when the amount of missed Delta Sync updates is more than twice the local Receiving Queue Size.

Possible mitigation:

  • If the counter's value is steady, this might indicate a one-time synchronization problem that can be resolved by running manual Full Sync. See sk37029.

  • If the counter's value keeps increasing, probable there are some networking issues. Increase the sizes of both the Receiving Queue and Sending Queue.

Oversized updates not sent

Shows how many oversized Delta Sync updates were discarded before sending them.

This counter increases when Delta Sync update is larger than the local Fragments Queue Size.

Possible mitigation:

  • If the counter's value is steady, increase the size of the Sending Queue.

  • If the counter's value keeps increasing, contact Check Point Support.

This section shows statistics that the Sending Queue is at full capacity and rejects Delta Sync retransmission requests.

Table: Description of the output fields

Field

Description

Sent reject notifications

Shows how many times this Cluster Member rejected Delta Sync retransmission requests from its peer Cluster Members, because this Cluster Member does not hold the requested Delta Sync update anymore.

Received reject notification

Shows how many reject notifications this Cluster Member received from its peer Cluster Members.

This section shows statistics for Delta Sync updates sent by this Cluster Member to its peer Cluster Members.

Table: Description of the output fields

Field

Description

Total generated sync messages

Shows how many Delta Sync updates were generated.

This counts the Delta Sync updates, Retransmission Requests, Retransmission Acknowledgments, and so on.

Sent retransmission requests

Shows how many times this Cluster Member asked its peer Cluster Members to retransmit specific Delta Sync update(s).

Retransmission requests are sent when certain Delta Sync updates (with a specified sequence number) are missing, while the sending Cluster Member already received Delta Sync updates with advanced sequences.

Note - Compare the number of Sent retransmission requests to the Total generated sync messages of the other Cluster Members.

A large counter's value can imply connectivity problems. If the counter's value is unreasonably high (more than 30% of the Total generated sync messages of other Cluster Members), contact Check Point Support equipped with the entire output and a detailed description of the network topology and configuration.

Sent retransmission updates

Shows how many times this Cluster Member retransmitted specific Delta Sync update(s) at the requests from its peer Cluster Members.

Peak fragments per update

Shows the peak amount of fragments in the Fragments Queue on this Cluster Member (usually, should be 1).

This section shows statistics for Delta Sync updates that were received by this Cluster Member from its peer Cluster Members.

Table: Description of the output fields

Field

Description

Total received updates

Shows the total number of Delta Sync updates this Cluster Member received from its peer Cluster Members.

This counts only Delta Sync updates (not Retransmission Requests, Retransmission Acknowledgments, and others).

Received retransmission requests

Shows how many retransmission requests this Cluster Member received from its peer Cluster Members.

A large counter's value can imply connectivity problems. If the counter's value is unreasonably high (more than 30% of the Total generated sync messages on this Cluster Member), contact Check Point Support equipped with the entire output and a detailed description of the network topology and configuration.

This section shows the sizes of the Delta Sync queues.

Table: Description of the output fields

Field

Description

Sending queue size

Shows the size of the cyclic queue, which buffers all the Delta Sync updates that were already sent until it receives an acknowledgment from the peer Cluster Members.

This queue is needed for retransmitting the requested Delta Sync updates.

Each Cluster Member has one Sending Queue.

Default: 512 Delta Sync updates, which is also the minimal value.

Receiving queue size

Shows the size of the cyclic queue, which buffers the received Delta Sync updates in two cases:

  • When Delta Sync updates are missing, this queue is used to hold the remaining received Delta Sync updates until the lost Delta Sync updates are retransmitted (Cluster Members must keep the order, in which they save the Delta Sync updates in the kernel tables).

  • This queue is used to re-assemble a fragmented Delta Sync update.

Each Cluster Member has one Receiving Queue.

Default: 256 Delta Sync updates, which is also the minimal value.

Fragments queue size

Shows the size of the queue, which is used to prepare a Delta Sync update before moving it to the Sending Queue.

Notes:

  • This queue must be smaller than the Sending Queue.

  • This queue must be significantly smaller than the Receiving Queue.

Default: 50 Delta Sync updates, which is also the minimal value.

This section shows the Delta Sync timers.

Field

Description

Delta Sync interval (ms)

Shows the interval at which this Cluster Member sends the Delta Sync updates from its Sending Queue.

The base time unit is 100ms (or 1 tick).

Default: 100 ms, which is also the minimum value.

See Increasing the Sync Timer.

Shows the date and the time of last statistics reset.

In parentheses, it shows how the last statistics was triggered - "manually", or "by fullsync".