Print Download PDF Send Feedback

Previous

Next

Monitoring Synchronization (fw ctl pstat)

To monitor the synchronization mechanism on ClusterXL:

Run this command on a cluster member: fw ctl pstat

The output of this command is a long list of statistics for the Security Gateway. At the end of the list there is a section called Sync, which applies to each Cluster member. Many of the statistics are counters that can only increase. For information about Delta Sync, see How State Synchronization Works.

A typical output:

Sync:

Version: new

Status: Able to Send/Receive sync packets

Sync packets sent:

total : 466729198, retransmitted : 1305, retrans reqs : 89, acks : 809

Sync packets received:

total : 77283541, were queued : 6715, dropped by net : 6079

retrans reqs : 37462, received 175 acks

retrans reqs for illegal seq : 0

dropped updates as a result of sync overload: 0

Delta Sync memory usage: currently using XX KB mem

Callback statistics: handled 138 cb, average delay : 2, max delay : 34

Number of Pending packets currently held: 1

Packets released due to timeout: 18

Explanations:

Output section

Explanation

Sync: off

Delta Sync is disabled: either Full Sync failed, or Delta Sync was disabled by cluster administrator.

Sync:
Live connections update: on

Active Mode tab is opened in SmartView Tracker. See sk30908.

Sync:
Version: new

Default text when Delta Sync is operational. The new refers to Check Point versions NG and higher.

Status: Able to Send/Receive sync packets

Delta Sync works correctly.

Status:
Able to send sync packets
Unable to receive sync packets

The problem is described in the output itself (requires cluster debugging).

Status:
Unable to send sync packets
Unable to receive sync packets

The problem is described in the output itself (requires cluster debugging).

Status:
Able to send sync packets
Saving incoming sync packets

The problem is described in the output itself (requires cluster debugging).

Status:
Unable to send sync packets
Saving incoming sync packets

The problem is described in the output itself (requires cluster debugging).

Status:
Able to send sync packets
Able to receive sync packets

The problem is described in the output itself (requires cluster debugging).

Status:
Unable to send sync packets
Able to receive sync packets

The problem is described in the output itself (requires cluster debugging).

Sync packets sent:
total : X, retransmitted : X, retrans reqs : X, acks : X

The TOTAL counter shows the total number of Delta Sync packets that were sent by this cluster member to peer cluster members. This value increases all the time.

The RETRANSMITTED counter shows the number of retransmitted Delta Sync packets in response to retransmission requests. This value can increase during traffic load.

The RETRANS REQS counter shows the number of retransmission request the were sent by this cluster member for Delta Sync packets that were received out of order. This value can increase during traffic load.

The ACKS counter shows the number of acknowledgments received for the cb request Delta Sync packets, which are Delta Sync packets with requests for acknowledgments.

Sync packets received:
total : X, were queued : X, dropped by net : X
retrans reqs : X, received X acks
retrans reqs for illegal seq : X
dropped updates as a result of sync overload: X

The TOTAL counter shows the number of Delta Sync packets that were received by this cluster member from peer cluster members. This value increases all the time.

The WERE QUEUED shows the number of received Delta Sync packets that meet one of these conditions (a nonzero value does not indicate a problem):

  • The sync Delta Sync packet is received with a sequence number that does not follow the previously processed Delta Sync packet.
  • The Delta Sync packet is fragmented. This is done to solve MTU restrictions.

The DROPPED BY NET counter counter shows the number of received Delta Sync packets with a sequence number, which is higher, than the expected sequence number. Meaning, Delta Sync packets with lower sequence numbers where lost somewhere along the way, and we should find out where. If this value increases fast, it indicates a congestion of Sync network.

The RETRANS REQS counter shows the number of retransmission request the were received by this cluster member for Delta Sync packets that were sent out of order. This value can increase during traffic load. When this value increases very fast, it may indicate that the traffic load on the cluster member is becoming too high for Delta Sync to handle.

The RETRANS REQS FOR ILLEGAL SEQ counter shows the number of retransmission request the were received by this cluster member for Delta Sync packets that are no longer in this cluster member's possession. This value indicates a problem with Delta Sync.

The DROPPED UPDATES AS A RESULT OF SYNC OVERLOAD counter shows the number of Delta Sync updates that this cluster member dropped. This value can increase during high traffic load.

Delta Sync memory usage: currently using X KB mem

This counter only appears for a non-zero value. It requires memory only while Full Sync is occurring. At other times, Delta Sync requires no memory.

Callback statistics: handled X cb, average delay : X, max delay : X

This counter only appears for a non-zero value.

This counter relates to the received Delta Sync packets that involve Flush and Ack.

If the AVERAGE DELAY counter shows how much the Delta Sync packet was delayed in this cluster member until it was released, when the cluster member received an ACK from all peer cluster members. The delay occurs because Delta Sync packets are held until all peer cluster members have acknowledged reception of that Delta Sync packet. Values greater than 1-5 packets can indicate an overload of Delta Sync traffic.

Number of Pending packets currently held: X

This counter only appears for a non-zero value. ClusterXL prevents out-of-state packets in non-sticky connections. It does this by holding packets until a TCP [SYN-ACK] packet is received from all other active cluster members. If for some reason a TCP [SYN-ACK] packet is not received, the cluster member will not release the packet, and the TCP connection will not be established.

Packets released due to timeout: X

This counter only appears for a non-zero value. If the value is large (more than 100 pending packets), and the counter Number of Pending packets currently held in the output of the cphaprob syncstat command shows small number, then you should take action to reduce the number of pending packets. See Reducing the Number of Pending Packets.