Viewing Critical Devices

Description

There are a number of built-in Critical Devices, and the Administrator can define additional Critical Devices.

When a Critical Device reports its state as a "problem", the Cluster Member reports its state as "DOWN".

To see the list of Critical Devices on a Cluster Member, and of all the other Cluster Members, run the commands listed below on the Cluster Member.

Table: Built-in Critical Devices

Critical Device

Description

Meaning of the "OK" state

Meaning of the "problem" state

Problem Notification

Monitors all the Critical Devices.

None of the Critical Devices on this Cluster Member report its state as problem.

At least one of the Critical Devices on this Cluster Member reports its state as problem.

Init

Monitors if "HA module" was initialized successfully. See sk36372.

This Cluster Member receives cluster state information from peer Cluster Members.

 

Interface Active Check

Monitors the state of cluster interfaces.

All cluster interfaces on this Cluster Member are up (CCP packets are sent and received on all cluster interfaces).

At least one of the cluster interfaces on this Cluster Member is down (CCP packets are not sent and/or received on time).

Load Balancing Configuration

Pnote is currently not used (see sk36373).

 

 

Recovery Delay

Monitors the state of a Virtual System (see sk92353).

State of a Virtual System can be changed on this Cluster Member.

State of a Virtual System cannot be changed yet on this Cluster Member.

CoreXL Configuration

Monitors CoreXL configuration for inconsistencies on all Cluster Members.

Number of configured CoreXL Firewall instances on this Cluster Member is the same as on all peer Cluster Members.

Number of configured CoreXL Firewall instances on this Cluster Member is different from peer Cluster Members.

Important - A Cluster Member with a greater number of CoreXL Firewall instances changes its state to DOWN.

Fullsync

Monitors if Full Sync on this Cluster Member completed successfully.

This Cluster Member completed Full Sync successfully.

This Cluster Member was not able to complete Full Sync.

Policy

Monitors if the Security Policy is installed.

This Cluster Member successfully installed Security Policy.

Security Policy is not currently installed on this Cluster Member.

fwd

Monitors the Security Gateway process called fwd.

fwd daemon on this Cluster Member reported its state on time.

fwd daemon on this Cluster Member did not report its state on time.

cphad

Monitors the ClusterXL process called cphamcset.
also see the $FWDIR/log/cphamcset.elg file.

cphamcset daemon on this Cluster Member reported its state on time.

cphamcset daemon on this Cluster Member did not report its state on time.

routed

Monitors the Gaia process called routed.

routed daemon on this Cluster Member reported its state on time.

routed daemon on this Cluster Member did not report its state on time.

cvpnd

Monitors the Mobile Access back-end process called cvpnd.
This pnote appears if Mobile Access Software Blade is enabled.

cvpnd daemon on this Cluster Member reported its state on time.

cvpnd daemon on this Cluster Member did not report its state on time.

ted

Monitors the Threat Emulation process called ted.

ted daemon on this Cluster Member reported its state on time.

ted daemon on this Cluster Member did not report its state on time.

VSX

Monitors all Virtual Systems in VSX Cluster.

On VS0, means that states of all Virtual Systems are not Down on this Cluster Member.

On other Virtual Systems, means that VS0 is alive on this Cluster Member.

Minimum of blocking states of all Virtual Systems is not "active" (the VSIDs will be printed on the line Problematic VSIDs:) on this Cluster Member.

Instances

This pnote appears in VSX HA mode (not VSLS) cluster.

The number of CoreXL Firewall instances in the received CCP packet matches the number of loaded CoreXL Firewall instances on this VSX Cluster Member or this Virtual System.

There is a mismatch between the number of CoreXL Firewall instances in the received CCP packet and the number of loaded CoreXL Firewall instances on this VSX Cluster Member or this Virtual System (see sk106912).

Hibernating

This pnote appears in VSX VSLS mode cluster with 3 and more Cluster Members. This pnote shows if this Virtual System is in "Backup" (hibernated) state. Also see sk114557.

This Virtual System is in "Backup" (hibernated) state on this Cluster Member.

 

admin_down

Monitors the Critical Device admin_down.

 

User ran the clusterXL_admin down command on this Cluster Member.
See The clusterXL_admin Script.

host_monitor

Monitors the Critical Device host_monitor.

User executed the $FWDIR/bin/clusterXL_monitor_ips script.
See The clusterXL_monitor_ips Script.

All monitored IP addresses on this Cluster Member replied to pings.

At least one of the monitored IP addresses on this Cluster Member did not reply to at least one ping.

A name of a user space process (except fwd, routed, cvpnd, ted)

User executed the $FWDIR/bin/clusterXL_monitor_process script.
See The clusterXL_monitor_process Script.

All monitored user space processes on this Cluster Member are running.

At least one of the monitored user space on this Cluster Member processes is not running.

Local Probing

Monitors the probing mechanism on the cluster interfaces (see the term Probing in the Glossary).

CCP packets are received on all cluster interfaces.

At least one of the cluster interfaces on this Cluster Member does not receive CCP packets for 5 seconds.

The probing started for the network connected to the affected interface.

Important:

  • The state of this Critical Device does not affect the cluster state of a Cluster Member. This Critical Device is only an indicator for the probing mechanism (instead of running a cluster debug).

  • If there is a real issue with a cluster interface, the Critical Device "Interface Active Check" reports its state as "problem".

Syntax

Shell

Command

Gaia Clish

show cluster members pnotes {all | problem}

Expert mode

cphaprob [-l] [-ia] [-e] list

Where:

Command

Description

show cluster members pnotes all

Shows the list of all Critical Devices

show cluster members pnotes problem

Shows the list of all the "Built-in Devices" and the "Registered Devices" that report their state as "problem"

cphaprob -l

Shows the list of all Critical Devices

cphaprob -i list

When there are no issues on the Cluster Member, shows:
There are no pnotes in problem state

When a Critical Device reports a problem, prints only the Critical Device that reports its state as "problem".

cphaprob -ia list

When there are no issues on the Cluster Member, shows:
There are no pnotes in problem state

When a Critical Device reports a problem, prints the Critical Device "Problem Notification" and the Critical Device that reports its state as "problem"

cphaprob -e list

When there are no issues on the Cluster Member, shows:
There are no pnotes in problem state

When a Critical Device reports a problem, prints only the Critical Device that reports its state as "problem"

Related topics

Examples

Critical Device fwd reports its state as problem because the fwd process is down.

[Expert@Member1:0]# cphaprob -l list
 
Built-in Devices:
 
Device Name: Interface Active Check
Current state: OK
 
Device Name: Recovery Delay
Current state: OK
 
Device Name: CoreXL Configuration
Current state: OK
 
Registered Devices:
 
Device Name: Fullsync
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 1753.7 sec
 
Device Name: Policy
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 1753.7 sec
 
Device Name: routed
Registration number: 2
Timeout: none
Current state: OK
Time since last report: 940.3 sec
 
Device Name: fwd
Registration number: 3
Timeout: 30 sec
Current state: problem
Time since last report: 1782.9 sec
Process Status: DOWN
 
Device Name: cphad
Registration number: 4
Timeout: 30 sec
Current state: OK
Time since last report: 1778.3 sec
Process Status: UP
 
Device Name: VSX
Registration number: 5
Timeout: none
Current state: OK
Time since last report: 1773.3 sec
 
Device Name: Init
Registration number: 6
Timeout: none
Current state: OK
Time since last report: 1773.3 sec
 
[Expert@Member1:0]#

Critical Device CoreXL Configuration reports its state as problem because the numbers of CoreXL Firewall instances do not match between the Cluster Members.

[Expert@Member1:0]# cphaprob -l list
 
Built-in Devices:
 
Device Name: Interface Active Check
Current state: OK
 
Device Name: Recovery Delay
Current state: OK
 
Device Name: CoreXL Configuration
Current state: problem (non-blocking)
 
Registered Devices:
 
Device Name: Fullsync
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 1753.7 sec
 
Device Name: Policy
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 1753.7 sec
 
Device Name: routed
Registration number: 2
Timeout: none
Current state: OK
Time since last report: 940.3 sec
 
Device Name: fwd
Registration number: 3
Timeout: 30 sec
Current state: OK
Time since last report: 1782.9 sec
Process Status: UP
 
Device Name: cphad
Registration number: 4
Timeout: 30 sec
Current state: OK
Time since last report: 1778.3 sec
Process Status: UP
 
Device Name: VSX
Registration number: 5
Timeout: none
Current state: OK
Time since last report: 1773.3 sec
 
Device Name: Init
Registration number: 6
Timeout: none
Current state: OK
Time since last report: 1773.3 sec
 
[Expert@Member1:0]#