Collecting System Diagnostics (smo verifiers)

Diagnostic Tests

Description

The smo verifiers commands in Gaia gClish The name of the global command line shell in Check Point Gaia operating system for Security Appliances connected to Check Point Quantum Maestro Orchestrators. Commands you run in this shell apply to all Security Appliances in the Security Group. run a specific set of diagnostic tests.

The full set of tests run by default. but you can manually select the tests you want to run.

The output shows the result of the test, Passed or Failed, and the location of the output log file.

Syntax

> show smo verifiers list

      [id <TestId1>,<TestId2>,...]

      [section <SectionName>]

> show smo verifiers report [except]

      [id <TestId1>,<TestId2>,...]

      [name <TestName>]

      [section <SectionName>]

> show smo verifiers print [except]

      [id <TestId1>,<TestId2>,...]

      [name <TestName>]

      [section <SectionName>]

> show smo verifiers

      periodic

      last-run report

      print

> delete smo verifiers purge [save <Num_Logs>]

Parameters

Parameter

Description

list

Shows the list of tests to run.

report

Runs tests and shows a summary of the test results.

print

Runs tests and shows the full output and summary of the test results.

except

Runs all tests except the specified tests.

Shows the requested results.

id <TestId1>,<TestId2>,...

Specifies the tests by their IDs (comma separated list).

To see a list of test IDs, run:

> show smo verifiers list

name <TestName>

Specifies the tests by their names.

Press the Tab key to see a full list of verifiers names.

section <SectionName>

Specifies the verifiers section by its name.

Press the Tab key to see a full list of the existing sections.

purge

Deletes the old smo verifiers logs.

Keeps the newest log.

save <Num_Logs>

Number of logs to save from the smo verifiers log files.

Default = 5.

periodic

Shows the latest periodic run results.

last-run

Shows the latest run results.

Showing the Tests

The show smo verifiers list command shows the full list of diagnostic tests.

The list shows the test ID, test Title (name), and the Command the smo verifiers command runs.

Example

> show smo verifiers list

--------------------------------------------------------------------------

| ID | Title              | Command                                      |

--------------------------------------------------------------------------

| System Components                                                      |

--------------------------------------------------------------------------

|  1 | System Health      | asg stat -v                                  |

|  2 | Resources          | asg resource                                 |

|  3 | Software Provision | asg_provision                                |

|  4 | Media Details      | transceiver_verifier -v                      |

|  5 | SSD Health         | asg resource --ssd                           |

--------------------------------------------------------------------------

| Policy and Configuration                                               |

--------------------------------------------------------------------------

|  6 | Distribution Mode  | distutil verify -v                           |

|  7 | DXL Balance        | dxl stat                                     |

|  8 | Policy             | asg policy verify -a                         |

|  9 | AMW Policy         | asg policy verify_amw -a                     |

| 10 | SWB Updates        | asg_swb_update_verifier -v                   |

| 11 | Installation       | installation_verify                          |

| 12 | Security Group     | security_group_util diag                     |

| 13 | Cores Distribution | cores_verifier                               |

| 14 | Clock              | clock_verifier -v                            |

| 15 | Licenses           | asg_license_verifier -v                      |

| 16 | IPS Enhancement    | asg_ips_enhance status                       |

| 17 | Configuration File | config_verify -v                             |

--------------------------------------------------------------------------

| VSX Configuration                                                      |

--------------------------------------------------------------------------

| 18 | USER KERNEL Dist   | distutil verify_vsx_dist                     |

| 19 | HW Utilization     | hw_utilization -d                            |

| 20 | BMAC VMAC verify   | mac_verifier -x                              |

--------------------------------------------------------------------------

| Networking                                                             |

--------------------------------------------------------------------------

| 21 | MAC Setting        | mac_verifier -v                              |

| 22 | ARP Consistency    | asg_arp -v                                   |

| 23 | Bond               | asg_bond -v                                  |

| 24 | IPv4 Route         | asg_route                                    |

| 25 | IPv6 Route         | asg_route -6                                 |

| 26 | Dynamic Routing    | asg_dr_verifier                              |

| 27 | Local ARP          | asg_local_arp_verifier -v                    |

| 28 | IGMP Consistency   | asg_igmp                                     |

| 29 | PIM Neighbors      | asg_pim_neighbors                            |

--------------------------------------------------------------------------

| Misc                                                                   |

--------------------------------------------------------------------------

| 30 | Core Dumps         | core_dump_verifier -v                        |

| 31 | Performance hogs   | asg_perf_hogs                                |

--------------------------------------------------------------------------

| Run "show smo verifiers print id <TestNum>" to display test output     |

--------------------------------------------------------------------------

Showing the Last Run Diagnostic Tests

The show smo verifiers last-run report command shows the default output for the last run diagnostic tests.

The show smo verifiers last-run print command shows verbose output for the last run diagnostic tests.

Example

> show smo verifiers last-run report

2019-02-07, 01:00:02

--------------------------------------------------------------------------------

| Tests Status                                                                 |

--------------------------------------------------------------------------------

| ID | Title              | Result     | Reason                                |

--------------------------------------------------------------------------------

| System Components                                                            |

--------------------------------------------------------------------------------

|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |

|  2 | Resources          | Passed     |                                       |

|  3 | Software Provision | Passed     |                                       |

|  4 | Media Details      | Passed     |                                       |

|  5 | SSD Health         | Passed     | (1)Failed to get SSD overall-health t |

|    |                    |            | est result from Member                |

--------------------------------------------------------------------------------

| Policy and Configuration                                                     |

--------------------------------------------------------------------------------

|  6 | Distribution Mode  | Failed (!) | (1)Verifier error - Check raw output  |

|  7 | DXL Balance        | Passed     |                                       |

|  8 | Policy             | Passed     |                                       |

|  9 | AMW Policy         | Passed     | (1)Not configured                     |

| 10 | SWB Updates        | Passed     | (1)Not configured                     |

| 11 | Installation       | Passed     |                                       |

| 12 | Security Group     | Passed     |                                       |

| 13 | Cores Distribution | Passed     |                                       |

| 14 | Clock              | Passed     |                                       |

| 15 | Licenses           | Failed (!) | (1)Trial license will expire within 2 |

|    |                    |            |  weeks                                |

|    |                    |            | (2)Execution error                    |

| 16 | IPS Enhancement    | Passed     |                                       |

| 17 | Configuration File | Passed     |                                       |

--------------------------------------------------------------------------------

| VSX Configuration                                                            |

--------------------------------------------------------------------------------

| 18 | USER KERNEL Dist   | Failed (!) |                                       |

| 19 | HW Utilization     | Failed (!) | (1)Execution error                    |

| 20 | BMAC VMAC verify   | Passed     |                                       |

--------------------------------------------------------------------------------

| Networking                                                                   |

--------------------------------------------------------------------------------

| 21 | MAC Setting        | Passed     |                                       |

... output was truncated for brevity ...

--------------------------------------------------------------------------------

| Misc                                                                         |

--------------------------------------------------------------------------------

| 30 | Core Dumps         | Failed (!) |                                       |

| 31 | Performance hogs   | Failed (!) |                                       |

--------------------------------------------------------------------------------

| Tests Summary                                                                |

--------------------------------------------------------------------------------

| Passed: 24/31 tests                                                          |

| Run: "show smo verifiers list id 1,6,15,18,19,30,31" to view a complete list |

|  of failed tests                                                             |

| Output file: /var/log/alert_verifier_sum.1-31.2019-02-07_01-00-02.txt        |

--------------------------------------------------------------------------------

Running all Diagnostic Tests

The show smo verifiers report command runs all diagnostic tests and shows their summary output.

When a test fails, the reasons for failure show in the Reason column.

Example

> show smo verifiers report

Duration of tests vary and may take a few minutes to complete

--------------------------------------------------------------------------------

| Tests Status                                                                 |

--------------------------------------------------------------------------------

| ID | Title              | Result     | Reason                                |

--------------------------------------------------------------------------------

| System Components                                                            |

--------------------------------------------------------------------------------

|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |

|  2 | Resources          | Passed     |                                       |

|  3 | Software Provision | Passed     |                                       |

|  4 | Media Details      | Passed     |                                       |

|  5 | SSD Health         | Passed     | (1)Failed to get SSD overall-health t |

|    |                    |            | est result from Member                |

--------------------------------------------------------------------------------

| Policy and Configuration                                                     |

--------------------------------------------------------------------------------

|  6 | Distribution Mode  | Failed (!) | (1)Verifier error - Check raw output  |

|  7 | DXL Balance        | Passed     |                                       |

|  8 | Policy             | Passed     |                                       |

|  9 | AMW Policy         | Passed     | (1)Not configured                     |

| 10 | SWB Updates        | Passed     | (1)Not configured                     |

| 11 | Installation       | Passed     |                                       |

| 12 | Security Group     | Passed     |                                       |

| 13 | Cores Distribution | Passed     |                                       |

| 14 | Clock              | Passed     |                                       |

| 15 | Licenses           | Failed (!) | (1)Trial license will expire within 2 |

|    |                    |            |  weeks                                |

|    |                    |            | (2)Execution error                    |

| 16 | IPS Enhancement    | Passed     |                                       |

| 17 | Configuration File | Passed     |                                       |

--------------------------------------------------------------------------------

| VSX Configuration                                                            |

--------------------------------------------------------------------------------

| 18 | USER KERNEL Dist   | Failed (!) |                                       |

| 19 | HW Utilization     | Failed (!) | (1)Execution error                    |

| 20 | BMAC VMAC verify   | Passed     |                                       |

--------------------------------------------------------------------------------

| Networking                                                                   |

--------------------------------------------------------------------------------

| 21 | MAC Setting        | Passed     |                                       |

... output was truncated for brevity ...

--------------------------------------------------------------------------------

| Misc                                                                         |

--------------------------------------------------------------------------------

| 30 | Core Dumps         | Failed (!) |                                       |

| 31 | Performance hogs   | Failed (!) |                                       |

--------------------------------------------------------------------------------

| Tests Summary                                                                |

--------------------------------------------------------------------------------

| Passed: 24/31 tests                                                          |

| Run: "show smo verifiers list id 1,6,15,18,19,30,31" to view a complete list |

|  of failed tests                                                             |

| Output file: /var/log/verifier_sum.1-31.2019-02-07_18-35-22.txt              |

| Run "show smo verifiers last-run print" to display verbose output            |

--------------------------------------------------------------------------------

Running Specific Diagnostic Tests

The show smo verifiers report name and the show smo verifiers report id commands run the specified diagnostic tests only.

Syntax to run a test by its name

> show smo verifiers report name <Test Name>

Note - Press the Tab key after the name parameter to see a full list of verifier names.

Syntax to run a test by its ID

> show smo verifiers report id <TestID1>,<TestID2>,...,<TestIDn>

Note - To see a list of test IDs, run the show smo verifiers list command.

Collecting Diagnostic Information for a Report Specified Section

The show smo verifiers report section command runs all diagnostic tests in the specified section.

Syntax

> show smo verifiers report section <Test Name>

Note - Press the Tab key after the section parameter to see a full list of verifier sections.

Error Types

The smo verifiers detects these errors:

Error Type	Error	Description
`System health`	`Chassis <X> error`	The Security Group A logical group of Security Appliances that provides Active/Active cluster functionality. A Security Group can contain one or more Security Appliances. Security Groups work separately and independently from each other. To the production networks, a Security Group appears a single Security Gateway. Every Security Group contains: (A) Applicable Uplink ports, to which your production networks are connected; (B) Security Appliances (the Quantum Maestro Orchestrator determines the applicable Downlink ports automatically); (C) Applicable management port, to which the Check Point Management Server is connected. quality grade is less than the defined threshold. We recommend that you correct this issue immediately.
`Hardware`	`<Component> is missing`	The component is not installed in the Chassis. Note - This applies only to 60000 / 40000 Appliances.
	`<Component> is down`	The component is installed in the Chassis, but is inactive. Note - This applies only to 60000 / 40000 Appliances.
`Resources`	`<Resource> capacity`	The specified resource capacity is not sufficient. You can change the defined resource capacity.
	`<Resource> exceed threshold`	The resource usage is greater than the defined threshold.
`CPU type`	`Non compliant CPU type`	At least one Security Group Member CPU type is not configured in the list of compliant CPUs. You can define the compliant CPU types.
`Security group`	`<Source> error`	The information collected from this source is different between the Security Group Members.
	`<Sources> differ`	The information collected from many sources is different.

Changing Compliance Thresholds

You can change some compliance thresholds that define a healthy, working system.

In the $SMODIR/conf/asg_diag_config file, change the threshold values.

These are the supported resources you can control:

Resource	Description
`Memory`	RAM memory capacity in GB
`HD: /`	Disk capacity in GB for `<disk>` - the root (`/`) partition
`HD:/var/log`	Disk capacity in GB for the `/var/log` partition
`HD: /boot`	Disk capacity in GB for the `/boot` partition

Changing the Default Test Behavior of the 'asg diag resource verifier'

By default, the asg diag resource verifier command only shows a warning about resource mismatches between Security Group Members.

The verification test results show as "Passed" in the output and no further action is taken.

You can change the default test behavior:

Step Instructions

Edit the $FWDIR/conf/asg_diag_config file:

# g_all vi $FWDIR/conf/asg_diag_config

Search for the parameter MismatchSeverity.

Set the value of this parameter to one of these values:

fail - Verification test result is set to "Failed"
warn - Verification test result is set to "Passed", and a warning is shown
ignore - Verification test result is set to "Ignore", and no errors are shown

Save the changes in the file and exit the Vi editor.

Troubleshooting Failures

Use the smo verifiers command to troubleshoot a failed diagnostic test.

Below is the example procedure based on the System Health test that failed.

Example

The System Health test failed:

> show smo verifiers report id 1

Duration of tests vary and may take a few minutes to complete

--------------------------------------------------------------------------------

| Tests Status                                                                 |

--------------------------------------------------------------------------------

| ID | Title              | Result     | Reason                                |

--------------------------------------------------------------------------------

| System Components                                                            |

--------------------------------------------------------------------------------

|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |

--------------------------------------------------------------------------------

| Tests Summary                                                                |

--------------------------------------------------------------------------------

| Passed: 0/1 test                                                             |

| Run: "show smo verifiers list id 1" to view a complete list of failed tests  |

| Output file: /var/log/verifier_sum.1.2019-02-07_20-12-07.txt                 |

| Run "show smo verifiers last-run print" to display verbose output            |

--------------------------------------------------------------------------------

Print the full report for this failed test:

> show smo verifiers print id 1

==============================

System Health:

==============================

--------------------------------------------------------------------------------

| VSX System Status - Maestro                                                  |

--------------------------------------------------------------------------------

| Up time                     | 06:44:48 hours                                 |

| SGMs                        | 3 / 3                                          |

| Virtual Systems             | 1                                              |

| Version                     | R80.30SP (Build Number xxx)                    |

--------------------------------------------------------------------------------

| VS ID:  0                     VS Name:  MyVSname                             |

--------------------------------------------------------------------------------

| SGM ID                              Chassis 1                                |

|                                     ACTIVE                                   |

--------------------------------------------------------------------------------

|  2                                    ACTIVE                                 |

|  3                                    ACTIVE                                 |

|  4                                    ACTIVE                                 |

--------------------------------------------------------------------------------

| Chassis Parameters                                                           |

--------------------------------------------------------------------------------

| Unit         |                          Chassis 1                            |

--------------------------------------------------------------------------------

| SGMs         |                            3 / 3                              |

| Ports        |                            0 / 0                              |

| SSMs         |                            1 / 2   !                          |

--------------------------------------------------------------------------------

| Synchronization                                                              |

|     Sync to Active chassis:    Enabled                                       |

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

| Tests Status                                                                 |

--------------------------------------------------------------------------------

| ID | Title              | Result     | Reason                                |

--------------------------------------------------------------------------------

| System Components                                                            |

--------------------------------------------------------------------------------

|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |

--------------------------------------------------------------------------------

| Tests Summary                                                                |

--------------------------------------------------------------------------------

| Passed: 0/1 test                                                             |

| Run: "show smo verifiers list id 1" to view a complete list of failed tests  |

| Output file: /var/log/verifier_sum.1.2019-02-07_20-12-14.txt                 |

--------------------------------------------------------------------------------

Examine which command produced the failed test:

> show smo verifiers list id 1

--------------------------------------------------------------------------

| ID | Title              | Command                                      |

--------------------------------------------------------------------------

| System Components                                                      |

--------------------------------------------------------------------------

|  1 | System Health      | asg stat -v                                  |

--------------------------------------------------------------------------

| Run "show smo verifiers print id <TestNum>" to display test output     |

--------------------------------------------------------------------------

Run the applicable command to understand what fails:

> asg stat -v

--------------------------------------------------------------------------------

| VSX System Status - Maestro                                                  |

--------------------------------------------------------------------------------

| Up time                     | 06:45:06 hours                                 |

| SGMs                        | 3 / 3                                          |

| Virtual Systems             | 1                                              |

| Version                     | R80.30SP (Build Number xxx)                    |

--------------------------------------------------------------------------------

| VS ID:  0                     VS Name:  MyVSname                             |

--------------------------------------------------------------------------------

| SGM ID                              Chassis 1                                |

|                                     ACTIVE                                   |

--------------------------------------------------------------------------------

|  2                                    ACTIVE                                 |

|  3                                    ACTIVE                                 |

|  4                                    ACTIVE                                 |

--------------------------------------------------------------------------------

| Chassis Parameters                                                           |

--------------------------------------------------------------------------------

| Unit        |                      Chassis 1                        | Weight |

--------------------------------------------------------------------------------

| SGMs        |                        3 / 3                          |   6    |

| Ports       |                                                       |        |

|   Standard  |                        0 / 0                          |  11    |

|   Bond      |                        0 / 0                          |  11    |

|   Other     |                        0 / 0                          |   6    |

| Sensors     |                                                       |        |

|   SSMs      |                        1 / 2   !                      |  11    |

|             |                                                       |        |

| Grade       |                       29 / 40  !                      |   -    |

--------------------------------------------------------------------------------

| Synchronization                                                              |

|     Sync to Active chassis:    Enabled                                       |

--------------------------------------------------------------------------------