Collecting System Diagnostics (smo verifiers)

Diagnostic Tests

Description

Use the "smo verifiers" commands in Gaia gClish to run a specific set of diagnostic tests.

The full set of tests run by default, but you can manually select the tests to run.

The output shows the result of the test, Passed or Failed, and the location of the output log file.

Syntax

show smo verifiers list

      [id <TestId1>,<TestId2>,...]

      [section <SectionName>]

show smo verifiers report [except]

      [id <TestId1>,<TestId2>,...]

      [name <TestName>]

      [section <SectionName>]

show smo verifiers print [except]

      [id <TestId1>,<TestId2>,...]

      [name <TestName>]

      [section <SectionName>]

show smo verifiers

      periodic

      last-run report

      print

delete smo verifiers purge [save <Num_Logs>]

Parameters

Parameter

Description

list

Shows the list of tests to run.

report

Runs tests and shows a summary of the test results.

print

Runs tests and shows the full output and summary of the test results.

except

Runs all tests except the specified tests.

Shows the requested results.

id <TestId1>,<TestId2>,...

Specifies the tests by their IDs (comma separated list).

To see a list of test IDs, run:

show smo verifiers list

name <TestName>

Specifies the tests by their names.

Press the Tab key to see a full list of verifiers names.

section <SectionName>

Specifies the verifiers section by its name.

Press the Tab key to see a full list of the existing sections.

purge

Deletes the old "smo verifiers" logs.

Keeps the newest log.

save <Num_Logs>

Number of logs to save from the "smo verifiers" log files.

Default: 5.

periodic

Shows the latest periodic run results.

last-run

Shows the latest run results.

Showing the Tests

The "show smo verifiers list" command shows the full list of diagnostic tests.

The list shows the test "ID", test "Title" (name), and the "Command" the "smo verifiers" command runs.

[Expert@HostName-ch0x-0x:0]# gclish
[Global] HostName-ch01-01 > show smo verifiers list
--------------------------------------------------------------------------
| ID | Title              | Command                                      |
--------------------------------------------------------------------------
| System Components                                                      |
--------------------------------------------------------------------------
|  1 | System Health      | asg stat -v                                  |
|  2 | Firmware Verifier  | firmware_verifier -v                         |
--------------------------------------------------------------------------
| Policy and Configuration                                               |
--------------------------------------------------------------------------
|  3 | Policy             | asg policy verify -a                         |
|  4 | AMW Policy         | asg policy verify_amw -a                     |
|  5 | SWB Updates        | asg_swb_update_verifier -v                   |
|  6 | Security Group     | security_group_util diag                     |
|  7 | Cores Distribution | cores_verifier                               |
|  8 | Clock              | clock_verifier -v                            |
|  9 | Licenses           | asg_license_verifier -v                      |
--------------------------------------------------------------------------
| VSX Configuration                                                      |
--------------------------------------------------------------------------
| 10 | BMAC VMAC verify   | mac_verifier -x                              |
--------------------------------------------------------------------------
| Networking                                                             |
--------------------------------------------------------------------------
| 11 | MAC Setting        | mac_verifier -v                              |
| 12 | ARP Consistency    | asg_arp -v                                   |
| 13 | Bond               | asg_bond -v                                  |
| 14 | IPv4 Route         | asg_route                                    |
| 15 | IPv6 Route         | asg_route -6                                 |
| 16 | Dynamic Routing    | asg_dr_verifier                              |
| 17 | Local ARP          | asg_local_arp_verifier -v                    |
| 18 | IGMP Consistency   | asg_igmp                                     |
| 19 | PIM Neighbors      | asg_pim_neighbors                            |
--------------------------------------------------------------------------
| Run "show smo verifiers print id <TestNum>" to display test output     |
--------------------------------------------------------------------------
[Global] HostName-ch01-01 >

Showing the Last Run Diagnostic Tests

The "show smo verifiers last-run report" command shows the default output for the last run diagnostic tests.

The "show smo verifiers last-run print" command shows verbose output for the last run diagnostic tests.

[Expert@HostName-ch0x-0x:0]# gclish
[Global] HostName-ch01-01 > show smo verifiers last-run report
2023-04-20, 01:00:05
--------------------------------------------------------------------------------
| Tests Status                                                                 |
--------------------------------------------------------------------------------
| ID | Title              | Result     | Reason                                |
--------------------------------------------------------------------------------
| System Components                                                            |
--------------------------------------------------------------------------------
|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |
|  2 | Firmware Verifier  | Passed     |                                       |
--------------------------------------------------------------------------------
| Policy and Configuration                                                     |
--------------------------------------------------------------------------------
|  3 | Policy             | Passed     |                                       |
|  4 | AMW Policy         | Passed     | (1)Not configured                     |
|  5 | SWB Updates        | Passed     | (1)Not configured                     |
|  6 | Security Group     | Passed     |                                       |
|  7 | Cores Distribution | Passed     |                                       |
|  8 | Clock              | Passed     |                                       |
|  9 | Licenses           | Failed (!) | (1)Trial license will expire within 2 |
|    |                    |            |  weeks                                |
|    |                    |            | (2)Execution error                    |
--------------------------------------------------------------------------------
| Networking                                                                   |
--------------------------------------------------------------------------------
| 10 | MAC Setting        | Passed     |                                       |
| 11 | ARP Consistency    | Passed     |                                       |
| 12 | Bond               | Passed     | (1)Not configured                     |
| 13 | IPv4 Route         | Passed     |                                       |
| 14 | IPv6 Route         | Passed     | (1)Not configured                     |
| 15 | Dynamic Routing    | Passed     | (1)Not configured                     |
| 16 | Local ARP          | Passed     | (1)Not configured                     |
| 17 | IGMP Consistency   | Passed     | (1)Not configured                     |
| 18 | PIM Neighbors      | Passed     | (1)Not configured                     |
--------------------------------------------------------------------------------
| Tests Summary                                                                |
--------------------------------------------------------------------------------
| Passed: 16/18 tests                                                          |
| Run: "show smo verifiers list id 1,9" to view a complete list                |
|  of failed tests                                                             |
| Output file: /var/log/alert_verifier_sum.1-18.2023-04-20_01-00-05.txt        |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >

Running all Diagnostic Tests

The "show smo verifiers report" command runs all diagnostic tests and shows their summary output.

When a test fails, the reasons for failure show in the Reason column.

[Global] HostName-ch01-01 > show smo verifiers report
Duration of tests vary and may take a few minutes to complete

--------------------------------------------------------------------------------
| Tests Status                                                                 |
--------------------------------------------------------------------------------
| ID | Title              | Result     | Reason                                |
--------------------------------------------------------------------------------
| System Components                                                            |
--------------------------------------------------------------------------------
|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |
|  2 | Firmware Verifier  | Passed     |                                       |
--------------------------------------------------------------------------------
| Policy and Configuration                                                     |
--------------------------------------------------------------------------------
|  3 | Policy             | Passed     |                                       |
|  4 | AMW Policy         | Passed     | (1)Not configured                     |
|  5 | SWB Updates        | Passed     | (1)Not configured                     |
|  6 | Security Group     | Passed     |                                       |
|  7 | Cores Distribution | Passed     |                                       |
|  8 | Clock              | Passed     |                                       |
|  9 | Licenses           | Failed (!) | (1)Trial license will expire within 2 |
|    |                    |            |  weeks                                |
--------------------------------------------------------------------------------
| Networking                                                                   |
--------------------------------------------------------------------------------
| 10 | MAC Setting        | Passed     |                                       |
| 11 | ARP Consistency    | Passed     |                                       |
| 12 | Bond               | Passed     | (1)Not configured                     |
| 13 | IPv4 Route         | Passed     |                                       |
| 14 | IPv6 Route         | Passed     | (1)Not configured                     |
| 15 | Dynamic Routing    | Passed     | (1)Not configured                     |
| 16 | Local ARP          | Passed     | (1)Not configured                     |
| 17 | IGMP Consistency   | Passed     | (1)Not configured                     |
| 18 | PIM Neighbors      | Passed     | (1)Not configured                     |
--------------------------------------------------------------------------------
| Tests Summary                                                                |
--------------------------------------------------------------------------------
| Passed: 16/18 tests                                                          |
| Run: "show smo verifiers list id 1,9" to view a complete list of failed tes |
| ts                                                                           |
| Output file: /var/log/verifier_sum.1-18.2023-04-20_12-24-37.txt              |
| Run "show smo verifiers last-run print" to display verbose output            |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >

Running Specific Diagnostic Tests

These commands run the specified diagnostic tests only:

show smo verifiers report name

show smo verifiers report id

Syntax to run a test by its name

show smo verifiers report name <Test Name>

Note - Press the Tab key after the "name" parameter to see a full list of verifier names.

[Expert@HostName-ch0x-0x:0]# gclish
[Global] HostName-ch01-01 > show smo verifiers report name System_Health
Duration of tests vary and may take a few minutes to complete
 
--------------------------------------------------------------------------------
| Tests Status                                                                 |
--------------------------------------------------------------------------------
| ID | Title              | Result     | Reason                                |
--------------------------------------------------------------------------------
| System Components                                                            |
--------------------------------------------------------------------------------
|  1 | System Health      | Passed     |                                       |
--------------------------------------------------------------------------------
| Tests Summary                                                                |
--------------------------------------------------------------------------------
| Passed: 1/1 test                                                             |
| Output file: /var/log/verifier_sum.1-18.2023-04-20_12-24-37.txt              |
| Run "show smo verifiers last-run print" to display verbose output            |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >

Syntax to run a test by its ID

show smo verifiers report id <TestID1>,<TestID2>,...,<TestIDn>

Note - To see a list of test IDs, run the "show smo verifiers list" command.

This example collects diagnostic information for specified tests 1 and 2.

[Expert@HostName-ch0x-0x:0]# gclish
[Global] HostName-ch01-01 > show smo verifiers report id 1,2
Duration of tests vary and may take a few minutes to complete
 
--------------------------------------------------------------------------------
| Tests Status                                                                 |
--------------------------------------------------------------------------------
| ID | Title              | Result     | Reason                                |
--------------------------------------------------------------------------------
| System Components                                                            |
--------------------------------------------------------------------------------
|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |
|  2 | Firmware Verifier  | Passed     |                                       |
--------------------------------------------------------------------------------
| Tests Summary                                                                |
--------------------------------------------------------------------------------
| Passed: 1/2 tests                                                            |
| Run: "show smo verifiers list id 1" to view a complete list of failed tests  |
| Output file: /var/log/verifier_sum.1-18.2023-04-20_12-24-37.txt              |
| Run "show smo verifiers last-run print" to display verbose output            |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >

Collecting Diagnostic Information for a Report Specified Section

The "show smo verifiers report section" command runs all diagnostic tests in the specified section.

Syntax

show smo verifiers report section <Test Name>

Note - Press the Tab key after the "section" parameter to see a full list of verifier sections.

[Expert@HostName-ch0x-0x:0]# gclish
[Global] HostName-ch01-01 > show smo verifiers report section System_Components
Duration of tests vary and may take a few minutes to complete
 
--------------------------------------------------------------------------------
| Tests Status                                                                 |
--------------------------------------------------------------------------------
| ID | Title              | Result     | Reason                                |
--------------------------------------------------------------------------------
| System Components                                                            |
--------------------------------------------------------------------------------
|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |
|  2 | Firmware Verifier  | Passed     |                                       |
--------------------------------------------------------------------------------
| Tests Summary                                                                |
--------------------------------------------------------------------------------
| Passed: 1/2 tests                                                            |
| Run: "show smo verifiers list id 1" to view a complete list of failed tests  |
| Output file: /var/log/verifier_sum.1-18.2023-04-20_12-24-37.txt              |
| Run "show smo verifiers last-run print" to display verbose output            |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >

Error Types

The "smo verifiers" command detects these errors:

Error Type

Error

Description

System health

Chassis <X> error

The Security Group quality grade is less than the defined threshold.

We recommend that you correct this issue immediately.

Hardware

<Component> is missing

The component is not installed in the Chassis.

Note - This applies only to 60000 / 40000 Appliances.

 

<Component> is down

The component is installed in the Chassis, but is inactive.

Note - This applies only to 60000 / 40000 Appliances.

Resources

<Resource> capacity

The specified resource capacity is not sufficient.

You can change the defined resource capacity.

 

<Resource> exceed threshold

The resource usage is greater than the defined threshold.

CPU type

Non compliant CPU type

CPU type is not configured in the list of compliant CPUs on at least one Security Group Member.

You can define the compliant CPU types.

Security group

<Source> error

The information collected from this source is different between the Security Group Members.

 

<Sources> differ

The information collected from many sources is different.

Changing Compliance Thresholds

You can change some compliance thresholds that define a healthy, working system.

Change the threshold values in the $SMODIR/conf/asg_diag_config file.

These are the supported resources you can control:

Resource

Instructions

Memory

RAM memory capacity in GB.

HD: /

Disk capacity in GB for <disk> - the root (/) partition.

HD:/var/log

Disk capacity in GB for the /var/log partition.

HD: /boot

Disk capacity in GB for the /boot partition.

Skew

The maximum permissible clock difference, in seconds, between the SGMs and CMMs.

Note - This resource applies only to 60000 / 40000 Appliances.

Certified cpu

Each line represents one compliant CPU type.

Note - This resource applies only to 60000 / 40000 Appliances

Changing the Default Test Behavior of the 'asg diag resource verifier'

By default, the "asg diag resource verifier" command only shows a warning about resource mismatches between Security Group Members.

The verification test results show as "Passed" in the output and no further action is taken.

You can change the default test behavior:

Step

Instructions

1

Connect to the command line on the Security Group.

2

Log in to the Expert mode.

3

Edit the $FWDIR/conf/asg_diag_config file:

g_all vi $FWDIR/conf/asg_diag_config

4

Search for this parameter:

MismatchSeverity

5

Set the value of this parameter to one of these values:

  • fail

    Verification test result is set to "Failed"

  • warn

    Verification test result is set to "Passed", and a warning is shown

  • ignore

    Verification test result is set to "Ignore", and no errors are shown

6

Save the changes in the file and exit the editor.

Troubleshooting Failures

Use the "smo verifiers" command to troubleshoot a failed diagnostic test.

Below is the example procedure based on the System Health test that failed.

  1. The System Health test failed:

    [Expert@HostName-ch0x-0x:0]# gclish
[Global] HostName-ch01-01 > show smo verifiers report id 1
Duration of tests vary and may take a few minutes to complete
 
--------------------------------------------------------------------------------
| Tests Status                                                                 |
--------------------------------------------------------------------------------
| ID | Title              | Result     | Reason                                |
--------------------------------------------------------------------------------
| System Components                                                            |
--------------------------------------------------------------------------------
|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |
--------------------------------------------------------------------------------
| Tests Summary                                                                |
--------------------------------------------------------------------------------
| Passed: 0/1 test                                                             |
| Run: "show smo verifiers list id 1" to view a complete list of failed tests  |
| Output file: /var/log/verifier_sum.1-18.2023-04-20_12-24-37.txt              |
| Run "show smo verifiers last-run print" to display verbose output            |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >

  2. Print the full report for this failed test:

    [Global] HostName-ch01-01 > show smo verifiers print id 1
==============================
System Health:
==============================
 
 
--------------------------------------------------------------------------------
| VSX System Status - Maestro                                                 |
--------------------------------------------------------------------------------
| Up time                     | 06:44:48 hours                                 |
| SGMs                        | 3 / 3                                          |
| Virtual Systems             | 1                                              |
| Version                     | R81.20 (Build Number xxx)                    |
--------------------------------------------------------------------------------
| VS ID:  0                     VS Name:  MyVSname                             |
--------------------------------------------------------------------------------
| SGM ID                              Chassis 1                                |
|                                     ACTIVE                                   |
--------------------------------------------------------------------------------
|  2                                    ACTIVE                                 |
|  3                                    ACTIVE                                 |
|  4                                    ACTIVE                                 |
--------------------------------------------------------------------------------
| Chassis Parameters                                                           |
--------------------------------------------------------------------------------
| Unit         |                          Chassis 1                            |
--------------------------------------------------------------------------------
| SGMs         |                            3 / 3                              |
| Ports        |                            0 / 0                              |
| SSMs         |                            1 / 2   !                          |
--------------------------------------------------------------------------------
| Synchronization                                                              |
|     Sync to Active chassis:    Enabled                                       |
--------------------------------------------------------------------------------
 
--------------------------------------------------------------------------------
| Tests Status                                                                 |
--------------------------------------------------------------------------------
| ID | Title              | Result     | Reason                                |
--------------------------------------------------------------------------------
| System Components                                                            |
--------------------------------------------------------------------------------
|  1 | System Health      | Failed (!) | (1)Chassis 1 error                    |
--------------------------------------------------------------------------------
| Tests Summary                                                                |
--------------------------------------------------------------------------------
| Passed: 0/1 test                                                             |
| Run: "show smo verifiers list id 1" to view a complete list of failed tests  |
| Output file: /var/log/verifier_sum.1-18.2023-04-20_12-24-57.txt              |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >

  3. Examine which command produced the failed test:

    [Global] HostName-ch01-01 > show smo verifiers list id 1
--------------------------------------------------------------------------
| ID | Title              | Command                                      |
--------------------------------------------------------------------------
| System Components                                                      |
--------------------------------------------------------------------------
|  1 | System Health      | asg stat -v                                  |
--------------------------------------------------------------------------
| Run "show smo verifiers print id <TestNum>" to display test output     |
--------------------------------------------------------------------------
[Global] HostName-ch01-01 >

  4. Run the applicable command to understand what failed:

    [Global] HostName-ch01-01 > asg stat -v
 
--------------------------------------------------------------------------------
| VSX System Status - Maestro                                                  |
--------------------------------------------------------------------------------
| Up time                     | 06:45:06 hours                                 |
| SGMs                        | 3 / 3                                          |
| Virtual Systems             | 1                                              |
| Version                     | R81.20 (Build Number xxx)                    |
--------------------------------------------------------------------------------
| VS ID:  0                     VS Name:  MyVSname                             |
--------------------------------------------------------------------------------
| SGM ID                              Chassis 1                                |
|                                     ACTIVE                                   |
--------------------------------------------------------------------------------
|  2                                    ACTIVE                                 |
|  3                                    ACTIVE                                 |
|  4                                    ACTIVE                                 |
--------------------------------------------------------------------------------
| Chassis Parameters                                                           |
--------------------------------------------------------------------------------
| Unit        |                      Chassis 1                        | Weight |
--------------------------------------------------------------------------------
| SGMs        |                        3 / 3                          |   6    |
| Ports       |                                                       |        |
|   Standard  |                        0 / 0                          |  11    |
|   Bond      |                        0 / 0                          |  11    |
|   Other     |                        0 / 0                          |   6    |
| Sensors     |                                                       |        |
|   SSMs      |                        1 / 2   !                      |  11    |
|             |                                                       |        |
| Grade       |                       29 / 40  !                      |   -    |
--------------------------------------------------------------------------------
| Synchronization                                                              |
|     Sync to Active chassis:    Enabled                                       |
--------------------------------------------------------------------------------
[Global] HostName-ch01-01 >