VSX Diagnostics and Troubleshooting

In This Section:

This chapter presents basic diagnostic and troubleshooting procedures that should be followed in the event you encountering a problem while working with VSX. This diagnostic routine will assist you in determining the source of the problem. This chapter presents several known issues and their solutions.

Most problems are caused by configuration errors occurring during the process of defining VSX Gateway, clusters and/or virtual devices. Another common source of problems involves networking and connectivity issues affecting VSX behavior. These problems are listed according to the order in which you will likely encounter them. Before reading and following a certain workaround, make sure you've read all the previous workarounds, and that those steps in the configuration were successful.

In some of the cases, one initial problem can cause problems in later stages of the configuration. For that reason, it is important to find the root of the problem when you are trying to understand what went wrong.

Introduction

General Troubleshooting Steps

If you suspect that there is a problem with your VSX configuration, there are several diagnostic procedures that you can follow to determine the source. These procedures utilize various commands documented in the Command Line section.

Perform a basic configuration check for each gateway or cluster member by running the fw vsx stat -v command. The output will allow you to:
1. Account for all Virtual Systems and verify that none are missing from the configuration.
2. Verify that all virtual devices are active
3. Verify that the correct security policy is installed for each Virtual System
4. Verify the SIC trust has been established with the management server
Run the cplic print command on each VSX Gateway, cluster member and management server to verify that you have the appropriate licenses installed.
Run the cphaprob stat command on each cluster member to verify its status. If a member is listed with a status other than Active, Standby, or Backup, refer to the "Troubleshooting" chapter in the R76 ClusterXL Administration Guide for additional troubleshooting assistance.
If you suspect that a Virtual System is experiencing connectivity problems, perform the following steps:
1. Run: vsenv to set the context to the appropriate Virtual System.
2. Run fw getifs to display the interface list for the Virtual System.
3. Examine connectivity status using standard operating system commands and tools such as: ping, traceroute, tcpdump, ip route, ftp, etc. Some of these run according to context (i.e. routing, source and destination IP addresses). .
You can also execute the ip route and ip link commands.

If these tests indicate that all interfaces and routers have connectivity, and appear to be functioning correctly, you should monitor the passage of packets through the system.
Execute the fw monitor -v <vsid> commands to capture details of packets at multiple points. This may return multiple reports on the same packet as it passes various capture points. This command does not report on Virtual Routers, except for packets destined to an external Virtual Router.
Note - The Performance Pack may have an adverse effect on the capabilities of the fw monitor command.
Execute the tcpdump command to display transmitted or received packets for specific interfaces, including Warp interfaces. This often provides valuable clues for resolving connectivity issues.

Troubleshooting Specific Problems

Cannot Establish SIC Trust for Gateway or Cluster

When creating a VSX Gateway or cluster, you cannot establish SIC trust. SmartDashboard gives an error message:

Certificate cannot be pushed. Connection error with wait agent.

Possible Causes	How to Resolve
Check that you have network connectivity between the gateway and the Security Gateway or Domain Management Server by pinging from the VSX system (A ping from the Domain Management Server/Security Management to the VSX system will not work because of the default security policy installed on the VSX Gateway/cluster.) Make sure the context is `vrf 0` first.	On all relevant machines, re-check the cables, routes, IP addresses and any intermediate networking devices (routers, switches, hubs, and so on) between the management and the gateway(s).
Check that all the Check Point processes on the VSX Gateway(s) are up and running by running `cpwd_admin list` and making sure each line has a non-zero value in the `PID` field.	If the gateway(s) has just rebooted, the Check Point processes might still be coming up. If this is not the case, and you are using Crossbeam X40, make sure you have executed the `application … start` command. (For more information refer to the Crossbeam documentation.)
Check that the CPD process is listening to the trust establishment port.	Run `netstat -an \| grep 18211` on the VSX Gateway(s), and make sure that output looks like this: `tcp 0 0 0.0.0.0:18211 0.0.0.0:* LISTEN`

Possible Causes

How to Resolve

Check that you have network connectivity between the gateway and the Security Gateway or Domain Management Server by pinging from the VSX system (A ping from the Domain Management Server/Security Management to the VSX system will not work because of the default security policy installed on the VSX Gateway/cluster.)

Make sure the context is vrf 0 first.

On all relevant machines, re-check the cables, routes, IP addresses and any intermediate networking devices (routers, switches, hubs, and so on) between the management and the gateway(s).

Check that all the Check Point processes on the VSX Gateway(s) are up and running by running cpwd_admin list and making sure each line has a non-zero value in the PID field.

If the gateway(s) has just rebooted, the Check Point processes might still be coming up. If this is not the case, and you are using Crossbeam X40, make sure you have executed the application … start command. (For more information refer to the Crossbeam documentation.)

Check that the CPD process is listening to the trust establishment port.

Run netstat -an | grep 18211 on the VSX Gateway(s), and make sure that output looks like this:
tcp 0 0 0.0.0.0:18211 0.0.0.0:* LISTEN

SIC Trust Problems with New virtual devices

When creating a new Virtual System, Virtual Router or Virtual Switch, you cannot establish SIC trust.

Possible Causes	How to Resolve
Time or time zone mismatch between the management and the gateway. For proper SIC operation, the time, date and time zone must be synchronized between the management server and Gateways/ cluster members. Execute the `/bin/date -u` command on all machines, to obtain the correct UTC/GMT time. The machines can be in different time zones, as long as their UTC/GMT times match.	Change the time, date and time zone on the management and/or the gateway so that their UTC/GMT times match. Refer to you operating system documentation for the exact commands needed to accomplish this.

Possible Causes

How to Resolve

Time or time zone mismatch between the management and the gateway.

For proper SIC operation, the time, date and time zone must be synchronized between the management server and Gateways/ cluster members.

Execute the /bin/date -u command on all machines, to obtain the correct UTC/GMT time. The machines can be in different time zones, as long as their UTC/GMT times match.

Change the time, date and time zone on the management and/or the gateway so that their UTC/GMT times match. Refer to you operating system documentation for the exact commands needed to accomplish this.

Re-establishing SIC Trust with virtual devices

In the event that you encounter connectivity problems due to the loss of SIC trust for a specific virtual device, you can use the following procedure to manually re-establish trust.

To manually re-establish SIC Trust with virtual devices:

Execute the following command from the VSX Gateway command line (In the expert mode): vsx sic reset <vsid>.
vsid: Identification number of the virtual device
Execute the following command(s) on the management server:
1. # mdsenv <target_domain_name>
  (Multi-Domain Security Management only)
2. # cpca_client revoke_cert -n <vs_sic_name>
vs_sic_name: virtual device SIC name. To determine the SIC name, run guidbedit.exe and search for the sic_name attribute on the virtual device network object.
In SmartDashboard, open the virtual device object and click OK. This action creates a new SIC certificate for the virtual device and saves it on the VSX Gateway.

Install Policy Error Using VSX Creation Wizard

After completing the VSX creation wizard, a failure occurs and the following message appears in the Operation Report window: Error: Default policy installation failed on VSX. Install policy manually using SmartDashboard.

Possible Causes	How to Resolve
Missing or invalid license on the management server. Execute `cplic check` on the management server to `verify tha`t you have the required licenses.	Obtain and install the appropriate licenses.
Missing or invalid VSX Gateway / cluster licenses. Run `fw vsx stat` on all Gateways, and make sure that the output says Number of Virtual Systems allowed by license: is greater than 0.	Obtain a VSX and install a valid license for each VSX Gateway or cluster member.
Time or time zone mismatch between the management and the gateway. For proper SIC operation, the time, date and time zone must be synchronized between the management server and Gateways / cluster members. Execute the `/bin/date -u` command on all machines, to obtain the correct UTC/GMT time. The machines can be in different time zones, as long as their UTC/GMT times match.	Change the time, date and time zone on the management and/or the gateways so that their UTC/GMT times match. Refer to you operating system documentation for the exact commands needed to accomplish this.

Possible Causes

How to Resolve

Missing or invalid license on the management server.

Execute cplic check on the management server to verify that you have the required licenses.

Obtain and install the appropriate licenses.

Missing or invalid VSX Gateway / cluster licenses. Run fw vsx stat on all Gateways, and make sure that the output says Number of Virtual Systems allowed by license: is greater than 0.

Obtain a VSX and install a valid license for each VSX Gateway or cluster member.

Time or time zone mismatch between the management and the gateway. For proper SIC operation, the time, date and time zone must be synchronized between the management server and Gateways / cluster members.

Execute the /bin/date -u command on all machines, to obtain the correct UTC/GMT time. The machines can be in different time zones, as long as their UTC/GMT times match.

Change the time, date and time zone on the management and/or the gateways so that their UTC/GMT times match. Refer to you operating system documentation for the exact commands needed to accomplish this.

Internal Host Cannot Ping Virtual System

After defining a Virtual System with an internal VLAN interface, an internal host on that VLAN cannot ping the Virtual System internal or external IP address.

Possible Causes	How to Resolve
A policy allowing the communication was not installed on the Virtual System. Note that after creating a Virtual System, it has a Default Policy that blocks all traffic.	Install a policy on the Virtual System that enables the traffic. Check with the SmartView Tracker that the Virtual System is allowing the traffic.
There is the VLAN configuration problem on a switch, or physical cable problem.	Check the switch configuration. Make sure that VLAN tag configured on the switch is the same as used for the Virtual System VLAN interface. Check the cables, and make sure that you have plugged the cable from the switch to the correct port on the VSX Gateway or cluster members.
Incorrect routing on adjacent routers or hosts.	Check the routing tables on intermediate routers and hosts. You can use `tcpdump` on the relevant VLAN interface on the VSX Gateway or cluster member to verify that the traffic is arriving to and leaving the VSX machine.
Incorrect IP address or net mask defined on the Virtual System VLAN interface.	Check the IP address and the net mask assigned to the Virtual System internal VLAN interface.