Kernel Debug Procedure

Alternatively, use the Kernel Debug Procedure with Connection Life Cycle.

Important:

Debug increases the load on the CPU on the Security Gateway Dedicated Check Point server that runs Check Point software to inspect traffic and enforce Security Policies for connected network resources. / Cluster Two or more Security Gateways that work together in a redundant configuration - High Availability, or Load Sharing. Members / Security Group Members Member of a Security Group in ElasticXL Cluster, Maestro, and Scalable Chassis. Acronym: SGM.. Schedule a maintenance window.
We strongly recommend to connect over serial console to your Security Gateway / each Cluster Member Security Gateway that is part of a cluster. / Scalable Platform Security Group A logical group of Security Appliances (in Maestro) / Security Gateway Modules (on Scalable Chassis) that provides Active/Active cluster functionality. A Security Group can contain one or more Security Appliances / Security Gateway Modules. Security Groups work separately and independently from each other. To the production networks, a Security Group appears a single Security Gateway. In Maestro, each Security Group contains: (A) Applicable Uplink ports, to which your production networks are connected; (B) Security Appliances (the Quantum Maestro Orchestrator determines the applicable Downlink ports automatically); (C) Applicable management port, to which the Check Point Management Server is connected. Members.

This is to prevent a possible issue when you cannot work with the CLI because of a high load on the CPU.
In Cluster, you must perform these steps on all the Cluster Members in the same way.
On Scalable Platforms (Maestro and Chassis), you must connect to the applicable Security Group.

Step

Instructions

Connect to the command line on the Security Gateway / each Cluster Member over SSH, or console.

Note - On Scalable Platforms (Maestro and Chassis), you must connect to the applicable Security Group.

Reset the kernel debug options.

On the Security Gateway / each Cluster Member, run:

fw ctl debug 0

On the Scalable Platform Security Group, run:

g_fw ctl debug 0

Reset the kernel debug filters.

On the Security Gateway / each Cluster Member, run:

fw ctl set int simple_debug_filter_off 1

On the Scalable Platform Security Group, run:

g_fw ctl set int simple_debug_filter_off 1

Configure the applicable kernel debug filters.

See Kernel Debug Filters.

Allocate the kernel debug buffer for each CoreXL Performance-enhancing technology for Security Gateways on multi-core processing platforms. Multiple Check Point Firewall instances are running in parallel on multiple CPU cores. Firewall instance.

On the Security Gateway / each Cluster Member, run:

fw ctl debug -buf 8200

On the Scalable Platform Security Group, run:

g_fw ctl debug -buf 8200

Make sure the kernel debug buffer was allocated.

On the Security Gateway / each Cluster Member, run:

fw ctl debug | grep buffer

On the Scalable Platform Security Group, run:

g_fw ctl debug | grep buffer

Enable the applicable debug flags in the applicable kernel modules.

On the Security Gateway / each Cluster Member, run:

fw ctl debug -m <module> {all | + <flags>}

On the Scalable Platform Security Group, run:

g_fw ctl debug -m <module> {all | + <flags>}

See Kernel Debug Modules and Debug Flags.

Important - The CPU load increases at this point because the Firewall kernel starts to write some debug messages to the /var/log/messages file and the dmesg buffer.

Examine the list of the debug flags that are enabled in the specified kernel modules.

On the Security Gateway / each Cluster Member, run:

fw ctl debug -m <module>

On the Scalable Platform Security Group, run:

g_fw ctl debug -m <module>

Save the kernel debug output to a file.

Note - For information about the new kernel debug mode (R82 and higher), see Kernel Debug Behavior on Security Gateways with 72 and more CPU Cores.

On the Security Gateway / each Cluster Member, run:

For the new kernel debug mode:

fw ctl ndebug -T -o /var/log/kernel_debug.txt

For the legacy kernel debug mode:

fw ctl kdebug -T -f > /var/log/kernel_debug.txt

On the Scalable Platform Security Group, run:

For the new kernel debug mode:

g_fw ctl ndebug -T -o /var/log/kernel_debug.txt

For the legacy kernel debug mode:

g_fw ctl kdebug -T -f > /var/log/kernel_debug.txt

Important - The CPU load increases even more at this point because the Firewall starts to write all debug messages to the output file.

Replicate the issue, or wait for the issue to occur.

Stop the kernel debug output:

Press the CTRL+C keys.

Important - This does not stop all CPU load yet because the Firewall kernel continues to write some debug messages to the /var/log/messages file and the dmesg buffer.

Reset the kernel debug options.

On the Security Gateway / each Cluster Member, run:

fw ctl debug 0

On the Scalable Platform Security Group, run:

g_fw ctl debug 0

Important - This stops all CPU load from the kernel debug.

Reset the kernel debug filters.

On the Security Gateway / each Cluster Member, run:

fw ctl set int simple_debug_filter_off 1

On the Scalable Platform Security Group, run:

g_fw ctl set int simple_debug_filter_off 1

Transfer this file from the Security Gateway / each Cluster Member / each Security Group Member to your computer:

/var/log/kernel_debug.txt

Best Practice - Compress this file with the "tar -zxvf" command and transfer it from the Security Gateway / each Cluster Member / each Security Group Members to your computer. If you transfer to an FTP server, do so in the binary mode.

Analyze the debug output file.

Example - Connection 192.168.20.30:<Any> --> 172.16.40.50:80

[Expert@GW:0]# fw ctl debug 0
Defaulting all kernel debugging options
Debug state was reset to default.
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set int simple_debug_filter_off 1
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set str simple_debug_filter_saddr_1 "192.168.20.30"
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set str simple_debug_filter_daddr_2 "192.168.20.40"
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set int simple_debug_filter_dport_1 80
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug -buf 8200
Initialized kernel debugging buffer to size 8192K
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug | grep buffer
Kernel debugging buffer size: 8192KB
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug -m fw + conn drop
Updated kernel's debug variable for module fw
Debug flags updated.
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug -m fw
Kernel debugging buffer size: 8192KB
Module: fw
Enabled Kernel debugging options: error warning conn drop
Messaging threshold set to type=Info freq=Common
[Expert@GW:0]#
[Expert@GW:0]# fw ctl kdebug -T -f > /var/log/kernel_debug.txt

... ... Replicate the issue, or wait for the issue to occur ... ...

... ... Press CTRL+C ... ...

[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug 0
Defaulting all kernel debugging options
Debug state was reset to default.
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set int simple_debug_filter_off 1
[Expert@GW:0]#
[Expert@GW:0]# ls -l /var/log/kernel_debug.txt
-rw-rw---- 1 admin root 1630619 Apr 12 19:49 /var/log/kernel_debug.txt
[Expert@GW:0]#