Kernel Debug Procedure

Alternatively, use the Kernel Debug Procedure with Connection Life Cycle.

Important:

Schedule a maintenance window.

Debug increases the load on the CPU on the Security Gateway Dedicated Check Point server that runs Check Point software to inspect traffic and enforce Security Policies for connected network resources. / Cluster Two or more Security Gateways that work together in a redundant configuration - High Availability, or Load Sharing. Members / Security Group Members Member of a Security Group in ElasticXL Cluster, Maestro, and Scalable Chassis. Acronym: SGM..
We strongly recommend to connect over serial console to your Security Gateway / each Cluster Member Security Gateway that is part of a cluster. / Scalable Platform Security Group A logical group of Security Appliances (in Maestro) / Security Gateway Modules (on Scalable Chassis) that provides Active/Active cluster functionality. A Security Group can contain one or more Security Appliances / Security Gateway Modules. Security Groups work separately and independently from each other. To the production networks, a Security Group appears a single Security Gateway. In Maestro, each Security Group contains: (A) Applicable Uplink ports, to which your production networks are connected; (B) Security Appliances (the Quantum Maestro Orchestrator determines the applicable Downlink ports automatically); (C) Applicable management port, to which the Check Point Management Server is connected. Members.

This is to prevent a possible issue when you cannot work with the CLI because of a high load on the CPU.
In Cluster, you must perform these steps on all the Cluster Members in the same way.
On Scalable Platforms (Maestro and Chassis), you must connect to the applicable Security Group.
The procedure below contains steps for SecureXL Check Point product on a Security Gateway that accelerates IPv4 and IPv6 traffic that passes through a Security Gateway. debug as well.

Follow these steps only if the traffic issue is caused by SecureXL.
See Kernel Debug Syntax.

Step

Instructions

Connect to the command line on the Security Gateway / each Cluster Member over SSH, or console.

Note - On Scalable Platforms (Maestro and Chassis), you must connect to the applicable Security Group.

Reset the kernel debug flags in all kernel debug modules.

On the Security Gateway / each Cluster Member, run:

fw ctl debug 0

On the Scalable Platform Security Group, run:

g_fw ctl debug 0

Reset the kernel debug flags in all SecureXL debug modules.

On the Security Gateway / each Cluster Member, run:

fwaccel dbg resetall

On the Scalable Platform Security Group, run:

g_fwaccel dbg resetall

Reset the kernel debug filters.

On the Security Gateway / each Cluster Member, run:

fw ctl set int simple_debug_filter_off 1

On the Scalable Platform Security Group, run:

g_fw ctl set int simple_debug_filter_off 1

Configure the applicable kernel debug filters.

See Kernel Debug Filters.

Allocate the kernel debug buffer for each CoreXL Performance-enhancing technology for Security Gateways on multi-core processing platforms. Multiple Check Point Firewall instances are running in parallel on multiple CPU cores. Firewall instance.

On the Security Gateway / each Cluster Member, run:

fw ctl debug -buf 8200

On the Security Gateway / each Cluster Member in the VSNext / Traditional VSX Virtual System Extension. Check Point virtual networking solution, hosted on a computer or cluster with virtual abstractions of Check Point Security Gateways and other network devices. These Virtual Devices provide the same functionality as their physical counterparts. mode, run:

fw ctl debug -buf 8200 -v {"<List of VSIDs>" | all} -k

On the Scalable Platform Security Group, run:

g_fw ctl debug -buf 8200

On the Scalable Platform Security Group in the VSNext / Traditional VSX mode, run:

g_fw ctl debug -buf 8200 -v {"<List of VSIDs>" | all} -k

Make sure the Security Gateway allocated the kernel debug buffer.

On the Security Gateway / each Cluster Member, run:

fw ctl debug | grep buffer

On the Scalable Platform Security Group, run:

g_fw ctl debug | grep buffer

Configure the applicable kernel debug modules and kernel debug flags.

On the Security Gateway / each Cluster Member, run:

fw ctl debug -m <Name of Kernel Debug Module> {all | + <Kernel Debug Flags>}

On the Security Gateway / each Cluster Member in the VSNext / Traditional VSX mode, run:

fw ctl debug -v {"<List of VSIDs>" | all} -k -m <Name of Kernel Debug Module> {all | + <Kernel Debug Flags>}

On the Scalable Platform Security Group, run:

g_fw ctl debug -m <Name of Kernel Debug Module> {all | + <Kernel Debug Flags>}

On the Scalable Platform Security Group in the VSNext / Traditional VSX mode, run:

g_fw ctl debug -v {"<List of VSIDs>" | all} -k -m <Name of Kernel Debug Module> {all | + <Kernel Debug Flags>}

See Kernel Debug Modules and Debug Flags.

Important - The CPU load increases at this point because the Firewall kernel starts to write some debug messages to the /var/log/messages file and the dmesg buffer.

Configure the applicable SecureXL debug modules and SecureXL debug flags.

On the Security Gateway / each Cluster Member, run:

fwaccel dbg -m <Name of SecureXL Debug Module> {all | + <SecureXL Debug Flags>}

On the Scalable Platform Security Group, run:

g_fwaccel dbg -m <Name of SecureXL Debug Module> {all | + <SecureXL Debug Flags>}

See the R82 Performance Tuning Administration Guide > SecureXL > SecureXL Commands and Debug > SecureXL Debug > SecureXL Debug Procedure.

Important - The CPU load increases at this point because the SecureXL starts to write some debug messages to the /var/log/messages file and the dmesg buffer.

Examine the the kernel debug configuration for kernel debug modules.

On the Security Gateway / each Cluster Member, run:

fw ctl debug -m <module>

On the Scalable Platform Security Group, run:

g_fw ctl debug -m <module>

Examine the SecureXL debug configuration for SecureXL debug modules

On the Security Gateway / each Cluster Member, run:

fwaccel dbg list

On the Scalable Platform Security Group, run:

g_fwaccel dbg list

Save the kernel debug output to a file.

Note - For information about the new kernel debug mode (R82 and higher), see Kernel Debug Behavior on Security Gateways with 72 and more CPU Cores.

Important - The CPU load increases even more at this point because the Firewall starts to write all debug messages to the output file.

On the Security Gateway / each Cluster Member, run:

For the new kernel debug mode:

fw ctl ndebug -T -o /var/log/kernel_debug.txt

For the legacy kernel debug mode:

fw ctl kdebug -T -f -o /var/log/kernel_debug.txt

On the Security Gateway / each Cluster Member in the VSNext / Traditional VSX mode, run:

For the new kernel debug mode:

fw ctl ndebug -v {"<List of VSIDs>" | all} -k -T -o /var/log/kernel_debug.txt

For the legacy kernel debug mode:

fw ctl kdebug -v {"<List of VSIDs>" | all} -k -T -f -o /var/log/kernel_debug.txt

On the Scalable Platform Security Group, run:

For the new kernel debug mode:

g_fw ctl ndebug -T -o /var/log/kernel_debug.txt

For the legacy kernel debug mode:

g_fw ctl kdebug -T -f -o /var/log/kernel_debug.txt

On the Scalable Platform Security Group in the VSNext / Traditional VSX mode, run:

For the new kernel debug mode:

g_fw ctl ndebug -v {"<List of VSIDs>" | all} -k -T -o /var/log/kernel_debug.txt

For the legacy kernel debug mode:

g_fw ctl kdebug -v {"<List of VSIDs>" | all} -k -T -f -o /var/log/kernel_debug.txt

Replicate the issue, or wait for the issue to occur.

Stop the kernel debug output:

Press the CTRL+C keys.

Important - This does not stop all CPU load yet because the Firewall kernel continues to write some debug messages to the /var/log/messages file and the dmesg buffer.

Reset all kernel debug flags in all kernel debug modules.

On the Security Gateway / each Cluster Member, run:

fw ctl debug 0

On the Scalable Platform Security Group, run:

g_fw ctl debug 0

Important - This stops all CPU load from the kernel debug.

Reset all the SecureXL debug flags in all SecureXL debug modules.

On the Security Gateway / each Cluster Member, run:

fwaccel dbg resetall

On the Scalable Platform Security Group, run:

g_fwaccel dbg resetall

Important - This stops all CPU load from the SecureXL debug.

Reset the kernel debug filters.

On the Security Gateway / each Cluster Member, run:

fw ctl set int simple_debug_filter_off 1

On the Scalable Platform Security Group, run:

g_fw ctl set int simple_debug_filter_off 1

Examine the kernel debug configuration to make sure it returned to the default.

On the Security Gateway / each Cluster Member, run:

fw ctl debug

On the Scalable Platform Security Group, run:

g_fw ctl debug

Examine the SecureXL debug configuration to make sure it returned to the default.

On the Security Gateway / each Cluster Member, run:

fwaccel dbg list

On the Scalable Platform Security Group, run:

g_fwaccel dbg list

Transfer these files from the Security Gateway / each Cluster Member / each Security Group Member to your computer:

/var/log/kernel_debug.txt

/var/log/messages*

$FWDIR/log/fwk.elg*

/var/log/usim_x86.elg*

Best Practice - Compress this file with the "tar -zxvf" command and transfer it from the Security Gateway / each Cluster Member / each Security Group Members to your computer. If you transfer to an FTP server, do so in the binary mode.

Analyze the debug output file.

Example - Connection 192.168.20.30:<Any> --> 172.16.40.50:80

[Expert@GW:0]# fw ctl debug 0
Defaulting all kernel debugging options
Debug state was reset to default.
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set int simple_debug_filter_off 1
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set str simple_debug_filter_saddr_1 "192.168.20.30"
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set str simple_debug_filter_daddr_2 "192.168.20.40"
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set int simple_debug_filter_dport_1 80
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug -buf 8200
Initialized kernel debugging buffer to size 8192K
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug | grep buffer
Kernel debugging buffer size: 8192KB
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug -m fw + conn drop
Updated kernel's debug variable for module fw
Debug flags updated.
[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug -m fw
Kernel debugging buffer size: 8192KB
Module: fw
Enabled Kernel debugging options: error warning conn drop
Messaging threshold set to type=Info freq=Common
[Expert@GW:0]#
[Expert@GW:0]# fw ctl kdebug -T -f -o /var/log/kernel_debug.txt

... ... Replicate the issue, or wait for the issue to occur ... ...

... ... Press CTRL+C ... ...

[Expert@GW:0]#
[Expert@GW:0]# fw ctl debug 0
Defaulting all kernel debugging options
Debug state was reset to default.
[Expert@GW:0]#
[Expert@GW:0]# fw ctl set int simple_debug_filter_off 1
[Expert@GW:0]#
[Expert@GW:0]# ls -l /var/log/kernel_debug.txt
-rw-rw---- 1 admin root 1630619 Apr 12 19:49 /var/log/kernel_debug.txt
[Expert@GW:0]#