Troubleshooting

In This Section: General Guidelines Log Files Viewing the Performance Statistics Automatically Saving Messages from the Falcon Acceleration Cards

General Guidelines

• Make sure the software loaded correctly.
• Make sure the Falcon Acceleration Cards are active and work correctly.

Log Files

Location	Description
/tmp/bcm_boot_log	Temporary boot log file. Contains additional information about the Falcon Acceleration Cards startup.
/var/log/sam_crash/	A fatal error in the Falcon Acceleration Card software triggers a panic on the Host Security Appliance and initiates a procedure to transfer relevant forensic data from the Falcon Acceleration Card to the Host Security Appliance. After the Host Security Appliance reboots, core dump files are located in this directory.
/var/log/sam_crash/crash_list	File that contain information about crashes and hangs of Falcon Acceleration Cards. This file does not exist until the first crash or hang.
/var/ktr_dump	File that shows the kernel trace information from Falcon Acceleration Cards. This file does not exist until the first crash or hang.
/var/log/messages	Gaia Operating System log file.
/var/log/dmesg	Log file for the Linux kernel ring buffer. To see the current messages, run the dmesg command in the Expert mode.

Viewing the Performance Statistics

For more information about the SecureXL, see the R80.20 Performance Tuning Administration Guide.

Step	Description
1	Connect to the command line on the Host Security Appliance.
2	Log in to the Expert mode.
3	Run the applicable commands (see below). Best Practice - Collect the complete information before and after the issue: Collect the output before the issue: [Expert@HostName:0]# (echo "$(hostname) , $(/bin/date +"%b %d %H:%M:%S %Y")" ; ipsctl -a) >> /var/log/ipsctl_before.txt Replicate the issue / wait for the issue to occur. Collect the output after the issue: [Expert@HostName:0]# (echo "$(hostname) , $(/bin/date +"%b %d %H:%M:%S %Y")" ; ipsctl -a) >> /var/log/ipsctl_after.txt

To see the packet drops on the interfaces:

ipsctl -a | grep "slot:" | \ egrep "_err|_drop" | \ egrep "fail_|ifdown_|invalid_if_|l2_filt_|too_big|nae_|no_link_|pcie_|rx_v"

Description of counters:

• fail_err - Failed to send an FMN message.
• filterin_drop - Forwarding is disabled.
• ifdown_drop - Interface is down.
• invalid_if_err - Invalid interface.
• l2_filt_drop - MAC address filtering failure.
• len_too_big_drop - Packet is too long.
• nae_egress - Total number of transmitted (TX) packets.
• no_link_drop - Outbound interface is down.
• pcie_corr_drop - PCIe correction drop between acceleration cards.
• pcie_jumbo_pkt_err - Jumbo packet DMA error from an acceleration card to the Host Security Appliance.
• pcie_reg_pkt_err - PCIe transmission packet drop.
• pcie_rx_err - Regular DMA error from an acceleration card to the Host Security Appliance.
• pcie_too_big_drop - PCIe packet is too large.
• pcie_tx_drop - PCIe transmission failure.
• pcie_xnp_rx_drop - Packet DMA error between acceleration cards.
• pcie_xnp_tx_drop - Nexthop output on PCIe failure.
• rx_v4_drop - IPv4 drops.
• rx_v6_drop - IPv6 drops.
• rx_vlan_drop - VLAN packets received on non-interface.

To see the packets in Slow Path (F2F):

ipsctl -a net:dev:adp:if:reasons

Most important counters are:

• es_noflow - No flow was found after decryption.
• es_wrong_mspi - MSPI was found wrong after decryption.
• frag - Fragment of a packet.
• ip_opts - Packets with IP Options.
• mtu_exceed - Packet MTU exceeds the interface MTU.
• no_flow - First packet of the connection or flow was not found.
• no_sa - No SAs found.
• route_err - Routing error.
• spoof - Anti-Spoofing checks failed.
• tcp_seq - TCP sequence adjustment failed.
• tcp_state - TCP state detection failed.
• tcp_syn - First SYN packet of a TCP connection.
• ttl - Packet TTL reached 0.
• unresolved - Neighbor was unresolved.

Packet drops because of route lookup failures:

ipsctl net:dev:adp:rt:stats:rt_reinject_drop

To see the packet drops because of ARP failures:

ipsctl -a | egrep "arp_dropped|arp_expired"

Description of the counters:

• arp_dropped - Packets dropped after ARP queue is full.
• arp_expired - Packets dropped after ARP resolution timed out.

To see the packet drops between the PCIe and acceleration cards:

ipsctl -a | grep tx_fulls

Description of the counters:

• tx_fulls - There is no transmit (TX) queue.

To see the packet drops between the PCIe and the Host Security Appliance:

ipsctl -a | egrep "no_jumbo_local|no_reg_local"

Description of the counters:

• no_reg_local - There is no receive (RX) buffer.
• no_jumbo_local - There is no Jumbo receive buffer.

To see the Deep Packet Inspection (DPI) drops:

ipsctl -a | grep ':dpi:' | \ egrep "enqueue_failure|packet_pool_failures|search_clamping|search_failure|start_failure"

Description of the counters:

• enqueue_failure - Failed to add to the DPI queue.
• packet_pool_failures- Before enqueue, allocation failed.
• search_clamping - Too many matching results.
• search_failure - Search failed.
• start_failure - After enqueue.

To see the Falcon Acceleration Cards packet pool:

ipsctl -a | grep usim | grep pkt_pool

To see the IPsec Security Associations (SAs) encryption and decryption statistics:

ipsctl -a | grep "slot:" | egrep "no_sa|oerr|olen_err|replay_err"

Description of the counters:

• no_sa - Is not updated.
• oerr - Is updated when errors are encountered during either encryption, or decryption by the acceleration card hardware (SAE).
• olen_err - Is not updated.
• replay_err - Is not updated.

To see the traffic paths distribution:

fwaccel [-i <SecureXL Instance ID>] stats

To see the PXL statistics:

fwaccel [-i <SecureXL Instance ID>] stats -x

To see the packet reordering prevention statistics:

fwaccel [-i <SecureXL Instance ID>] stats -o

To see the CoreXL Dispatcher statistics:

fw ctl pstat -m

Most important counters are:

• Async ADP call - Asynchronous calls from acceleration cards.
• Async index req - Index request for asynchronous calls.
• Etm multik chain - QoS messages.
• F2P packet kernel - Medium Path packets handled in kernel.
• F2P packet userspace - Medium Path packets handled in userspace.
• Multik message kernel - CoreXL messages sent between CoreXL FW instances in kernel.
• Multik message userspace - CoreXL messages sent between CoreXL FW instances in userspace.
• Notification Packet - CPAQ messages as packet.
• SXL Device State Info - SecureXL device state updates.
• Vs message - VSX messages.
• Vs_kill - VSX messages.

To see the CPAS drop statistics:

fw ctl fwcpasstat | grep drop

To see the PSL drop statistics:

fw ctl fwtcpstrstat -p | grep drop

Automatically Saving Messages from the Falcon Acceleration Cards

You can configure the Falcon Acceleration Cards to generate additional important messages automatically.

In this case, you must configure the Host Security Appliance to save these messages automatically.

The Host Security Appliance saves the messages in the Linux kernel ring buffer - dmesg.

To control this feature, use these parameters on the Host Security Appliance:

Parameter Value	Description
net:dev:bp:log_messages=0	This is the default value. The Falcon Acceleration Cards do not generate additional important messages automatically.
net:dev:bp:log_messages=1	The Falcon Acceleration Cards generate additional important messages automatically.

Best Practice - In cluster, configure the same value of this parameter on all the Cluster Members.

To examine the current Host Security Appliance configuration:

Step	Description
1	Connect to the command line on your Host Security Appliance.
2	Log in to the Expert mode.
3	Check the current value of the Falcon Acceleration Cards logging parameter: [Expert@MyAppliance:0]# ipsctl net:dev:bp:log_messages Example: [Expert@MyAppliance:0]# ipsctl net:dev:bp:log_messages ipsctl net:dev:bp:log_messages = 0 [Expert@MyAppliance:0]#

To enable the feature temporarily on the Host Security Appliance (does not survive reboot):

Step	Description
1	Connect to the command line on your Host Security Appliance.
2	Log in the Expert mode.
3	Set the current value of the Falcon Acceleration Cards logging parameter: [Expert@MyAppliance:0]# ipsctl -w net:dev:bp:log_messages 1 Example: [Expert@MyAppliance:0]# ipsctl -w net:dev:bp:log_messages 1 net:dev:bp:log_messages 0 -> 1 [Expert@MyAppliance:0]#

To enable the feature permanently on the Host Security Appliance (survives reboot):

Step	Description
1	Connect to the command line on your Host Security Appliance.
2	Log in to the Expert mode.
3	Set the value of the Falcon Acceleration Cards logging parameter: Back up the current script: [Expert@MyAppliance:0]# cp -v /etc/rc.d/rc.local{,_BKP} Edit the current script: [Expert@MyAppliance:0]# vi /etc/rc.d/rc.local Add these two lines at the bottom of the script: # Falcon Acceleration Cards - Additional logging during the card crash ipsctl -w net:dev:bp:log_messages 1 Important - After the second line, you must press Enter to start a new line. Save the changes in the file and exit the Vi editor. Examine the file (it must end with an empty line): [Expert@MyAppliance:0]# cat -n /etc/rc.d/rc.local
5	Reboot the Host Security Appliance. Important - In cluster, this can cause a failover.
6	Connect to the command line on your Host Security Appliance.
7	Log in to the Expert mode.
8	Make sure the new value of the Falcon Acceleration Cards logging parameter is set: [Expert@MyAppliance:0]# ipsctl net:dev:bp:log_messages

To see the messages saved in the Linux kernel ring buffer:

Step	Description
1	Connect to the command line on your Host Security Appliance.
2	Log in to the Expert mode.
3	Examine the saved messages from the Linux kernel ring buffer: [Expert@MyAppliance:0]# less /var/log/dmesg
4	Print the contents of the current Linux kernel ring buffer. To print the messages on the screen: [Expert@MyAppliance:0]# dmesg To print the messages into a file: [Expert@MyAppliance:0]# dmesg > /<path>/<filename> Example: [Expert@MyAppliance:0]# dmesg > /var/log/acceleration_card_dmesg.txt

For more information, see https://linux.die.net/man/8/dmesg.