Troubleshooting SD-WAN "Overlay"

For background information, see SD-WAN Connection Type - "Overlay".

Symptom:

After an SD-WAN link failover, the "Overlay" traffic is disconnected and stops passing.

Solution:

  1. In Infinity Portal > Quantum SD-WAN application > Network view > SD-WAN Policy page:

    1. In the Steering Behavior object "Overlay", make sure you selected the WAN link, to which you expected the traffic to fall back.

      As a result, this "Overlay" traffic and probing traffic would match the "Overlay" steering behavior that uses the WAN link.

    2. Make sure that this "Overlay" traffic and probing traffic do not match a rule with the Steering Behavior object "Breakout".

  2. On the Security Gateway, after the SD-WAN link failover:

    1. Connect to the command line on the Security Gateway.

    2. Log in to the Expert mode.

    3. Make sure that you have at least one VPN tunnel in the status "Up" with the VPN peer:

      1. Run:

        cpview

      2. At the top, click Advanced tab > SD-WAN page.

      3. Refer to the section Overlay Probing Results > column State.

    4. Make sure that you have at least one VPN tunnel in the status "Up" with the VPN peer:

      1. Run:

        cpview

      2. At the top, click Advanced tab > SD-WAN page.

      3. Refer to the section Overlay Probing Results > column State.

    5. Examine the routing table:

      clish -c "show route"

      After the previously active SD-WAN interface goes down, you must make sure that there is still another active route (either the default route or a static route) that includes the destination for the "Overlay" connection.

      Without such an active route, the operating system cannot route the "Overlay" traffic.

      The SD-WAN selects the route to the "Overlay" destination (and not the operating system based on a static route).

      This route in the operating system is necessary only for the "Overlay" traffic to flow from the inbound chain to the outbound chain, where the Security Gateway encrypts the traffic, and SD-WAN selects the best VPN transport to the VPN peer.

    6. Examine the VPN connections and their VPN tunnels:

      Syntax:

      vpn tu conn <Source IP> <Source Port> <Destination IP> <Destination Port> <Protocol>

      Note - The minus character "-" is a wildcard "any".

      For more information, refer to the CLI Reference Guide for your version > Chapter "VPN Commands" > Section "vpn" > Section "vpn tu".

      Example 1:

      vpn tu conn - - - - -

      Example 2:

      vpn tu conn 192.168.10.10 - 192.168.20.10 443 6

Symptom:

CPView > Advanced > SD-WAN page shows that the state of the "Overlay" probing is "DOWN".

Solution:

  1. Make sure the nexthop device for the SD-WAN interface is up.

  2. Connect to the command line on the Security Gateway.

  3. Log in to the Expert mode.

  4. Make sure the VPN tunnel is established and "Up" from the SD-WAN interface to the required interface on the VPN peer:

    For more information about these commands, refer to the CLI Reference Guide for your version > Chapter "VPN Commands" > Section "vpn" > Section "vpn tu".

    1. Run:

      vpn tu

      and enter the digit 2 for the option "List all IPsec SAs".

    2. Run:

      vpn tu tlist

      If the command output does not show Outbound SA or IPsec SA, then the VPN tunnel is "down".

      Example of a "vpn tu tlist" output:

      • The top section shows "No outbound SA". Meaning, this VPN tunnel is "down".

      • The bottom section shows a value in the "Out SPI" field. Meaning, this VPN tunnel is "up".

    3. Run:

      vpn tu conn <Source IP Address> – <Destination IP Address> – 1

      Note - The minus character "-" is a wildcard "any" because ICMP does not use traffic ports.

      Example: vpn tu conn 192.168.3.57 – 172.16.4.68 – 1

      Run this command with the IP addresses of the overlay probing connection that is down.

      The connection must be encrypted and must pass through a VPN tunnel that is "up".

  5. If the VPN tunnel is "down", then understand why:

    1. Examine the Security Gateway logs in SmartConsole / SmartView.

      Filter for the applicable Source IP address, Destination IP address, and Ports 500 (IKE) and 4500 (NAT-T).

      Determine whether the VPN peers send and respond to the IKE negotiation traffic.

    2. Capture the traffic on the Security Gateway with FW Monitor (sk30583).

      Filter for the applicable Source IP address, Destination IP address, and Ports 500 (IKE) and 4500 (NAT-T).

      Determine whether the VPN peers send and respond to the IKE negotiation traffic.

    3. Examine the Security Gateway logs in SmartConsole / SmartView.

      Filter for the action "Key Install".

      Determine the reason for the failure. For example, an invalid IPsec certificate. The Source IP address of a VPN Gateway is not known to its VPN peer (Invalid ID).

      For additional information, see: sk164840, sk88780, sk112054.

    4. Collect the VPN debug as described in sk180488.

    5. Contact Check Point Support and include the CLI outputs, VPN debug outputs, and log files.

  6. If VPN tunnel is "up", but CPView shows it as "down", then the Security Gateway does not receive the ICMP Replies from the VPN peer.

    1. Make sure the Security Policy on each VPN peer explicitly allows the ICMP Echo Request traffic between these VPN peers on their external interfaces.

    2. Make sure the SD-WAN Policy:

      1. Matches the ICMP Echo Request traffic between these VPN peers on the "Overlay" steering object.

      2. Has all available WAN interfaces selected.

    3. Make sure the Encryption Domain of the Security Gateway does not include the subnets of the SD-WAN peer.

      If the Encryption Domain contains the VPN peer subnets, the Security Gateway does not encrypt the traffic it sends to the VPN peer.

    4. Capture the traffic on the Security Gateway with FW Monitor (sk30583).

      Use the "-F" parameter to filter for the ICMP package between the Security Gateway and its VPN peer.

      Example syntax (port "0" means "any"):

      fw monitor -F "192.168.22.33,0,141.10.60.1,0,1" -F "172.20.44.55,0,192.168.22.33,0,1"

      The FW Monitor traffic capture must show:

      1. The ICMP Echo Request traffic from the Security Gateway (encrypted in the Pre-Outbound chain "oE") to the VPN peer.

      2. The ICMP Echo Reply traffic from the VPN peer to the Security Gateway (decrypted in the Pre-Inbound chain "iD").

      If this ICMP traffic is not encrypted, then:

      1. In SmartConsole, examine the Access Control policy.

        Make sure there is an explicit rule to accept this ICMP traffic.

      2. Open the Security Gateway object > expand Network Management > click VPN Domain.

        Make sure you did not exclude the external interface that establish the VPN tunnel from the VPN Domain.

      3. Make sure you did not exclude this ICMP traffic in the crypt.def file (see sk25675).

      4. Make sure you did not exclude this ICMP traffic in the corresponding VPN Community object on the Excluded Services page.

      If the ICMP Echo Reply traffic from the VPN peer to the Security Gateway does not get to the Post-Inbound chain "I", then determine why the Security Gateway drops it:

      1. Examine the Security Gateway logs in SmartConsole / SmartView.

      2. Collect the kernel debug:

        For more information about the kernel debug procedure on a Security Gateway, see the Security Gateway Guide for your version > chapter "Kernel Debug".

        fw ctl zdebug -m fw + drop | grep 'value'

Symptom:

CPView > Advanced > SDWAN > Probing > the section "Steering Objects Results" does not show the "Overlay" steering objects.

Cause:

This is by design.

CPView shows only the "Local Breakout" and "Backhaul" steering objects.

Symptom:

FW Monitor (sk30583) traffic capture of the "Overlay" traffic between a client and a server does not show the expected outbound interface on Post-Outbound chain "O".

Cause:

FW Monitor shows the outbound interface based on the routing table.

This does not mean that it was the actual interface that routed the packet.

Solution:

To see the real interface that routed the traffic, do one of these:

  • In the applicable Security Gateway logs, examine the "SD-WAN" section.

  • Capture the ESP / NAT-T traffic with the CPPCAP tool (sk141412) in the Expert mode:

    cppcap -i <Name of Interface> -f "esp or port 4500" -o /var/log/cppcap_ESP_NATT.cap

  • Examine the VPN connections and their VPN tunnels:

    Syntax:

    vpn tu conn <Source IP> <Source Port> <Destination IP> <Destination Port> <Protocol>

    Note - The minus character "-" is a wildcard "any".

    For more information, refer to the CLI Reference Guide for your version > Chapter "VPN Commands" > Section "vpn" > Section "vpn tu".

    Example 1:

    vpn tu conn - - - - -

    Example 2:

    vpn tu conn 192.168.10.10 - 192.168.20.10 443 6

Symptom:

  • CPView > Advanced > SDWAN > Probing > the section "Overlay Probing Results" does not show the expected VPN Transports to VPN peers.

  • Infinity Portal > Quantum SD-WAN > Monitor view > Dashboard page > Overlay Tunnels Monitoring tab does not shows the expected VPN transports to VPN peers.

For example, there are no VPN transports to a specific VPN Peer, or to a specific interface of a VPN peer.

The kernel debug "fw ctl zdebug -m fw + drop" on the Security Gateway (see Troubleshooting SD-WAN with Kernel Debug) may show these traffic drops between the IP addresses of the interfaces on the local Security Gateway and the IP addresses of the interfaces on the VPN Peer Security Gateway:

fw_log_drop_ex: Packet proto=1 <Local IP Address>:0 -> <Peer IP Address>:<Port> dropped by vpn_encrypt_chain Reason: encrypt drop; sdwan_get_vpn_transport_id: Could not find transport id, vpn_transport=[<Peer IP Address>, 12, <Peer IP Address>] kiss_htab_bl_get_ret_val=-1 (_set1); vpn_sdwan_get_transport: Could not find transport entry in defaults transports for peer gateway <Peer IP Address>, conn=<dir 1, <Local IP Address>:0 -> <Peer IP Address>:<Port>IPP 1> rule_uuid=<UUID>.;

Solution:

  1. In SmartConsole, make sure the names of interfaces in the object of each involved VPN Peer Security Gateway match the names of interfaces in the Operating System.

    • In the object for a Security Gateway that runs the Gaia OS, go to the page Network Management.

    • In the object for a Quantum Spark Gateway that runs the Gaia Embedded OS, go to the page Topology.

    Important - In a cluster object, the names of the Cluster Virtual Interfaces must be identical to the corresponding physical / virtual interfaces on the cluster members. For example, if you paired the interfaces on cluster members with the names "eth1", then the Cluster Virtual Interface must also have the name "eth1".

  2. In SmartConsole, make sure the involved VPN Peer Security Gateways / Clusters participate in the correct VPN Community.

  3. Make sure the involved VPN Peer Security Gateways / Cluster Members have SD-WAN enabled and the Nano-Agent is running and communicating with the Internet:

    Note - In a cluster, run these commands on each cluster member.

    1. Run:

      cpsdwan stat

      The output must show the SD-WAN policy is installed.

    2. Run:

      cpnano -s

      The output must show the SD-WAN Nano-Services are running, and the connectivity to the Internet did not fail.

  4. If you recently made changes in the objects of the involved VPN Peer Security Gateways / Clusters, then install the Access Control policy on each VPN Peer Security Gateway / Cluster.

  5. On each VPN Peer Security Gateway / Cluster Member, run:

    jq . $FWDIR/state/local/SDWAN/sdwan_steering_policy.json

    The output must show the relevant Security Gateway / each relevant Cluster Member in the section "sdwan_steering_vpn_peers".

    1. If the VPN peer does not appear in the output at all, it most probably means one of these (other causes are also possible):

      • The local Security Gateway / Cluster does not have the SD-WAN information about that VPN peer from Infinity Portal.

      • The local Security Gateway / Cluster does not participate in the same VPN community with this VPN peer.

    2. If the VPN peer appears in the output, then in the section of this VPN peer, the sub-section "interfaces" must show the configured interfaces and must show the "Circuit ID" number for each interface.

      Important - Make sure the same interface name appears in all these places:

      1. In the Gaia operating system on the Security Gateway / each Cluster Member.

      2. In SmartConsole > Security Gateway / Cluster object > Network Management page.

      3. In the sdwan_steering_policy.json file > in the section "interfaces" > in the "name" value.

    3. In the upper section of the output, the section "sdwan_steering_vpn_local" must show the configured interfaces and must show the "Circuit ID" number for each interface for the local Security Gateway / Cluster Member.

    4. When the VPN Peer Security Gateways / Clusters are managed by the same Security Management Server / Domain Management Server:

      • The section "sdwan_steering_vpn_local" and the section "sdwan_steering_vpn_peers" must show the same "Domain UUID" value. If the value is not the same in these sections, then these VPN Peers behave if they are managed by different Management Servers.

      • If you configured a Global VPN Community on a Multi-Domain Security Management Server, and the VPN Peer Security Gateways / Cluster are managed by different Domain Management Servers:

        • The section "sdwan_steering_vpn_local" and the section "sdwan_steering_vpn_peers" will show different "Domain UUID" values.

        • The section "sdwan_steering_vpn_peers" will show the names of interfaces on the VPN peers managed as "ifc-<Number>".

        Support for "Overlay" in a Global VPN Community is available in:

    Important:

    • If the "Circuit ID" numbers for the local Security Gateway / Cluster Member and for the VPN Peer are different, then the VPN Transport cannot be created between them. This is by design.

    • It is not supported to use a dummy object that represents a Management Server.

      For example, in the Security Gateway / Cluster object properties > on the Fetch Policy page.

  6. Examine the $FWDIR/log/sdwan_steering.elg file for errors about the VPN Peer.

  7. Examine the VPN tunnel details with dedicated tools:

    1. Run:

      vpn tu tlist

      The output must show tunnels with "From:" and "To:" fields indicating the SD-WAN interfaces on the local Security Gateway / Cluster Member and on the VPN Peer.

    2. Run:

      vpn tu conn - - - - -

      The output must show connections that are passing between these:

      • Encryption Domain of the local Security Gateway / Cluster Member

      • Encryption Domain of the VPN Peer

      The output must show connections with "From:" and "To:" fields indicating the SD-WAN interfaces on the local Security Gateway / Cluster Member and on the VPN Peer.

    If the above dedicated tools show the VPN tunnels and VPN connections, then this may be a display issue in the CPView tool.

  8. If the issue persists, then contact Check Point Support and include these files from each the involved Security Gateway:

    1. CPinfo file (sk92739)

    2. $FWDIR/log/sdwan_steering.elg

    3. $FWDIR/log/cpsdwan.elg

    4. $FWDIR/state/local/SDWAN/sdwan_steering_policy.json

    5. /var/log/nano_agent/cp-nano-sdwan.dbg

    6. /var/log/nano_agent/cp-nano-orchestration.dbg

    7. Run this command and include the output file "/var/log/vpn_tu_tlist.txt":

      vpn tu tlist >> /var/log/vpn_tu_tlist.txt

    8. Run this command and include the output file "/var/log/vpn_tu_conn.txt":

      vpn tu conn - - - - - >> /var/log/vpn_tu_conn.txt

    9. Screenshot from CPView:

      1. Run: cpview

      2. At the top, click Advanced > SDWAN > Probing.

      3. If needed, scroll to down to the relevant section.

      4. Take a screenshot - press the "C" key.

      5. Press the CTRL+C keys to exit the CPView tool.

      6. Include the screenshot file "cpview_<PID>.cap<Number>"

        This file is created in the current working directory, in which you ran the cpview command.

        Example: cpview_25117.cap0

Symptoms:

After enabling SD-WAN "Overlay" on a Security Gateway, or after adding multiple SD-WAN VPN Peers:

  1. Output of the "top" command shows significant increase in the CPU load in the processes "fw_worker".

    CPView (sk101878) > CPU > Overview shows significant increase in the CPU load in the "CoreXL_FW" instances.

  2. Output of the "fw tab -t connections -u -s" command (see sk65133) shows significant increase in the number of concurrent connections (the #VALS column).

  3. Output of the "fwaccel conns" command shows that ICMP connections are forwarded to the Firewall kernel.

Cause:

  1. Each SD-WAN "Overlay" Peer sends ICMP Echo Request packets to probe the configured next hop (see SD-WAN Probing and Configuring Steering Behavior).

  2. When the value of the kernel parameter "fw_allow_simultaneous_ping" is set to "1" (one), the Security Gateway handles each ICMP Echo Request packet as a new ICMP connection. Therefore, the Security Gateway does not accelerate these ICMP connections in SecureXL, but forwards them to the Firewall kernel for complete inspection.

Important - Understand why the value of this kernel parameter was set to "1". There are cases when this value is required, so traffic sent directly to Cluster Members over a Site to Site VPN tunnel can reach them. See sk26874.

Solution:

  1. Connect to the command line on the Security Gateway.

  2. Log in to Gaia Clish or the Expert mode.

  3. Get the current value of the kernel parameter "fw_allow_simultaneous_ping":

    fw ctl get int fw_allow_simultaneous_ping

  4. If the current value of this kernel parameter is "1" (one), then set its value to "0" (zero) temporarily:

    fw ctl set int fw_allow_simultaneous_ping 0

  5. Examine the CPU load in one of these ways:

    • Run "cpview" > go to CPU > Overview > examine the CPU load in the "CoreXL_FW" instances.

    • Run "top" > examine the CPU load in the processes "fw_worker".

  6. To set the value of this kernel parameter to "0" (zero) permanently, run:

    fw ctl set -f int fw_allow_simultaneous_ping 0

Symptoms:

  • Traffic that should be routed through the VTI is not encrypted and is sent in clear-text.

  • Local outgoing connections from the SD-WAN Security Gateway's VTI to the remote VPN peer's VTI (for example, BGP traffic) are not encrypted and are sent in clear-text.

  • Kernel debug on the SD-WAN Security Gateway may show this message (when the debug flag "fw ctl debug -m VPN + policy" is enabled):

    init_encryption_params: Traffic accepted by a breakout rule - change to clear

Cause:

The traffic is matching a Local Breakout rule, which by design disables VPN encryption.

Solution:

Make sure that all VPN traffic matches an SD-WAN rule with the "Overlay" behavior and not with the "Internet > Local Breakout Only" behavior.