Troubleshooting SD-WAN "Local Breakout Only"

Symptom:

Inbound NAT stopped working with SD-WAN.

Solution:

Capture the traffic on the Security Gateway with FW Monitor (sk30583).

Filter for the applicable traffic to see what is the routing behavior for the connections.

If the traffic capture shows that the packet enters the Security Gateway and the Security Gateway sends the packet back - towards the ISP Next Hop (instead of passing it through), then it is possible the connection matches a "Local Breakout" / "Backhaul" rule that marks the packet to be routed to an ISP.

Available options:

• Option 1:

  You can use the Dynamic Object "SD-WAN Internet" that makes sure to match the traffic against the destination after NAT, and not against the original destination.

  Support for the Dynamic Object "SD-WAN Internet" is available in:

  • R82 and higher

  • R81.20 Jumbo Hotfix Accumulator Take 79 and higher

  • Quantum Spark R81.10.15 and higher

  For information about this Dynamic Object "SD-WAN Internet", refer to the Configuring SD-WAN Policy.

• Option 2:

  In the SD-WAN Policy, make sure the traffic that is directed to your public NAT Pool IP addresses does not match a rule with a "Local Breakout" / "Backhaul" steering behavior.

  If such a rule is necessary in your SD-WAN Policy, then configure another rule above it to match the traffic to all your public NATed IP addresses with an "Overlay - VPN" action.

  As a result, because the traffic is not encrypted, the operating system would make the routing decision and not SD-WAN.

Symptom:

The "SD-WAN" section in the Security Gateway logs for connections the Security Gateway created (to Cloud / Infinity Portal applications / Smart-1 Cloud / the public IP address of the on-premises Management Server), shows an SD-WAN routing decision that perhaps is not the correct one.

Cause:

You can safely ignore the "SD-WAN" section in the Security Gateway logs for connections initiated by the Security Gateway.

SD-WAN does not enforce any action on such connections.

These logs just show the theoretical decision.

To see the outgoing interface, capture the traffic with FW Monitor (sk30583) or CPPCAP (sk141412).

Symptom:

SD-WAN routing decision overrides my directly connected networks / Gaia OS routes on the Security Gateway.

Cause:

This is by design.

If traffic matches an SD-WAN Policy rule with the Steering Behavior "Local Breakout Only" or "Backhaul Only", then the Security Gateway sends the traffic to the selected ISP, regardless of all the other routes.

If traffic matches an SD-WAN Policy rule with the Steering Behavior "Overlay - VPN", and the traffic is not encrypted to an SD-WAN peer, then the Security Gateway sends the traffic based on the active kernel routes.

Important - Make sure only the required traffic can match the SD-WAN Policy rules, especially with the Steering Behavior "Local Breakout" or "Backhaul".

Symptom:

CPView > Advanced > SD-WAN page > "Local Breakout" Probing Results section shows that the state of the "Local Breakout" target probing is "DOWN".

Cause:

This usually means that the Security Gateway did not receive the ICMP Echo Reply from the nexthop probing target.

Solution:

1. Make sure the state of the applicable interface is "UP" on the nexthop probing target.

2. Make sure the nexthop probing target is configured to respond to ARP Requests or to ICMP Echo Requests from the Security Gateway.

   Important - If the nexthop probing target does not respond to ARP Requests and ICMP Echo Requests (for example, it is currently down), then the Security Gateway does not try to probe the "Local Breakout" targets.

   The Security Gateway uses ARP Probing by default in these versions:

   • R82 and higher

   • R81.20 Jumbo Hotfix Accumulator Take 79 and higher

   • Quantum Spark R81.10.15 and higher

3. Connect to the command line on the Security Gateway.

4. Log in to the Expert mode.

5. Examine the state of the applicable ISP.

   Important - If the Security Gateway detects the state of the ISP as "DOWN", the Security Gateway does not try to probe the "Local Breakout" / Internet target for that ISP.

   1. Run:

      cpview

   2. Go to Advanced tab > SD-WAN page > Nexthop Probing Results section, and refer to the ISP State.

6. Continue the troubleshooting:

   • If the nexthop is "Down":

     1. Use the CPPCAP tool (sk141412) to capture the ICMP traffic between the Security Gateway (on the applicable interface) and the nexthop probing target.

        cppcap -i <Name of Interface> -f "icmp and host <IP Address of Nexthop Probing Target>" -o /var/log/cppcap_probing.cap

     2. Make sure the Security Gateway does not drop this ICMP traffic to and from the nexthop probing target:

        For more information about the kernel debug procedure on a Security Gateway, see the Security Gateway Guide for your version > chapter "Kernel Debug".

        fw ctl zdebug -m fw + drop

   • If the nexthop is "UP", but the "Local Breakout" probe is "DOWN":

     1. Use the CPPCAP tool (sk141412) to capture the ICMP traffic between the Security Gateway (on the applicable interface) and the "Local Breakout" probing target (such as 8.8.8.8).

        cppcap -i <Name of Interface> -f "icmp and host <IP Address of "Local Breakout" Probing Target>" -o /var/log/cppcap_probing.cap

     2. Make sure the Security Gateway does not drop this ICMP traffic to and from the "Local Breakout" probing target:

        fw ctl zdebug -m fw + drop

Symptom:

Infinity Portal > Quantum SD-WAN > Monitor view > Dashboard page does not show the SLA information from a specific Security Gateway. This issue occurs all the time, or randomly from time to time.

Solution:

Follow these steps on the Security Gateway:

1. Make sure the timezone, the date, and the time are configured correctly.

2. Check if the Security Gateway provides the SD-WAN information.

   In the Expert mode, run (this command is available in R82 and higher, see sk101878):

   cpview -mf SDWAN

   A JSON output must appear after several seconds.

3. Make sure the "sdwan_steering" process is running.

   Run:

   cpwd_admin list | grep -E "PID|SDWAN_STEERING"

   In the output:

   1. The APP column must show SDWAN_STEERING

   2. The STAT column must show "E" (executing)

   3. The #START column must show the digit "1" (started only one time)

4. Make sure the Nano-Agent is running without errors:

   cpnano -s

5. Make sure the disk partitions have free disk space:

   df -kh

   On a Quantum Spark Gateway, if the /tmp partition is full, then these options are available:

   • Temporary option:

     Reboot the Quantum Spark Gateway.

   • Permanent option:

     If the /tmp partition repeatedly gets full, then it is necessary to understand which files are created that fill this partition.

     As an immediate step, you can increase the site of the /tmp partition in Gaia Clish:

     1. Run:

        set temp-dir-size tmp-dir-size <Size in Megabytes>

        Example for 60 MB: set temp-dir-size tmp-dir-size 60

     2. Reboot:

        reboot

6. If the issue persists, then contact Check Point Support.

Symptoms:

• My nexthop device does not respond to pings, which brings the SD-WAN interface to the "Down" state for "Local Breakout"

• CPView > Advanced > SD-WAN page > Nexthop Probing Results section shows that the state of the SD-WAN interface is "DOWN".

Cause:

The Security Gateway probes the next hop of all SD-WAN interfaces. If a next hop is not responding to the ICMP Echo Request, the Security Gateway considers the state of an SD-WAN interface as "DOWN" (from theSD-WAN perspective). As a result, the Security Gateway does not use this SD-WAN interface for SD-WAN steering, and does not to try to probe the "Local Breakout" targets through this SD-WAN interface.

Important - The Security Gateway uses ARP Probing by default in these versions: R82 and higher R81.20 Jumbo Hotfix Accumulator Take 79 and higher Quantum Spark R81.10.15 and higher

Solution

This solution applies to:

• Security Gateway versions that use only ICMP Probing.

• Configurations, in which the nexthop device does not respond to ICMP Probing / ARP Probing and requires a static ARP entry on the Security Gateway.

Steps:

1. Connect to the command line on the Security Gateway.

2. Log in to the Expert mode.

3. If the nexthop device should respond, make sure the Security Gateway learned the MAC Address for this next hop:

   arp -an

4. Use the CPPCAP tool (sk141412) to capture the ICMP and ARP traffic between the Security Gateway (on the applicable interface) and the nexthop device.

5. Make sure the Security Gateway does not drop this ICMP traffic to and from the nexthop device:

   For more information about the kernel debug procedure on a Security Gateway, see the Security Gateway Guide for your version > chapter "Kernel Debug".

   fw ctl zdebug -m fw + drop

6. Configure the Security Gateway to skip the nexthop probing.

   You can disable the nexthop probing completely, so the SD-WAN can still probe "Local Breakout" targets on that SD-WAN interface, regardless of the nexthop device.

   1. If the file $FWDIR/conf/sdwan/sdwan_steering_params.json exists, create a backup copy.

      Otherwise, create this file.

      Run this long command:

      if [ -f $FWDIR/conf/sdwan/sdwan_steering_params.json ] ; then echo "Creating the backup ..." ; cp -v $FWDIR/conf/sdwan/sdwan_steering_params.json{,BKP} ; else echo "Creating the file ..." ; touch $FWDIR/conf/sdwan/sdwan_steering_params.json ; stat $FWDIR/conf/sdwan/sdwan_steering_params.json ; fi

   2. Edit the $FWDIR/conf/sdwan/sdwan_steering_params.json file:

      vi $FWDIR/conf/sdwan/sdwan_steering_params.json

   3. Add these lines:

      { "sdw_nexthop_probe_enabled" : 0 }

   4. Save the changes in the file and exit the editor.

   5. Restart the SD-WAN service:

      sdwan_steering_stop ; sdwan_steering_start