Troubleshooting traffic that does not match SD-WAN rules

Symptom:

  • In SmartConsole, the "Accept" / "Encrypt" log for the relevant connection shows no SD-WAN rule information, or an unexpected SD-WAN rule as the match.

  • On the Security Gateway in Expert mode, the "fw ctl zdebug -m SDWANRB + all | grep PROB" command shows the connection with "SD-WAN, no match", or an unexpected SD-WAN rule match.

  • The Security Gateway is enforcing the latest SD-WAN policy, as confirmed by cpsdwan stat.

  • In SmartConsole -> Infinity Services tab, the Configuration Sharing is enabled, it's status is Active, the last sync time is after the last session publish & SD-WAN policy was enforced after the last sync. For Configuration sharing issues or missing objects see Missing network objects in Check Point Portal SD-WAN.

Background:

The SD-WAN Internet object is a smart object that, from the gateway's perspective, excludes:

  • Reserved IP address ranges.

  • Directly connected networks.

  • Local VPN Domain.

  • Peer VPN Domains.

In other words Internet is "everything" minus reserved, local, and VPN-domain ranges.

Key behavior:

  • If the connection destination IP falls inside the excluded ranges table, the traffic will not match the SD-WAN Internet object.

  • If the connection source IP falls inside the SD-WAN Internet table, the traffic will not match the SD-WAN Internet object, even if the rule's Source column includes that source IP.

  • If the traffic is subject to Destination NAT, SD-WAN matching is done against the post-NAT (translated) destination of the connection.

Notes:

Solution:

  1. On-the-fly check: Is the destination IP address considered as "Internet"?

    1. To check if a specific destination IP address is treated as Internet by the Security Gateway, run in the Expert mode:

      fw ctl zdebug -m NRB + all | grep sdwanrb_is_ip_in_internet

    2. While this debug runs, initiate the problematic connection.

    You should see log lines showing the result of "sdwanrb_is_ip_in_internet" evaluation for the destination IP address.

  2. To see the content of the kernel tables used by the SD-WAN Internet object, run on the Security Gateway in the Expert mode:

    fw tab -t sdwan_internet_ranges1 -u

    fw tab -t sdwan_internet_ranges2 -u

    Only one of these tables should contain data .

    If the destination IP address that you expect to be considered "Internet" is not in the table (or the table is empty), proceed with the steps below.

  3. If the SD-WAN Internet Ranges table contain wrong data or is empty, run on the Security Gateway in the Expert mode:

    1. Start the SD-WAN Steering debug:

      fw debug sdwan_steering on TDERROR_SDWAN_SDWAN=3

    2. Fetch the SD-WAN policy again:

      cpsdwan fetch_local

    3. Return the SD-WAN steering debug level to the normal value:

      fw debug sdwan_steering on TDERROR_SDWAN_SDWAN=1

    4. To check the SD-WAN steering log for Internet ranges calculation:

      1. Open the $FWDIR/log/sdwan_steering.elg file.

      2. Search for: SDWInternetRanges

      3. Verify that the ranges make sense and include your expected Internet ranges.

    5. Collect these log files for Check Point Support:

      $FWDIR/log/sdwan_steering.elg*

To collect SD-WAN policy direction files:

Background:

  • "My VPN Domain" and "Peer VPN Domain" SD-WAN are dynamic objects that are populated from the kernel table "vpn_routing".

  • The kernel table "vpn_routing" is built from the VPN Domain configuration in the Security Gateway objects, on the VPN Community objects, as well as from definitions in the relevant *.def files on the Management Server.

Notes:

  • The Security Gateway runs a version that supports these dynamic objects.

  • For detailed behavior and version support, refer to Dynamic Objects in SD-WAN Policy.

  1. To check the content of these dynamic objects, run on the Security Gateway in the Expert mode:

    1. Show the list of all dynamic objects:

      dynamic_objects -l

    2. Check if a specific destination IP address is listed in any of these dynamic objects:

      dynamic_objects -ip <IP_ADDRESS>

    If the IP address is included in the kernel table "vpn_routing" but not included in the expected dynamic object (for example, "My_VPN_Domain" or "Peer_VPN_Domain"), proceed to the next step.

  2. To check the kernel table "vpn_routing", run on the Security Gateway:

    1. Show the entries in the kernel table:

      fw tab -t vpn_routing -u [-f]

    2. Verify if the problematic IP address is included in the "vpn_domain" range of the local Security Gateway, or the relevant remote SD-WAN peer.

    3. For remote peers, confirm if the peer is recognized as an SD-WAN peer:

      cpview

      Navigate to Advanced > SD-WAN > refer to Overlay probing results.

      If the peer is not identified as SD-WAN, its ranges will not be included in "Peer VPN Domain" dynamic object.

    4. If the IP address is missing from the right "vpn_domain" range in the kernel table "vpn_routing", change the VPN Domain configuration in SmartConsole for the local Security Gateway object, and/or the remote gateway / VPN Community.

    5. Install the Access Control policy.

    6. Check again.

    If the IP address is included in kernel table "vpn_routing" but not included in the expected dynamic object (for example, "My_VPN_Domain" or "Peer_VPN_Domain"), proceed to the next step.

  3. IP address is included in the kernel table "vpn_routing" but is missing from Dynamic Objects.

    Note - Global ranges that are not explicitly defined, but are only implicitly derived from the third option in the VPN Routing tab of the VPN Community, are not added to the Peer VPN Domain dynamic object.

    If the IP address is present in the kernel table "vpn_routing", but does not appear in the relevant dynamic object, run on the Security Gateway in the Expert mode:

    1. Start the SD-WAN steering debug:

      fw debug sdwan_steering on TDERROR_SDWAN_SDWAN=5

    2. Fetch the SD-WAN policy locally:

      cpsdwan fetch_local

    3. Return the SD-WAN steering debug level to the normal value:

      fw debug sdwan_steering on TDERROR_SDWAN_SDWAN=1

    4. Collect these log files for Check Point Support:

      $FWDIR/log/sdwan_steering.elg*

      Look for references such as "CreateDynamicObject" in the "sdwan_steering.elg" files.

  • Collect the directory with the SD-WAN policy direction files: $FWDIR/state/local/SDWAN

    You can inspect these files to look for the relevant IP range in the installed policy files.

    Note - IP addresses appear in policy files only when explicitly defined. IP addresses included via dynamic or special objects (for example, “My VPN Domain”, “Peer VPN Domain”, “SD-WAN Internet”) are translated on the gateway and therefore do not appear in the policy files.

    Download the original SD-WAN profile policy from Check Point Portal:

    1. From the left navigation panel, click Network.

    2. In the middle section, click Profiles.

    3. Click the relevant profile.

    4. Click Advanced tab.

    5. Click Download.

    Compare the SD-WAN policy on the Security Gateway with the profile in Check Point Portal.

  • Kernel Debug for SD-WAN Rule Matching:

    Use this when traffic does not match the expected SD-WAN rule and policy or object checks give no explanation.

    1. Connect to the command line on the Security Gateway.

    2. Reset any previous debug configuration:

      fw ctl debug 0

      fw ctl set int simple_debug_filter_off 1

      fwaccel dbg resetall

    3. Allocate the debug buffer:

      fw ctl debug -buf 8200

    4. Enable the required kernel debug flags:

      1. FW module:

        fw ctl debug -m fw + drop conn

      2. SDWAN module:

        fw ctl debug -m SDWAN all

      3. SDWANRB module:

        fw ctl debug -m SDWANRB all

      4. NRB module:

        fw ctl zdebug -m NRB all

      5. If the issue involves an overlay (VPN) connection, also enable VPN module:

        fw ctl debug -m VPN all

    5. Enable the SecureXL Debug.

      1. Default module:

        fwaccel dbg -m default all

      2. SDWAN module:

        fwaccel dbg -m sdwan all

      3. If Overlay / VPN is relevant:

        fwaccel dbg -m vpn all

      4. Optional – API module for extra details:

        fwaccel dbg -m api all

    6. Start the kernel debug:

      fw ctl kdebug -T -f > /var/log/kernel_debug.txt

    7. Reproduce the issue - initiate the connection that does not match the expected SD-WAN rule.

    8. Stop the kernel debug:

      Press the CTRL+C keys and run:

      fw ctl debug 0

      fwaccel dbg resetall

    9. Collect the debug output files.

      Provide these files to Check Point Support for further analysis.

      Also mention the connection details that was tested during the kernel debug (source, destination IP, port, protocol).

      1. Kernel debug output:

        /var/log/kernel_debug.txt

      2. SD-WAN steering logs:

        $FWDIR/log/sdwan_steering.elg

      3. SD-WAN policy files - the entire directory $FWDIR/state/local/SDWAN

      4. Name of the relevant SD-WAN profile in Check Point Portal.