Manual Configuration of Alerts
To configure alerts manually, refer to the Grafana documentation for setting up alerts.
Use this checklist when creating the same alerts manually from the Grafana Web UI instead of provisioning the YAML file.
-
Create or select the SD-WAN folder and the SD-WAN evaluation group.
Use a 1-minute evaluation interval unless the customer requires a different interval.
-
For each alert rule, select the Prometheus data source that receives Skyline SD-WAN metrics.
-
Paste the PromQL query from the alert package into query A.
Keep Instant enabled for rules that evaluate the current state.
-
Add expression C as a Threshold expression that evaluates query A.
Use the same threshold value from the YAML rule.
-
Add labels Importance and Severity with the values mentioned in the alert below.
These labels are used by the dashboard alert counters and by notification routing.
-
Add the same summary, description, and runbook URL from the YAML.
-
Select an existing contact point or notification policy receiver.
-
Link the rule to the dashboard and panel only after the dashboards are imported in the target environment and the dashboard UID and panel ID are known.
Configuration in Grafana Web UI
-
Open the Grafana Web UI.
-
Navigate to Alerts > Alerts Rules.
-
In the top right corner, click New alert rule.
-
In Step 1, enter the alert name.
-
Select your data source.
-
Configure the alert query and expression.
-
In Step 4, create a folder or assign the alert to an existing folder.
-
In Step 5, either select an existing or create new evaluation Group which defines the sample rate for all your alerts in the group.
Configure the applicable pending mode.
-
Give the alert a summary and attach it to your dashboard widgets if needed.
Example Configuration Queries for Alerts
Query for alert - Security Gateway is "Down"
Configuration
PromQL syntax for the query:
|
|
Expression:
Expression C:
-
Type: Threshold -
Input: A -
Is above: 0
Alert Summary:
No telemetry received from the Security Gateway {{ $labels.host_name }} in the last 5 minutes. The Security Gateway may be down or unreachable.
Description:
Why you see this alert:
The Security Gateway {{ $labels.host_name }} was up in the last 5 days but not sending telemetry over the past 3 minutes.
Remediation:
-
Verify the Security Gateway is powered on.
-
Check connectivity, modem status and reachability with ping / SSH.
-
Check connectivity and skyline configuration between the Security Gateway and the monitoring system.
-
Contact the ISP Support or Check Point Support based on the results of previous steps.
Labels:
-
Importance: Critical
-
Severity: Critical
Query for alert - ISP "Down"
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above or equal to: 1
Alert Summary:
SD-WAN detected that an ISP is unreachable based on nexthop probes.
This may be caused by a physical link failure, upstream outage, or unresponsive next hop.
Description:
Why do I see this alert:
SD-WAN detected that ISP {{ $labels.isp_name }} on the Security Gateway {{ $labels.host_name }} is unreachable based on nexthop probes.
This may be caused by a physical link failure, upstream outage, or unresponsive next hop.
Remediation:
-
Verify interface and modem status on the Security Gateway.
-
Review recent changes in the ISP link configuration (speed/duplex, VLANs, etc.).
-
Follow the ISP troubleshooting steps.
-
If the link remains down, contact the ISP Support or Check Point Support.
Labels:
-
Importance: Critical
-
Severity: Critical
Query for alert - Link QOE is low
Configuration
PromQL syntax for the query:
|
|
Expression:
Is below: 3.8
Alert Summary:
WAN link QOE score is lower than 3.8, which indicates degraded performance.
Description:
Link QOE score for the ISP {{ $labels.isp_name }} on the Security Gateway {{ $labels.host_name }} is {{ $values.A.Value }}, below the threshold of 3.8.
Remediation:
-
Check for high traffic volume, latency, jitter, or packet loss on the interface.
-
If degradation persists, open a ticket with the ISP and provide the time/metrics.
Labels:
-
Importance: High
-
Severity: High
Query for alert - VPN tunnel is "Down" while ISP link is "UP"
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 0
Alert Summary:
On the Security Gateway, the Overlay VPN tunnel from the local ISP to the peer ISP is down, although the ISP link remains operational.
Description:
Why Do you see the Alert:
SD-WAN detected that the ISP {{ $labels.vpn_isp }} on the Security Gateway {{ $labels.host_name }} is operational, but the Overlay VPN tunnel endpoint is not responding to SD-WAN overlay probing.
This can be a result of configuration mismatches, peer-side issues, or encryption failures.
Remediation:
-
Check interface / VPN configuration changes.
-
Follow the applicable troubleshooting steps.
If the tunnel remains down, contact Check Point Support.
Labels:
-
Importance: Low
-
Severity: Low
Query for alert - All public ISPs exceeded thresholds of one steering object
Configuration
PromQL syntax for the query:
|
|
Expression:
Is equal to: 1
Alert Summary:
On the Security Gateway (check the label to see the Security Gateway name), all public ISPs assigned to the steering object {{ $labels.steering_policy }} exceeded the configured performance thresholds.
Description:
Why do you see this alert:
SD-WAN detected that every public ISP link assigned to this steering object {{ $labels.steering_policy }} on the Security Gateway {{ $labels.host_name }} exceeded one or more thresholds (latency, packet loss, or jitter).
No healthy public ISP path was available during this time.
Remediation:
-
Investigate local network metrics and utilization, Security Gateway resource utilization, and last-mile connectivity.
-
Check the probing target / application.
-
Contact the ISP Support if degradation persists.
Labels:
-
Importance: Medium
-
Severity: Medium
Query for alert - All public ISPs exceeded thresholds for multiple steering objects
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 1
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, all public ISPs exceeded performance thresholds across multiple steering objects.
Remediation:
-
Review the Security Gateway health (CPU, drops, link utilization, shaping) and local network conditions.
-
Review ISP metrics and recent changes.
-
Contact the ISP Support if degradation persists.
Labels:
-
Importance: High
-
Severity: High
Query for alert - ISP exceeded threshold on one steering object
Configuration
PromQL syntax for the query:
|
|
Expression:
Is equal to: 1
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, the ISP {{ $labels.isp_name }} exceeded the configured threshold for one steering object.
Description:
Why you see this alert:
SD-WAN continuously monitors ISP performance. This alert was generated because one of the metrics in the ISP {{ $labels.isp_name }} on the Security Gateway {{ $labels.host_name }} exceeded the configured threshold for the specified steering object.
Remediation:
-
Review current and historical ISP performance metrics.
-
Verify whether the issue is temporary or persistent.
-
Check application or probing target for issues.
-
If degradation continues beyond a short time window, contact the ISP.
Labels:
-
Importance: Low
-
Severity: Low
Query for alert - ISP exceeded thresholds for multiple steering objects
Configuration
PromQL syntax for the query:
Query A:
|
|
Query B:
|
|
Expression:
-
Expression C.
Type: ThresholdInput: AIs above: 1
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, the ISP {{ $labels.isp_name }} exceeded configured thresholds in multiple steering objects.
Description:
Why you see this alert:
SD-WAN detected threshold violations for the same ISP {{ $labels.steering_policy }} on the Security Gateway {{ $labels.host_name }} across multiple steering objects within short time.
This pattern indicates a shared ISP issue.
Remediation:
-
Review ISP metrics, link utilization, and recent changes.
-
Contact the ISP Support if degradation persists.
Labels:
-
Importance: Medium
-
Severity: Medium
Query for alert - ISP state changes frequently
Configuration
Scenario:
When the status of an ISP link changes from Down to Up, or from Up to Down, more than 5 times in 1 last hour.
PromQL syntax for the query:
|
|
Expression:
Is above: 5
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, the ISP {{ $labels.isp_name }} experienced frequent UP / DOWN transitions during the monitored time window.
Description:
Why you see this alert:
SD-WAN detected that the ISP {{ $labels.isp_name }} on the Security Gateway {{ $labels.host_name }} is unreachable based on nexthop probes.
This may be caused by a physical link failure, upstream outage, or unresponsive next hop.
Remediation:
-
Review link flap timing and correlate with environmental or power outages.
-
Verify cabling, modem health, and local infrastructure.
-
If instability continues, contact the ISP Support.
Labels:
-
Importance: High
-
Severity: High
Query for alert - Overlay VPN tunnel is down (ISP down)
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 0
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, VPN tunnels that are using the ISP {{ $labels.vpn_isp }} are down due to ISP unavailability.
Description:
Why do you see the Alert:
SD-WAN detected that the ISP {{ $labels.vpn_isp }} is down on the Security Gateway {{ $labels.host_name }}.
This causes all Overlay VPN tunnels routed through this ISP to go down.
Remediation:
-
Restore ISP connectivity.
-
Follow the applicable troubleshooting steps.
-
If the tunnels remain down after recovery, contact Check Point Support.
Labels:
-
Importance: Medium
-
Severity: Medium
Query for alert - All Overlay VPN tunnels to a peer are down
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 0
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, all Overlay VPN tunnels to the peer {{ $labels.vpn_main_peer_ip }} with the peer IP {{ $labels.vpn_peer_ip }} are down.
Description:
Why do you see the Alert:
SD-WAN detected that all Overlay VPN tunnels to the specified peer {{ $labels.vpn_main_peer_ip }} with the peer IP {{ $labels.vpn_peer_ip }} on the Security Gateway {{ $labels.host_name }} are unreachable.
Remediation:
-
Check remote gateway status.
-
Check VPN configuration.
-
Follow troubleshooting steps in the Administration Guide.
-
If the tunnel remains down, contact Check Point Support.
Labels:
-
Importance: Critical
-
Severity: Critical
Query for alert - Overlay VPN tunnel state changes frequently
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 5
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, the Overlay VPN tunnel from the local ISP {{ $labels.vpn_isp }} to the peer IP {{ $labels.vpn_peer_ip}} experienced frequent UP / DOWN transitions during the monitored time window.
Description:
Why you see this alert:
On the Security Gateway, the Overlay VPN tunnel from the local ISP {{ $labels.vpn_isp }} to the peer IP {{ $labels.vpn_peer_ip }} changed the status at least 6 times over the last day, indicating unstable connectivity.
Remediation:
-
Review timing of the flapping events, verify link statuses, cabling and modem health, and assess environmental or power-related factors.
-
Follow the applicable troubleshooting steps.
-
If the issue continues, contact Check Point Support.
Labels:
-
Importance: High
-
Severity: High
Query for alert - High CPU utilization
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 75
Alert Summary:
CPU utilization on the Security Gateway is greater than 75%.
Description:
Why you see this alert:
CPU utilization on the Security Gateway {{ $labels.host_name }} is greater than 75%.
Remediation:
-
Check for spikes in traffic volume or new large flows.
-
Review CPU utilization by processes on the Security Gateway.
-
Consider optimizing security policies, inspection profiles, or logging if relevant.
-
If the issue persists, evaluate the Security Gateway capacity (scale-up / scale-out).
Labels:
-
Importance: High
-
Severity: High
Query for alert - High memory utilization
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 75
Alert Summary:
Memory utilization on the Security Gateway is greater than 75%.
Description:
Why you see this alert:
Memory utilization on the Security Gateway {{ $labels.host_name }} is greater than 75%.
Remediation:
-
Review memory utilization by processes on the Security Gateway.
-
Review logs for memory-related errors or crashes.
-
Restart non-critical services during maintenance windows if needed.
-
If the issue persists, plan for software upgrade or hardware resource increase.
Labels:
-
Importance: High
-
Severity: High
Query for alert - WAN link exceeded thresholds for more than 10% of time
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 0
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, the WAN link {{ $labels.isp_name }} exceeded the threshold for the steering object {{ $labels.steering_policy }} during more than 10% of the monitored time window.
Description:
SD-WAN detected that the WAN link {{ $labels.isp_name }} on the Security Gateway {{ $labels.host_name }} exceeded one or more thresholds for the steering object {{ $labels.steering_policy }} during more than 10% of the monitored window. This indicates persistent degradation rather than a short spike.
Labels:
-
Importance: Medium
-
Severity: Medium
Query for alert - WAN link was down for more than 10% of time
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 0
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, the WAN link {{ $labels.isp_name }} was down during more than 10% of the monitored time window.
Description:
SD-WAN detected that the WAN link {{ $labels.isp_name }} on the Security Gateway {{ $labels.host_name }} was down for more than 10% of the monitored window.
This indicates unstable WAN connectivity rather than a single isolated failure.
Labels:
-
Importance: High
-
Severity: High
Query for alert - All overlay tunnels to multiple peers are down.
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 1
Alert Summary:
All overlay tunnels to several peers are down within the same time window on the Security Gateway {{ $labels.host_name }}, affecting {{ $values.A.Value }} peers.
Description:
All overlay tunnels to several peers are down within the same time window on the Security Gateway {{ $labels.host_name }}, affecting {{ printf "%.0f" $values.A.Value }} peers.
This alert is triggered when SD-WAN detects concurrent overlay tunnel failures to multiple peers. This may indicate a local Security Gateway issue, WAN connectivity problem, or shared infrastructure failure.
Remediation:
-
Follow the applicable troubleshooting steps.
-
Verify WAN connectivity and tunnel status for the affected Security Gateway.
-
If the tunnels remain down, contact Check Point Support.
Labels:
-
Importance: High
-
Severity: High
Query for alert - WAN link is frequently breaching and recovering from steering threshold
Configuration
PromQL syntax for the query:
|
|
Expression:
Is above: 10
Alert Summary:
On the Security Gateway {{ $labels.host_name }}, the WAN link {{ $labels.isp_name }} is frequently exceeding (and then recovering) the acceptable threshold for the steering object {{ $labels.steering_policy }}.
Description:
On the Security Gateway {{ $labels.host_name }}, the WAN link {{ $labels.isp_name }} exceeded the threshold state {{ printf "%.0f" $values.A.Value }} times in the last 24 hours for the steering object {{ $labels.steering_policy }}.
Why you see this alert:
This alert is triggered when SD-WAN detects at least 3 threshold state changes within 24 hours. The link may be unstable, repeatedly moving between breaching and recovering from the acceptable steering threshold because latency, jitter, or packet loss crosses and then recovers from the configured threshold.
Remediation:
-
On the Security Gateway {{ $labels.host_name }}, check the WAN link quality for the ISP {{ $labels.isp_name }}.
-
Review latency, jitter, and packet-loss behavior for the steering object {{ $labels.steering_policy }}.
-
Verify whether the ISP link is experiencing intermittent degradation, congestion, or packet loss.
-
Check for physical link issues, modem/router instability, ISP maintenance, or upstream provider problems. 5.
If the issue continues, contact the ISP Support or Check Point Support.
Labels:
-
Importance: High
-
Severity: High