HyperFlow

Overview

Elephant flows are large (in total number of bytes) continuous connections that the TCP or UDP establishes.

For example, a download of a large file (such as a Linux ISO file) over the HTTP, HTTPS, FTP, or NFS protocol.

These large continuous connections consume the network capacity significantly in comparison to other types of data sessions.

Without the HyperFlow feature, a Security Gateway Dedicated Check Point server that runs Check Point software to inspect traffic and enforce Security Policies for connected network resources. uses only one CPU core (one CoreXL Performance-enhancing technology for Security Gateways on multi-core processing platforms. Multiple Check Point Firewall instances are running in parallel on multiple CPU cores. Firewall instance) to inspect one elephant connection. In addition, traffic throughput decreases gradually as the CPU utilization increases on the Security Gateway.

The HyperFlow feature on Security Gateways R81.20 and higher handles such elephant connections on more than one CPU core in parallel.

The HyperFlow feature breaks the whole inspection task into smaller tasks and dispatches these smaller tasks to the available CPU cores:

The tasks without the HyperFlow	The tasks with the HyperFlow
Packet retrieval Inbound Streaming Protocol parsers Context Management Interface (1) Interface on a Gaia Security Gateway or Cluster member, through which Management Server connects to the Security Gateway or Cluster member. (2) Interface on Gaia computer, through which users connect to Gaia Portal or CLI. / Infrastructure (CMI) Pattern Match (PM) and Hash (MD5, SHA) Software Blade Specific security solution (module): (1) On a Security Gateway, each Software Blade inspects specific characteristics of the traffic (2) On a Management Server, each Software Blade enables different management capabilities. logic Outbound Streaming Routing Packet transmission	Inbound processing in CoreXL Firewall: Packet retrieval Inbound Streaming Protocol parsers Context Management Interface / Infrastructure (CMI) Internal PPE processing (on many CPU cores): Pattern Match (PM) and Hash (MD5, SHA) Packet transmission Outbound processing in CoreXL Firewall: Software Blade logic Outbound Streaming Routing

The tasks without the HyperFlow

The tasks with the HyperFlow

Packet retrieval
Inbound Streaming
Protocol parsers
Context Management Interface (1) Interface on a Gaia Security Gateway or Cluster member, through which Management Server connects to the Security Gateway or Cluster member. (2) Interface on Gaia computer, through which users connect to Gaia Portal or CLI. / Infrastructure (CMI)
Pattern Match (PM) and Hash (MD5, SHA)
Software Blade Specific security solution (module): (1) On a Security Gateway, each Software Blade inspects specific characteristics of the traffic (2) On a Management Server, each Software Blade enables different management capabilities. logic
Outbound Streaming
Routing
Packet transmission

Inbound processing in CoreXL Firewall:
1. Packet retrieval
2. Inbound Streaming
3. Protocol parsers
4. Context Management Interface / Infrastructure (CMI)
Internal PPE processing (on many CPU cores):
1. Pattern Match (PM) and Hash (MD5, SHA)
2. Packet transmission
Outbound processing in CoreXL Firewall:
1. Software Blade logic
2. Outbound Streaming
3. Routing

As a result, the HyperFlow feature:

Increases throughput of elephant connections when Threat Prevention Software Blades are enabled (the Security Gateway takes less time to inspect elephant connections).

This is possible only if the network infrastructure is not a "bottleneck".
Automatically detects and dynamically allocates the CPU cores between main tasks on a Security Gateway.
Improves response time from the CoreXL FWK processes while they inspects elephant connections (the idle time of the corresponding CPU cores increases).

Important:

By default, the HyperFlow feature is enabled on Check Point Appliances that meet the requirements.
By design, the HyperFlow feature works only in the User Space Firewall (USFW).
By design, the HyperFlow feature engages only when needed, and when the total CPU load allows it.

The total throughput has priority over elephant connections.

Notes:

By design, a manual allocation of CPU cores is not necessary. Therefore, it is not possible.

You can configure thresholds to control when HyperFlow is active or passive.
By default, HyperFlow works in the standby mode.

HyperFlow is triggered (becomes active) when a heavy connection is detected.

HyperFlow becomes passive when the heavy connection is closed.

For additional information, see sk178070.

Watch the video:

Requirements

Check Point Appliance models with at least 8 CPU logical cores.

For the list of supported models, see sk178070.
Firewall in User Mode (USFW). See sk167052.

Enable the CoreXL Dynamic Balancing (see Dynamic Balancing of CoreXL Instances):

dynamic_split –o enable

Configure SecureXL Check Point product on a Security Gateway that accelerates IPv4 and IPv6 traffic that passes through a Security Gateway. to work in Kernel Mode (KPPAK) (see Configuring SecureXL).
Enable the applicable Software Blades from one of these categories:

Glossary

Term

Description

CoreXL_FW

A CoreXL Firewall instance that handles the traffic concurrently.

Each CoreXL Firewall instance is independent and replicated multiple times (see CoreXL).

Each replicated CoreXL Firewall instance runs on one processing CPU core.

Note - CPView shows this string on the CPU > Overview > Host page > in the column Type.

CoreXL_SND

A CoreXL Secure Network Distributor (SND) responsible for:

Processing incoming traffic from the network interfaces
Securely accelerating authorized packets (when SecureXL is running)
Distributing non-accelerated packets among FW instances

Note - CPView shows this string on the CPU > Overview > Host page > in the column Type.

CoreXL_FW_RESERVED

A logical sibling of a CoreXL Firewall instance (FW worker) that handles a heavy connection.

When a logical CPU core is utilized at a high level because it handles heavy connections, its logical sibling can be stopped to decrease its utilization of resources from the physical CPU core.

Chain on events:

CoreXL assigns a CoreXL Firewall instance to inspect an elephant connection.
This CoreXL Firewall instance runs on a logical CPU core of a physical CPU.
To improve the internal performance of the physical CPU, the CoreXL Dynamic Balancing feature can stop the CoreXL Firewall instances on the sibling logical CPU cores of the original logical CPU core.

PPE

Parallel Processing Engine architecture.

This is the HyperFlow dynamic infrastructure that allocates CPU cores as required to increase the throughput of Elephant connections.

This is the thread that polls the interface queues and retrieves packets.

Also known as Dual Mode Job Dispatcher (DMD).

PPE_MGR

Parallel Processing Engine Manager.

Receives packet payload and dispatches jobs to PPE.

Works on a complete physical CPU core.

Note - CPView shows this string on the CPU > Overview > Host page > in the column Type.

This string appears in CPView only when the HyperFlow is enabled, and an elephant connection passes through the Security Gateway.

PPE_MGR_RESERVED

A sibling of PPE_MGR.

Because PPE_MGR works on a complete physical CPU core, it is not possible to use other logical CPU cores on that physical CPU core. The logical siblings have the status "PPE_MGR_RESERVED"

Note - CPView shows this string on the CPU > Overview > Host page > in the column Type.

This string appears in CPView only when the HyperFlow is enabled, and an elephant connection passes through the Security Gateway.

PPE_WT

Parallel Processing Engine Worker Thread.

Receives packet handling jobs from the Parallel Processing Engine Manager.

Dispatches jobs to Worker Threads (WTs.)

Works on a complete physical CPU core.

Note - CPView shows this string on the CPU > Overview > Host page > in the column Type.

This string appears in CPView only when the HyperFlow is enabled, and an elephant connection passes through the Security Gateway.

DPDK

Data Plane Development Kit.

PMD

Poll Mode Driver.

Worker Thread.

The thread that executes the packet handling logic.

In plural: WTs

Syntax

Important:

You must run this command in the Expert mode.
In a Cluster Two or more Security Gateways that work together in a redundant configuration - High Availability, or Load Sharing., you must configure all the Cluster Members in the same way.
On Scalable Platforms (Maestro and Chassis), you must connect to the applicable Security Group A logical group of Security Appliances (in Maestro) / Security Gateway Modules (on Scalable Chassis) that provides Active/Active cluster functionality. A Security Group can contain one or more Security Appliances / Security Gateway Modules. Security Groups work separately and independently from each other. To the production networks, a Security Group appears a single Security Gateway. In Maestro, each Security Group contains: (A) Applicable Uplink ports, to which your production networks are connected; (B) Security Appliances (the Quantum Maestro Orchestrator determines the applicable Downlink ports automatically); (C) Applicable management port, to which the Check Point Management Server is connected..

{connection_pipelining | g_connection_pipelining}

advanced

on

off

heaviest_conn

pipelined

status

{connection_pipelining | g_connection_pipelining} advanced

allow_accelerated_pipeline

async

default

prevent_accelerated_pipeline

sleep

sync

wake_up

Parameters

Parameter

Description

No Parameters

Shows the built-in help.

connection_pipelining

Must enter this command only on Security Gateways other than Scalable Platforms.

g_connection_pipelining

Must enter this command only on Scalable Platforms.

advanced

Shows the advanced options.

allow_accelerated_pipeline

Allows new connections to be opened as accelerated pipeline connections - the Security Gateway uses the new asynchronous parsers for connections.

This is the default.

Notes:

This command applies only to elephant connections that opened after you run this command.
When you run this command, the Security Gateway deletes all SecureXL Connection Templates with this command:

fw tab -t cphwd_tmpl -x -y
This command does not require a reboot.

async

Configures the asynchronous flow mode (this is the default).

In this mode, CoreXL Firewall instances send jobs to the PPE.

Notes:

This command applies to existing elephant connections.
This command does not require a reboot.

default

Restores default settings

heaviest_conn

Shows the statistics for the heaviest connection with the maximum duration (number of packets and bytes).

off

Disables the feature.

Important - This change requires a reboot.

on

Enables the feature.

This is the default (on Check Point Appliances that meet the requirements).

Important - This change requires a reboot.

pipelined

Shows the accelerated elephant connections in the pipeline.

prevent_accelerated_pipeline

Prevents new connections from being opened as accelerated pipeline connections.

In this mode, the Security Gateway uses the legacy parsers for connections.

Notes:

Use this command only to troubleshoot issues with elephant connections.
This command applies only to elephant connections that opened after you run this command.
This command does not require a reboot.
When you run this command, the Security Gateway deletes all SecureXL Connection Templates with this command:

fw tab -t cphwd_tmpl -x -y

sleep

Configures the Job Dispatcher (PPE) and Working Threads (WTs) to sleep.

Notes:

Use this command only to troubleshoot issues with elephant connections, if the issue persists after you configured the synchronous flow mode.
This command does not require a reboot.

status

Shows the status and configuration of the feature.

The output shows these lines with the applicable values:

Status of connection pipelining: <Status>
Flow mode: <Mode>
Status of PPE_MGR and PPE_WT: <Status>
Status of accelerated pipeline: <Status>

sync

Configures the synchronous flow mode.

In this mode, CoreXL Firewall instances do not send jobs to the PPE.

Notes:

Use this mode only to troubleshoot issues with elephant connections.
This command applies to existing elephant connections.
This command does not require a reboot.

wake_up

Wakes up the Job Dispatcher (PPE) and Working Threads (WTs) from sleep.

The PPE gets CPU cores available for allocation.

Monitoring in CPView

You can monitor how the HyperFlow performs on the Security Gateway.

Viewing the CPU load from specific elephant connections

On the Security Gateway, examine the elephant connections in the past 24 hours and which CoreXL Firewall instance inspects them (see the "[fw_<Number>]" in the beginning of each line):

fw ctl multik print_heavy_conn

Example:

;[fw_5]: Conn: 192.168.10.20:60478 -> 172.30.40.50:80 IPP: 6; Instance load: 63%; Connection instance load: 99%; ...<truncated>...

In CPView, go to CPU > Top-Connections > Instances<X>-<Y> (for example, Instances0-5) > Instance<Z> (for example, Instance5)

In the last section Top Connections, examine the columns "% out of CPU" and "% out of WT CPU"

Example (this is only the applicable part of the output):

Top Connections
 
Connection                                 Protocol   % out of CPU % out of WT CPU
192.168.10.20:60478 -> 172.30.40.50:80     TCP:http           70.81%        48.97%

Viewing the status of the PPE

Go to Advanced > HyperFlow > Overview > PPE_0
Click the applicable tab

Examine these sections:

Section	Gauge	Description
PPE overview	PPE state	Shows the state of the feature - Active or Asleep
Pipeline status	Free	The number of free slots in the pipeline (maximum is 320)
	Active	Slots that PPE is currently using
	Job pending	Slots whose execution depends on another job
	Slot pending	Jobs whose execution depends on an available slot in the pipeline
Overload indicators	No pipeline entry	PPE_MGR had no free slots in the pipeline
	WT slot unavailable	The WTs have no available slots

Viewing the PPE messages from the Firewall

Go to Advanced > HyperFlow > Firewall-messages

Examine these sections:

Section	Gauge	Description
Sessions and data	New session	Number of new opened sessions
	Update session	Number of updated session (for example, because of policy installation)
	End session	Number of sessions that ended
	Current session	Number of currently opened sessions
	Data	Number of received data buffers
Errors		Various internal PPE errors
Messages received in a single read loop (Histogram)		In each loop, the PPE_MGR can read up to 32 items from the receiving queue. This section shows a histogram of the times the PPE_MGR read items from the queue and how many were "waiting" in the queue (up to 32)

Limitations

Firewall in Kernel Mode (KFW) is not supported.
Open Servers and Virtual Machines are not supported.
Cloud platforms are not supported.
HyperFlow is not supported when the Security Gateway / Cluster is configured to work as an HTTP/HTTPS Proxy.
When an elephant connection triggers the HyperFlow, the outputs of the "top" and "ps" commands may show that the HyperFlow user space processes consume some CPU cores at 99-100%.

This happens because HyperFlow is constantly polling its queues to handle the incoming jobs. After the elephant connection closes, the output of these commands shows that the user space "us" consumption goes back to regular levels because HyperFlow stops processing the jobs.

To see the actual load on the CPU, use CPView (CPU > Overview > Host), SNMP, or SmartConsole Check Point GUI application used to manage a Check Point environment - configure Security Policies, configure devices, monitor products and events, install updates, and so on..

This does not trigger inspection bypass because of a high CPU load.
When the HyperFlow feature is enabled, the speed of an elephant flow may not increase in these cases:
- This is a complex connection that requires special parsers – CIFS, SMB, FTP, SCP, VoIP
- This connection undergoes the Content Awareness Check Point Software Blade on a Security Gateway that provides data visibility and enforcement. See sk119715. Acronym: CTNT. or a Data Loss Prevention Check Point Software Blade on a Security Gateway that detects and prevents the unauthorized transmission of confidential information outside the organization. Acronym: DLP. policy
- This connection has dynamic content and goes through the Pattern Matcher 1st tier
- Strict Hold is enabled (the $FWDIR/conf/malware_config file on the Security Gateway contains "strict_hold_enable=1")
- This connection is HTTPS and the HTTPS Inspection is disabled
When you enable only the Firewall Software Blade in the Security Gateway object, the HyperFlow feature does not improve the performance.

This is because SecureXL accelerates connections that do not undergo the inspection, while the HyperFlow accelerates connections that undergo the inspection by various Software Blades.

Troubleshooting

Log files (the main file is rotated every 10 megabytes):
- $FWDIR/log/dsd.elg
- $FWDIR/log/dmd.elg
- $FWDIR/log/dmd_controller.elg
- $FWDIR/log/connection_pipelining.elg
Internal configuration files (you must not edit these files):
- $FWDIR/conf/connection_pipelining.conf
- $FWDIR/conf/connection_pipelining_params.conf

For additional information, see sk178070.