Advanced Multi-Queue settings

Description

Advanced Multi-Queue settings include:

• Controlling the number of queues
• IRQ Affinity
• Viewing the CPU utilization

To see the current number of active RX queues:

On the Security Gateway, run:

cpmq get rx_num igb ixgbe i40e mlx5_core

To configure the specified number of RX queues:

The number of RX queues depends on the interface driver:

Interface Driver	Queues	Recommended number of RX queues
igb	When you configure the Multi-Queue for an igb interface, it calculates the number of TX and RX queues based on the number of active RX queues. Note - The number of queues for the on-board interfaces (Mgmt and Sync) on Check Point appliances is limited to just two queues (hardware restriction).	4
ixgbe	When you configure the Multi-Queue for an ixgbe interface, it creates an RxTx queue for each CPU core. You can control the number of active RX queues with this command: cpmq set rx_num ixgbe {default | <Value>} All TX queues are active.	16
i40e	When you configure the Multi-Queue for an i40e interface, it calculates the number of TX and RX queues based on the number of active RX queues with a maximum queue value set to 14.	14
mlx5_core	When you configure the Multi-Queue for an mlx5_core interface, it calculates the number of TX and RX queues based on the number of active RX queues with a maximum queue value set to 10.	10

Notes:

• By default, Security Gateway calculates the number of active RX queues based on this formula:

Active RX queues = (Number of CPU cores) - (Number of CoreXL FW instances)

• By default, VSX Gateway calculates the number of active RX queues based on this formula:

Active RX queues = The lowest CPU ID, to which an FWK process is assigned

On the Security Gateway, run:

cpmq set rx_num {igb | ixgbe | i40e | mlx5_core} <Number of Active RX Queues>

To configure the recommended number of RX queues:

On a Security Gateway, the number of active queues changes automatically when you change the number of CoreXL FW instances in the cpconfig menu.

The number of active RX queues does not change, if you configure the number of RX queues manually.

On the Security Gateway, run:

cpmq set rx_num {igb | ixgbe | i40e | mlx5_core} default

IRQ Affinity of the RX and TX queues:

The Security Gateway configures the IRQ affinity of the queues automatically when it boots.

The configuration depends on the number of CPU cores.

Examples:

SMT on Appliance	Example
SMT (HyperThreading) is disabled	If you configured rx_num to 3 on an appliance with 4 CPU cores: rxtx-0 -> CPU 0 rxtx-1 -> CPU 1 rxtx-2 -> CPU 2 rxtx-3 -> CPU 3 This is also true in cases, where you assign the RX and TX queues with a separated IRQ: rx-0 -> CPU 0 tx-0 -> CPU 0 rx-1 -> CPU 1 tx-1 -> CPU 1 and so on.
SMT (HyperThreading) is enabled (see sk93000)	If you configured rx_num to 3 on an appliance with 8 CPU cores: rxtx-0 -> CPU 0 rxtx-1 -> CPU 4 rxtx-2 -> CPU 1 rxtx-3 -> CPU 5

Notes:

• You cannot use the sim affinity or the fw ctl affinity commands to change and query the IRQ affinity of the Multi-Queue interfaces.
• You can reset the affinity of Multi-Queue IRQs. Run: cpmq set affinity
• You can view the affinity of Multi-Queue IRQs. Run: cpmq get -v

  Important - Do not change the IRQ affinity of queues manually. This can negatively affect the performance of your Security Gateway.

To see the CPU utilization:

1. Find the CPU cores assigned to Multi-Queue IRQs.

   Run:

   cpmq get -v

   Example:

   [Expert@GW:0]# cpmq get -v   Active mlx5_core interfaces: eth2-01 [On] Active i40e interfaces: eth5-01 [On] eth5-02 [Off] Active ixgbe interfaces: eth4-01 [On] eth4-02 [On] Active igb interfaces: Mgmt [On] The rx_num for mlx5_core is: 10 (default) The rx_num for i40e is: 10 The rx_num for ixgbe is: 16 (default) The rx_num for igb is: 2 multi-queue affinity for mlx5_core interfaces: CPU | TX | Vector | RX Bytes ------------------------------------------------------------- 0 | 0 | eth2-01-0 (211) | 0 1 | 2 | eth2-01-2 (227) | 0 2 | 4 | eth2-01-4 (52) | 0 3 | 6 | eth2-01-6 (68) | 0 4 | 8 | eth2-01-8 (84) | 0 5 | 10 | | multi-queue affinity for i40e interfaces: CPU | TX | Vector | RX Bytes ------------------------------------------------------------- 0 | 0 | i40e-eth5-01-TxRx-0 (99) | 0 1 | 2 | i40e-eth5-01-TxRx-2 (115) | 0 2 | 4 | i40e-eth5-01-TxRx-4 (131) | 0 3 | 6 | i40e-eth5-01-TxRx-6 (147) | 0 4 | 8 | i40e-eth5-01-TxRx-8 (163) | 0 5 | 0 | | multi-queue affinity for ixgbe interfaces: CPU | TX | Vector | RX Bytes ------------------------------------------------------------- 0 | 0 | eth4-01-TxRx-0 (156) | 0 | | eth4-02-TxRx-0 (157) | 1 | 2 | eth4-01-TxRx-2 (172) | 0 | | eth4-02-TxRx-2 (173) | 2 | 4 | eth4-01-TxRx-4 (188) | 0 | | eth4-02-TxRx-4 (189) | 3 | 6 | eth4-01-TxRx-6 (204) | 0 | | eth4-02-TxRx-6 (205) | 4 | 8 | eth4-01-TxRx-8 (220) | 0 | | eth4-02-TxRx-8 (221) | 5 | 10 | eth4-01-TxRx-10 (236) | 0 | | eth4-02-TxRx-10 (237) | 6 | 12 | eth4-01-TxRx-12 (61) | 0 | | eth4-02-TxRx-12 (62) | 7 | 14 | eth4-01-TxRx-14 (77) | 0 | | eth4-02-TxRx-14 (78) | [Expert@GW:0]#

2. Run:

   top

3. Press 1 to show all the CPU cores.

   Example - The CPU utilization of Multi-Queue CPU cores is approximately 50%, because CPU0 and CPU1 handle the queues:

   top - 18:02:33 up 28 days, 1:18, 1 user, load average: 1.22, 1.38, 1.48 Tasks: 137 total, 3 running, 134 sleeping, 0 stopped, 0 zombie Cpu0 : 2.0%us, 0.0%sy, 0.0%ni, 42.7%id, 5.9%wa, 0.0%hi, 49.4%si, 0.0%st Cpu1 : 0.0%us, 1.0%sy, 0.0%ni, 55.2%id, 0.0%wa, 0.0%hi, 43.8%si, 0.0%st Cpu2 : 2.0%us, 2.0%sy, 0.0%ni, 45.5%id, 0.0%wa, 4.0%hi, 46.5%si, 0.0%st Cpu3 : 1.0%us, 2.0%sy, 0.0%ni, 74.5%id, 0.0%wa, 0.0%hi, 22.5%si, 0.0%st Cpu4 : 5.0%us, 1.0%sy, 0.0%ni, 42.6%id, 0.0%wa, 0.0%hi, 51.5%si, 0.0%st Mem: 12224020k total, 70005820k used, 5218200k free, 273536k buffers Swap: 14707496k total, 0k used, 14707496k free, 484340k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3301 root 15 0 0 O 0 R 17 0.0 2747:04 [fw_worker_3] 3326 root 15 0 0 O 0 R 16 0.0 2593:35 [fw_worker_0] ... ... ...

For more information, run:

cpmq get -vv

Example:

[Expert@GW:0]# cpmq get -vv Active i40e interfaces: eth5-01 [On] eth5-02 [Off] Active ixgbe interfaces: eth4-01 [On] eth4-02 [On] Active igb interfaces: Mgmt [On] The rx_num for i40e is: 10 The rx_num for ixgbe is: 16 (default) The rx_num for igb is: 2 multi-queue affinity for i40e interfaces: CPU | TX | Vector | RX Packets | RX Bytes -------------------------------------------------------------------- 0 | 0 | i40e-eth5-01-TxRx-0 (220) | 0 | 0 1 | 2 | i40e-eth5-01-TxRx-2 (236) | 0 | 0 2 | 4 | i40e-eth5-01-TxRx-4 (61) | 0 | 0 3 | 6 | i40e-eth5-01-TxRx-6 (77) | 0 | 0 4 | 8 | i40e-eth5-01-TxRx-8 (93) | 0 | 0 5 | 0 | | | multi-queue affinity for ixgbe interfaces: CPU | TX | Vector | RX Packets | RX Bytes -------------------------------------------------------------------- 0 | 0 | eth4-01-TxRx-0 (234) | 0 | 0 | | eth4-02-TxRx-0 (187) | | 1 | 2 | eth4-01-TxRx-2 (59) | 0 | 0 | | eth4-02-TxRx-2 (203) | | 2 | 4 | eth4-01-TxRx-4 (75) | 0 | 0 | | eth4-02-TxRx-4 (219) | | 3 | 6 | eth4-01-TxRx-6 (91) | 0 | 0 | | eth4-02-TxRx-6 (235) | | 4 | 8 | eth4-01-TxRx-8 (107) | 0 | 0 | | eth4-02-TxRx-8 (60) | | 5 | 10 | eth4-01-TxRx-10 (123) | 0 | 0 | | eth4-02-TxRx-10 (76) | | 6 | 12 | eth4-01-TxRx-12 (139) | 0 | 0 | | eth4-02-TxRx-12 (92) | | 7 | 14 | eth4-01-TxRx-14 (155) | 0 | 0 | | eth4-02-TxRx-14 (108) | | multi-queue affinity for igb interfaces: CPU | TX | Vector | RX Packets | RX Bytes -------------------------------------------------------------------- 0 | 0 | Mgmt-TxRx-0 (172) | 2752 | 176674 1 | 0 | | | [Expert@GW:0]#

Overriding RX queue and interface limitations

Warning - We do not recommend to change this configuration. The Multi-Queue is intended to work with up to eight RX queues and up to five interfaces.

• The number of RX queues is limited by the number of CPU cores and the type of the interface driver.

  To add more RX queues, run:

  cpmq rx_num {igb | ixgbe | i40e | mlx5_core} <number of active RX queues> -f

• Due to IRQ limitations, you can configure a maximum of five interfaces with Multi-Queue.

  To add more interfaces, run:

  cpmq set -f