Advanced QoS Policy Management

Overview

This chapter covers more advanced QoS Policy management procedures that let you to refine the basic QoS Policies described in Basic Policy Management.

Examples: Guarantees and Limits

The QoS Action properties defined in the rules and sub-rules of a QoS Policy Rule Base decide bandwidth allocation.

The guidelines and examples in the sections that follow show how to use effectively guarantees and limits.

Per Rule Guarantees

• The bandwidth allocated to the rule equals the guaranteed bandwidth plus the bandwidth allocated to the rule because of its weight. To uphold the guarantee, the guaranteed bandwidth is subtracted from the total bandwidth and set aside. The remaining bandwidth is then distributed according to the weights specified by all the rules.

  The bandwidth guaranteed to a rule is the guaranteed bandwidth plus the rule's share of bandwidth according to weight.

  Total Rule Guarantees

  Rule Name	Source	Destination	Service	Action
  Rule A	Any	Any	ftp	Rule Guarantee - 100KBps Weight 10
  Rule B	Any	Any	http	Weight 20

  • The link capacity is 190KBps.
  • In this example, Rule A receives 130KBps, 100KBps from the guarantee, plus (10/30) * (190-100).
  • Rule B receives 60KBps, which is (20/30) x (190-100).
• If a guarantee is defined in a sub-rule, then a guarantee must be defined for the rule above it. The guarantee of the sub-rule can also not be greater than the guarantee of the rule above it.

  Guarantee is Defined in Sub-rule A1, But Not in Rule A Making the Rule Incorrect

  Rule	Source	Destination	Service	Action
  Rule A	Any	Any	ftp	Weight 10
  Start of Sub-Rule
  Rule A1	Client-1	Any	ftp	Rule Guarantee - 100KBps Weight 10
  Rule A2	Client-2	Any	ftp	Weight 10
  End of Sub-Rule
  Rule B	Any	Any	http	Weight 30

  This Rule Base is not correct. The guarantee is defined in sub-rule A1, but not in Rule A. To correct this, add a guarantee of 100KBps or more to Rule A.

• A rule guarantee must not be smaller than the sum of guarantees defined in its sub‑rules.

  Example of an Incorrect Rule Base

  Rule	Source	Destination	Service	Action
  Rule A	Any	Any	ftp	Rule Guarantee - 100KBps Weight 10
  Start of Sub-Rule
  Rule A1	Client-1	Any	ftp	Rule Guarantee - 80KBps Weight 10
  Rule A2	Client-2	Any	ftp	Rule Guarantee - 80KBps Weight 10
  Rule A3	Client-3	Any	ftp	Weight 10
  End of Sub-Rule
  Rule B	Any	Any	http	Weight 30

  This Rule Base is incorrect. The sum of guarantees in Sub-Rules A1 and A2 is (80 + 80) = 160, which is greater that the guarantee defined in Rule A (100KBps). To correct this, define a guarantee not smaller than 160KBps in Rule A, or decrease the guarantees defined in A1 and A2.

• If a rule's weight is low, connections that match the rule might receive little bandwidth.

  If a Rule's Weight is Low, Some Connections Might Receive Little Bandwidth

  Rule	Source	Destination	Service	Action
  Rule A	Any	Any	ftp	Rule Guarantee - 100KBps Weight 1
  Start of Sub-Rule
  Rule A1	Client-1	Any	ftp	Rule Guarantee - 100KBps Weight 10
  Rule A2	Client-2	Any	ftp	Weight 10
  End of Sub-Rule
  Rule B	Any	Any	http	Weight 30

  The link capacity is 190KBps.

  Rule A is entitled to 103KBps, which are the 100KBps guaranteed, plus (190-100) x (1/31). FTP traffic classified to Sub-Rule A1 receives the guaranteed 100KBps which is almost all the bandwidth to which Rule A is entitled. All connections classified to Sub‑Rule A2 together receive only 1.5KBps, which is half of the remaining 3KBps.

• The sum of guarantees in rules in the top level must not be more than 90% of the capacity of the link.
• The guarantee rule reserves the bandwidth only if a connection matches the guarantee rule. If no connection matches the guarantee rule, the bandwidth is not reserved.
• When the connection speed is less than the bandwidth guarantee, the guarantee rule makes unused bandwidth available to other connections.

  For example, if the guarantee is 5MB and the connection speed is 3MB. The unused 2MB reserved by the rule is made available for other connections.

Per Connections Guarantees

1. If the Accept additional connections is checked, connections exceeding the number defined in the Number of guaranteed connections are opened. If the field adjacent to Accept additional connections is empty, additional connections receive bandwidth allocated according to the defined Rule Weight.
2. You can define Per connection guarantees for a rule and for its sub-rule. The Per connection guarantee of the sub-rule must not be greater than the Per connection guarantee of the rule.

   When such a Rule Base is defined, a connection classified to the sub-rule receives the Per connection guarantee that is defined in the sub-rule. If a sub-rule does not have a Per connection guarantee, it still receives the Per connection guarantee defined in the parent rule.

Limits

A rule can have both a Rule limit and a Per connection limit. But the Per connection Limit must not be greater than the Rule Limit.

If a limit is defined in a rule with sub-rules, and limits are defined for all the sub-rules, the rule limit has a restriction. The rule limit must not be greater than the sum of limits defined in the sub-rules. It is not possible to give more bandwidth to a rule than the bandwidth determined by the sum of the limits of its sub-rules.

Guarantee - Limit Interaction

• If a Rule Limit and a Guarantee per rule are defined in a rule, the limit must not be less than the guarantee.
• If both a Limit and a Guarantee are defined in a rule, and the Limit is equal to the Guarantee, connections might not receive bandwidth.

Example:

No Bandwidth Received:

The Guarantee in sub-rule A1 equals the Guarantee in rule A (100KBps). When there is sufficient traffic on A1 to use the full Guarantee, traffic on A2 does not receive bandwidth from A. (There is a limit on A of 100KBps).

In this example:

• A rule has both a guarantee and a limit, such that the limit equals the guarantee.
• The rule has sub-rules with Total Rule Guarantees that add up to the Total Rule Guarantee for the rule.
• The rule also has sub-rule(s) with no guarantee.

In such a case, the traffic from the sub-rule(s) with no guarantee might receive little or no bandwidth.

Differentiated Services (DiffServ)

Overview

DiffServ is an architecture for giving different types or levels of service for network traffic.

When on the enterprise network, packets are marked in the IP header TOS byte as belonging to some Class of Service (QoS Class). When outside on the public network, these packets are granted priority according to their class.

DiffServ markings have meaning on the public network, not on the enterprise network. Good implementation of DiffServ requires that packet markings be recognized on all public network segments.

DiffServ Markings for IPSec Packets

When DiffServ markings are used for IPSec packets, the DiffServ mark can be copied between headers by setting these properties in: $FWDIR/conf/objects_5_0.c.

• :ipsec.copy_TOS_to_inner

  The DiffServ mark is copied from the IPSec header to the IP header of the packet after decapsulation/decryption.

• :ipsec.copy_TOS_to_outer

  The DiffServ mark is copied from the packet's IP header to the IPSec header of the encrypted packet after encapsulation.

  The default setting are:

  :ipsec.copy_TOS_to_inner (false)

  :ipsec.copy_TOS_to_outer (true)

Interaction Between DiffServ Rules and Other Rules

Just like QoS Policy Rules, a DiffServ rule specifies not only a QoS Class, but also a weight. These weights are enforced only on the interfaces on which the rules of this class are installed.

For example, if a DiffServ rule specifies a weight of 50 for FTP connections. That rule is installed only on the interfaces for which the QoS Class is defined. On other interfaces, the rule is not installed. FTP connections routed through the other interfaces do not get the weight specified by the rule. To specify a weight for all FTP connections, add a rule below "Best Effort."

DiffServ rules can be installed only on interfaces for which the related QoS Class has been defined. QoS class is defined on the QoS tab of the Interface Properties window. For more, see: Define the QoS Properties for the Interfaces.

"Best Effort" rules (that is, non-DiffServ rules) can be installed on all interfaces of gateways with QoS gateways installed. Only rules installed on the same interface interact with each other.

Note:

• QoS supports adding diffserv markings to packets that match a rule
• QoS does not support matching packets based on diffserv tagging.

Low Latency Queuing

Overview

For most traffic on the Web (most TCP protocols), the WFQ (Weighted Fair Queuing, see Intelligent Queuing Engine) paradigm is sufficient. Packets reaching QoS are put in queues and forwarded according to the interface bandwidth and the priority of the matching rule.

Using this standard Policy, QoS avoids dropping packets. Dropped packets adversely affect TCP. Avoiding drops means holding (possibly) long queues, which can lead to non-negligible delays.

For some types of traffic, such as voice and video, bounding this delay is important. Long queues are inadequate for these types of traffic. Long queues can result in substantial delay. For most "delay sensitive" applications, it is not necessary to drop packets from queues to keep the queues short. The fact that the streams of these applications have a known, bounded bit rate can be utilized. If QoS is configured to forward as much traffic as the stream delivers, only a small number of packets are queued and delay is negligible.

QoS Low Latency Queuing makes it possible to define special Classes of Service for "delay sensitive" applications like voice and video. Rules below these classes can be used together with other rules in the QoS Policy Rule Base. Low Latency classes require you to specify the maximal delay that is tolerated and a Constant Bit Rate. QoS then guarantees that traffic matching rules of this type is forwarded within the limits of the bounded delay.

Low Latency Classes

For each Low Latency class defined on an interface, a constant bit rate and maximal delay must be specified for active directions. QoS checks packets matched to Low Latency class rules to make sure they have not been delayed for longer than their maximal delay permits. If the maximal delay of a packet has been exceeded, it is dropped. Otherwise, it is transmitted at the defined constant bit rate for the Low Latency class to which it belongs.

If the Constant Bit Rate of the class is not smaller than the expected arrival rate of the matched traffic, packets are not dropped. The maximal delay must also exceed some minimum. For more, see Computing Maximal Delay).

When the arrival rate is higher than the specified Constant Bit Rate, packets exceeding this constant rate are dropped. This is to make sure that transmitted packets comply with the maximal delay limitations.

Low Latency Class Priorities

In most cases, one Low Latency class is sufficient for all bounded delay traffic. In some cases, it might be necessary to define more than one Low Latency class. For this reason, Low Latency classes are assigned one out of five priority levels (not including the Expedited Forwarding class, see Low Latency versus DiffServ). These priority levels are relative to other Low Latency classes.

As a best practice, define more than one Low Latency class if different types of traffic require different maximal delays.

The class with the lower maximal delay must get a higher priority than the class with the higher delay. When two packets are ready to be forwarded, one for each Low Latency class, the packet from the higher priority class is forwarded first. The remaining packet (from the lower class) then encounters greater delay. The maximal delay that can be set for a Low Latency class depends on the Low Latency classes of higher priority.

Other Low Latency classes can affect the delay incurred by a class. Other Low Latency classes must be taken into consideration when determining the minimal delay that is possible for the class. This is best done by:

• Initially setting the priorities for all Low Latency classes according to maximal delay
• Defining the classes according to descending priority

When you define class two, for example, class one must already be defined.

For more on the effects of class priority on calculating maximal delay, see: Computing Maximal Delay.

Logging LLQ Information

SmartView Tracker logs data for all aspects of LLQ. For more, see SmartView Tracker.

Calculating the Correct Constant Bit Rate and Maximal Delay

Limits on Constant Bit Rate

For the inbound or outbound interface direction, the sum of the constant bit rates of all the Low Latency classes has a limit. This sum cannot exceed 20% of the total designated bandwidth rate. This 20% limit makes sure that "Best Effort" traffic does not suffer substantial jitter because of the existing Low Latency class(es).

Calculating Constant Bit Rate

To calculate the Constant Bit Rate of a Low Latency class, you must know the bit rate of one application stream in traffic that matches the:

• Class
• Number of expected streams that are simultaneously opened.

The Constant Bit Rate of the class equals the bit rate of one application multiplied by the expected number of streams opened at the same time.

If the number of streams is greater than the number you expected, the total incoming bit rate will exceed the Constant Bit Rate. Many drops will occur. To prevent drops, limit the number of concurrent streams. For more, see Ensuring that Constant Bit Rate is Not Exceeded (Preventing Unwanted Drops).

Calculating Maximal Delay

To calculate the maximal delay of a Low Latency class, take into account the:

• Maximal delay that streams matching the class can tolerate in QoS
• Minimal delay that QoS can guarantee this stream

It is important not to define a maximal delay that is too small, which can result in unwanted drops. The delay value defined for a class determines the number of packets that can be queued in the Low Latency queue before drops occur. The smaller the delay, the shorter the queue. A maximal delay that is not sufficient can cause packets to be dropped before they are forwarded. Allow for some packets to be queued, as explained in the steps below.

If you are using SmartView Tracker, it is recommended to use the default Class Maximal Delay defined in the LLQ log. To obtain this default number:

• First configure the correct Constant Bit Rate for the Class
• Give an estimation for the Class Maximal Delay

For more information, see SmartView Tracker.

You can also set the Class Maximal Delay by obtaining estimates for the upper and lower bounds. Set the delay to a value between the bounds.

1. Estimate the greatest delay that you can set for the class:
   1. Refer to the technical details of the streaming application and find the delay that it can tolerate.

      For voice applications, the user generally starts to experience irregularities when the overall delay exceeds 150 ms.

   2. Find or estimate the bound on the delay that your external network (commonly the WAN) imposes. Many Internet Service Providers publish Service Level Agreements (SLAs) that guarantee some bounds on delay.
   3. The maximal delay must be set at no more than:

      (i) The delay that the streaming application can tolerate minus

      (ii) The delay that the external network introduces

   This makes sure that the delay introduced by QoS plus the delay introduced by the external network is no more than the delay tolerated by the streaming application.

2. Estimate the smallest delay that you can set for the class:
   • Find the bit rate of the streaming application in the application properties, or use SmartView Monitor. (For more, see: R77.10 SmartView Monitor Administration Guide.

     Note: Even if you set the Constant Bit Rate of the class to accommodate multiple simultaneous streams, do the next calculations with the rate of a single stream:

   • Estimate the typical packet size in the stream.
     • Find it in the application properties, or monitor the traffic.
     • If you do not know the packet size, use the size of the MTU of the LAN behind QoS. For Ethernet, this number is 1500 Bytes.
   • Many LAN devices, switches and NICs, introduce some burstiness to flows of constant bit rate by changing the delay between packets. For constant bit rate traffic generated in the LAN and going out to the WAN, monitor the stream packets on the QoS gateway. To get an estimate of burst size, monitor the internal interface that precedes the QoS gateway.
   • If no burstiness is detected, the minimal delay of the class must be no smaller than:

3 x packet size

---------------

   bit rate 

This enables three packets to be held in the queue before drops can occur.

The bit rate must be the bit rate of one application, even if the Constant Bit Rate of the class is for multiple streams.

• If burstiness is detected, set the minimal delay of the class to at least:

(burst size + 1) x packet size

------------------------------

            bit rate 

The maximal delay that you select for the class must be between the smallest delay (step 2) and the greatest delay (step 1). Setting the maximal delay near to one of these values is not recommended. If you expect the application to burst occasionally, or if you don't know whether the application generates bursts at all, set the maximal delay close to the value of the greatest delay.

This error message can show after you enter the maximal delay: "The inbound/outbound maximal delay of class... must be greater than... milliseconds." The message shows if Class of Service that you define is not of the first priority (see Low Latency Class Priorities). The delay value displayed in the error message depends on the Low Latency classes of higher priority, and on interface speed.

Set the maximal delay to a value no smaller than the one printed in the error message.

Making sure that Constant Bit Rate is not Exceeded

If the total bit rate going through the Low Latency class exceeds the Constant Bit Rate of the class, then drops occur. (See: Logging LLQ Information.)

This occurs when the number of streams opened exceeds the number you expected when you set the Constant Bit Rate.

To limit the number of streams opened through a Low Latency Class:

1. Define one rule under the class, with a per connection guarantee as its Action.
2. In the Per Connection Guarantee field of the QoS Action Properties window, define the per connection bit rate that you expect
3. In the Number of guaranteed connections field, define the maximal number of connections that you allow in this class.

   Do not select the Accept additional non-guaranteed connections option.

The number of connections is limited to the number you used to calculate the Constant Bit Rate of the class.

Interaction between Low Latency and Other Rule Properties

To activate a Low Latency class, define at least one rule below it in the QoS Policy Rule Base. Traffic matching a Low Latency class rule receives the delay and Constant Bit Rate properties defined for the specified class. The traffic is handled according to the rule properties (weight, guarantee and limit).

You can use all types of properties in the rules below the Low Latency class:

• Weight
• Guarantee
• Limit
• Per Connection Guarantee
• Per Connection Limit.

Think of the Low Latency class with its rules as a separate network interface:

• Forwarding packets at a rate defined by the Constant Bit Rate with delay bounded by the class delay
• With the rules defining the relative priority of the packets before they reach the interface

If a rule has a relatively low priority, then packets matching it are entitled to a small part of the Constant Bit Rate. More packets will be dropped if the incoming rate is not sufficiently small.

Note:

• Using sub-rules under the low latency class is not recommended. Sub-rules make it difficult to calculate streams that suffer drops and the drop pattern.
• Guarantees and limits are not recommended for the same reason. Except for Per Connection Guarantees, as described in Ensuring that Constant Bit Rate is Not Exceeded (Preventing Unwanted Drops).

When to Use Low Latency Queuing

Use Low Latency Queuing when:

• Low delay is important, and the bit rate of the incoming stream is known. For example video and voice applications. In such cases, specify both the maximal delay and the Constant Bit Rate of the class.
• Controlling delay is important, but the bit rate is unknown. For example, Telnet requires quick responses, but the bit rate is not known. If the stream occasionally exceeds the Constant Bit Rate, you do not want to experience drops. A longer delay is recommended.
  • Set the Constant Bit Rate of the class to a high estimate of the stream rate.
  • Set a large maximal delay (such as 99999 ms).

  The large delay makes sure that packets are not dropped if a burst exceeds the Constant Bit Rate. The packets are queued and forwarded according to the Constant Bit Rate.

  Note - When the incoming stream is smaller than the Constant Bit Rate, the actual delay is much smaller than 99999 ms. (As in the example above). Packets are forwarded almost as soon as they arrive. The 99999 ms bound is effective only for large bursts.

Do not use a Low Latency Class when controlling delay is not of primarily importance. For most TCP protocols (such as HTTP, FTP and SMTP) the other type of QoS rule is more applicable. Use Weights, Limits and Guarantees. The correct priority is imposed on traffic without having to adjust bit rate and delay.

QoS enforces the policy with minimal drops. Weights and guarantees dynamically fill the pipe when expected traffic is not present. Low Latency Queuing limits traffic according to the Constant Bit Rate.

Low Latency versus DiffServ

Low Latency classes are different from DiffServ classes in that they do not receive type of service (TOS) markings. Not all packets are marked as Low Latency. Preferential treatment is guaranteed only while the packets are passing through the QoS gateway.

The exception to this rule is the Expedited Forwarding DiffServ class. A DiffServ class defined as an Expedited Forwarding class automatically becomes a Low Latency class of highest priority. Such a class receives the conditions afforded it by its DiffServ marking both in QoS and on the network.

Note: To use the Expedited Forwarding class as DiffServ only, without delay being enforced, specify a Maximal Delay value of 99999 in the Interface Properties tab (see Low Latency Classes).

When to Use DiffServ and When to Use LLQ

Do not use Low Latency Queuing to delay traffic when your ISP:

• Supports DiffServ

  Despite the DiffSev marking that you apply, the IP packets might get a different QoS level from the ISP.

• Offers you a number of Classes of Service using MPLS

  DiffServ marking communicate to your ISP the Class of Service that you expect all packets to receive.

For these two cases, mark your traffic using a DiffServ class (see When to Use Low Latency Queuing):

Authenticated QoS

Check Point Authenticated QoS gives Quality of Service (QoS) for end-users in dynamic IP environments, such as remote access and DHCP environments. This lets priority users, such as corporate CEOs, to receive priority service when remotely connecting to corporate resources.

Authenticated QoS dynamically prioritizes end-users, based on information gathered during network or VPN authentication. The feature leverages Check Point UserAuthority technology to classify both inbound and outbound user connections. The User Authority Server (UAS) maintains a list of authenticated users. When you query the UAS, QoS retrieves the data and allocates bandwidth accordingly.

QoS supports Client Authentication, Encrypted Client Authentication, and SecuRemote/SecureClient Authentication. User and Session Authentication are not supported.

For about Client Authentication, see the R77 Security Gateway Technical Administration Guide.

Note: Authenticated QoS:

• Is available for backward compatibility
• Only works in QoS policy mode but does not support CoreXL or SecureXL acceleration technologies

Citrix MetaFrame Support

This section covers support for Citrix Metaframe.

Overview

Citrix MetaFrame is a client/server software application that enables a client to run a published application on a Citrix server farm from the client's desktop. Citrix MetaFrame:

• Load balances by:
  • Automatically directing a client to the server with the lightest load
  • Allows publishing and application management from only one server in that farm.
• Supplies a secure encryption option using the ICA (Independent Computing Architecture) protocol developed by Citrix.

Note:

• Uncontrolled printing traffic on Citrix ICA networks with a slow internet connection can remove bandwidth from mission critical applications. On slow networks, it is necessary to differentiate between Citrix traffic and other types of:
  • Network traffic
  • Traffic in the same layer (layer 7)
• The Citrix Print Manager service can only be used in QoS policy mode when SecureXL or CoreXL acceleration technologies are not enabled.

QoS solves the problem of uncontrolled printing traffic on Citrix ICA networks by:

• Identifying all ICA applications running over Citrix through layer 7.
• Differentiating between the Citrix traffic based on ICA published applications, ICA printing traffic (Priority Tagging) and NFuse.

For more, see: Managing QoS for Citrix ICA Applications.

To manage printing over Citrix, QoS uses the Citrix_ICA_printing service. The Citrix ICA printing traffic service is supported in NG with Application Intelligence (R55) and higher, but is not supported:

• In Express policy mode
• When SecureXL or CoreXL acceleration technologies are enabled.

  In QoS policy mode, when SecureXL or CoreXL is enabled on the gateway, you cannot install a Policy that uses the Citrix_ICA_printing service in a rule. Policy installation will fail.

For more, see: Managing QoS for Citrix Printing.

Limitations

• Citrix services are supported in QoS policy mode only, but not when SecureXL or CoreXL are enabled on the gateway.
• Session Sharing must be disabled.
• The inspection infrastructure detects a maximum of 2048 applications. After the 2048 limit, console errors are sent. These errors do not affect your system. To stop the errors, restart the machine.
• Versions of MetaFrame prior to 1.8 are not supported because there is no packet tagging in these versions.
• Only one Citrix TCP service can be allocated per rule.

Load Sharing

Overview

Load Sharing is a mechanism that distributes traffic in a cluster of gateways to increase the total throughput. QoS architecture guarantees that Load Sharing uses either:

• Two-way Stickiness - all packets of a connection use the same gateway in both directions.
• Conversation Stickiness - all packets of control/data connections in a conversation use the same gateway in both directions.

In Load Sharing configurations, all functioning gateways in the cluster are active, and handle network traffic (Active/Active operation). If one of the member gateways fails, its connections are redistributed to other members of the cluster.

If one gateway in the cluster becomes unreachable, transparent failover occurs to the other members. Connections are shared between the remaining gateways without interruption.

All cluster servers share the same set of virtual interfaces. Each virtual interface corresponds to an outgoing link. The next example shows a typical cluster.

QoS gives a fault-tolerant QoS solution for cluster Load Sharing that deploys a unique, distributed WFQ bandwidth management technology. The user specifies a unified QoS policy for each virtual interface of the cluster. The resulting bandwidth allocation is the same as that obtained by installing the policy on one server.

QoS Cluster Infrastructure

This section describes the cluster infrastructure needed for QoS load sharing.

Cluster State

ClusterXL introduces a member's load value. A member's load, calculated in percentages, is assigned to each member by the cluster. The load is different for ClusterXL multicast and unicast modes.

Usually, the load for N members in the cluster equals (100 / N)%. If the number of cluster members changes dynamically (due to failover or recovery) the load is dynamically adjusted by the cluster to the applicable value.

Changes in Cluster State

All cluster members recalculate their rates when the load on one of the members changes. If a member fails, on the next rate calculation the members bandwidth is divided between the active cluster members.

Rates Calculation Algorithm

QoS Load Sharing uses a member's load value to get the correct rates allocation for QoS rules. QoS calculates the actual rate according to these criteria:

For a centralized policy rule:

• The rate, limit and guarantee values in the rule are proportionally divided between each cluster member according to their load value
• The rate is equally divided between each connection that matches the rule

For a physical network interface, the limit is:

• A fraction of the cluster-interface limit value
• Proportional to the cluster-member's load value

The rates, limits and guarantees are recalculated each time the cluster's state changes.

Per-connection guarantees are processed separately (per-connection limit implementation remains unchanged by the Load Sharing mechanism).

Per-Connection Guarantee Allocation

Each rule with a per connection guarantee manages its rate budget. A rule's budget is the sum of all per connection guarantee rates over the number of per-connection guarantee connections allowed by this rule.

To determine if a new connection receives its per-connection guarantee, the overall rate (already granted to the matched rule's per-connection guarantee) is checked. If this rate is below the rule's budget then the new connection is granted its per-connection guarantee.

This budget is also divided between cluster members proportionally to their load. Generally, each member will get only half the allowed per-connection guarantee matched to the rule. The cluster grants per-connection guarantee service according to the QoS policy.

Example of Rates Calculation

This example shows a cluster consisting of two members with one virtual interface configured to the rate of 125KBps. The two members of the cluster equally share the load. The centralized scheduling policy and the corresponding local scheduling policies look like this:

Conclusion

The decision function distributes traffic between all cluster members. The resultant load sharing allocates the same rates to the rules/connections as would be done by a centralized policy.