Understanding Logging

Security Gateways / ClusterClosed Two or more Security Gateways that work together in a redundant configuration - High Availability, or Load Sharing. Members generate network logs, and the Management ServerClosed Check Point Single-Domain Security Management Server or a Multi-Domain Security Management Server. generates audit logs, which are a record of actions taken by administrators. The Security PolicyClosed Collection of rules that control network traffic and enforce organization guidelines for data protection and access to resources with packet inspection. that is installed on each Security GatewayClosed Dedicated Check Point server that runs Check Point software to inspect traffic and enforce Security Policies for connected network resources. / Cluster determines which rules generate logs.

Logs can be stored on a:

Note - Logs can be automatically forwarded to the Security Management ServerClosed Dedicated Check Point server that runs Check Point software to manage the objects and policies in a Check Point environment within a single management Domain. Synonym: Single-Domain Security Management Server. or Log Server, according to a schedule, or manually imported with the Remote File Management operation via CLI (with the "fw fetchlogs" command). The management servers and log servers can also forward logs to other servers.

To find out how much storage is necessary for logging, see the R81.20 Release Notes.

A Log Server handles log management activities:

An administrator can configure Backup Log Servers:

  • If all Primary Log Servers are disconnected, the Security Gateway / Cluster starts to send logs only to the first configured Backup Log Server.

  • If the first Backup Log Server is also disconnected, the Security Gateway / Cluster sends logs to the second configured Backup Log Server, and so on.

Configuring the Age of Log Files to Migrate During an Upgrade

Starting in R81.20, when you upgrade a Management Server / Log Server to a new version, log file migration to the new version is limited to logs for the past 180 days.

You can change the age of log files to a number between 30 and 360 days.

To configure the number of days for log file migration:

Important - You must do this procedure before you upgrade.

  1. Connect to the command line on the server (Security Management Server, Multi-Domain Server, Multi-Domain Log Server, dedicated Log Server, dedicated SmartEvent ServerClosed Dedicated Check Point server with the enabled SmartEvent Software Blade that hosts the events database.):

  2. Log in to the Expert mode.

  3. Create the required XML file:

    touch $FWDIR/conf/upgradeLogData.xml

  4. Edit this XML file:

    vi $FWDIR/conf/upgradeLogData.xml

  5. This XML file must contain these lines:

    Copy 
    <?xml version="1.0"?>
      <config>
        <ImportLogDays>NUMBER_OF_DAYS</ImportLogDays>
      </config>

    The number of days must be an integer between 30 and 360.

  6. Save the changes in the file and exit the editor.

Dedicated Log Servers and Domain Dedicated Log Servers

To decrease the load on the Management Server, you can install a dedicated Log Server and configure the Security Gateways to send their logs to this Log Server.

To see the logs from all Log Servers, connect to the Management Server with SmartConsoleClosed Check Point GUI application used to manage a Check Point environment - configure Security Policies, configure devices, monitor products and events, install updates, and so on., and go to the Logs & Monitor view > Logs tab.

See:

Dynamic Log Distribution

In R81 and lower versions, each Log Server received a copy of every log. If one Log Server was disconnected, the Security Gateway / Cluster connected to the backup Log Server and sent it a copy of every log.

Starting in R81.10, with Dynamic Log Distribution, you can configure the Security Gateway / Cluster to distribute its logs between the active Log Servers.

If all the primary Log Servers are disconnected, logs are distributed between backup Log Servers.

If no Log Servers are connected, the Security Gateway / Cluster writes the logs locally.

Use Case - Log distribution reduces:

  • The high utilization of resources on the Log Servers.

  • The log traffic on each network that connects the Security Gateway / Cluster to its Log Server.

  1. Connect with SmartConsole to the Management Server that manages this Security Gateway / Cluster.

  2. From the left navigation panel, click Gateways & Servers.

  3. Open the Security Gateway / Cluster object.

  4. From the left tree, click Logs > Log Distribution.

  5. In the section Log Distribution, select Distribute logs between log servers for improved performance (applies to primary and backup log servers).

  6. In the section Log Servers > Primary log server, select the applicable primary Log Server object(s).

  7. In the section Log Servers > Backup log server, select the applicable backup Log Server object(s).

  8. Click OK.

  9. Publish the SmartConsole session.

  10. Install the Access Control policy on the Security Gateway / Cluster object.

Log Storage

SmartEvent Server and Log Server use an optimization algorithm to manage disk space and other system resources. When the Logs and Events database becomes too large, the server automatically deletes the oldest logs and events based on the configured thresholds.

  1. Connect with SmartConsole to the Management Server.

  2. From the left navigation panel, click Gateways & Servers.

  3. Open the object of the Management Server / SmartEvent Server / Log Server.

  4. From the left tree, go to Logs > Storage.

    Note - The Logs section appears only if you enabled the Logging & Status Software BladeClosed Specific security solution (module): (1) On a Security Gateway, each Software Blade inspects specific characteristics of the traffic (2) On a Management Server, each Software Blade enables different management capabilities. on the General Properties page > Management tab.

  5. In the Disk Management section, configure these settings:

    Field

    Description / Instructions

    Measure free disk space in

    Select MBytes or Percentage.

    When disk space is below <number> Mbytes, issue alert <type>

    Get an alert when the available disk space for logs and log index files is below this threshold.

    This value must be at least 5 MB greater than the value in the When disk space is below <number> Mbytes, stop logging field on the Additional Logging Configuration page.

    When disk space is below <number> Mbytes, start deleting old files

    Delete the oldest logs and log index files when the available disk space is below this threshold.

    The server examines the available space in the log partition every 1 minute. When the threshold is reached, the log disk maintenance occurs- deleting the oldest day of log and index data and repeating until reaching the available space above the configured threshold.

    This value must be at least 5 MB greater than the value in the When disk space is below <number> Mbytes, issue alert <type> field on this page.

    Run the following script before deleting old files

    Enter an absolute path to the shell script (path and the file name).

    This shell script must exist on the server.

  6. In the Daily Logs Retention Configuration section, configure these settings:

    For more information, see Daily Logs Retention.

    First, select Apply the following logs retention policy.

    Field

    Description / Instructions

    Keep indexed logs for no longer than <number> days

    Occurs daily at midnight.

    Deleting oldest index files by days, keeping today + the configured number of index days (14 = 14 days + today).

    Keep log files for an extra <number> days

    Occurs daily at midnight.

    Deleting oldest log files by days, keeping today + the configured number of index days + extra log days (3664 = 14 [from index settings] + 3650 days + today).

    As 3664 is more than 10 years, effectively keeping all log files.

    Note - The maximum total value of both indexed logs and log files is 3664 days.
  7. Click OK.

  8. Publish the SmartConsole session.

  9. Install the database (click Menu > Install database > select all server objects > click Install)

  1. Connect with SmartConsole to the Management Server.

  2. From the left navigation panel, click Gateways & Servers.

  3. Open the object of the Security Gateway / Cluster.

  4. From the left tree, go to Logs > Storage.

  5. In the Disk Management section, configure these settings:

    Field

    Description / Instructions

    Measure free disk space in

    Select MBytes or Percentage.

    When disk space is below <number> Mbytes, issue alert <type>

    Get an alert when the available disk space for logs and log index files is below this threshold.

    This value must be at least 5 MB greater than the value in the When disk space is below <number> Mbytes, stop logging field on the Additional Logging Configuration page.

    When disk space is below <number> Mbytes, start deleting old files

    Delete the oldest logs and log index files when the available disk space is below this threshold.

    The server examines the available space in the log partition every 1 minute. When the threshold is reached, the log disk maintenance occurs- deleting the oldest day of log and index data and repeating until reaching the available space above the configured threshold.

    This value must be at least 5 MB greater than the value in the When disk space is below <number> Mbytes, issue alert <type> field on this page.

    Run the following script before deleting old files

    Enter an absolute path to the shell script (path and the file name).

    This shell script must exist on the server.

    Reserve <number> <units> for packet capturing

    Some types of logs can also capture the packets that created the log eventClosed Record of a security or network incident that is based on one or more logs, and on a customizable set of rules that are defined in the Event Policy.. Set the amount, in megabytes or percent, that you want to use for captured packets.

  6. In the Daily Logs Retention Configuration section, configure these settings:

    For more information, see Daily Logs Retention.

    First, select Apply the following logs retention policy.

    Field

    Description / Instructions

    Keep indexed logs for no longer than <number> days

    Occurs daily at midnight.

    Deleting oldest index files by days, keeping today + the configured number of index days (14 = 14 days + today).

    Keep log files for an extra <number> days

    Occurs daily at midnight.

    Deleting oldest log files by days, keeping today + the configured number of index days + extra log days (3664 = 14 [from index settings] + 3650 days + today).

    As 3664 is more than 10 years, effectively keeping all log files.

    Note - The maximum total value of both indexed logs and log files is 3664 days.
  7. Click OK.

  8. Publish the SmartConsole session.

  9. Install the Access Control policy on the Security Gateway / Cluster object.

For these examples, the administrator enables these thresholds:

  • When disk space is below [5000] Mbytes, start deleting old files

  • Daily logs retention

    • Keep indexed logs for 14 days

    • Keep log files for an extra 6 days (6 + 14 = 20 days of log files)

Example

Description

1

The server has 3000 MBytes of free disk space, and 5 days of logs and index files.

The server deletes logs and index files, one day at a time, until there is 5000 Mbytes of free disk space.

2

The server has 10 GBytes of free disk space and 30 days of logs and index files.

The server deletes all log files older than 20 days ago (6 + 14), each day at midnight.

The server deletes all index files older than 14 days ago, each day at midnight.

3

A server produces 1GB of logs and 1GB of index files each day. The server now has 35 days of logs and 30 days of index files and only 2.5GB of free disk space left. The configured disk space threshold is 5GB, which means the server is now 2.5GB below the threshold.

The index files threshold is 14 days.

The log file threshold is 20 days.

When the disk space threshold (5GB) is reached, disk space maintenance deletes logs and index data until there is again more than 5GB of free space.

In this example:

  1. Logs from day one are deleted first, as they are older. Three days of the oldest logs are deleted to clear 3GB of logs and leave 6GB of free space on the drive, 1GB above the threshold, leaving the server with 32 log days and 30 index days.

  2. The server still has more than 14 days of index files - an extra 16 days (30 days of index files now)

    And more than 20 days of logs – an extra 12 days (32 days of log files now).

    At midnight, the extra log & index files are deleted until only the current day’s log files plus the last 20 days remain.

    Index days are deleted until only the current day’s index plus the last 14 days remain.

    The deletion of three days of logs left 5.5GB of free space.

    The deletion of 12 log file days + 16 index file days frees up a total of 28GB (12 + 16) of space.

    33.5GB of space is now free.

    The daily logs retention occurs every day at midnight keeping the chosen number of days of log + index data.

    Most likely, this means it will never reach the log disk space threshold. But if the log disk space threshold is again reached, the log disk maintenance process repeats to make sure space never runs out.

Daily Logs Retention

In R80.40 and higher, daily logs retention refers to how long logs are stored before they are deleted. Configure this value to help you manage free disk space.

The Management Server does not delete audit log files, even in a case of emergency disk space maintenance, regardless of the configured log retention value. You cannot configure the daily retention for the Management Server audit logs.

The Management Server does not delete audit indexes as part of daily maintenance regardless of the value configured in SmartConsole. The Management Server deletes audit log indexes (not the log files) only in a disk space emergency. In a Multi-Domain environment, you can change this behavior only for the Global SmartEvent Server in the log_maintenance_domain_conf.csv file (see the corresponding section below).

Note - The server deletes old logs daily at midnight.

You can configure log retention policy on different servers:

  1. Connect with SmartConsole to the applicable server:

    • Security Management Server if managed Security Gateways send their logs to it

    • Security Management Server that manages the dedicated SmartEvent Server or dedicated Log Server

  2. From the left navigation panel, click Gateways & Servers.

  3. Open the object of the Management Server / dedicated SmartEvent Server / dedicated Log Server.

  4. From the left tree, go to Logs > Storage.

  5. In the section Daily Logs Retention Configuration:

    1. Select Apply the following logs retention policy.

    2. In the field Keep indexed logs for no longer than <number> days, configure the required number of days.

    3. In the field Keep log files for an extra <number> days, configure the required number of days.

      Notes:

      • When this value is 0, the servers keeps the indexed logs and the log files for the same number of days.

      • If you configure a value greater than 0, the server keeps the log files for the additional configured number of days (after the configured number of days for indexed logs).

    Note - The maximum total value of both indexed logs and log files is 3664 days.

  6. Click OK.

  7. Publish the SmartConsole session.

  8. Install the database (click Menu > Install database > select all server objects > click Install).

Notes:

  • Only Super User can configure these settings.

  • This configuration applies to all Domain Management Servers and Domain Log Servers that are not configured explicitly (see the corresponding section).

  • The daily index deletion on the Multi-Domain Server / Multi-Domain Log Server is enforced based on the greatest value configured between the Domain and the Multi-Domain Server levels.

  1. Connect with SmartConsole to the applicable Multi-Domain Server / Multi-Domain Log Server to the MDS context.

  2. From the left navigation panel, click Multi-Domain > Domains.

  3. In the table, locate the column for this Multi-Domain Server / Multi-Domain Log Server.

  4. Right-click the cell for this Multi-Domain Server / Multi-Domain Log Server and click Edit.

    The Multi-Domain Server window opens.

  5. From the left tree, go to Log Settings > General.

  6. In the section Daily Logs Retention Configuration:

    1. Select Apply the following logs retention policy.

    2. In the field Keep indexed logs for no longer than <number> days, configure the required number of days.

    3. In the field Keep log files for an extra <number> days, configure the required number of days.

      Notes:

      • When this value is 0, the servers keeps the logs and the indexed logs for the same number of days.

      • If you configure a value greater than 0, the server keeps the logs for the additional configured number of days.

    Note - The maximum total value of both indexed logs and log files is 3664 days.
  7. Click OK.

  8. Publish the SmartConsole session.

Notes:

  • Only Super User can configure these settings.

  • By default, all Domain Management Servers use the settings a Super User configured in the Multi-Domain Server / Multi-Domain Log Server object.

  • The Multi-Domain Server / Multi-Domain Log Server deletes a log index only when no Domains use this log index.

    For example, if you configured one Domain to keep its log index for 5 days and another Domain to keep its log index for 30 days, then the server deletes the log index only after 30 days.

  1. Connect with SmartConsole to the applicable Domain Management Server.

  2. From the left navigation panel, click Gateways & Servers.

  3. Open the object of the Domain Management Server / Domain Log Server.

  4. From the left tree, go to Logs > Storage.

  5. In the section Daily Logs Retention Configuration:

    1. Select Override Multi-Domain Settings.

    2. In the field Keep indexed logs for no longer than <number> days, configure the required number of days.

    3. In the field Keep log files for an extra <number> days, configure the required number of days.

      Notes:

      • When this value is 0, the servers keeps the logs and the indexed logs for the same number of days.

      • If you configure a value greater than 0, the server keeps the logs for the additional configured number of days.

    Note - The maximum total value of both indexed logs and log files is 3664 days.
  6. Click OK.

  7. Publish the SmartConsole session.

  8. Install the database (click Menu > Install database > select all server objects > click Install)

You must configure the required settings only in the corresponding configuration file:

Settings

Configuration File

Comment

General settings that apply to all Domain Management Servers that use this Global SmartEvent Server

log_policy_extended.C

See sk117317

Settings that apply to only to a specific Domain Management Server that uses this Global SmartEvent Server

log_maintenance_domain_conf.csv

See below

Note - If you do not configure settings explicitly, then the default values apply.

To configure settings for specific Domain Management Servers:

  1. Connect to the command line on the Multi-Domain Server over SSH.

  2. Log in to the Expert mode.

  3. Back up the current file:

    cp -v $RTDIR/conf/log_maintenance_domain_conf.csv{,_ORIGINAL}

  4. Get the contents of the current file:

    cat $RTDIR/conf/log_maintenance_domain_conf.csv

  5. On your computer, copy the two lines from this file (from the SSH session) into a text editor or table editor (like Microsoft Excel, or LibreOffice Calc).

  6. Save the file in the CSV format with this name:

    log_maintenance_domain_conf.csv

  7. Configure the names of Domains and the required number of days to keep the logs.

    Best Practice - Add the row with the Domain name "default" and configure the default values. Each new Domain you create automatically uses these default values.

    Example for "Domain1" and "Domain2":

    Domain_name

    audit

    files

    firewallandvpn

    other

    other-smartlog

    resources

    smartevent

    Domain1

    3650

    20

    15

    15

    14

    14

    30

    Domain2

    3650

    20

    30

    30

    14

    14

    14

    default

    3650

    30

    14

    14

    14

    14

    14

    Note - If you do not configure a Domain explicitly, then it takes the greatest values from each column. In the example, if there is a Domain called "Domain3", but you do not configure it explicitly in this file, then this Domain uses the values "3650, 20, 30, 30, 14, 14, 14, 30".

  8. Copy the modified CSV file from your computer to the Multi-Domain Server to some directory (for example, /var/log/).

  9. Go back to the SSH session on the Multi-Domain Server.

  10. If you edited this CSV file on Windows OS, then convert the file from the DOS format to the UNIX format:

    dos2unix /var/log/log_maintenance_domain_conf.csv

  11. Replace the current file with the modified file:

    cp -f -v /var/log/log_maintenance_domain_conf.csv $RTDIR/conf/log_maintenance_domain_conf.csv

    cat $RTDIR/conf/log_maintenance_domain_conf.csv

  12. Restart Check Point services:

    mdsstop ; mdsstart

Log Receive Rate

To learn how to monitor the Log Receive Rate on the Management Server / Log Server, see sk120341.