Introducing SmartReporter

The SmartReporter Solution

Check Point SmartReporter delivers a user-friendly solution for monitoring and auditing traffic. You can generate detailed or summarized reports in the format of your choice (list, vertical bar, pie chart etc.) for all events logged by Check Point Security Gateway, SecureClient and IPS.

SmartReporter implements a Consolidation Policy, which goes over your original, "raw" log file. It compresses similar logs into events and writes the compressed list of events into a relational database (the SmartReporter Database). This database enables quick and efficient generation of a wide range of reports. The SmartReporter solution provides a balance between keeping the smallest report database possible and retaining the most vital information with the most flexibility.

A Consolidation Policy is similar to a Security Policy in terms of its structure and management. For example, both Rule Bases are defined through the SmartDashboard's Rules menu and use the same network objects. In addition, just as Security Rules determine whether to allow or deny the connections that match them, Consolidation Rules determine whether to store or ignore the logs that match them. The key difference is that a Consolidation Policy is based on logs, as opposed to connections, and has no bearing on security issues.

The Log Consolidation Solution diagram illustrates the Consolidation process, defined by the Consolidation Policy. After the Security Gateways send their logs to the Security Management server, the Log Consolidator Engine collects them, scans them, filters out fields defined as irrelevant, merges records defined as similar and saves them to the SmartReporter Database.

The SmartReporter server can then extract the consolidated records matching a specific report definition from the SmartReporter Database and present them in a report layout.

Two types of reports can be created: Standard Reports and Express Reports. The Standard Reports are generated from information in log files through the Consolidation process to yield relevant analysis of activity. Standard reports that are listed under “Event Management” are based on SmartEvent events database and require SmartEvent-generated events. Express Reports are generated from SmartView Monitor History files and are produced faster.

SmartReporter Standard Reports are supported by two Clients:

• SmartDashboard Log Consolidator — manages the Log Consolidation rules.
• SmartReporter Client — generates and manages reports.

The interaction between the SmartReporter client and Server components applies both to a distributed installation, where the Security Management server and SmartReporter's Server components are installed on two different machines, and to a standalone installation, in which these Software Blades are installed on the same machine.

Log Consolidation Process

It is recommended to use the Log Consolidator's predefined Consolidation Policy (the Out of the Box Policy), designed to filter out irrelevant logs and store the most commonly requested ones (such as blocked connection, alert or web activity logs). The Log Consolidator Engine scans the Consolidation Rules sequentially and processes each log according to the first Rule it matches.

Figure 1‑3 illustrates how the Consolidation Policy processes logs: when a log matches a Consolidation Rule, it is either ignored or stored. If it is ignored, no record of this log is saved in the SmartReporter system, so its data is not available for report generation. If it is stored, it is either saved as is (so all log fields can later be represented in reports), or consolidated to the level specified by the Rule.

The consolidation is performed on two levels: the interval at which the log was created and the log fields whose original values should be retained. When several logs matching a specific Rule are recorded within a predefined interval, the values of their relevant fields are saved "as is", while the values of their irrelevant fields are merged (for example, "consolidated") together.

How to interpret Computer names in DHCP enabled networks

In DHCP address mapping is used. Assuming the DNS knows how to resolve dynamic addresses, the information you see in the report reflects the correct resolving results for the time the reported log events have been processed by the SmartDashboard Log Consolidator and inserted into the database.

Because of the dynamic nature of DHCP address distribution, there is no guarantee that consolidation of old log files will produce correct address name resolving.

When DHCP is in use, consolidating log files close to the time of their creation will improve address-resolving accuracy.

DBsync

DBsync enables SmartReporter to synchronize data stored in different parts of the network. After SIC is established, DBsync connects to the management server to retrieve all the objects. After the initial synchronization, it gets updates whenever an object is saved In distributed information systems DBsync provides one-way synchronization of data between the Security Management servers object database and the SmartReporter machine, and supports configuration and administration of distributed systems.

With DBsync, initial synchronization is established between the SmartReporter machine and the Management server machine (for example, Security Management Server or Multi-Domain Server). In a Multi-Domain environment, you can choose which domains to synchronize in the SmartReporter client, in the Domain Activation menu. If the initial synchronization is not complete the administrator will receive a warning informing him that the GUI will open in read-only mode. Once initial synchronization is complete SmartReporter will open in Read/Write mode.

As a result of DBsync, whenever an object is saved (that is, a new object is created or an existing object is changed) on a Management machine the object is automatically synchronized in SmartEvent.

Synchronization can take time up to 30 minutes, although this is usually the time needed for a very large database.

Basic Concepts and Terminology

• Automatic Maintenance - the process of automatically deleting and/or archiving older database records into a backup file.
• Consolidation - the process of reading logs, combining instances with the same key information to compress data and writing it to the database.
• Consolidation Policy - the rules to determine which logs the consolidator will accept and how to consolidate them. We recommend that you use the out-of-the-box policy without change.
• Consolidation Session - an instance of the consolidation process. There can be one active session for every log server.
• Express Reports - reports based on the SmartView Monitor counters and SmartView Monitor History files. These reports are not as flexible as standard reports but are generated quickly.
• Log Sequence - the series of log files as specified by fw.logtrack. When a log switch is performed, the log file is recorded in the sequence of files. The log consolidator can follow this sequence.
• Report - a high-level view of combined log information that provides meaning to users. Reports are comprised of sections.
• Standard Reports - reports based on consolidated logs.
• $RTDIR - the installation directory of the SmartReporter.

Predefined Reports

The SmartReporter client offers a wide selection of predefined reports for both Standard and Express reporting, designed to cover the most common network queries from a variety of perspectives.

SmartReporter Standard Reports

The Log Consolidation process results in a database of the most useful, relevant records, known as the SmartReporter Database. The information is consolidated to an optimal level, balancing the need for data availability with the need for fast and efficient report generation.

Reports are generated based on a single database table, specified in the Reports view > Standard Reports > Input tab. By default, all consolidated records are saved to the CONNECTIONS table and all reports use it as their data source. However, each time you create a new consolidation session, you have the option of storing records in a different table.

Dividing the consolidated records between different tables allows you to set the SmartReporter client to use the table most relevant to your query, thereby improving the SmartReporter server's performance. In addition, dividing records between tables facilitates managing the SmartReporter Database: you can delete outdated tables, export tables you are not currently using to a location outside of the SmartReporter Database and import them back when you need them.

SmartReporter Express Reports

Express Reports are based on data collected by Check Point system counters and SmartView Monitor History files. Standard Reports, in contrast, are based on Log Consolidator logs. Because Express Reports present historical data, they cannot be filtered, but they can be generated at a faster rate.

Express Reports are supported by one Client, the SmartReporter. To configure your system to generate Express Reports, see Express Reports Configuration.

The Express Report Architecture diagram illustrates the SmartReporter architecture for Express Network Reports:

Report Structure

Each report consists of a collection of sub-topics known as sections, which cover various aspects of the report. For example, the User Activity report consists of sections such as User Activity by Date, Top Users and Top Services for User Related Traffic.

Customizing Predefined Reports

You can easily customize the report that is closest to your needs (by changing its date range, filters etc.) to provide the desired information. Changing the filters of a predefined report constitutes a change in the nature of the report and the report must therefore be saved in a different location or under a different name. You can save the customized report under a different name in the report subject dedicated to user-defined reports, My Reports.

SmartReporter Considerations

SmartReporter's default options have been designed to address the most common reporting needs. To maximize the product's benefits, it is recommended that you adapt it to your specific profile. This section describes the considerations you should take into account before starting to use SmartReporter.

Standalone vs. Distributed Deployment

In a standalone deployment, all SmartReporter server components (the Log Consolidator Engine, the SmartReporter Database and the SmartReporter server) are installed on the Security Management server. In a distributed deployment, the SmartReporter server components and the Security Management server are installed on two different machines. They communicate through standard Check Point protocols such as LEA and CPMI.

In a standalone deployment, you can use one server for all of the management components. In a distributed deployment, the SmartReporter performance is significantly improved.

SmartReporter Backward Compatibility

In a standalone deployment, you can install SmartReporter on a Security Management server of the same version. In a distributed deployment, you can install SmartReporter on a Log server and manage it with a Security Management server of any supported version.

Log Availability vs. Log Storage and Processing

Since all SmartReporter operations are performed on the logs you have saved, the extent to which you can benefit from this product depends on the quality of the available logs. Therefore, you must ensure your Security Policy is indeed tracking (logging) all events you may later wish to see in your reports.

In addition, you should consider how accurately your logs represent your network activity. If only some of your Rules are tracking events that match them, the events' proportion in your reports will be distorted. For example, if only the blocked connections Rule is generating logs, the reports will give you the false impression that 100% of the activity in your network consisted of blocked connections.

On the other hand, tracking multiple connections results in an inflated log file, which not only requires more storage space and additional management operations, but significantly slows down the Consolidation process.

Log Consolidation Phase Considerations

Record Availability vs. Database Size

Reports are a direct reflection of the records stored in the SmartReporter Database. To generate detailed, wide-ranging and accurate reports, the corresponding data must be available in the database. You must configure the database settings to make sure the database does not exceed the available space.

Carefully consider which type of logs you store and how much you consolidate them.

Saving Consolidated Records to One vs. Multiple Database Tables

A report is generated based on a single table. If you save all consolidated records to the same table, all the data is readily accessible and you are saved the trouble of moving records between tables and selecting the appropriate source table for each report you wish to generate.

Dividing the records between different tables reduces the report generation time and allows you to maintain a useful database size by exporting tables you are not currently using to an external location.

High Availability

SmartReporter supports Security Management server High Availability.

In High Availability, the active Security Management server always has one or more standby Security Management servers that are ready to take over from the active Security Management server. These Security Management servers must all be of the same Operating System (for instance, all Windows NT), and have to be of the same version. The existence of the standby Security Management server allows for crucial backups to be in place:

• For the Security Management server - the various databases in the corporate organization, such as the database of objects and users, policy information and ICA files are stored on both the Standby SCSs as well as the active Security Management server. These Security Management servers are synchronized so data is maintained and ready to be used. If the active Security Management server is down, a standby Security Management server needs to become Active in order to be able to edit and install (that is, enforce) the Security Policy.
• For the gateway - certain operations that are performed by the gateways via the active Security Management server, such as fetching a Security Policy, or retrieving a CRL from the Security Management server, can be performed on standby Security Management server.

In a High Availability deployment the first installed Security Management server is specified as the Primary Security Management server. This is a regular Security Management server used by the system administrator to manage the Security Policy. When any subsequent Security Management server is installed, these must be specified as Secondary Security Management servers. Once the Secondary Security Management server has been installed and manually synchronized, the distinctions between Primary versus Secondary is no longer significant. These servers are now referred to according to their role in the Management High Availability scenario as Active or Standby, where any Security Management server can function as the active Security Management server.

When changes are made to report definitions (including report schedules), consolidation sessions and their settings, automatic maintenance configuration and report configuration, the information is stored in the active Security Management server and will be synchronized to the secondary Security Management server when a user synchronizes the Security Management servers.

The report generation results are not synchronized between Security Management servers. For instance, when SmartReporter generates a report connected to Security Management server A, a record of its generation will be stored in Security Management server A. When SmartReporter generates a report connected to Security Management server B, a record of its generation will be stored in Security Management server B. The Activity Log in Security Management server A will not be visible in Security Management server B and vice versa. However, even though the Activity Log in the inactive Security Management server A is not visible, it is still possible to connect to the inactive Security Management server A in read-only mode to access the report generations that are not visible in Security Management server B.

Report Generation Phase Considerations

Adapting the Report's Detail Level to your Needs

When a report is very detailed, it may become difficult to sort out the most significant results and understand it. To achieve the optimal balance between getting the right level of detail in your reports, closely examine the report's date range, filters (source, destination, service etc.) and filter values, and adjust them to pinpoint details.

Generating Only Selected Sections

By default, specific sections are included in the report generation and sections that require a great deal of resources (that is, report generation time and the report's size) are not selected. However, you can generate any sections in the list provided by checking them in the Content tab associated with the selected report in the Reports > Definitions view.

Scheduling Reports

The Schedule feature allows you to set both delayed and periodic report generations.

If you wish to produce a detailed and lengthy report, you should consider postponing its generation and scheduling it so that it does not run at time of peak log creation activity since such a report generation might slow down your system.

In addition, it is useful to identify the reports you require on a regular basis (for example, a daily alerts report or a monthly user activity report) and schedule their periodic generations.

Report Filters

Reports are based on records of the most commonly required filters, such as Source and Destination. Specifying the appropriate filter settings is the key to extracting the information you are looking for.

For each filter you choose, specify the values, such as network objects or services, to be matched out of all values available for that filter. The available values are taken from the Security Management server and are refreshed on a regular basis. If you cannot see a value you have added through SmartDashboard in the available values list, refresh the list by selecting a different filter and then return to the previous one.

The SmartReporter client also allows you to include additional objects by manually adding them to the matched values list.

Filters and their values can be specified for all sections of a report using the Filter tab, or for individual sections by editing the section from the Content tab. Filters for individual section set in the Content tab will override conflicting filters set for all sections using the Filter tab.

Report output (Email, FTP Upload, Web Upload, Custom)

All report results are displayed on your screen and saved to the SmartReporter server.

By default, the report is saved in HTML output in an index.htm file; and in CSV (Comma Separated Values) format in a tables.csv file. The HTML file includes descriptions and graphs, but the CSV file contains only the report table units, without a table of contents, descriptions or graphs. The tables.csv is provided in order to conveniently import tables into applications like Excel.

Report Files and Formats

Before generating a report, determine whether you want it to be saved or sent to additional or different targets. For example, when you generate a user activity-related report, you may wish to make it available to all managers in your organization by sending them the output via email or by placing it on your intranet.

SmartReporter Database Management

All database management operations are performed through the SmartReporter Database Maintenance view.

Tuning the SmartReporter Database

To improve performance, adjust available RAM memory for MySQL usage (see UpdateMySQLConfig -R option for additional information). In addition, place the database data and log files on different hard drives (physical disks), if available. Moving the temporary directory to a different hard drive will improve the performance of report generation and will avoid the possible clash between the temporary database directory and the space intended for the data directory.

Modifying SmartReporter Database Configuration

You can change the SmartReporter database settings by modifying the my.cnf file, located in the $RTDIR/Database/conf directory (in Windows: my.ini). Run the UpdateMySQLConfig application. Note that before running this application you must stop all SmartReporter services: run evstop -reporter.

When you run the UpdateMySQLConfig application, it creates a backup of the database configuration file.

There are a number of factors that can improve performance of the SmartReporter database. Most of these factors can be changed with the UpdateMySQLConfig utility.

• RAM - The database needs substantial amounts of RAM to buffer data up to 1200 MB. This can be set using UpdateMySQLConfig -R
• Temporary directories - The database uses temporary disk space to perform intermediate operations (such as sorting and grouping during report generation and during the table import operation) and may require up to 50% of the current database size to generate large reports. After report generation the temporary directory is emptied.
  Generating a substantial report may fail to execute the required SQL query if there is not enough disk space for the temporary directory. The temporary directory can be moved to a new location using UpdateMySQLConfig -T.
• Log files - The database log files ensure that changes persist in the event of a system crash. Place these files on a device that is separate from the database's data files using the UpdateMySQLConfig -L option.
• Database data files - these files should be put on a large, fast disk. The database's data files can be placed on several disks. Use UpdateMySQLConfig -A to add a new file to the set of database files and use UpdateMySQLConfig -M to move an existing file to a new location. Do not place database files on a network drive since performance may suffer and in some instances the database will not work.

  The default database file is ibdata1. If this file needs to be moved to a new absolute directory (for example, d:/Database/data), verify that the directory exists and run:

  UpdateMySQLConfig -M -src=ibdata1 -dst="d:/Database/data/ibdata1"

  If you want to remove an absolute directory (for example, d:/Database/data2 to d:/Database/data2), verify that the directory exists and run the following:

  UpdateMySQLConfig -M -src="d:/Database/data/ibdata1" -dst="d:/Database/data2/ibdata1"

• An alternative way to enlarge database capacity is to enlarge the maximum size of the default data file (ibdata1). Use the $RTDIR/Database/conf/my.cnf file (in Windows, my.ini) for the required configuration. In order to enlarge the maximum size of ibdata1 edit the value innodb_data_file_path and change its maximum. For example, change innodb_data_file_path=ibdata1:10M:autoextend:max:40G to innodb_data_file_path=ibdata1:10M:autoextend:max:60G. This will enable ibdata1 to grow up to 60G.

• Default data directory - this is the directory that contains the MySQL table definitions and data.

Changing the Database Data Directory

1. Run the command cpstop.
2. Move database files.

   The location of the database data files is specified in the mysql configuration file my.ini (Windows) or my.cnf (all other platforms).

   Open the mysql configuration file located in the directory $RTDIR/Database/conf/.

3. Locate the lines that begin as follows:

   - datadir=

   - innodb_data_file_path=

   The directories indicated by these entries are the directories and subdirectories that should be copied to the new location. The following example shows how these directories appear in the mysql configuration file.

   [mysqld]

   datadir="C:/Program Files/CheckPoint/EventiaSuite/R76/ReportingServer/Database/data"

   innodb_log_group_home_dir="C:/Program Files/CheckPoint/EventiaSuite/R76/ReportingServer/Database/log"

   innodb_data_file_path = ibdata1:10M:autoextend:max:40G

   The entry innodb_data_file_path, records database files that were added or moved to absolute locations. Make sure that these recorded database files are copied to a new location so that they are not forgotten.

4. Modify the following fields in the mysql configuration file so that they match the new locations of the database data files: datadir,innodb_data_file_path.

   Make sure that the paths are written in Unix format, with forward (/) slashes between directories.

5. Run the command cpstart.

UpdateMySQLConfig Syntax

The UpdateMySQLConfig Options table contains the usage of the UpdateMySQLConfig application.

UpdateMySQLConfig

[-A -f=string -s=number -auto[=true|=false] [ -m=number ] ]

[-R=number ]

[-M -src=string -dst=string ]

[-T=string ]

[-L=string ]

[-h ]

-A

-R

-M

-T

-L

-h

Automatically Maintaining the Size of the Database

The Log Consolidator process continuously adds new records into the database as they are generated from the Security Gateway. Eventually, the space allocated for the database will fill up. Typically, users can manually archive or delete older, less pertinent records from the database to provide space for the newest records. Automatic Maintenance performs this process automatically. With Automatic Maintenance, the user selects a maintenance operation (whether it is deleting records or archiving them to an external file) and specifies high and low watermarks to trigger when Automatic Maintenance should occur.

The High Watermark value represents the percentage of space that can occupy the database and/or the age of database records (that is, how many days old the records are). When the database occupies too much space or the records are older than the specified age, then the conditions are right to trigger an Automatic Maintenance operation. The High Watermark values are checked once a day and if the percentage of space or the age of the database records is higher than the assigned values, the Automatic Maintenance operation is triggered.

The Automatic Maintenance operation will delete records from the database until it reaches the Low Watermark. For example, if you specify that the High Watermark is 80% and the Low Watermark is 70% then the operation will begin to delete the oldest records when the occupied space is over 80%.

Typically, it is recommended that 80% would be the High Watermark to avoid reaching 100% capacity in certain cases.

In addition, it is possible to specify which database tables will participate in Automatic Maintenance. Since some of the tables are created for special purposes (for example, a table created from an external log file), Automatic Maintenance should not be performed on them.

When deletion of records occurs during automatic maintenance, you may see that the database size grows at first. This is normal behavior since the database needs to keep duplicate information in case of a server crash. The database will recover the disk space allocated for logs for about an hour after the maintenance operation is complete.

Backing Up the SmartReporter Database

The SmartReporter Database system consists of a set of files that can be copied, compressed or backed up like any other file. Backup files require the same disk space as the original files. It is highly recommended to save backup copies of the SmartReporter Database files, which can later be used to recover from an unexpected database corruption. Proceed as follows:

1. Stop the SmartReporter services by running: evstop -reporter.
2. From the SmartReporter Database directories, copy the entire data directory tree (as specified by the datadir parameter in the my.cnf or my.ini file) to the backup location. You may compress them to save disk space. Copy any database and log files that may have been moved to a different location using the UpdateMySQLConfig utility.
3. Restart the SmartReporter services and run rmdsstart.