Appendix: Manual Syslog Parsing

Many third-party devices use the syslog format to log. The Log ServerClosed Dedicated Check Point server that runs Check Point software to store and process logs. reformats the raw data to the Check Point log format to process third-party syslog messages. SmartEvent can take the reformatted logs and convert them into security events.

You can use the Log Parsing Editor to make a parsing file (see Importing Syslog Messages). As an alternative you can manually create a parsing file. This section shows you how to do that.

Warning - Manual modifications to out-of-the-box parsing files cannot be preserved automatically during an upgrade. Mark your modifications with comments so you can remember what changed.

Planning and Considerations

  1. Learn the accurate structure of the logs the device generates with these guides.

    1. The vendor logging guide, or other documentation that specifies the logs the device can generate and their structure. Documentation is important to make sure that you found all possible logs. Usually it is sufficient to write the parsing file.

    2. Log samples, as many as possible. Use logs generated from the actual devices to be used with SmartEvent. Samples are important to examine the parsing file and to tune it accordingly.

  2. Learn and know The Free Text Parsing Language and the necessary parsing files and their location on the Log Server (see The Parsing Procedure).

  3. Compare existing parsing files of an equivalent product.

  4. Select the fields to extract from the log. The fields to extract are different from one device to another. But devices of the same category usually have equivalent log fields. For example:

    Device Type

    Typical Log Fields

    Firewall, router and other devices that send connection based logs

    source IP address, destination IP address, source port, destination port, protocol, accept/reject indication

    IDS / IPSClosed Check Point Software Blade on a Security Gateway that inspects and analyzes packets and data for numerous types of risks (Intrusion Prevention System)., application Firewall and other devices that send attack logs

    attack name/ID

The Parsing Procedure

The procedure occurs on the Log Server and starts with the syslog daemon. The syslog daemon that runs on the Log Server receives the syslogs and calls for their parsing. The parsing involves many parsing files, which contain the different parsing definitions and specifications, and can be found in the $FWDIR/conf/syslog/ directory. In these files there are the device-specific parsing files, which define the actual parsing and extraction of fields, according to each device specific syslog format.

The parsing starts with the syslog_free_text_parser.C file. This file defines the different Dictionary terms and parses the syslog. The file extracts fields, which are common to all syslog messages (such as PRI, date and time), and the machine and application that generated the syslog.

The syslog_free_text_parser.C file uses the allDevices.C file (which refers to two files: UserDefined/UserDefinedSyslogDevices.C and CPdefined/CPdefinedSyslogDevices.C).

  • The first file (UserDefined/UserDefinedSyslogDevices.C) contains the names of the devices parsing files that the user defines.

  • The second file (CPdefined/CPdefinedSyslogDevices.C) contains devices parsing files that Check Point defines.

The allDevices.C file goes over the device parsing files, and tries to match the incoming syslog with the syslog format parsed in that file.

After the parsing-file succeeds in the preliminary parsing of the syslog (that is, it matches the syslog format and is therefore the syslog origin), the remaining of the syslog is parsed in that file. If a match is not found, the file will continue to go over the Check Point device parsing files until it finds a match.