Appendix: Manual Syslog Parsing

Many third-party devices use the syslog format to log. The Log Server reformats the raw data to the Check Point log format to process third-party syslog messages. SmartEvent can take the reformatted logs and convert them into security events.

You can use the Log Parsing Editor to make a parsing file. As an alternative you can manually create a parsing file. This section shows you how to do that.

Warning - Manual modifications to out-of-the-box parsing files cannot be preserved automatically during an upgrade. Mark your modifications with comments so you can remember what changed.

Planning and Considerations

1. Learn the accurate structure of the logs the device generates with these guides.
   1. The vendor logging guide, or other documentation that specifies the logs the device can generate and their structure. Documentation is important to make sure that you found all possible logs. Usually it is sufficient to write the parsing file.
   2. Log samples, as many as possible. Use logs generated from the actual devices to be used with SmartEvent. Samples are important to examine the parsing file and to tune it accordingly.
2. Learn and know the Free Text Parsing Language and the necessary parsing files and their location on the Log Server.
3. Compare existing parsing files of an equivalent product.
4. Select the fields to extract from the log. The fields to extract are different from one device to another. But devices of the same category usually have equivalent log fields. For example:

   Device Type	Typical Log Fields
   Firewall, router and other devices that send connection based logs	source IP address, destination IP address, source port, destination port, protocol, accept/reject indication
   IDS / IPS, application Firewall and other devices that send attack logs	attack name/ID

The Parsing Procedure

The procedure occurs on the Log Server and starts with the syslog daemon. The syslog daemon that runs on the Log Server receives the syslogs and calls for their parsing. The parsing involves many parsing files, which contain the different parsing definitions and specifications, and can be found in the $FWDIR/conf/syslog/ directory. In these files there are the device-specific parsing files, which define the actual parsing and extraction of fields, according to each device specific syslog format.

The parsing starts with the syslog_free_text_parser.C file. This file defines the different dictionaries and parses the syslog. The file extracts fields, which are common to all syslog messages (such as PRI, date and time), and the machine and application that generated the syslog.

The syslog_free_text_parser.C file uses the allDevices.C file (which refers to two files: UserDefined/UserDefinedSyslogDevices.C and CPdefined/CPdefinedSyslogDevices.C).

• The first file (UserDefined/UserDefinedSyslogDevices.C) contains the names of the devices parsing files that the user defines.
• The second file (CPdefined/CPdefinedSyslogDevices.C) contains devices parsing files that Check Point defines.

The allDevices.C file goes over the device parsing files, and tries to match the incoming syslog with the syslog format parsed in that file.

After the parsing-file succeeds in the preliminary parsing of the syslog (that is, it matches the syslog format and is therefore the syslog origin), the remaining of the syslog is parsed in that file. If a match is not found, the file will continue to go over the Check Point device parsing files until it finds a match.

Manual Syslog Parsing

To parse a syslog file:

1. Create a new parsing file called <device product name>.C.
2. Put this file in the directory $FWDIR/conf/syslog/UserDefined on the Log Server.
3. On the Log Server, edit the file $FWDIR/conf/syslog/UserDefined/UserDefinedSyslogDevices.C to add a line that includes the new parsing file. For example:

   : ( :command ( :cmd_name (include) :file_name ("snortPolicy.C") ) )

4. Optional: If required:
   1. Create a new dictionary file called <device product name>_dict.ini.
   2. Put it in the directory $FWDIR/conf/syslog/UserDefined on the Log Server.

      A dictionary translates values with the same meaning from logs from different devices into a common value. This common value is used in the Event Definitions.

   3. Edit the file $FWDIR/conf/syslog/UserDefined/UserDefinedSyslogDictionaries.C on the Log Server.
   4. Add a line to include the dictionary file. For example:

   :filename ("snort_dict.ini")

5. To examine the parsing, send syslog samples to a Check Point Log Server.

To send syslog samples:

1. To configure the Log Server to accept syslogs, connect to the Security Management Server with SmartConsole.
2. In Logs and Masters > Additional Logging Configuration, enable the property Accept Syslog messages.
3. Edit the Log Server network object.
4. Run the commands cpstop & cpstart, or fw kill fwd & fwd –n.

   The fwd procedure on the Log Server restarts.

5. Send syslogs from the device itself, or from a syslog generator.

   For example: Kiwi Syslog Message Generator, available at http://www.kiwisyslog.com/software_downloads.htm#sysloggen.

Troubleshooting:

If SmartConsole does not show the logs as expected, there can be problems with the parsing files:

• If there is a syntax error in the parsing files, an error message shows. To read a specified error message, set the TDERROR_ALL_FTPARSER value to 5 before you run the procedure fwd -n.
• If the syslogs show in SmartConsole with 'Product syslog', the log was not parsed properly, but as a general syslog.
• If the Product field contains another product (not the one you have just added) this means there is a problem with the other product parsing file. Report this to the Check Point SmartEvent team.
• If the product reports correctly in the log, look for all the fields you extracted. Some of them are in the Information section. Some fields can be seen only when you select More Columns.

The Free Text Parsing Language

The free text parsing language enables to parse an input string, extract information, and define log fields. These log fields which show as part of the Check Point log in the Log Server. They are used in the definition of events. Each parsing file contains a tree of commands. Each command examines or parses part of the input string (sometimes it adds fields to the log as a result), and decides if to continue to parse the string (according to the success/failure of its execution).

The Commands

Each command consists of these parts:

• cmd_name - the name of the command.
• command arguments - arguments that define the behavior of the command.
• on_success (optional) - the next command executed if the current command execution succeeds.
• on_fail (optional) - the next command executed if the current command execution fails.

Sample

:command (       :cmd_name (try)       :try_arguments            .            .       :on_success (              :command()       )       :on_fail (              :command()       ) )

Try

The try command matches a regular expression against the input string.

Try Command Parameters

Argument	Description
parse_from	start_position - run the regular expression from the start of the input string. last_position - run the regular expression from the last position of the previous successful command.
regexp	The regular expression to match.
add_field	One or more fields to add to the result (only if the regular expression is successful).

Try Command Sample

:command (      :cmd_name (try)      :parse_from (start_position)      :regexp ("([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)")      :add_field (              :type (index)              :field_name (Src)              :field_type (ipaddr)              :field_index (1)      ) )

In the above example, we try to match the regular expression ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) that looks at the entire log (parse_from (start_position)) - parse from the start of the log). If the regular expression is matched, we add a source field.

Group_try

The command group_try executes one or more commands in one of these modes:

• try_all tries all commands in the group, and ignores the return code of the commands.
• try_all_successively tries all the commands in the group, and ignores the return code of the commands.

  Each command tries to execute from the last position of the earlier successful command.

• try_until_success tries all the commands until one succeeds.
• try_until_fail tries all the commands until one fails.

The command group_try is commonly used when it parses a "free-text" piece of a log, which contains a number of fields we want to extract. For example:

%PIX-6-605004: Login denied from 194.29.40.24/4813 to outside:192.168.35.15/ssh for user 'root'

When you look at see this section of the log, you can use this structure:

Group_try Command Sample 1

:command (       :cmd_name (group_try)       :mode (try_all_successively)       :(             # A "try" command for the source.             :command ()       )       :(             # A "try" command for the destination.             :command ()       )       :(             # A "try" command for the user.             :command ()       )                  .                  .                  . )

In this example, the first try command in the group_try block (for the source) is executed.

If the source, destination and user are not in a specified sequence in the syslog, use the try_all mode instead of try_all_successively.

Group_try Command Sample 2

In this example, the regular expressions in the different commands try to match more specified logs. At most, one command in the group_try block will be successful. When it is found, it is not necessary to examine the others:

:command (     :cmd_name (group_try)     :mode (try_until_success)     :(          :command (          .          .          .            :regexp ("(\(|)(login|su)(\)|).* session (opened|closed) for user ([a-z,A-Z,0-9]*)")          )     )     :(          :command (              .              .              .            :regexp ("(\(|)su(\)|).* authentication failure; logname=([a-zA-Z0-9]*).* user=([a-zA-Z0-9]*)")          )     )          .          .          . )

Note - When you add a new device, the first try command in the parsing file must use the try until success parameter:

:cmd_name (group_try)
:mode (try_until_success)
: (
….
)

Switch

This command enables to compare the result of a specified field against a list of predefined constant values.

Switch Command Parameters

Parameter	Description
field_name	The field name whose value is checked.
case	One or more case attributes followed by the value with which to compare.
default	Execute only if no relevant case is available. The default value is optional.

Switch Command Sample

:command (      :cmd_name (switch)      :field_name (msgID)      :(            :case (302005)            :command ()      )      :(            :case (302001)            :case (302002)            :command ()      )      :default (            :command()      ) )

Unconditional _try

This command is an "empty" command that allows you to add fields to the result without any conditions.

Unconditional _try Command Sample 1

:command (       :cmd_name (unconditional_try)       :add_field (             :type (const)             :field_name (product)             :field_type (string)             :field_value ("Antivirus")       ) )

A common usage of unconditional_try is with the switch command. In example 2, each message ID is attached with its corresponding message field which denotes its meaning.

Unconditional _try Command Sample 2

:command (       :cmd_name (switch)       :field_name (msgID) ( :case (106017) :command ( :cmd_name (unconditional_try) :add_field ( :type (const) :field_name (message) :field_type (string_id) :field_value ("LAND Attack") ) ) ) :( :case (106020) :command ( :cmd_name (unconditional_try) :add_field ( :type (const) :field_name (message) :field_type (string_id) :field_value ("Teardrop Attack") ) ) ) . . . )

Include

This command enables the inclusion of a new parsing file.

file_name

The full path plus the file name of the file to be included.

Include Command Sample

:command (      :cmd_name (include)      :file_name ("c:\freeTextParser\device\antivirusPolicy.C") )

add_field

Each add_field has some parameters:

• Type - The type of the add_field command. This parameter has these possible values:
  • Index - Part of the regular expression will be extracted as the field. The field_index value denotes which part will be extracted (see field_index bullet).
  • Const - Add a constant field whose value does not depend on information extracted from the regular expression. See field_value bullet.

Field_name - the name of the new field. There are some fields, which have corresponding columns in SmartConsole Logs & Monitor > Logs. This table shows the names to give these fields to show in their Logs & Monitor > Logs column (and not in the Information field, where other added fields appear):

Field Name to be Given	Column in Logs & Monitor > Logs
Src	Source
Dst	Destination
proto	Protocol
s_port	Source Port
product	Product
service	Service (when resolved includes the port and protocol.)
Action	Action
ifname	Interface
User	User

When you name the above fields accordingly, they are placed in their correct column in Logs & Monitor > Logs. This enables them to participate in all filtering done on these columns. These fields automatically take part in existing event definitions with these field names.

Field_type - the type of the field in the log. This table shows the possible field types.

Field Type	Comment
int
uint
string
ipaddr	For IP addresses used with the Src and Dst fields.
pri	Includes the facility and severity of a syslog.
timestmp	Includes the date and time of the syslog. Supports the format 'Oct 10 2004 15:05:00'.
time	Supports the format '15:05:00'.
string_id	For a more efficient usage of strings. Used when there is a finite number of possible values for this field.
action	Supports these actions: drop, reject, accept, encrypt, decrypt, vpnroute, keyinst, authorize, deauthorize, authcrypt, and default.
ifdir	0 - inbound 1 - outbound
ifname	For an interface name (used with the "ifname" field).
protocol	The field name should be "proto".
port	For "service", "s_port" or "port" fields.

The field type of the field names in this table must be as mentioned:

Field Name	Field Type
Src	ipaddr
Dst	ipaddr
proto	protocol
s_port	port
service	port
Action	action
ifname	ifname

• field_index or field_value - The parameter used depends on the value of the "type" field. If it is index, field_index shows. If it is const, field_value shows.

  field_index denotes which part of the regular expression is extracted, according to the grouping of the patterns. To make this grouping, write a certain expression in brackets. In this expression, the number in field_index denotes the bracket number whose pattern is taken into account.

Add_field Command Sample

:command (      :cmd_name (try)      :parse_from (last_position)      :regexp ("Failed password for ([a-zA-Z0-9]+) from ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) port ([0-9]+)")      :add_field (         :type (index)         :field_name (User)         :field_type (string)         :field_index (1)      )      :add_field (         :type (index)         :field_name (Src)         :field_type (ipaddr)         :field_index (2)      )      :add_field (         :type (index)         :field_name (port)         :field_type (port)         :field_index (3)      ) )

The pattern for the User, [a-zA-Z0-9]+, is located in the first pair of brackets. Therefore, the field_index is one. The pattern for the Source address, [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+, is located in the second pair of brackets. Therefore, the index is two. The pattern for the port is in the third pair of brackets.

In each parsed regular expression the maximum number of brackets must be up to nine. To extract more than nine elements from the regular expression, break the expression into two pieces. The first regular expression contains the first nine brackets. The remaining of the regular expression is in the on_success command.

:command ( :cmd_name (try) :parse_from (start_position) :regexp ("access-list (.*) (permitted|denied|est-allowed) ([a-zA-Z0-9_\([a-zA-Z0-9_\\.[0-9]+\.[0-9]+\.[0-9]+)\(([0-9]*)\) -> ") :add_field ( :type (index) :field_name (listID) :field_type (string) :field_index (1) ) :add_field ( :type (index) :field_name (action) :field_type (action) :field_index (2) ) :add_field ( :type (index) :field_name (proto) :field_type (protocol) :field_index (3) ) :add_field ( :type (index) :field_name (ifname) :field_type (ifname) :field_index (4) ) :add_field ( :type (index) :field_name (Src) :field_type (ipaddr) :field_index (5) ) :on_success ( :command ( :cmd_name (try) :parse_from (last_position) :regexp ("([a-zA-Z0-9_\\.[0-9]+\.[0-9]+\.[0-9]+)\(([0-9]*)\) hit-cnt ([0-9]+) ") :add_field ( :type (index) :field_name (destination_interface) :field_type (string) :field_index (1) ) ) ) )

field_value is the constant value to be added.

:command (       :cmd_name (try)       :parse_from (last_position)       :regexp ("%PIX-([0-9])-([0-9]*)"))       :add_field (              :type (const)              :field_name (product)              :field_type (string_id)              :field_value ("CISCO PIX")       ) )

Dict_name is the name of the dictionary to use to convert the value. If the value is not found in the dictionary, the value is the result. See Dictionary.

Dictionary

The free text parser enables us to use dictionaries to convert values from the log. These conversions are used to translate values from logs from different devices, with the same meaning, into a common value, which is used in the event definitions.

Each dictionary file is defined as an .ini file. In the ini file the section name is the dictionary name and the values are the dictionary values (each dictionary can include one or more sections).

[dictionary_name] Name1 = val1 Name2 = val2 cisco_action]          [3com_action] permitted = accept      Permit    = accept denied = reject         Deny      = reject

Dictionary Sample

The reference to a dictionary in the parsing file is shown in this table:

Dictionary Command Sample 2

:command (       :cmd_name (try)       :parse_from (start_position)       :regexp ("list (.*) (permitted|denied) (icmp) ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) -> ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).* packet")        :add_field (                :type (index)                :field_name (action)                :field_type (action)                :field_index (2)                :dict_name (cisco_action)        ) )