Print Download PDF Send Feedback

Previous

Appendix: Manual Syslog Parsing

Many third-party devices use the syslog format to log. The Log Server reformats the raw data to the Check Point log format to process third-party syslog messages. SmartEvent can take the reformatted logs and convert them into security events.

You can use the Log Parsing Editor to make a parsing file. As an alternative you can manually create a parsing file. This section shows you how to do that.

Warning - Manual modifications to out-of-the-box parsing files cannot be preserved automatically during an upgrade. Mark your modifications with comments so you can remember what changed.

Planning and Considerations

  1. Learn the accurate structure of the logs the device generates with these guides.
    1. The vendor logging guide, or other documentation that specifies the logs the device can generate and their structure. Documentation is important to make sure that you found all possible logs. Usually it is sufficient to write the parsing file.
    2. Log samples, as many as possible. Use logs generated from the actual devices to be used with SmartEvent. Samples are important to examine the parsing file and to tune it accordingly.
  2. Learn and know the Free Text Parsing Language and the necessary parsing files and their location on the Log Server.
  3. Compare existing parsing files of an equivalent product.
  4. Select the fields to extract from the log. The fields to extract are different from one device to another. But devices of the same category usually have equivalent log fields. For example:

Device Type

Typical Log Fields

Firewall, router and other devices that send connection based logs

source IP address, destination IP address, source port, destination port, protocol, accept/reject indication

IDS / IPS, application Firewall and other devices that send attack logs

attack name/ID

The Parsing Procedure

The procedure occurs on the Log Server and starts with the syslog daemon. The syslog daemon that runs on the Log Server receives the syslogs and calls for their parsing. The parsing involves many parsing files, which contain the different parsing definitions and specifications, and can be found in $FWDIR\conf\syslog\. In these files there are the device-specific parsing files, which define the actual parsing and extraction of fields, according to each device specific syslog format.

The parsing starts with the syslog_free_text_parser.C file. This file defines the different dictionaries and parses the syslog. The file extracts fields which are common to all syslog messages (such as PRI, date and time), and the machine and application that generated the syslog.

syslog_free_text_parser.C uses the allDevices.C file (which refers to two files: UserDefined/UserDefinedSyslogDevices.C and CPdefined/CPdefinedSyslogDevices.C). The first (UserDefined/UserDefinedSyslogDevices.C) contains the names of the devices parsing files that the user defines. The second (CPdefined/CPdefinedSyslogDevices.C) contains devices parsing files that Check Point defines. allDevices.C goes over the device parsing files, and tries to match the incoming syslog with the syslog format parsed in that file.

After the parsing-file succeeds in the preliminary parsing of the syslog (that is, it matches the syslog format and is therefore the syslog origin), the remaining of the syslog is parsed in that file. If a match is not found, the file will continue to go over the Check Point device parsing files until it finds a match.

Manual Syslog Parsing

To parse a syslog file:

  1. Create a new parsing file called <device product name>.C.
  2. Put this file in the directory $FWDIR/conf/syslog/UserDefined on the Log Server.
  3. On the Log Server, edit the file $FWDIR/conf/syslog/UserDefined/UserDefinedSyslogDevices.C to add a line that includes the new parsing file. For example:

    : (
    :command (
    :cmd_name (include)
    :file_name ("snortPolicy.C")
    )
    )

  4. Optional: If required:
    1. Create a new dictionary file called <device product name>_dict.ini.
    2. Put it in the directory $FWDIR/conf/syslog/UserDefined on the Log Server.

      A dictionary translates values with the same meaning from logs from different devices into a common value. This common value is used in the Event Definitions.

    3. Edit the file $FWDIR/conf/syslog/UserDefined/UserDefinedSyslogDictionaries.C on the Log Server.
    4. Add a line to include the dictionary file. For example:

    :filename ("snort_dict.ini")

  5. To examine the parsing, send syslog samples to a Check Point Log Server.

To send syslog samples:

  1. To configure the Log Server to accept syslogs, connect to the Security Management Server with SmartConsole.
  2. In Logs and Masters > Additional Logging Configuration, enable the property Accept Syslog messages.
  3. Edit the Log Server network object.
  4. Run the commands cpstop & cpstart, or fw kill fwd & fwd –n.

    The fwd procedure on the Log Server restarts.

  5. Send syslogs from the device itself, or from a syslog generator.

    For example: Kiwi Syslog Message Generator, available at http://www.kiwisyslog.com/software_downloads.htm#sysloggen.

Troubleshooting:

If SmartConsole does not show the logs as expected, there can be problems with the parsing files:

The Free Text Parsing Language

The free text parsing language enables to parse an input string, extract information, and define log fields. These log fields which show as part of the Check Point log in the Log Server. They are used in the definition of events. Each parsing file contains a tree of commands. Each command examines or parses part of the input string (sometimes it adds fields to the log as a result), and decides if to continue to parse the string (according to the success/failure of its execution).

The Commands

Each command consists of these parts:

Sample

:command (
      :cmd_name (try)
      :try_arguments
           .
           .

      :on_success (
             :command()
      )
      :on_fail (
             :command()
      )
)

Try

The try command matches a regular expression against the input string.

Try Command Parameters

Argument

Description

parse_from

start_position - run the regular expression from the start of the input string.

last_position - run the regular expression from the last position of the previous successful command.

regexp

The regular expression to match.

add_field

One or more fields to add to the result (only if the regular expression is successful).

Try Command Sample

:command (
     :cmd_name (try)
     :parse_from (start_position)
     :regexp ("([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)")
     :add_field (
             :type (index)
             :field_name (Src)
             :field_type (ipaddr)
             :field_index (1)
     )
)

In the above example, we try to match the regular expression ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) that looks at the entire log (parse_from (start_position)) - parse from the start of the log). If the regular expression is matched, we add a source field.

Group_try

The command group_try executes one or more commands in one of these modes:

The command group_try is commonly used when it parses a "free-text" piece of a log, which contains a number of fields we want to extract. For example:

%PIX-6-605004: Login denied from 194.29.40.24/4813 to outside:192.168.35.15/ssh for user 'root'

When you look at see this section of the log, you can use this structure:

Group_try Command Sample 1

:command (
      :cmd_name (group_try)
      :mode (try_all_successively)
      :(
            # A "try" command for the source.
            :command ()
      )
      :(
            # A "try" command for the destination.
            :command ()
      )
      :(

            # A "try" command for the user.
            :command ()
      )
                 .
                 .
                 .
)

In this example, the first try command in the group_try block (for the source) is executed.

If the source, destination and user are not in a specified sequence in the syslog, use the try_all mode instead of try_all_successively.

Group_try Command Sample 2

In this example, the regular expressions in the different commands try to match more specified logs. At most, one command in the group_try block will be successful. When it is found, it is not necessary to examine the others:

:command (
    :cmd_name (group_try)
    :mode (try_until_success)
    :(
         :command (
         .
         .
         .
           :regexp ("(\(|)(login|su)(\)|).* session (opened|closed) for
user ([a-z,A-Z,0-9]*)")
         )
    )
    :(
         :command (
             .
             .
             .
           :regexp ("(\(|)su(\)|).* authentication failure; logname=([a-zA-Z0-9]*).*
user=([a-zA-Z0-9]*)")
         )
    )
         .
         .
         .
)

Note - When you add a new device, the first try command in the parsing file must use the try until success parameter:

:cmd_name (group_try)
:mode (try_until_success)
: (
….
)

Switch

This command enables to compare the result of a specified field against a list of predefined constant values.

Switch Command Parameters

Parameter

Description

field_name

The field name whose value is checked.

case

One or more case attributes followed by the value with which to compare.

default

Execute only if no relevant case is available. The default value is optional.

Switch Command Sample

:command (

     :cmd_name (switch)

     :field_name (msgID)

     :(
           :case (302005)
           :command ()
     )
     :(
           :case (302001)
           :case (302002)
           :command ()
     )
     :default (
           :command()
     )
)

Unconditional _try

This command is an "empty" command that allows you to add fields to the result without any conditions.

Unconditional _try Command Sample 1

:command (
      :cmd_name (unconditional_try)
      :add_field (
            :type (const)
            :field_name (product)
            :field_type (string)
            :field_value ("Antivirus")
      )
)

A common usage of unconditional_try is with the switch command. In example 2, each message ID is attached with its corresponding message field which denotes its meaning.

Unconditional _try Command Sample 2

:command (
      :cmd_name (switch)
      :field_name (msgID)
(
:case (106017)
:command (
:cmd_name (unconditional_try)
:add_field (
:type (const)
:field_name (message)
:field_type (string_id)
:field_value ("LAND Attack")
)
)
)
:(
:case (106020)
:command (
:cmd_name (unconditional_try)
:add_field (
:type (const)
:field_name (message)
:field_type (string_id)
:field_value ("Teardrop Attack")
)
)
)
.
.
.
)

Include

This command enables the inclusion of a new parsing file.

file_name

The full path plus the file name of the file to be included.

Include Command Sample

:command (
     :cmd_name (include)
     :file_name ("c:\freeTextParser\device\antivirusPolicy.C")
)

add_field

Each add_field has some parameters:

Field_name - the name of the new field. There are some fields, which have corresponding columns in SmartConsole Logs & Monitor > Logs. This table shows the names to give these fields to show in their Logs & Monitor > Logs column (and not in the Information field, where other added fields appear):

Field Name to be Given

Column in Logs & Monitor > Logs

Src

Source

Dst

Destination

proto

Protocol

s_port

Source Port

product

Product

service

Service (when resolved includes the port

and protocol.)

Action

Action

ifname

Interface

User

User

When you name the above fields accordingly, they are placed in their correct column in Logs & Monitor > Logs. This enables them to participate in all filtering done on these columns. These fields automatically take part in existing event definitions with these field names.

Field_type - the type of the field in the log. This table shows the possible field types.

Field Type

Comment

int

 

uint

 

string

 

ipaddr

For IP addresses used with the Src and Dst fields.

pri

Includes the facility and severity of a syslog.

timestmp

Includes the date and time of the syslog. Supports the format 'Oct 10 2004 15:05:00'.

time

Supports the format '15:05:00'.

string_id

For a more efficient usage of strings. Used when there is a finite number of possible values for this field.

action

Supports these actions: drop, reject, accept, encrypt, decrypt, vpnroute, keyinst, authorize, deauthorize, authcrypt, and default.

ifdir

0 - inbound

1 - outbound

ifname

For an interface name (used with the "ifname" field).

protocol

The field name should be "proto".

port

For "service", "s_port" or "port" fields.

The field type of the field names in this table must be as mentioned:

Field Name

Field Type

Src

ipaddr

Dst

ipaddr

proto

protocol

s_port

port

service

port

Action

action

ifname

ifname

Add_field Command Sample

:command (
     :cmd_name (try)
     :parse_from (last_position)
     :regexp ("Failed password for ([a-zA-Z0-9]+) from
([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) port ([0-9]+)")
     :add_field (
        :type (index)
        :field_name (User)
        :field_type (string)
        :field_index (1)
     )
     :add_field (
        :type (index)
        :field_name (Src)
        :field_type (ipaddr)
        :field_index (2)
     )
     :add_field (
        :type (index)
        :field_name (port)
        :field_type (port)
        :field_index (3)
     )
)

The pattern for the User, [a-zA-Z0-9]+, is located in the first pair of brackets. Therefore, the field_index is one. The pattern for the Source address, [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+, is located in the second pair of brackets. Therefore, the index is two. The pattern for the port is in the third pair of brackets.

In each parsed regular expression the maximum number of brackets must be up to nine. To extract more than nine elements from the regular expression, break the expression into two pieces. The first regular expression contains the first nine brackets. The remaining of the regular expression is in the on_success command.

:command (
:cmd_name (try)
:parse_from (start_position)
:regexp ("access-list (.*) (permitted|denied|est-allowed)
([a-zA-Z0-9_\([a-zA-Z0-9_\\.[0-9]+\.[0-9]+\.[0-9]+)\(([0-9]*)\) -> ")
:add_field (
:type (index)
:field_name (listID)
:field_type (string)
:field_index (1)
)
:add_field (
:type (index)
:field_name (action)
:field_type (action)
:field_index (2)
)
:add_field (
:type (index)
:field_name (proto)
:field_type (protocol)
:field_index (3)
)
:add_field (
:type (index)
:field_name (ifname)
:field_type (ifname)
:field_index (4)
)
:add_field (
:type (index)
:field_name (Src)
:field_type (ipaddr)
:field_index (5)
)
:on_success (
:command (
:cmd_name (try)
:parse_from (last_position)
:regexp
("([a-zA-Z0-9_\\.[0-9]+\.[0-9]+\.[0-9]+)\(([0-9]*)\) hit-cnt ([0-9]+) ")
:add_field (
:type (index)
:field_name (destination_interface)
:field_type (string)
:field_index (1)
)
)
)
)

field_value is the constant value to be added.

:command (
      :cmd_name (try)
      :parse_from (last_position)
      :regexp ("%PIX-([0-9])-([0-9]*)"))
      :add_field (
             :type (const)
             :field_name (product)
             :field_type (string_id)
             :field_value ("CISCO PIX")
      )
)

Dict_name is the name of the dictionary to use to convert the value. If the value is not found in the dictionary, the value is the result. See Dictionary.

Dictionary

The free text parser enables us to use dictionaries to convert values from the log. These conversions are used to translate values from logs from different devices, with the same meaning, into a common value, which is used in the event definitions.

Each dictionary file is defined as an .ini file. In the ini file the section name is the dictionary name and the values are the dictionary values (each dictionary can include one or more sections).

[dictionary_name]
Name1 = val1
Name2 = val2
cisco_action]          [3com_action]
permitted = accept      Permit    = accept
denied = reject         Deny      = reject

Dictionary Sample

The reference to a dictionary in the parsing file is shown in this table:

Dictionary Command Sample 2

:command (
      :cmd_name (try)
      :parse_from (start_position)
      :regexp ("list (.*) (permitted|denied) (icmp)
([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) -> ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).* packet")
       :add_field (
               :type (index)
               :field_name (action)
               :field_type (action)
               :field_index (2)
               :dict_name (cisco_action)
       )
)