The Free Text Parsing Language

The free text parsing language enables to parse an input string, extract information, and define log fields. These log fields which show as part of the Check Point log in the Log ServerClosed Dedicated Check Point server that runs Check Point software to store and process logs.. They are used in the definition of events. Each parsing file contains a tree of commands. Each command examines or parses part of the input string (sometimes it adds fields to the log as a result), and decides if to continue to parse the string (according to the success/failure of its execution).

The Commands

Each command consists of these parts:

  • cmd_name - the name of the command.

  • command arguments - arguments that define the behavior of the command.

  • on_success (optional) - the next command executed if the current command execution succeeds.

  • on_fail (optional) - the next command executed if the current command execution fails.

Sample

:command (
   :cmd_name (try)
   :try_arguments
        .
        .
   :on_success (
       :command()
   )
   :on_fail (
       :command()
   )
)

Try

The try command matches a regular expression against the input string.

'Try' Command Parameters

Argument

Description

parse_from

start_position - run the regular expression from the start of the input string.

last_position - run the regular expression from the last position of the previous successful command.

regexp

The regular expression to match.

add_field

One or more fields to add to the result (only if the regular expression is successful).

:command (
     :cmd_name (try)
     :parse_from (start_position)
     :regexp ("([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)")
     :add_field (
       :type (index)
       :field_name (Src)
       :field_type (ipaddr)
       :field_index (1)
     )
)

In the above example, we try to match the regular expression "([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)" that looks at the entire log (parse_from (start_position)) - parse from the start of the log). If the regular expression is matched, we add a source field.

Group_try

The command group_try executes one or more commands in one of these modes:

  • "try_all" tries all commands in the group, and ignores the return code of the commands.

  • "try_all_successively" tries all the commands in the group, and ignores the return code of the commands.

    Each command tries to execute from the last position of the earlier successful command.

  • "try_until_success" tries all the commands until one succeeds.

  • "try_until_fail" tries all the commands until one fails.

The command "group_try" is commonly used when it parses a "free-text" piece of a log, which contains a number of fields we want to extract.

For example:

%PIX-6-605004: Login denied from 194.29.40.24/4813 to outside:192.168.35.15/ssh for user 'root'

When you look at see this section of the log, you can use this structure:

:command (
   :cmd_name (group_try)
   :mode (try_all_successively)
   :(
      # A "try" command for the source.
      :command ()
   )
   :(
      # A "try" command for the destination.
      :command ()
   )
   :(
      # A "try" command for the user.
      :command ()
   )
           .
           .
           .
)

In this example, the first try command in the "group_try" block (for the source) is executed.

If the source, destination and user are not in a specified sequence in the syslog, use the "try_all" mode instead of "try_all_successively".

In this example, the regular expressions in the different commands try to match more specified logs.

At most, one command in the group_try block will be successful.

When it is found, it is not necessary to examine the others:

:command (
   :cmd_name (group_try)
   :mode (try_until_success)
   :(
      :command (
      .
      .
      .
        :regexp ("(\(|)(login|su)(\)|).* session (opened|closed) for user ([a-z,A-Z,0-9]*)")
      )
    )
   :(
      :command (
       .
       .
       .
        :regexp ("(\(|)su(\)|).* authentication failure; logname=([a-zA-Z0-9]*).* user=([a-zA-Z0-9]*)")
      )
    )
      .
      .
      .
)

Note - When you add a new device, the first "try" command in the parsing file must use the "try_until_success" parameter:

:cmd_name (group_try)
:mode (try_until_success)
: (
?
)

Switch

This command enables to compare the result of a specified field against a list of predefined constant values.

'Switch' Command Parameters

Parameter

Description

Parameter

Description

field_name

The field name whose value is checked.

case

One or more case attributes followed by the value with which to compare.

default

Execute only if no relevant case is available. The default value is optional.

:command (
   :cmd_name (switch)
   :field_name (msgID)
   :(
      :case (302005)
      :command ()
     )
   :(
      :case (302001)
      :case (302002)
      :command ()
     )
   :default (
      :command()
     )
)

Unconditional_try

This command is an "empty" command that allows you to add fields to the result without any conditions.

:command (
   :cmd_name (unconditional_try)
   :add_field (
      :type (const)
      :field_name (product)
      :field_type (string)
      :field_value ("Antivirus")
   )
)

A common usage of unconditional_try is with the switch command.

In this example, each message ID is attached with its corresponding "message" field which denotes its meaning.

:command (
   :cmd_name (switch)
   :field_name (msgID)
   (
   :case (106017)
   :command (
      :cmd_name (unconditional_try)
      :add_field (
      :type (const)
      :field_name (message)
      :field_type (string_id)
      :field_value ("LAND Attack")
      )
   )
   )
   :(
   :case (106020)
   :command (
      :cmd_name (unconditional_try)
      :add_field (
      :type (const)
      :field_name (message)
      :field_type (string_id)
      :field_value ("Teardrop Attack")
      )
   )
   )
   .
   .
   .
)

Include

This command enables the inclusion of a new parsing file.

file_name

The full path plus the file name of the file to be included.

:command (
   :cmd_name (include)
   :file_name ("c:\freeTextParser\device\antivirusPolicy.C")
)

Add_field

Each "add_field" has some parameters:

  • Type - The type of the "add_field" command. This parameter has these possible values:

    • Index - Part of the regular expression will be extracted as the field. The "field_index" value denotes which part will be extracted (see "field_index" bullet).

    • Const - Add a constant field whose value does not depend on information extracted from the regular expression. See field_value bullet.

  • field_name - the name of the new field.

    There are some fields, which have corresponding columns in SmartConsoleClosed Check Point GUI application used to manage a Check Point environment - configure Security Policies, configure devices, monitor products and events, install updates, and so on. > Logs & Monitor > Logs.

    This table shows the names to give these fields to show in their Logs & Monitor > Logs column (and not in the Information field, where other added fields appear):

    Field Name to be Given

    Column in Logs & Monitor > Logs

    Src

    Source

    Dst

    Destination

    proto

    Protocol

    s_port

    Source Port

    product

    Product

    service

    Service (when resolved includes the port

    and protocol.)

    Action

    Action

    ifname

    Interface

    User

    User

    When you name the above fields accordingly, they are placed in their correct column in Logs & Monitor > Logs.

    This enables them to participate in all filtering done on these columns. These fields automatically take part in existing eventClosed Record of a security or network incident that is based on one or more logs, and on a customizable set of rules that are defined in the Event Policy. definitions with these field names.

  • field_type - the type of the field in the log.

    This table shows the possible field types.

    Field Type

    Comment

    int

     

    uint

     

    string

     

    ipaddr

    For IP addresses used with the Src and Dst fields.

    pri

    Includes the facility and severity of a syslog.

    timestmp

    Includes the date and time of the syslog. Supports the format 'Oct 10 2019 15:05:00'.

    time

    Supports the format '15:05:00'.

    string_id

    For a more efficient usage of strings. Used when there is a finite number of possible values for this field.

    action

    Supports these actions: drop, reject, accept, encrypt, decrypt, vpnroute, keyinst, authorize, deauthorize, authcrypt, and default.

    ifdir

    0 - inbound

    1 - outbound

    ifname

    For an interface name (used with the "ifname" field).

    protocol

    The field name should be "proto".

    port

    For "service", "s_port" or "port" fields.

    The field type of the field names in this table must be as mentioned:

    Field Name

    Field Type

    Src

    ipaddr

    Dst

    ipaddr

    proto

    protocol

    s_port

    port

    service

    port

    Action

    action

    ifname

    ifname

  • field_index or field_value - The parameter used depends on the value of the "type" field.

    • If the "type" field is index, the "field_index" shows.

    • If the "type" field is const, the "field_value" shows.

    The "field_index" denotes which part of the regular expression is extracted, according to the grouping of the patterns.

    To make this grouping, write a certain expression in brackets.

    In this expression, the number in the "field_index" denotes the bracket number whose pattern is taken into account.

    :command (
       :cmd_name (try)
       :parse_from (last_position)
       :regexp ("Failed password for ([a-zA-Z0-9]+) from ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) port ([0-9]+)")
       :add_field (
          :type (index)
          :field_name (User)
          :field_type (string)
          :field_index (1)
         )
       :add_field (
          :type (index)
          :field_name (Src)
          :field_type (ipaddr)
          :field_index (2)
       )
       :add_field (
          :type (index)
          :field_name (port)
          :field_type (port)
          :field_index (3)
       )
    )

    The pattern for the User, "[a-zA-Z0-9]+", is located in the first pair of brackets. Therefore, the "field_index" is one.

    The pattern for the Source address, "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+", is located in the second pair of brackets. Therefore, the index is two.

    The pattern for the port is in the third pair of brackets.

    In each parsed regular expression the maximum number of brackets must be up to nine.

    To extract more than nine elements from the regular expression, break the expression into two pieces.

    The first regular expression contains the first nine brackets.

    The remaining of the regular expression is in the "on_success" command.

    :command (
       :cmd_name (try)
       :parse_from (start_position)
       :regexp ("access-list (.*) (permitted|denied|est-allowed) ([a-zA-Z0-9_\([a-zA-Z0-9_\\.[0-9]+\.[0-9]+\.[0-9]+)\(([0-9]*)\) -> "))
       :add_field (
          :type (index)
          :field_name (listID)
          :field_type (string)
          :field_index (1)
       )
       :add_field (
          :type (index)
          :field_name (action)
          :field_type (action)
          :field_index (2)
       )
       :add_field (
          :type (index)
          :field_name (proto)
          :field_type (protocol)
          :field_index (3)
       )
       :add_field (
          :type (index)
          :field_name (ifname)
          :field_type (ifname)
          :field_index (4)
       )
       :add_field (
          :type (index)
          :field_name (Src)
          :field_type (ipaddr)
          :field_index (5)
       )
       :on_success (
          :command (
             :cmd_name (try)
             :parse_from (last_position)
             :regexp ("([a-zA-Z0-9_\\.[0-9]+\.[0-9]+\.[0-9]+)\(([0-9]*)\) hit-cnt ([0-9]+) ")
             :add_field (
                :type (index)
                :field_name (destination_interface)
                :field_type (string)
                :field_index (1)
             )
          )
       )
    )

    The "field_value" is the constant value to be added.

    :command (
       :cmd_name (try)
       :parse_from (last_position)
       :regexp ("%PIX-([0-9])-([0-9]*)"))
       :add_field (
           :type (const)
           :field_name (product)
           :field_type (string_id)
           :field_value ("CISCO PIX")
       )
    )

  • dict_name is the name of the dictionary to use to convert the value. If the value is not found in the dictionary, the value is the result.

    The free text parser enables us to use dictionaries to convert values from the log. These conversions are used to translate values from logs from different devices, with the same meaning, into a common value, which is used in the event definitions.

    Each dictionary file is defined as an .ini file.

    In the .ini file the section name is the dictionary name and the values are the dictionary values (each dictionary can include one or more sections).

    [dictionary_name]

    Name1 = val1

    Name2 = val2

    [cisco_action]       [3com_action]

    permitted = accept   Permit    = accept

    denied = reject      Deny   = reject

    :command (
       :cmd_name (try)
       :parse_from (start_position)
       :regexp ("list (.*) (permitted|denied) (icmp) ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) -> ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+).* packet")
       :add_field (
             :type (index)
             :field_name (action)
             :field_type (action)
             :field_index (2)
             :dict_name (cisco_action)
        )
    )