What is parsing, and why is it important?

There is no standard format for how vendors create events and logs in their applications, and they can vary widely. Parsing enables Helix to use the data in these events by aligning the data within them to a taxonomy of defined values. By applying this standard taxonomy to event data, Helix normalizes the log data, allowing it to be interpreted by its various detection engines, and allowing users to query data systematically without requiring any knowledge of the original log format. Parsers are patterns that match a log format, extract content from the event, and store it aligned with the Helix taxonomy.

Logs from vendors can change over time as the product is updated, or as a result of changes to the logging configuration. This can cause the events to no longer match the patterns of existing parsers, marking them as class=unknown, and as a result, they are not analyzed against Trellix rules, Analytics or Threat Intelligence. Trellix will create new patterns upon request, but for those industrious folks who don’t want to wait, or who want to create their own pattern, Helix’s Self-Service Parsing is a method to upload a custom parser to run against events being marked as class=unknown.

HOW TO WRITE A PARSER

1. Overview

The syslog parsing engine in Helix is based on Syslog-ng and patterndb.xml. Helix comes with parsing support for hundreds of products out of the box. When a new parser is added to Helix through self-service parsing, the parser can safely be run against unparsed class=unknown logs. It will not damage any existing parsing functionality.

2. BASIC DEFINITIONS

Before we start talking about the development process, you need to be familiar with some concepts that will be mentioned on the next pages. These are simple explanations just to give you an idea to what we are referring while you are reading. Some of them will be explained in detail on the following sections.

Parser: Predefined operators that match a certain type of information based on their function and a user-defined criteria. Think of them as pre-programmed functions for each type of data. You will read more about it in the CREATING A PATTERN section.

Pattern: Set of parsers joined to extract specific information from a log. Each pattern is created to match a certain type of log (with a specific structure), and each pattern should be as specific as possible.

Rule: Contains a pattern and a set of values, such as: metaclass, IF statements, operations, etc.

Class: Denotes the device or app producing the log. If the device or application produces more than one type of log according to its functionalities, for example, firewall traffic and http traffic, then the type of log is included in the class name: checkpoint_firewall and checkpoint_http_proxy classes.

Metaclass: Represents a classification for the logs according to the class. Each class can belong to one or more metaclasses, depending on the type of device or application that we are talking about.

Rule Set: A set of rules inside a predefined XML structure ordered from the most specific rule to the least specific. The order of the rules inside the rule set really matters; from it depends the results during the parsing. If a generic rule (a rule that just looks for common things, such as commas, spaces, dots, etc.) is added at the beginning of the rule set, it will match all the logs, and it won't deliver any useful information to the end user.

Program: Logs produced by a device or application sometimes come with a program keyword that identifies them in the source, and that keyword needs to be added to the set of rules. When the Helix platform is looking for a rule to parse the logs, if the logs come with a program keyword, the system will look for the keyword in the rule sets and then will try to match the logs with one of the rules.

Values: The part of the rule where the metaclass is defined. Also, here we can add IF statements, operations, and concatenations needed to present the information parsed by the rule in the appropriate format.

3. TAXONOMY

For patterns development, you must follow a standard taxonomy used by Helix. This taxonomy is like a dictionary of field names that can be used in the patterns, and it is available online in the cloud integrations portal’s JSON tile here: https://helix-integrations.cloud.aws.apps.fireeye.com/#/install/json/?preview=true. Only this list of fields can be used in a pattern. If any other field name is used, it will cause an internal error on the system.

- Classes: The class name and the associated fields.

- MetaClasses: The metaclass name, a brief description of its meaning and the classes that belong to each metaclass. You can consider this as a general classification for the logs.

- Fields: The field name and the field characteristics (Type, Format, Max Length, Usage, Used for). It is really important that you use the correct pattern parser to parse the information on the fields. For example, if you are parsing a srcport (source port), according to the taxonomy, this field type is integer, so you must use the @NUMBER@ pattern parser.

- Field Aliases: You can find the alias for a specific field (if the field has one).

4.    ANALYZING LOGS

Sometimes you need to analyze many different logs on the same file to create your patterns, and that will complicate the identification of log formats. In that case, you should use Notepad++ to help you to separate the logs that have the same log format, which makes it easier for you create the patterns to cover all the logs.

The first thing that you need to do is read the first pattern, compare that line with the second one, and try to identify any difference between them. In some cases, you will find a different word or symbol that can help you to separate the different logs on different files. If you can’t identify a difference between those two lines, repeat the process with the next line until you find a difference.

Once you have identified a word, symbol or a combination of both, select it and open the search using Ctrl+f. Go to the Mark tab, check the checkbox Bookmark line and click on the button Mark All. The application will search for all the logs that match your pattern and will add a bookmark to those lines. After that, go to Search -> Bookmark -> Cut Bookmarked Lines, so you can put your lines in another file and create the pattern more easily.

Once you have all the logs separated, you are ready to start with the pattern creation.

5.    CREATING A PATTERN

Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers are enclosed between @ characters. The syntax of parsers is the following:

• A beginning @ character;

• the type of the parser written in capitals;

• optionally a variable name between double dots;

• parameters of the parser, if any;

• a closing @ character.

You can use 3 principal options when making your parser. These options are the most used, but feel free to use anyone of the available options according to the field data type and the information that you will match (see Pattern Parsers List after Example 2):

Copyright © 2023 Musarubra US LLC

• ESTRING = a string containing anything. This type has a required parameter that acts as the stop character: the parser parses everything until it finds the stop character or a string. If you don’t specify a stop character, the parser parses everything until the end. Examples:

  @ESTRING:: @ -> parses everything until it finds a space

  @ESTRING::src:@ -> parses everything until it finds src:

• NUMBER = must be a number.

• IPv4 = must be an IPv4 address.

Patterns and literals can be mixed together. For example, to parse a message that begins with Host: followed by an IP address (e.g. Host: 192.168.1.1), the following pattern can be used: Host: @IPv4::@. (This assumes that Host is a standard word in all the logs. If it is not, you cannot add the word as a static part of the pattern. In that case the pattern should be: @ESTRING:: @@IPv4::@.)

Special characters should be placed in source code like any other alphanumeric characters according to the Special Characters List¹. For example, > instead of >. Double quotation marks mustn’t be encoded.

To assign a value to a variable, put the variable name between colons (::). All the value parsed will be assigned to the variable. Examples:

@ESTRING:action: @ -> A word is assigned to action
@IPv4:srcip:@ -> An IP address is assigned to srcip
@NUMBER:seconds:@ -> A number is assigned to seconds

Example 1

We have a log that needs to be parsed:

List CELERA-EXCHANGE-NAC2 permitted icmp 10.12.6.113:80 -> 10.20.6.179:137, 1 packet

The client requires some fields:

        Protocol: icmp
        Source IP: 10.12.6.113
        Source port: 80
        Destination IP: 10.20.6.179
        Destination port: 137    

This is the final pattern:

@ESTRING:: @@ESTRING:: @@ESTRING:: @@ESTRING:protocol: @
@IPv4:srcipv4:@:@NUMBER:srcport:@ -> @IPv4:dstipv4:@:@NUMBER:dstport:@@ESTRING::@

Parts parsed:

@ESTRING:: @ -> parses until a spacing, List @ESTRING:: @ -> parses until a spacing, CELERA-EXCHANGE-NAC2 @ESTRING:: @ -> parses until a spacing, permitted @ESTRING: protocol: @ -> parses until a spacing and is assigned to protocol without the whitespace, icmp @IPv4:srcipv4:@: -> parses an IP and is assigned to srcipv4 and the IP must be followed by ":", 10.12.6.113 @NUMBER:srcport:@ -> parses a number and is assigned to srcport and the number must be followed by " ->", 80 @IPv4:dstipv4:@: -> parses an IP and is assigned to dstipv4 and the IP must be followed by ":", 10.20.6.179 @NUMBER:dstport:@ -> parses a number and is assigned to dstport, 137 @ESTRING::@ -> parses everything until the end    

@ESTRING:: @@ESTRING:: @@ESTRING:: @@IPv4::@ >@ESTRING:interface: @@ESTRING::src: @@IPv4:srcipv4:@; @ESTRING::dst: @@IPv4:dstipv4:@;@ESTRING::;@

srcipv4: 216.207.71.1 dstipv4:
23.74.145.174
interface: eth1-06

srcipv4: 216.207.71.1 dstipv4:
23.74.145.174
interface: 7Jan2017

@ESTRING:: @@NUMBER::@:@NUMBER::@:@NUMBER::@ @@ESTRING:: @@IPv4::@ >@ESTRING:interface: @src: @@IPv4:srcipv4:@; @ESTRING::dst: @@IPv4:dstipv4:@;@ESTRING::;@

Logs will match the most specific pattern in the pattern file. Therefore, make each pattern as specific as possible to avoid having it parsed incorrectly. Using NUMBER and IPv4 are a couple of ways to help make your pattern more specific.

Pattern Parsers List []

@ANYSTRING::@

Parses everything to the end of the message; you can use it to collect everything that is not parsed specifically to a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.

@EMAIL@

This parser matches an e-mail address. The parameter is a set of characters to strip from the beginning and the end of the e-mail address. That way e-mail addresses enclosed between other characters can be matched easily (for example, <user@example.com> or "user@example.com"). Characters that are valid for a hostname are not stripped from the end of the hostname. This includes a trailing period if present. For example, the @EMAIL:email:"[<]>@ parser will match any of the following e-mail addresses: <user@example.com>, [user@example.com], "user@example.com", and set the value of the email field to user@example.com.

@ESTRING::@

This parser has a required parameter that acts as the stop character. The parser parses everything until it finds the stop character. For example, to stop at the next " (double quote) character, use @ESTRING::"@. To stop at a colon (:), the colon has to be escaped with another colon, like this: @ESTRING:::::@. As of syslog-ng 3.1, it is possible to specify a stop string instead of a single character, for example, @ESTRING::stop_here.@. The @ character cannot be a stop character, nor can line breaks or tabs.

@FLOAT::@

A floating-point number that may contain a dot (.) character. (Up to syslog-ng 3.1, the name of this parser was @DOUBLE@.)

@HOSTNAME@

Parses a generic hostname. The hostname must contain only alphanumeric characters (A-Z, a-z, 0-9), hyphen (-), or dot (.).

@IPv4::@

Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).

@IPv6::@

Parses any valid IPv6 IP address.

@IPvANY::@

Parses any IP address.

@LLADDR@

Parses a Link Layer Address in the xx:xx:xx:... form, where each xx is a 2-digit hexadecimal number (an octet). The parameter specifies the maximum number of octets to match and defaults to 20. The MACADDR parser is a special wrapper using the LLADDR parser. For example, the following parser parses up to 10 octets and stores the results in the link-level-address macro: @LLADDR:link-level-address:10@

@MACADDR@

Parses the standard format of a MAC-48 address, consisting of six groups of two hexadecimal digits, separated by colons. For example, 00:50:fc:e3:cd:37.

@NUMBER::@

A sequence of decimal (0-9) numbers (for example, 1, 0687, and so on). If the number starts with the 0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x. A leading hyphen (–) is accepted for non-hexadecimal numbers, but other separator characters (for example, dot or comma) are not. To parse floating-point numbers, use the @FLOAT@ parser.

@PCRE@

Use Perl-Compatible Regular Expressions (as implemented by the PCRE library). This pattern parser accepts the use of very simple regular expressions only, including: \s, \S, \w, \W, \d, \D, .+, .*, *, +, {} (this doesn’t accept ranges, but only one number). Syntax: @PCRE:name:regexp@

@QSTRING::@

Parse a string between the quote characters specified as parameter. The quote character can be different at the beginning and the end of the quote. For example: @QSTRING:"@" parses everything between two quotation marks ("), while @QSTRING:<>@ parses from an opening bracket to the closing bracket. The @ character cannot be a quote character, nor can line breaks or tabs.

@SET@

Parse any combination of the specified characters until another character is found. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings. For example, the @SET::"@ parser will parse any combination of whitespaces and double quotes. Available in syslog-ng OSE 3.4 and later.

@STRING::@

A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other accepted characters can be listed as parameters (for example, to parse a complete sentence, add the whitespace as parameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line breaks or tabs.

RESOLVED EXAMPLES

(-- 1 --)

Log

list CELERA-EXCHANGE-NAC2 permitted 250 10.6.6.138 -> 10.20.6.103, 1 packet

Fields required

• action

• protocol

• srcipv4

• dstipv4

Pattern

list @ESTRING:aclname: @@ESTRING:action: @@ESTRING:protocol: @@IPv4:srcipv4:@ -> @IPv4:dstipv4:@, @ANYSTRING::@

Results

• aclname=CELERA-EXCHANGE-NAC2

• action= permitted

• protocol=250

• srcipv4=250 10.6.6.138

• dstipv4=10.20.6.103

(-- 2 --)

Log

Deny tcp source 10.1.1.1/5555 destination 10.2.2.2/80 access-group "outside-in"

Fields required

• ✓ protocol

• ✓ srcipv4

• ✓ srcport

• ✓ dstipv4

• ✓ dstport

• ✓ access group

Pattern

@ESTRING:action: @@ESTRING:protocol: @ source @IPv4:srcipv4:@/@@NUMBER:srcport:@ destination @IPv4:dstipv4:@/@@NUMBER:dstport:@ access-group "@ESTRING:accessgroup:"@

Results

• ✓ action=Deny

• ✓ protocol= tcp

• ✓ srcipv4= 10.1.1.1

• ✓ srcport= 5555

• ✓ dstip= 10.2.2.2

• ✓ dstport= 80

• ✓ accessgroup= outside-in

(-- 3 --)

Log

7Jan2014 18:25:23    192.168.1.1 >  default_device_message: <166>CPFW:  7Jan2014 18:25:08 accept  10.142.160.4 >eth1 user: Security LondonPL (93251@client.com)(+); src_user_name: Security LondonPL (93251@client.com)(+); src_machine_name: 46ysf5jd@client.com; snid: faf79dca; inzone: Internal; outzone: Internal; rule: 91; rule_uid: {A2DACDA4-6EF4-4054-986B-96C2805586FD}; src: 10.134.11.43; dst: 10.132.151.81; proto: udp; xlatesrc: 10.142.160.4; NAT_rulenum: 337; NAT_addtnl_rulenum: 390; product: VPN1 & FireWall-1, service: 49300; s_port: 63895; xlatesport: 38598; product_family: Network;

Fields required

• ✓ action

• ✓ interface

• ✓ user name

✓ calling user name

✓ source host

✓ source zone

✓ destination zone

✓ rule

✓ rule id

✓ srcipv4

✓ dstipv4

✓ protocol

✓ transsrcip

✓ product

✓ service

✓ srcport

✓ transsrcport

✓ family

Pattern

@PCRE::.*\w+\d{2}:\d{2}:\d{2}: @@ESTRING:action: @@ESTRING::>@@ESTRING:interface: @user: @ESTRING:username:;@ src_user_name: @ESTRING:callingusername:;@ src_machine_name: @ESTRING:srchost;@@ESTRING::inzone: @@ESTRING:srczone:;@ outzone: @ESTRING:dstzone:;@ rule: @ESTRING:rule:;@ rule_uid: @ESTRING:ruleid;@@ESTRING::src: @@IPv4:srcipv4:@;@ESTRING::dst: @@IPv4:dstipv4:@;@ESTRING::proto: @@ESTRING:protocol:;@@ESTRING::xlatesrc: @@IPv4:transsrcip:@;@ESTRING::product: @@ESTRING:product:;@ service: @@ESTRING:service:;@@ESTRING::s_port: @@NUMBER:srcport:@;@@ESTRING::xlatesport: @@NUMBER:transsrcport:@;@ESTRING::product_family: @@ESTRING:family:;@

Results

• action=accept

• interface= eth1

• user name= Security LondonPL (93251@client.com)(+)

• calling user name= Security LondonPL (93251@client.com)(+)

• source host=46ysf5jd@client.com

• source zone= Internal

• destination zone= Internal

• rule=91

• rule id={A2DACDA4-6EF4-4054-986B-96C2805586FD}

• srcipv4=10.134.11.43

• dstipv4=10.132.151.81

• protocol= udp

• transsrcip=10.142.160.4

• product= VPN-1 & FireWall-1

• service=49300

• srcport=63895

• transsrcport=38598

• family= Network

6. XML FILE

Now we will talk about how we manage the patterns already created. The following rule structure is an example of how a pattern or set of patterns can be embedded into an xml file. The xml then serves as the customer parsing rules.

Rule Set Structure

1. patterndb : The start tag of the xml in pattern db xml file. The file can include multiple rulesets to allow 1 class per ruleset section.

2. id(ruleset_name): This field identifies what kind of events the rule set parsers from the incoming logs or events to. E.g.: aws_cloudfront-1, bluecoat_http_proxy-1, etc. This field is formed by class name followed by the order of the rule among rules that parse the same kind of events. For example, the first rule set parsing out events of class name “class” would be class-1, followed by class-2 and so on.

3. description: A description of the ruleset. This field can be left empty.

4. url: Empty

5. provider: A name of your choice, such as a customer name or company name.

6. pattern tag/ program_name: This field identifies what rules the incoming traffic identifies. The program_name can be identified from the csv file that contains the logs. This field may be the same as the class_name, but not always. For example: aws_cloudfront, bluecoat_http_proxy.

7. Class_name: The name of the logs we are classifying. For example: aws_cloudfront, bluecoat_http_proxy.

8. id(rule_id): When you create a new rule, its rule ID is formed by the class name in lowercase, followed by a number that denotes the creation order (1, 2, 3, and so on). The rule ID can also be directly searched within Helix via (meta_rule:<rule_id>).

9. Pattern: The original pattern that we built, as described in the section CREATING A PATTERN.

10. Metaclass (metaclass_name): The metaclass name can be any of the following, depending on the type of transaction. See the section basic definitions for details.

• antivirus

• app_transaction

• auth

• connection

• dhcp

• dns

• email

• file_xfer

• firewall

• http_proxy

• http_server

• ids

• nat

• posix

• privilege

• vpn

• windows

Example:

<rule id="ms_windows_event-491-1" class="ms_windows_event">

When you modify an existing rule, the CRC32 value is not changed but the ticket number is added with the word “EDIT” after the number.

Example:

<rule id=" ms_windows_event-491EDIT-23179844660" class="ms_windows_event">

7. ADDING PATTERNS/RULE SETS TO XML

At the beginning of the process, you locate any existing rule set related to the class that you are working on. If you find a rule set in the repository, you can copy the file to another location and add the new patterns or any other update to it. This file will be used for testing. Don’t forget that the PATTERN ORDER MATTERS to the result of the matching.

Important:

• If you are adding a new rule to a ruleset or adding a new ruleset to the existing rulesets, you must export the existing ruleset before you can edit it.

• if the rulesets are being added for the first time, importing the xml file with the pattern should be sufficient.

8. UNIT TESTING

As you create your patterns, you will test them many times to ensure they are working correctly. Here are the steps to manually test a pattern using pdbtool (tool to test patterns to confirm they work correctly).

Run it against the test_rules.xml file using the following syntax:

/usr/local/syslog-ng/bin/pdbtool match -p test_rules.xml -P "program/class_name" -M "log to test here"

Example (testing a CHECKPOINT log with test_rules.xml file):

/usr/local/syslog-ng/bin/pdbtool match -p test_rules.xml -P "checkpoint_http_proxy" -M "23Jan2014 22:11:24"

allow 10.14.160.5 <eth1-02 src: 10.2.169.41; dst: 199.233.57.41; proto: tcp; appi_name: ******; app_desc: ******; app_id: 944265405; app_category: ******; matched_category: ******; app_properties: ******; app_risk: ******; app_rule_id: ******; app_rule_name: ******; web_client_type: Other: iPhone%20Free/184890 CFNetwork/672.0.8 Darwin/14.0.0; proxy_src_ip: 10.2.169.41; resource: http://bcp.crwdcntrl.net/5/c=2215/mid=4FE885E0AADA4A7AAE6B01BA4481EB97/e=app/seg=mobile/pe=y; product: URL Filtering; service: http; s_port: 59662; product_family: Network; "

Specify the program with –P, if the log has a program keyword. If not, enter the class name there. Use –p to specify your XML rules, and use –M to send a log message to test any rule.

After you run the command, pdbtool will show the result. You will be able to see the log without special characters, the program name, class ID, rule ID, and all the variables with their values.

References

Special Characters List http://www.utexas.edu/learn/html/spchar.html

Pattern Parsers http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guideadmin/html/reference-patterndb-parsers.html

20