The Enterprise Log Manager (ELM) uses bloom indexes to optimize queries. While most Perl Compatible Regular Expressions (PCRE) can be used for ELM searches, not every PCRE can be optimized to use the bloom.
The bloom regex optimizer performs pre-tuning to provide optimal searches, but you can obtain even better performance from your queries by keeping a few things in mind.
You can only use mandatory parts of the regular expression for bloom filtering. The bloom filter only uses substrings in the regular expression that exist in every matching string. The one exception is that you can use a one-level deep OR grouping such as
(seth|matt|scott|steve).You can't use mandatory parts of a regular expression that are shorter than four characters. For example,
seth.*groverusessethandgroverwith the bloom, buttom.*wilsononly useswilsonbecausetomis too short.OR groupings that contain non-constant substrings or a substring that is too-short can't be used. For example,
(start|\w\d+|ending)can't be used because the middle item in the OR list is not a constant that can be searched for in the bloom. As another example,(seth|tom|steve)can't be used becausetomis too short; but(seth|matt|steve)can be used.
The optimizer process for the database runs the regex-to-bloom query. That optimizer deconstructs the regex and finds the mandatory constant substrings.
As an example, the original regular expression is:
\|\|(626|629|4725|4722)\|\|.*\|\|(bbphk)\|\|
The only part that the bloom uses from this expression is bbphk. This change reduces the search set from over a million files down to 20,000.
The regular expression can be further optimized in the following way:
(\|\|626\|\||\|\|629\|\||\|\|4725\|\||\|\|4722\|\|).*\|\|bbphk\|\|
In this example, the \|\| has been moved from before and after the first group to the front and back of each element in the group, which does two things:
It allows the pipe characters to be included.
It makes the elements in the first group, which were ignored because they were only three characters, longer than four characters so they can be used.
In addition, the parentheses around bbphk have been removed as they were not needed and indicated to the bloom filter that this is a new subgroup. Performing these types of manual adjustments to the regular expression can effectively reduce the search even further to only about 2,000 files.
Note
Running complex searches over long time spans can cause the search process to stop working. Consider breaking searches for long periods into smaller time spans.