The content of the data source file is tokenized and normalized before hashed into fingerprints. The same rules are applied to the data that is scanned for exact data matching.
User data is considered equal regardless of the case. For example, DATA, Data, and data are considered the same.
Non-word characters (equivalent to PCRE \W) aren't considered for matching. This makes underscore and its variants a special case where an identifier separator is considered for matching.
Underscore matching:
Ignored, if a word is surrounded by underscore.
Considered as a word separator, if a word contains an underscore in between. For example, in the mail ID xxxx_yy@zz.com, underscore is considered as a word breaker, and the mail ID is divided into two tokens, “xxxx” and “yy@zz.com”. An incident is reported even when the sample or scanned document has an underscore anywhere in a word, but the fingerprinted document doesn't.
Note
To apply the change in underscore tokenization, reindex exact data match fingerprint with the EDMTrain tool.
Japanese address area codes (chōme-ban-gō) are normalized such that, 二丁目２１番９号, ２の２１の９, and 2-21-9 are considered the same.
Applies ICU NFKC normalization (Unicode® Standard, Annex #15). For example, half-width and full-width Japanese characters are converted to their normal equivalent forms.
Applies ICU rules for text segmentation (Unicode® Standard, Annex #29 and ASCII single word identifiers) with these minor deviations, so that the text is broken into more meaningful tokens for exact data matching:
Comma and semicolon variants are considered as word breakers when located between numbers. Also, in contrast, '/' is not considered as a word breaker when located between numbers so that dates are treated as single word.
'-', '+', and '@' variants aren't considered as word breakers when located between numbers and/or letters.
ICU rules for text segmentation with ASCII single word identifiers permit these characters and separators:
Numeric: [0-9] separated by [./@_+-]
Alphabetical: [a-zA-Z] separated by [.@_+-]
Alphanumeric: [a-zA-Z0–9] separated by [@_+-] ('.' and '/' are permitted as mentioned if both sides are numeric, similarly for '.', if both sides are alphabetical).
EDM boundaries
The data source file can be a .csv (comma-separated value) file or .tsv (tab-separated value) file. It isn't possible to escape the delimiter character, so the delimiter character must not appear in the data.
Automatic fingerprint creation is supported in data source files with up to 6 billion cells, subject to a maximum of 4,294,967,295 (about 4 billion user records) rows and 32 columns.
The first line of the data source file can be a header of column names, as the headers are extracted into the .props file. The
-h(--header) option used in theedmtraincommand indicates that the first line is a header line. If the header option isn't specified, the tool generates the output file with the column names, such as Column 1, Column 2, and Column 3.A column name can't contain more than 100 characters.
Phone numbers in the fingerprint file only match identical numbers in text files. For example, a phone number entered in the CSV without a country code doesn't trigger a match with the same number in the text file if it includes the country code.
Exact data matching supports multibyte using Unicode to enable processing of data in all supported languages including Chinese, Japanese, and Korean.