Regular Expressions

Lasernet supports regular expressions in several modules. They are mostly used in the Text Filter modifier and engine and the Binary Filter modifier; however, regular expressions can also be very useful in JavaScript.

Regular expressions are built up from expressions, quantifiers and assertions. The simplest form of expression is simply a character, for example, x or 5. An expression can also be a set of characters. For example [ABCD] will match an A, B, C or D. In shorthand it could be written as [A-D]. If we want to match any of the capital letters in the English alphabet we can write [A-Z]. A quantifier tells the regular expression engine how many occurrences of the expression we want e.g., x{1,1} means match an x which occurs at least once and at most once. Assertions and more complex expressions will be discussed later.

In general, regular expressions cannot be used to check for balanced brackets or tags. For example, if you want to match an opening html <b> and its closing </b> you can only use a regular expression if you know that these tags are not nested; the html fragment, <b>bold <b>bolder</b></b> will not match as expected. If you know the maximum level of nesting it is possible to create a regular expression that will match correctly, but for an unknown level of nesting, regular expressions will fail.

Example 1

We will start by writing a regular expression to match integers in the range 0 to 99. We require at least one digit, so we start with [0-9]{1,1} which means match a digit just once. This regular expression matches integers in the range 0 to 9. To match one or two digits we can increase the maximum number of occurrences. The regular expression then becomes [0-9]{1,2} meaning match a digit at least once and at most twice.

However, this regular expression will not match correctly as its scope needs to be properly defined. To ensure that we match against the whole string we must use anchor assertions. By placing a ^ (caret) symbol first, the regular expression must match from the beginning of the string. By placing a $ (dollar) symbol last, the regular expression must match until the end of the string. So our regular expression now looks like this ^[0-9]{1,2}$. Note that assertions such as ^ and $, do not match any actual characters.

If you have seen regular expressions elsewhere, they may have looked different from the ones above. This is because some sets of characters and some quantifiers are so common that they have special symbols to represent them. [0-9] can be replaced with the symbol \d. The quantifier to match exactly one occurrence, {1,1}, can be replaced with the symbol itself. This means that x{1,1} is the equivalent of just using x. So our 0 to 99 matcher could be written ^\d{1,2}$. Another way of writing it could be ^\d\d{0,1}$; in other words, from the start of the string, match a digit followed by zero or one digits. In practice, most people would write it ^\d\d?$. The ? is a shorthand for the quantifier {0,1}, i.e., a minimum of no occurrences and a maximum of one occurrence. This is used to make an expression optional. The regular expression ^\d\d?$ means “from the beginning of the string match one digit followed by zero or one digit and then the end of the string”.

Example 2

Our second example matches the words 'mail', 'letter' or 'correspondence' without matching 'email', 'mailman', 'mailer', 'letterbox', and so on. We'll start by just matching 'mail'. In full, the regular expression is m{1,1}a{1,1}i{1,1}l{1,1}, but since each expression itself is automatically quantified by {1,1} we can simply write this as mail; an 'm' followed by an 'a' followed by an 'i' followed by an 'l'. The symbol '|' (bar) is used for alternation, so our regular expression now becomes mail|letter|correspondence which means match 'mail' or 'letter' or 'correspondence'.

Whilst this regular expression will find the words we want, it will also find words we don't want such as 'email'. We will start by putting our regular expression in parentheses (mail|letter|correspondence). Parentheses have two effects: firstly, they group expressions together, and secondly, they identify parts of the regular expression that we wish to capture. Our regular expression still matches any of the three words but now they are grouped together as a unit. This is useful for building up more complex regular expressions. It is also useful because it allows us to examine which of the words actually matched.

We need to use another assertion, this time \b “word boundary”: \b(mail|letter|correspondence)\b. This regular expression means “match a word boundary followed by the expression in parentheses followed by another word boundary”. The \b assertion matches a position in the regular expression not a character in the regular expression. A word boundary is any non-word character such as a space, a newline or the beginning or end of the string.

Example 3

For our third example we want to replace ampersands with the HTML entity '&'. The regular expression to match is simple: &, i.e. match one ampersand. Unfortunately, this will ruin our text if some of the ampersands have already been turned into HTML entities. So what we really want to say is ‘replace an ampersand providing it is not followed by 'amp;'’. For this we need the negative lookahead assertion. Our regular expression now becomes: &(?!amp;). The negative lookahead assertion is introduced with '(?!' and finishes with ')'. It means that the text it contains, 'amp;' in our example, must not follow the expression that precedes it.

Regular expressions provide a rich language that can be used in a variety of ways. For example, suppose we want to count all the occurrences of 'Eric' and 'Eirik' in a string. Two valid regular expressions to match these are \b(Eric|Eirik)\b and \bEi?ri[ck]\b. We need the word boundary '\b' so we don't get 'Ericsson' etc. The second regular expression actually matches more than we want, 'Eric', 'Erik', 'Eiric' and 'Eirik'.

Characters and Abbreviations for Sets of Characters

Sets of Characters

Square brackets are used to match any character in the character set that is contained within them. All the character set abbreviations described above can be used within square brackets. Apart from the character set abbreviations and the following two exceptions, no characters have special meanings in square brackets.

The predefined character set abbreviations are more universal than using character ranges across platforms and languages. For example, [0-9] matches a digit in Western alphabets but \d matches a digit in any alphabet.

Note

In most regular expression literature, sets of characters are called “character classes”.

Quantifiers

By default, an expression is automatically quantified by {1,1}, i.e. it should occur exactly once. In the following list E stands for any expression. An expression is a character, an abbreviation for a set of characters, a set of characters in square brackets or any parenthesized expression.

(MAXINT is implementation dependent, but will not be smaller than 1024.)

If we wish to apply a quantifier to more than just the preceding character, we can use parentheses to group characters together in an expression. For example, tag+ matches a 't' followed by an 'a' followed by at least one 'g', whereas (tag)+ matches at least one occurrence of 'tag'.

Note

Quantifiers are “greedy”. They will match as much text as they can. For example, 0+ will match as many zeros as it can from the first zero it finds; for example, 2.0005. Quantifiers can be made non-greedy.

Assertions

Assertions make a statement about the text at the point where they occur in the regular expression, but they do not match any characters. In the following list E stands for any expression.